I'm interested in this for robotics. Probably need physics to do that well.
Thanks for sharing - I'm actually working with my kid on something similar but with physics modeling for First Lego League. Cool to see your approach!
Fun idea. Congrats on building and releasing!
That said, LLMs encroaching into the pure-imagination world of Lego gives me a bit of the icks...
The next step after this is to have AI and bots build a real house. Great work!
This is great
Nice job. I actually read a paper on Arxiv last year about this exact thing - using LLMs to drive model building through LDraw that might be worth reading [1].
They had some interesting experiments like giving the LLM a fixed set of parts as a constraint and asked it to "build" tools out of those available lego.
I think they were using Opus 4.5 so I'm sure with even more powerful models it'll be even better.
Hi there, thanks!
I'm taking a look at this paper right now, didn't came across it before.
I think it's really good, and also an approach I tried.
In my case, it was working pretty well in terms of geometry and correctness, but not in terms of creativity: too many constraints and an agentic LLM would end up building very similar buildings all the time. Too few ones, and it would drift into unbuildable models.
In the end, I think the current iterative process gives it a good balance: let it try things first and be creative, parts will be placed incorrectly at the beginning, but it will fix their placements, colours and so on bit by bit, until the end result is (hopefully!) correct.
And regarding Opus, according to my experience, I'd say definitely models quality is way better when produced by Opus 5 and even better by Opus 5.5
Up next: a tool that converts a photograph of your dumped bin of old Lego into a catalogue of all the Lego in your collection..
Any estimates for token costs to generate these models? Curious about costs for Astra, Opus and Jev for some of the examples you have in the video.
Well, that greatly depends on the thinking required to produce a single LDraw model, not so much regarding how much of a "thinker" a model is, but how much thinking it must do to get a model right.
Technic models, for example, are usually expensive, given that initial placements are usually further from the right position than non-Technic models, the "bricks" ones.
A (wild!) estimate: Astra, building a Technic mechanism, around 5$...
This is one of the reasons for creating this project: making a tooling that can be used by low-end agents to iteratively build, while high-end agents, like e.g. Astra or Opus, would do the planning, vision models the inspection... etc.
Well, who knows :)
Can it create functional mechanisms using LEGO Technic parts?
That's still WIP, started already with Technic assemblies, not mechanisms yet, in order to first produce the required scaffolding to attach things like e.g. cogs, transmissions, etc. to them.
Getting mechanisms to work is one of my goals, though =)
It seems to be able to! https://github.com/anteloc/ldraw-nova/blob/master/examples/m...
That's unfinished work, however... didn't get good results with Technic models yet, except for very specific cases, like e.g. the Atlas Crane.
Thats pretty cool, i wonder how far it goes with mechanisms where the parts actually have to move correctly, rather than just being assembled correctly
Neat! Perhaps I'll finally get around to doing something with the 51515 Lego set I have laying around collecting dust.
> I did this in order to get agentic LLMs capable of designing buildable, physical things!
I've been experimenting with FreeCAD + freecad-mcp + Claude recently and have been similarly surprised how well this works. I've produced three useful parts from Claude's designs: a 3d printed case for an OLED display + some inputs + an ESP32, and a milled aluminium hotend mount adapter for a 3d printer, and a printed pen + pressure sensor mount to use the printer as a plotter. I just provided part/model numbers plus a couple of caliper measurements Claude wasn't able to find online and went through two or three cad revisions before making the parts. All three worked on the first hardware revision.
The biggest thing I've found it seems to struggle with is understanding how the parts will be assembled. The pen mount in particular was tricky to put together, having a part which needed to fit through a slightly too small space to get to its spot. Although it's hard to blame the AI for that when I didn't notice it either I suppose.
> the 51515 Lego set
On that note- have you considered adding constraints for part availability? It would be useful to be able to create designs using only parts which are on hand.
I recall seeing a project a while ago which could take a photo of a pile of lego as input and produce models that could be made with those parts. Although I think in that case the models were sorta just random arrangements.
> I've been experimenting with FreeCAD + freecad-mcp + Claude recently and have been similarly surprised how well this works
Yup. I’ve been getting some pretty decent 3d models directly on stl format + the slicer file with all the right configs for my printer
It’s cool to create a simulation, run some visualizations on web directly from Claude desktop, and then choosing a model to print and build
I also added some genetic algorithms to make it fun iterating on different shapes/mechanisms
I really enjoy how much creativity these tools are unlocking by enabling bridging so many elements together
Thanks!
I've been experimenting with FreeCAD and freecad-mcp too, even managed for Claude to create a new LDraw benchmark for it.
However, given the amount of parts some LDraw models have, FreeCAD performance would not be enough, something to do with its solver trying to work out too many constraints at once.
The thing about how to assemble the parts, yes, that drove me crazy for quite a while.
A way I found that helps is to provide the agents information about contacts, e.g. what parts of a brick are actually connectors.
With that in place, it can create tools to automatically check if two parts are connected "the right way" ;-)
I got LEGO connectors info from here: https://www.melkert.net/LDCad/tech/shadowLib
I just added another comment under my original as you were replying- I'll try not to spam you with too many questions, this is just so cool though!
Do you find the models generally manage to avoid making errors when placing parts (e.g. placing parts which intersect, don't connect properly, etc) or do they rely on validation and correction cycles?
A mix of both, I think.
When giving them enough examples, like e.g. houses, they tend to do well in that category, but not perfect.
My guess is that what happens is, given that most LEGO parts have a very limited, discrete set of legal positions and rotations, expressed as integer numbers, the agent finds patterns between different models, like e.g. a brick close to the center of a model usually has very similar (integer) coordinates and rotation components to another central brick from another model.
However, it still messes up very often, with things like e.g. +/- signs that flip parts due to rotations becoming -90deg when they should be +90deg; then, validation and correction cycles, running tools that verify the presence/absence of collisions, really helps in finishing the job.
Hi there :-)
New on HN, first time posting, and (I guess) my post on Show HN was rejected...
Anyway, I hope I'm not bothering anyone by posting this outside Show HN, or posting almost the same thing twice.
Past year, around December, I started experimenting with making ChatGPT and Claude generate source code in LDraw language.
This LDraw is literally an "assembly" language, a low-level programming language that describes how to assemble LEGO pieces together into models, one placement instruction at a time.
When executed by specific tools, like e.g. LDView, LeoCAD, Studio... these instructions become LEGO CAD models, that can be interacted with, modified, etc.
Or, in other words: one LDraw source file in .mpd or .ldr format is equivalent to one LEGO CAD model.
So, the idea I had was: if I manage for maybe ChatGPT or Claude to generate high-quality LDraw source files... then, they would actually be generating high-quality LEGO CAD models, right?
Then, after months of iterations and trying one thing after the other... it worked!!!
Long story short: using GPT-6 Astra and Opus 5.5, I've managed to create a python toolset, instructions, and docs for agents in general. Now, these can be used by them to generate LDraw models.
I've packed it all as a dockerized web app for others to try and experiment, with several providers (and agents) to choose from: OpenAI, Claude and OpenRouter.
I'd really appreciate feedback and comments, let's see where this goes =)
This project is perfect for a Show HN! I moved (most of) your text to the top and added "Show HN" to the title. I'll respond to this bit here:
> and (I guess) my post on Show HN was rejected... Anyway, I hope I'm not bothering anyone by posting this outside Show HN, or posting almost the same thing twice.
Unfortunately we had to put some barriers in place (https://news.ycombinator.com/showlim) because we got flooded, and still are, with Show HNs from new accounts which lack the spark of curiosity that is most important here. The hope is people will take the time to read the site, participate in threads, get a feel for the culture, and eventually post a Show HN that is a better fit.
None of that applies to your project, and if we had a better way to screen these things, we'd write software that let posts like yours through the first time. In any case I'm glad I randomly saw this one, and welcome to HN!
Thanks dang!
I really appreciate you helping me with this, and totally get the bit about getting flooded.
Anyway, I hope HN community will like this, and maybe contribute, have some crazy ideas about how and where to move forward =)
(Normally I'd email, but maybe it's fun for others to be in on the conversation.)
I think this post would be even better if there were more visual examples to look at. Would it be possible to link to more pictures, or even a demo video, showing different cool and/or weird things produced by the tool?
(In terms of video, you don't need anything fancy - what works best is raw and direct, with zero production values.)
Also, another thing that isn't clear to me is whether you've built any of these structures physically. If so, it would be particularly cool to see that in action! If not, it would be interesting to know what it would take to do so.