LegoGPT can design stable structures using standard LEGOs from text prompts

A team of engineers and AI specialists at Carnegie Mellon University has developed an AI application that can design stable structures from standard LEGOs using text prompts. In their study published on the arXiv preprint server, the group repurposed a large language model (LLM) created by META to build their new system.

The research team sought to improve on 3D generative models that currently tend to produce designs that will not work in the real world due to gravity or disconnection issues. To that end, they turned to LEGOs, the brick-like plastic toys that allow children to use their imagination to build structures.

To build their system, the team started with META’s LLM, LLaMA-3.2-1B-Instruct. To use it to design structures made of bricks, they swapped next-word predictions for next-brick predictions. They also added a separate math-based module that assured physical stability by taking account of structural forces and gravity. Next, they created a training dataset of 47,000 stable LEGO structures and characteristics, with captions generated by another AI system, and then used it to train their AI model.

To create a design, the system takes a recursive approach—bricks are placed and then tested to see if adding them causes instability; if so, the brick is removed and another approach is taken. The researchers tested their system without the rollback feature and found that just 24% of the designs would be stable if built in the real world; with it, the rate went up to 98.8%. The team next added color and texture abilities.

To further test their design, the researchers used a pair of robots that could be programmed to build LEGO structures given a design and generated several test structures. They also built some by hand.

The team says that their system is capable of producing a wide variety of stable LEGO structures. They also note that they tested their system against other AI systems that have been built to create 3D objects, and found that theirs produced a higher percentage of stable structures.

© 2025 Science X Network