MIT Engineers Develop AI "CAD Co-Pilot" That Learns From 41,000 Design Videos
- MM24 News Desk
- 5 hours ago
- 2 min read
MIT engineers have created an AI agent that can use computer-aided design (CAD) software just like a human, transforming a simple 2D sketch into a detailed 3D model by clicking buttons and selecting menu options.
The breakthrough, powered by a massive new dataset called VideoCAD containing more than 41,000 instructional videos, could dramatically lower the barrier to entry for complex design work and boost productivity for experienced engineers.
The research aims to address the steep learning curve associated with professional CAD software. The team’s AI model doesn't just generate a 3D shape; it learns the precise sequence of user-interface actions—clicks, drags, and menu selections—required to build an object within the software itself.
“There’s an opportunity for AI to increase engineers’ productivity as well as make CAD more accessible to more people,” says Ghadi Nehme, a graduate student in MIT’s Department of Mechanical Engineering, reported in the team's presentation.
The project expands on recent developments in AI-driven user interface agents, but CAD presents a far more complex challenge due to its vast array of features. The key innovation was translating high-level design commands like "sketch line" or "extrude" into the low-level pixel-perfect mouse movements and keyboard shortcuts a human would use.
“For example, let’s say we drew a sketch by drawing a line from point 1 to point 2,” Nehme explains. “We translated those high-level actions to user-interface actions, meaning we say, go from this pixel location, click, and then move to a second pixel location, and click.”
The ultimate goal is an intelligent "CAD co-pilot." This envisioned tool would not only create 3D models from sketches but also collaborate with a human designer by suggesting logical next steps or automatically executing tedious, repetitive build sequences.
“This is significant because it lowers the barrier to entry for design, helping people without years of CAD training to create 3D models more easily and tap into their creativity,” adds Faez Ahmed, associate professor of mechanical engineering at MIT.
The research, conducted by Professor Ahmed, Ghadi Nehme, graduate student Brandon Man, and postdoc Ferdous Alam, will be presented at the prestigious Conference on Neural Information Processing Systems (NeurIPS) in December.
The team trained their model on the VideoCAD dataset, which contains over 41,000 videos showing the step-by-step construction of objects ranging from simple brackets to complex house designs. By learning from these human demonstrations, the AI learns to control the CAD software directly.
External experts see significant potential in the work. “VideoCAD is a valuable first step toward AI assistants that help onboard new users and automate the repetitive modeling work that follows familiar patterns,” says Mehdi Ataei, a senior research scientist at Autodesk Research, who was not involved in the study.
He notes this is an early foundation for tools that could eventually span multiple CAD systems and handle more complex, real-world design workflows. The MIT team is now training the model on more complex shapes, paving the way for AI assistants that could revolutionize how we design everything from consumer products to architectural structures.
Comments