Programming as Dialogue: LLMs and Designers in Collaborative 3D Modeling

Antonio Sitong Li: Symposium Poster
Like

Share this post

Choose a social network to share with, or copy the URL to share elsewhere

This is a representation of how your post may appear on social media. The actual post will vary between social networks

Abstract:

Recent advances in artificial intelligence have expanded generative models from text and images into three-dimensional design. Yet most 3D generation efforts rely on diffusion methods that output imprecise point clouds or meshes, making results difficult to edit and poorly suited for integration with professional modeling software. This paper proposes a programming-centered alternative: positioning large language models (LLMs) as generators of executable code in familiar languages such as Python, which is then used to construct 3D models within established 3D design softwares like Rhinoceros 3D. By making code the foundational representation, the framework ensures outputs remain parametric, editable, and compatible with design workflows, while also enabling gradual improvement of generative quality through iterative prompting. Crucially, this approach introduces bi-directionality: programming code becomes the shared language through which both human designers and AI can collaborate. Designers can refine a model either by editing the code directly or by manipulating the 3D model in the design environment, with those changes translated back into code. The LLM can then generate, interpret, and modify that same code, creating a continuous feedback loop of co-design. Multi-cycle prompting and iterative sampling further enhance this process by balancing creativity with geometric clarity and encoding constraints for scale, form, and program-specific requirements. Results show that code-based generation produces models more robust to iteration and customization than diffusiononly methods, opening pathways for interactive, collaborative AI–human 3D modeling. Challenges remain in scaling spatial reasoning and reducing geometric overlaps in complex prompts, but the study highlights the promise of aligning 3D generation with the strengths of LLM programming, effectively allowing 3D modeling to “piggyback” on advances in code generation.

Please sign in

If you are a registered user on Laidlaw Scholars Network, please sign in