I have always been fascinated by the idea of building a new 3D engine or game engine. Recently, as models have rapidly improved at code generation and understanding 3D worlds, this idea returned to my mind and gradually became a ghost I could not get rid of.
Maybe it is time to rebuild the 3D stack for AI.
The 3D vision research I conducted during my PhD and the consumer product experience I gained while building a startup suddenly seem to intersect around this question.
Traditional 3D Tools
Today’s 3D software can be divided into several broad categories:
- Digital content creation tools such as Blender, where people model objects, create animations, and render images or videos offline.
- Game engines such as Unreal Engine and Unity, where people build interactive 3D worlds that run in real time.
- Industrial design tools such as CAD software, which emphasize precision, engineering constraints, and manufacturability.
- Robotics simulation platforms such as Isaac Sim, which emphasize physics, sensors, and interactions between robots and their environments.
These tools serve different industries, but they all address a similar set of problems: how to create a 3D world, how to make it move and respond, and how to present it. More broadly, the 3D stack consists of Modeling, Simulation/Animation, Rendering, and Runtime.
Different tools optimize different parts of this stack. CAD focuses on precision, Blender on artistic creation, game engines on real-time interaction, and Isaac Sim on physical fidelity. Yet they share the same creation paradigm: people must understand complex software and manually do almost everything.
AI + Traditional 3D Tools?
The most obvious approach is to let AI operate Blender, Unity, or Unreal Engine through natural language. An agent can call tools, change parameters, and write scripts. This is useful, but the underlying representations were designed for human workflows rather than AI.
A traditional 3D asset is usually stored as a mesh. Meshes are excellent for rendering, but they often preserve only the final geometry. A mesh of a tree may contain vertices, triangles, and materials, but not which parts are the trunk, branches, and leaves, how they should move, or how the tree was constructed.
In other words, a mesh often loses the structure, process, and design intent behind the final result. AI can manipulate it, but may struggle to understand and continuously redesign it.
An AI-native 3D system needs representations that can be rendered by machines while also being understood, generated, and modified by AI.
Code as a 3D Representation
AI can generate 3D content in two main ways. It can directly generate meshes, 3D Gaussians, NeRFs, or other visual representations. It can also generate code that constructs objects and scenes through tools such as Three.js.
Code may currently produce rougher visuals, but it preserves how an object is built. A bird can be represented as a body, wings, tail feathers, and a beak, with each part having its own geometry and behavior. AI can then change its proportions or movement without rebuilding the entire object.
Code also makes it possible to reuse graphics know-how. A bird might use simple geometry, SDFs, or parametric surfaces. A bush might use geometric leaves up close and billboards from a distance. These techniques can be accumulated and reused across many scenes.
This does not mean everything should become code. Generated meshes may work better for detailed objects, while code may work better for objects that need structure, variation, or behavior. The key is to combine multiple representations and let AI choose the right one.
Why Not Just Use Codex and Three.js?
Users can already ask Codex to create a Three.js scene and turn it into a video or a simple game. But there is still a large gap between one-off code generation and a complete creation product.
First, good procedural 3D content requires specialized know-how. AI will not always choose the right modeling, animation, or rendering technique. A dedicated product can accumulate these techniques as reusable assets, behaviors, templates, and generation strategies.
Second, every scene needs the same infrastructure: a scene tree, asset management, camera controls, a timeline, input handling, version history, preview, runtime, and export. A product can provide these capabilities once instead of regenerating them for every project.
Turning It Into a Product
If this direction works, it could eventually become an AI-native 3D studio where people create, direct, play, and share 3D worlds.
But trying to replace Blender, Unity, and Unreal Engine from day one would be too broad. We still need a concrete entry point: a task where traditional 3D tools are too complex, while general coding agents remain too unreliable.
There are three possible directions: an AI 3D Director, an AI 3D Game Studio, or a broader AI 3D World Builder.
Option 1: AI 3D Director
An AI 3D Director would help people create short films, music videos, memes, game trailers, and other visual content. Users generate an editable 3D scene, adjust its objects, animation, timing, and cameras, and then render it directly or enhance it with a video generation model.
The main value is control. Unlike pure video generation, a 3D scene gives creators explicit control over placement, motion, composition, and camera paths. It is also the easier product to build: a first version mainly needs scene generation, object editing, simple animation, cameras, and video export.
The main risk is output quality. Video generation may improve materials and visual style without fixing weak motion or cinematography. The pipeline may work while the final video still fails to meet users’ expectations. This direction may therefore be better suited to stylized content, social videos, previs, and game trailers than professional filmmaking.
Option 2: AI 3D Game Studio
An AI 3D Game Studio would let people generate, modify, and share playable 3D web games through natural language. Users could describe a game, play it immediately, modify its world and rules, and share the result as a remixable link.
Its advantage is that rough 3D content can still work. Low-poly assets and simple animation may look unfinished in a film, but they can become a valid game style. This also creates a natural loop: Create → Play → Share → Remix.
However, games are harder to build reliably. The system must handle input, physics, collisions, game states, UI, and unpredictable player behavior. A first version would probably need to focus on constrained formats such as flying games, driving games, physics playgrounds, or small exploration worlds.
Option 3: AI 3D World Builder
Instead of choosing immediately, the product could begin as an AI 3D World Builder: Use AI to create 3D worlds you can render and play.
Here, the world—not the video or game—is the main creative object. Users first generate and edit a structured environment, then add cameras and animation to render it, or add controls and interactions to make it playable.
This does not mean building a full film tool and a full game engine at the same time. The first version could combine Render Lite—simple cameras, animation, and video export—with Play Lite—basic controls, collision, interaction, and a shareable web link.
This approach allows user behavior to guide the next step. If people mainly create videos, the product can deepen its Director capabilities. If they spend more time building playable experiences, it can evolve toward a Game Studio. The main risk is that “building a world” may be less concrete than making a video or game, so the value should be communicated through the outputs: Create a 3D world from an idea. Turn it into a video, or make it playable.
What the Product Needs to Accumulate
A simple wrapper around Codex, Three.js, and a video model will become easier to replace as the underlying models improve. A lasting product needs to accumulate capabilities that make creation more reliable and reusable.
| Layer | Core capability |
|---|---|
| Scene Runtime | Organize objects, relationships, states, and time |
| Reusable Assets | Preserve useful modeling, animation, and interaction techniques |
| Agent Runtime | Translate user intent into reliable local edits |
| Interactive Runtime | Support input, rules, physics, and feedback |
| Generative Rendering | Turn structured 3D scenes into better visuals |
The Scene Runtime, reusable assets, and Agent Runtime form the shared foundation. Interactive Runtime supports Play, while Generative Rendering supports Render. Together, they turn one-off code generation into a persistent creation system.
From Assets to Worlds
As the product grows, people could share not only static models, but also procedural assets, behaviors, game mechanics, and complete World Templates. These would not be finished scenes, but structured worlds that AI could continue to modify and extend.
This reflects a broader shift in 3D creation: from static meshes to programs that preserve structure, behavior, and intent. A world created for a video can become interactive by adding input and rules. A world created for a game can become a video by adding cameras and directorial intent.
Both paths could eventually lead to the same form:
An AI-native World Studio for worlds that can be rendered, played, and continuously extended.
The larger question is:
If we rebuilt the 3D stack for AI today, how should we represent and create a world?