Utariga — a procedural island city that Claude and I both edit
Utariga is a workflow exploration for gen AI worlds. It's a low res procedural island city in Three.js that Claude and I both edit. About 300 buildings, each one a JSON spec regenerated from one seed. I take buildings into Blender with Claude and the dcc-mcp, light them in Cycles or Octane, and bring the results back into the city.
The newest work is baked lighting: per-light lightmaps and lightprobes from Octane, on meshes and on splats, and sim ready assets for a Genesis robot sim. Most everything is deep linkable, so a link carries the camera, the render and the light mix.
Disclaimer: this is a workflow/tool exploration. Some assets and splats are low quality to save tokens, and it hasn't had an optimization pass.
1. Baked lightmaps and lightprobes
Lightmaps are baked in Octane, one AOV per light: sun, ambient, and a channel per
light ID. In the browser the channels are summed, so every light can be remixed live
with sliders, a Kelvin gel per light, MIDI faders, or the mix= URL param.
- One Run Script press in Blender joins the mesh, unwraps it and bakes the AOVs. A node script denoises, splits the atlas, packs and publishes it.
- Channels ship as packed log8 PNGs instead of 300–450 MB of EXRs.
- Lightprobes are made in the browser from the lightmap on screen. A cube camera captures each mixer channel and projects it to SH9, so the probes follow the sliders too. Turn a lamp down and a character under it gets darker.
- Probe leaking is handled like Unity's APV: probes inside walls are culled, neighbours have to see each other, and the SH is deringed.
- A reflection cube captured from the lightmap gives characters specular.
- Moving props and the Panda robot are lit by the probes, with their baked copies cut out of the lightmap.
Joey's Apartment → bld_910c2a Octane lightmap, probe lit characters
In the live app
Ostraka → bld_5920g9 artist compound · per-fixture light channels
Ostraka is a sculptor's walled compound that Claude built: a long studio, a kiln, a maquette archive and a helix stair. Its lights are split into channels for the fire glow, the side room, the courtyard lights and the sphere lights, so each group can be mixed on its own.
In the live app
2. The same lighting on splats
The splats aren't trained from photos. They're generated in the browser from the same mesh, with Three.js's mesh sampler, so the normal, color and lighting at each point are already known.
- Each gaussian samples the Octane AOV atlas at its UV and stores a value per light
channel, so the same sliders and
mix=drive the splat and the lightmap. - Spherical harmonics hold only the glossy part, Cycles COMBINED minus DIFFUSE, assigned directly instead of trained.
- A splat avatar is lit by the same probes as the mesh avatar.
- Splats are around 10x the size of the lightmapped GLB. No reflection probe on the splat yet, and there are sorting problems, so it doesn't look as good as the mesh.
In the live app
- Joey's splat with a splat avatar →
- Joey's 8M splat with a mesh avatar →
- Ostraka courtyard, 16M relightable splat →
- Ostraka long studio, 16M relightable splat →
The first version of this was the Kip House, loosely modeled after the house in Berkeley, with a Cycles lightmap transferred to the splat.
3. Sim ready assets: the Panda sim
A Franka Panda arm on Joey's desk picks up the mug, sets it down and sweeps the bottle off the desk. The sim runs in Genesis and plays back in the browser, lit by the room's lightmap and probes.
- Assets are made sim ready in Blender. My addon writes Blender's Rigid Body settings into the GLB extras: body type, collider shape, mass, friction, restitution, joints and world settings. Collections go into the GLB too.
- Genesis reads the GLB, settles the props and runs the Panda with IK.
- The 60 Hz pose stream is baked to an animated sidecar GLB that targets the
props by name. The
simclip=param loads it. - Genesis has no browser build, so it's record then play. No Python on the public site.
In the live app
Cake sim (WIP)
A procedural cake made with Geometry Nodes in Blender, with zones for crust, crumb, filling and frosting, cut by a Genesis MPM sim. It works in Blender. Getting it into the world, on the desk next to the Panda, is WIP.
4. Buildings to Blender and back
Any building exports as a GLB from the browser and goes into a live Blender session through the MCP. Claude models, set dresses and lights the interior from plain language art direction. I clean up its mistakes, edit layouts, add lights and cameras, and send it back. The GLB re-binds to the same lot, and Blender cameras come back as shots stored in building space, so they move with the building.
In the live app
5. Renders and compares on the authored camera
Cycles and Octane renders publish from Blender back into the app, keyed by building and camera name. Opening one puts the camera where the Blender camera was and overlays the render with a blend slider and a wipe. The link carries the time of day and the post settings.
In the live app
6. Imagine and Lightmatch
The ✨ Imagine panel sends the current frame and a prompt to Flux Kontext dev in local ComfyUI, or Qwen Image Edit on a Hugging Face Space. The result is published as a render on that camera, so it opens in the same blend/wipe compare.
Lightmatch tunes the real-time lights to match a target image. It runs Nelder–Mead over six multipliers (exposure, ambient, point/spot, area, warmth, emissive), one evaluation per frame. The loss compares image statistics, not pixels, since the AI targets change the content too. Doing it at a few times of day gives a light curve that is saved per building.
7. Marble splats from panoramas
The app renders a 360° panorama and a depth plate from a point in a room and sends both to World Labs' Marble, which returns a splat. It lands back in the scene aligned. The depth is encoded from percentile bounds so a bit of city through a window doesn't eat the depth range, and the heading is solved locally since Marble doesn't report it. Marble 1.1 is much better than 1.0.
8. Splat to object
Click an object in a splat and the app sends the click points to SAM 3 in local ComfyUI for a mask, then SAM 3D Objects makes a GLB that's placed back in the scene. Texture baking doesn't fit on a 24 GB 4090, so the GLBs are vertex colored, about a million triangles, and need retopology.
9. Editing the city live
Roads and buildings are editable in the browser. Drag a road and the terrain, pedestrians and street addresses follow it. Move or raise a building and the ground follows the footprint. Edit a building's JSON and the city rebakes around it.
10. Buildings and stories Claude made
I had Claude pick buildings with no interior and give them the full treatment: a name, a backstory, the Blender pass, Cycles cameras, relights, solved light grades and a narrated tour. It likes bars, old shops and spiral staircases. The in-game walk tests caught float precision gaps at stair and floor seams that Blender's raycasts missed, and those checks went back into the build scripts.
Hidden features and known issues
Hidden features
- Frame stats and performance capture.
- Record stories from camera shots or driving paths.
- Save panoramas, depth panoramas, segmentation and mask images.
- MIDI faders for the light mixer.
Known issues
- Performance: some views make my 4090 run hot at 4K. Desktop is capped at 60 FPS, mobile at 30, both can be uncapped.
- Mobile: works on my Pixel 10, even with SSGI. Splats on mobile are rough.
- Panda sim: the planner has a bug so moves interpolate in joint space, the mug lands a few cm high, and the desk lamp isn't a collider.
- Blender MCP: crashes occasionally.
How it's built
Three.js for rendering, React for the UI, Vite to
build. An Express server with a JSON file fallback. Post is
pmndrs postprocessing with N8AO,
bloom, SMAA and an experimental SSGI. Splats render with Spark.
Blender authoring is Claude driving a live Blender session through an MCP gateway. Lightmaps and renders come from Cycles and Octane. Sims run in Genesis. Image generation is Flux Kontext dev in ComfyUI or Qwen Image Edit on a Hugging Face Space. Marble splats from World Labs. Object extraction is SAM 3 and SAM 3D Objects in local ComfyUI.