What I Learned Exploring Open-Source Photo-to-3D Models (And the App I Built Along the Way)

A few weeks ago I set out to answer a narrow question: how good are today's open-source image-to-3D models, really, if you point one at a single photo instead of a clean product render? That question turned into Dioramic — an open-source app that uploads a photo, runs it through a GPU-hosted 3D pipeline, and hands you back a textured mesh you can rotate in the browser and download.

This post is less a launch announcement and more a field report: what the open-source photo-to-3D landscape looks like right now, which models actually fit the constraints I had, and where the pipeline still falls short.

Live demo: https://dioramic.vercel.app/ Source code: https://github.com/Hitesh-s0lanki/3d-opensource-playground

The starting constraint: single object, single photo

Most of the interesting open-source reconstruction models — TripoSR, InstantMesh, Hunyuan3D — are trained on one thing: an object centered on a blank background, seen from one or more angles. Feed one a photo of a whole bedroom and, as the project's own model notes put it, "you get garbage." These models have no concept of a room; they orbit whatever you give them like it's a single centered object, which is fine for a chair and useless for a chair sitting inside four walls.

So the pipeline this project ships today deliberately narrows to what actually works: one photo of one object, roughly centered, background removed automatically. The result comes back as a .glb — a textured mesh at roughly 40k triangles and 4–5 MB, or an untextured one at up to 1.5M triangles if you skip the paint pass.

What's actually running under the hood

The generation model is Tencent's Hunyuan3D-2.1, hosted on a rented GPU (Modal) rather than run locally — it needs two CUDA extensions compiled from source and enough VRAM that "runs on my laptop" was never the goal. A run walks through:

1. Background removal — rembg / u2net strips the object from its background before the mesh model ever sees it.
2. Shape generation — Hunyuan3D-Shape produces the raw geometry.
3. Texture painting — Hunyuan3D-Paint bakes a texture onto that geometry.

A generation takes about 4–5 minutes end to end. Because that's too long to hold an HTTP request open, the app doesn't try: uploading a photo spawns the Modal job and returns immediately, and a poll on the frontend picks up the finished mesh whenever it lands. Close the tab mid-generation and nothing is lost — there's no in-memory state to lose.

Why open source mattered for this specific project

I could have called a hosted photo-to-3D API and shipped a thinner wrapper faster. The reason I didn't: with an API you get a mesh back and no visibility into why a given photo produced a bad one. With an open model, every stage — detection, background removal, shape, texture — is a file you can read, swap, or instrument.

That transparency ended up mattering in a concrete way. This project actually implements two pipelines, and only one of them is live:

• Single object (what the hosted app does today): photo → Hunyuan3D-2.1 → textured mesh. Straightforward, and it fits in a Modal container.
• Whole room: detect every object with GroundingDINO, reconstruct each one separately with TripoSR, estimate where they sit in the room from a camera model and known furniture sizes, then assemble the scene in Blender. This part is implemented and it works, but it drives Blender as a subprocess and hasn't been packaged into a container yet — so in the live app, "Whole room" is visible in the UI but disabled.

Building the room pipeline is what forced the harder lessons, because a single photo of a room has no depth information at all. Two assumptions fill that gap: furniture has roughly known real-world sizes (a double bed is about 2.0 × 1.6 m), and an object's apparent height in the frame — not the bottom of its bounding box — gives you distance once you assume a focal length. Get the assumed field-of-view wrong and the whole room's scale drifts; the project's own notes describe a sample bedroom coming out 9×10 meters at one FOV assumption and correctly sized at another.

What surprised me about the model landscape

A few things stood out while surveying what's available (details in docs/models.md):

• Licensing varies more than quality does. TripoSR (MIT) is the weakest of the object reconstructors but the only one that fits a 4 GB card. InstantMesh and MIDI-3D are both Apache-2.0 and clearly better, but need roughly 10–30 GB of VRAM. One promising whole-scene project bundles several sub-models under different licenses each — worth checking before any commercial use.
• Consistency, not resolution, is the real failure mode for anything that generates multiple views of an object and reconstructs from those. A 3D-aware multi-view model (Zero123++) keeps views agreeing with each other; a generic image model (SDXL, FLUX) has no such constraint and produces views that quietly disagree, which is why an entire sub-field (Carve3D, MVDiff) exists just to correct for it.
• Retrieval may beat reconstruction for design tools. Generated furniture meshes are lumpy and hard to edit. Matching a detected object against a CAD library instead of reconstructing it gives you something clean, correctly scaled, and swappable — reconstruction gets you layout, retrieval gets you quality.

Try it yourself

The app is live and open source:

• Demo: https://dioramic.vercel.app/ — every account starts with 5 free generations, no card required, and six sample photos are built in if you don't have one handy.
• Code: https://github.com/Hitesh-s0lanki/3d-opensource-playground — Next.js 16 frontend, a three.js viewer, Clerk for auth, Neon Postgres for run/job/credit state, Vercel Blob for photo and mesh storage, and the Modal-hosted Hunyuan3D-2.1 worker.

If you're also poking at open-source 3D generation, or you spot something the pipeline gets wrong, I'd genuinely like to hear about it — issues and stars on the repo are both welcome.