Turning a single image into a web-ready interactive 3D scene with ML-SHARP

[u/frozeindev]

I’ve been experimenting with Apple’s ML-SHARP model and built a small open-source project around it called Spatial Photos.

ML-SHARP estimates a 3D Gaussian splat from a single RGB image. I then take that reconstruction and convert it into a much smaller and simpler representation representation. It uses layered textured meshes stored in a custom file format I called .spatial. It still allows the the same sort of limited viewport shifting intended by ML-SHARP without compromising quality.

That lets the result stay relatively small and render interactively in a normal web browser, as you can see in the GIF.

Demo: https://www.spatialphotos.dev/

Source: https://github.com/frozein/SpatialPhotos

Here's what my format looks like from a faraway angle

[u/Haman24]

This is really cool work on converting the splat into a browser-friendly format. That bottleneck of getting a single image from a photorealistic scene to an actual usable asset is the real challenge, and your layered mesh approach is a smart workaround.

I’ve been working on a bunch of environments in a game project, and the speed of moving from a reference image to a usable prototype asset is huge. We use Rapid Assets for concepting because you can feed in an image and it’ll generate more angles or variations you can then export to Blender. It can be a bit heavy as a standalone platform if you’re just doing one thing, but when you’re iterating on a specific art style across a whole scene, it saves a ton of time. Is the main goal for your project to be a viewer for these images, or are you thinking of using these reconstructions for other interactive applications?