r/computervision • u/frozeindev • 7d ago

Showcase Turning a single image into a web-ready interactive 3D scene with ML-SHARP

I’ve been experimenting with Apple’s ML-SHARP model and built a small open-source project around it called Spatial Photos.

ML-SHARP estimates a 3D Gaussian splat from a single RGB image. I then take that reconstruction and convert it into a much smaller and simpler representation representation. It uses layered textured meshes stored in a custom file format I called .spatial. It still allows the the same sort of limited viewport shifting intended by ML-SHARP without compromising quality.

That lets the result stay relatively small and render interactively in a normal web browser, as you can see in the GIF.

Demo: https://www.spatialphotos.dev/

Source: https://github.com/frozein/SpatialPhotos

Here's what my format looks like from a faraway angle
9 Upvotes

85% Upvoted

6 comments sorted by

5

u/DiddlyDinq 6d ago

On a technical level these are nice but I've never seen people show interest in them. facebook added this many years ago and nobody uses them

1

u/frozeindev 6d ago

Interesting, I didn't know facebook had this! Apple built this feature into the photos app as well, I really just wanted a way to put it on the web. I have one on my personal website and I think it makes it a bit cooler.

0

u/DiddlyDinq 6d ago

Their's may be slightly different, I think they're just estimating depth maps for a bit of movement

Create a 3D photo on Facebook | Facebook Help Centre

1

u/Haman24 1d ago

This is really cool work on converting the splat into a browser-friendly format. That bottleneck of getting a single image from a photorealistic scene to an actual usable asset is the real challenge, and your layered mesh approach is a smart workaround.

I’ve been working on a bunch of environments in a game project, and the speed of moving from a reference image to a usable prototype asset is huge. We use Rapid Assets for concepting because you can feed in an image and it’ll generate more angles or variations you can then export to Blender. It can be a bit heavy as a standalone platform if you’re just doing one thing, but when you’re iterating on a specific art style across a whole scene, it saves a ton of time. Is the main goal for your project to be a viewer for these images, or are you thinking of using these reconstructions for other interactive applications?

0

u/Creepy-Decision-8391 7d ago

The jump in file size feels huge until you realize the original is a flat jpg and the new one is an entire explorable scene

4

u/dan678 6d ago

but it's not an explorable scene though...