Replica: Room2Dream

Hey guys, I'm Cyril and this is replica.

Essentially what we built is a radio to three dimensional foundational model that allows you to go ahead and take video and create it into a monocular depth estimation using vanilla Dino V2 encoder and then takes that and puts it in a Cartesian frame for you to visualize through the part that is being mapped.

The reason why is there's not much context for robotics.

So we want to train and collect data for robotics which enable others to label, use or actually help with implementation of robotic situations.

That being said, here's a quick video which I'm going to walk you through.

As you can see, this is a video of the hack app Conduct, and you can just walk through and see the different spacing conductor.

I'm going to scrub through because it takes a while and essentially we take this one shot video, with no reference frames, just video and we create a three dimensional pose and representation in point cloud or Gaussian splat and then we label it.

The reason why it's valuable is because you can have environments and spaces that typically don't have particular use cases and implement them with robotic uses and applications.

One of them being Gemini Robotics.

Asimov with the benchmark and using gemini to provide 3D spatial understanding.

The whole point was to augment it.

I wasn't able to finish implementation with Gemini, but we used Modal and we also used Pydantic to provide live evals as well as live AI to kind of debug on the fly and figure out what they were things were wrong.

Your training as well as your inference and what happens in terms of streaming the data if you have any asynchronous problems or any debug problems in that context.

this is the output that came out.

So there's a map point clouds as well as also you can also view as a splat or Gaussian splat.

This is the table, this is the chair.

You kind of see these points, then we play them through Rerun.

Rerun is a browser based render that allows you to render three things 3D 3 dimension.

Here you can kind of see different points from the map as you're going through.

It gives you a depth estimation of each point.

Right.

So if you see depth estimation you can click on those points and provide you estimation.

You see that of the image in real time.

And then we go to Pydantic.

Pydantic will give us log file specifically where the error was in terms of running our eval.

So we're running eval specifically how well the model trained or how fast it was.

And actually trained pretty fast.

It took about a minute 50 to training and to actually run a model.

And you can see provided the weights but deploying the weights had an error.

So I could go in and debug, figure out what keyboard interrupt was for the model.

and then you see batch jobs.

It batches each job and runs in Rerun.

So you can actually view real time.

And it cost 10 cents.

That's how much the computer was to kind of do a map of the environment.

Our use cases for world models.

So we can train multiple world models and over time you can build really really fast single world models from just one camera or just one phone.

So you can just have a video online.

You can train it and it creates a 3D representation of your world model.

For robotics.

and one thing we started using was AZIMOR benchmarks and we didn't have time to finish training and Gemini Robotics to create 3D representations Scene of the environment.

but the next time we hope we can implement and publish the 3D segmentation as well as also semantics for the object and scene in the environment.