points by rao-v 2 days ago | 0 comments
I don’t understand why these world models aren’t designed to be backed by an updatable world state. It could probably just lay out the world onetime and reference it as needed vs trying to invent from wholecloth via diffusion. Object persistence!
[–] metalrain 1 day ago
[–] rao-v 1 day ago
I’d do this as an RL post training step maybe…
The model already ingests text, so you just to attach a tiny VLM head that reviews the image and read / writes scene continuity text tokens.
Once you have that you can probably just RL with scoring on continuity issues and eventually the VLM head will learn to update dense continuity tokens?
So many fun wierd experiments to run if you have some compute to mess with. There are competent open video diffusion models that you could test this concept easily enough.
Kind of makes me think that autonomous vehicles will eventually gather better dataset (although limited to roads!), since they do have 360 degree cameras and lidar, so they can track objects before & after they are shown on certain view.