points by metalrain 1 day ago | 0 comments
It must be due the training data. YouTube videos don't have the metadata of all the objects and where they are/were.
Kind of makes me think that autonomous vehicles will eventually gather better dataset (although limited to roads!), since they do have 360 degree cameras and lidar, so they can track objects before & after they are shown on certain view.
The model already ingests text, so you just to attach a tiny VLM head that reviews the image and read / writes scene continuity text tokens.
Once you have that you can probably just RL with scoring on continuity issues and eventually the VLM head will learn to update dense continuity tokens?
So many fun wierd experiments to run if you have some compute to mess with. There are competent open video diffusion models that you could test this concept easily enough.