Object permancence is apparently a task so difficult that even our own brains and those of many species of animals struggle with it. Maybe the problem domain lies in the fact that compared to a world model, our world is continous regardless of the observer's capabilities. If a tree falls in a forest with no one there to see it, it still falls. The same isn't true in a world model's view.