For most objects, you access it directly and nothing weird happens, right?

Can you not mostly cache whether anything weird needs to happen and do the direct access with minimal overhead?

You said elsewhere "But that doesn't help much if/when you're doing two hash lookups per iteration just to find two small integers to add together." but looking up two fields isn't inherently a very big overhead. And you can cache the actual hash, and if you want to you can make sure the fields on similar objects have a consistent offset and cache that offset to make it even faster. This code using lots of objects is presumably not the core number crunching on arrays; we're doing more logic per access and we can handle a few instructions to get a value.