Hacker News

points by vikramkr 4 days ago | 0 comments

Tokens per second? That only really matters for things like chat and sessions where you are monitoring a session as it runs which is going to be an increasingly small percentage of usage. That's bottlenecked by human attention. Agents in the background, long running tasks, unattended tasks, all that stuff is what is even capable of driving significant usage because you can just increase consumption massively by parallelizing without being the bottleneck. I wouldn't be shocked if already the vast majority of token usage was from unattended sessions. TPS is not that important there

[–] imtringued 3 days ago

You're ignoring the fact that a waiting human costs money which is actually the primary reason why AI hasn't resulted in significant productivity increases, while overly focusing on the human being the bottleneck. The point isn't to monitor the session, the point is to verify the output with human inspection. If the agent is fast enough, attending to the agent can make financial sense.

Human memory has a finite capacity and decays over time. If you do many inspection tasks in parallel you are splitting that finite memory. If you do the same tasks sequentially, the human can give instant feedback to the model without costly context switching. This in turn increases iteration speed and reduces total labor costs. This is no different from developers preferring JavaScript or Python for immediate feedback and fast iterations.

[–] cactusplant7374 4 days ago

It is impossible to make sophisticated software products at a high level with a low TPS. Try making a modern emulator or an award winning game. Both require a lot of iteration and not everything works well with multi agent.