Hacker News
points by torben-friis 3 days ago | 0 comments
I think the main problem is surfacing two topics:

- Understanding how effort and class interact (Is a high effort terra above a low effort Sol? Are larger models mostly always better regardless of effort?)

- understanding when (if ever) a new model tends to be better than an older but higher class one. Is an old astra better than a new terra?

If increasing effort and increasing model class produce improved results in a different way, I miss making it explicit in which way. In other words: What kind of issue warrants going up in effort? what kind of issue warrants going up in model class? What kind of issue is unlikely to be solved by either?

[–] tedd4u 3 days ago
OpenAI itself cannot answer these questions. A prospective price-sensitive (or speed-sensitive) user has to experiment to find the cheapest (or fastest) model that is sufficient for each task.
[–] someguyiguess 3 days ago
Right. Because they don’t know. They’re just pumping out crap desperately trying to stop hemorrhaging billions.
[–] andai 3 days ago
>effort / performance curves by model

Oh, easy:

https://deepswe.datacurve.ai/

Artificial Analysis has their own curves which are pretty similar.

[–] irthomasthomas 3 days ago
That's good, but they haven't updated it for a month. It is missing all of the current frontier models.
[–] aftbit 3 days ago
Seems to be missing all of the 6.0 and 6.1 models?
[–] dist-epoch 3 days ago
> Is a high effort terra above a low effort Sol

You can literally see this in the charts from the article