points by teiferer 15 hours ago | 0 comments
It's odd because I wonder if anybody actually cares about such scores. It's not like millions of end users look at the score and use it to make a decision what to buy. Subscriptions can easiliy be canceled and people anyway use the models they "feel" work best for them. Hardly anybody I know uses an even remotely scientific method to evaluate model performance. It's mostly vibes with a little bit of (mostly flawed) A/B testing sprinkled in.
So ... why bother to cheat?
[–] InsideOutSanta 15 hours ago
[–] conception 6 hours ago
I’m sure carbon is great but also Google stock just took a pretty big hit so…
[–] dist-epoch 14 hours ago
From all of the terrible benchmarks of AIs, the AA one is the least terrible.
[–] kennywinker 13 hours ago
Only because it aggregates all the other benchmarks. It still feels off. It felt off before the last two re-rankings, and it still feels off. Probably because the benchmarks its based on aren’t measuring the right things.
[–] teiferer 13 hours ago
Not questioning that it's off/biased/..., but what are you basing your "feel" on? Anything empirical or really just a feel?
[–] kennywinker 13 hours ago
Entirely vibes based for the AA rankings.
For specific benchmarks, I’ve downloaded and used basically every model out there under 35b parameters - especially the tiny models: ling-3.0-tiny, spark-x2.6-4b, nanbeige, etc. etc. some of those tiny models are ranked VERY high on benchmarks like deepswe or terminalbench but are obviously incapable when you try to use them for swe or terminal based tasks.
If that’s benchmaxxing or a bad benchmark, idk, but it’s made me not trust the benchmarks much. If it benchmarks poorly, it’s probably bad. If it benchmarks well, it might be good.
I think part of it is to justify capex. "Yeah, we spent x billions on this, but look, we have the best models, we're beating OpenAI and Anthropic, we're winning, it's all worth it!"