Hacker News

points by parsimo2010 16 hours ago | 0 comments

Just so everyone is aware of the "coincidences" that happen with this index:

- Two days ago AA announced their v5 Index which added a few new benchmarks and tweaked a few things.

- Today the newest model announced from a big lab tops the charts.

I'm not saying that AA is definitely taking money from the big labs and changing the benchmark to make the labs' new models look better, but it kind of looks that way. This pattern of behavior has happened before, where the index gets tweaked right around when a new model comes out, and the new model is on top. AA does not, AFAIK, provide a way to see how the new models would have scored under the older index versions. You could probably calculate it yourself, but I'm not going to do that much legwork just for an HN comment.

It's interesting because they publish the index methodology, so you can't really accuse them of not being transparent, but I don't really believe that the choices to update the index were completely independent of the model they happened to be evaluating at the time.

Related: The DXOMark camera scoring method also had similar behavior, where they would release new testing methodology right before flagship smartphones were released. Again, they are technically being transparent about the process, but it kind of smells like the "independent" testers may not be as independent as the average user would hope.

[–] rbehrends 16 hours ago

I had a look at the changes and the biggest impact seems to be from adding Terminal-Bench-Science 0.1 and having it contribute 15% to the overall score.

Example problems are:

- https://hub.harborframework.com/tasks/terminal-bench-science...

I have no doubt that these are very relevant task for people in those respective fields, but I suspect they have next to no relevance to some 99.9% of model users. Hence why especially smaller labs with smaller models didn't really train for such use cases and you can see it.

Personally, it's not something that matters at all to me, as I don't work in cheminformatics or bioinformatics.

This is also arguably more of a test of a model's knowledge base than reasoning ability.

But most importantly, there is also the question to which extent the aggregate score then remains a meaningful result when you mix up too many too disparate benchmarks.

The coding agents subindex, in contrast, looks at least superficially plausible [1].

[1] https://artificialanalysis.ai/agents/coding-agents#coding-ag...

[–] WarmWash 15 hours ago

Ironically coding has no relevance to something like 95% of users (I forget the exact OAI statistic), but it is the mainstay LLM benchmark.

[–] rbehrends 15 hours ago

True, but coding is where something like 80%-90% of their revenue comes from; few other use cases are as token-hungry, and none of those are nearly as widespread. Many users will of course use the models just as chatbots, but they don't generate much in terms of revenue. And that is ultimately what determines where labs focus their resources.

[–] montroser 16 hours ago

Yeah, it is very suspicious. Each time they update their ranking algorithm, the closed models from US labs jump up and any previously high ranking open model jumps down. It happened with MiMo, Kimi, GLM. All were right there amongst the very top, and each time that happened, AA followed quickly with an index update that shunted them dramatically lower.

Check out vals.ai for an alternative. Not sure how independent they really are either, but it's at least good for a cross reference.

[–] parsimo2010 16 hours ago

I also look at the ECI from Epoch.ai as well, but they haven’t evaluated Carbon yet. Probably because the model isn’t widely available yet.

[–] vessenes 14 hours ago

This could be another point informing your prior as to how much benchmaxxing and distilling of closed models is being performed by the open weights publishers as well.

[–] dist-epoch 16 hours ago

Have you considered the other direction? They test a new model, and realize their current index is broken, so they improve it?

At the previous iterations they were accused of favoring OpenAI. Now they are accused of favoring Google.

[–] parsimo2010 16 hours ago

We’re literally talking about the same thing, but ascribing different motivations.

I do think the test a new model and realize their index is broken- the new model isn’t on top, so they change the weights to make sure it’s on top.

The difference is that you seem to think they are doing so out of some innocent feeling that the new model is actually better, while I think they are doing so because they were paid to do so. I think my interpretation is more likely, but that’s just me.

[–] dist-epoch 16 hours ago

So explain why the Chinese are not paying to put their model on top, surely that would drive a shit ton of business to them.

Or are you implying that the Chinese can't afford the amounts paid by OpenAI/Google.

[–] kennywinker 14 hours ago

Can’t afford isn’t the only option. Unwilling to pay the bribe, or not invited to the party are also possible reasons.

[–] pu_pe 16 hours ago

And how do they realize their current index is broken? They either have some recency bias, or indeed favor the big American labs (more likely, since a new Chinese model topping the charts triggers a shakeup).

[–] teiferer 16 hours ago

It's odd because I wonder if anybody actually cares about such scores. It's not like millions of end users look at the score and use it to make a decision what to buy. Subscriptions can easiliy be canceled and people anyway use the models they "feel" work best for them. Hardly anybody I know uses an even remotely scientific method to evaluate model performance. It's mostly vibes with a little bit of (mostly flawed) A/B testing sprinkled in.

So ... why bother to cheat?

[–] InsideOutSanta 16 hours ago

> So ... why bother to cheat?

I think part of it is to justify capex. "Yeah, we spent x billions on this, but look, we have the best models, we're beating OpenAI and Anthropic, we're winning, it's all worth it!"

[–] conception 7 hours ago

I’m sure carbon is great but also Google stock just took a pretty big hit so…

[–] dist-epoch 16 hours ago

From all of the terrible benchmarks of AIs, the AA one is the least terrible.

[–] kennywinker 14 hours ago

Only because it aggregates all the other benchmarks. It still feels off. It felt off before the last two re-rankings, and it still feels off. Probably because the benchmarks its based on aren’t measuring the right things.

[–] teiferer 14 hours ago

Not questioning that it's off/biased/..., but what are you basing your "feel" on? Anything empirical or really just a feel?

[–] kennywinker 14 hours ago

Entirely vibes based for the AA rankings.

For specific benchmarks, I’ve downloaded and used basically every model out there under 35b parameters - especially the tiny models: ling-3.0-tiny, spark-x2.6-4b, nanbeige, etc. etc. some of those tiny models are ranked VERY high on benchmarks like deepswe or terminalbench but are obviously incapable when you try to use them for swe or terminal based tasks.

If that’s benchmaxxing or a bad benchmark, idk, but it’s made me not trust the benchmarks much. If it benchmarks poorly, it’s probably bad. If it benchmarks well, it might be good.

[–] vessenes 14 hours ago

I don't understand this conspiracy/suspicion vibe around AI benchmarks -- the best benchmarks get maxed quickly in today's world, especially if the datasets are open. We're in a situation where the AA's of the world show 0-1 point difference between models, so by definition they are NOT good discriminators. These tests NEED improvement.

Do they work with frontier labs? Probably. They should work with them at least a little so that we can get an independent read on the model quality. Do they inject new tests that specifically favor one particular lab? Clearly no, as different labs have moved up over time. Do they inject new tests that help to provide some separation on scoring? God, I hope so.

Nota bene, another change they announced was dramatically increasing the amount of held back test data; this would be a simple explanation for seeing open weights models, all of which are known to distill from frontier labs and benchmarks, drop in the rankings.

[–] applicative 16 hours ago

My paranoid impression was more that the updating and addition of tests — obviously essential in this game — was delayed to provide advertisement for Kimi K3.1. The same happened for K3 — and this is why it was famous — but its position had collapsed by the time of 3.1