Hacker News

points by satvikpendem 14 hours ago | 0 comments

Artificial Analysis is not trustworthy anymore. Every time an open-weight model, especially Chinese, looks to be nearing the top, AA updates the index to (ostensibly) specifically push closed models higher. Just recently with v5, OpenAI models were pushed up.

[–] cbg0 14 hours ago

Or it could be that the benchmaxxed models are getting pushed down by the introduction of unsaturated benchmarks. Muse Wonder also went down 52 -> 43 after the update.

[–] asenna 14 hours ago

I don't think that's the right read. Deepseek Flash 4.1 1017 did not get affected much.

I think what's happening is Chines models genuinely try to benchmax on it and hence they suffer when a new version comes out.

[–] disgruntledphd2 14 hours ago

> I think what's happening is Chines models genuinely try to benchmax on it and hence they suffer when a new version comes out.

I read somewhere on Substack that a lot of this was driven by the inclusion of more knowledge work benchmarks, where the US labs are stronger, versus coding which is where most Chinese models appear to put their energies for now.

[–] jbellis 14 hours ago

It's not that hard to create your own uncontaminated eval for the problems you care about, and compare.

Here's mine for coding in Rust, and I definitely see the same pattern of "most chinese models do much worse than on public evals that have been around long enough to train on" that AA v5 surfaces: https://slopcop.com/power-ranking