GLM 5.3-flash: N/A

https://artificialanalysis.ai/models/glm-5-3-flash/

Qwen 3.8-flash-next: N/A

https://artificialanalysis.ai/models/qwen3-8-flash-next

An issue I think all benchmarks suffer from is regarding precision. What does a one point difference mean? How many points of difference is needed to translate into a meaningful distinction between models?

Among fellow humans, we often find ourselves picking what we consider to be most intelligent and then designing the tests around that. I've noticed a similar thing in AI, where whatever OAI/Anthropic decide to focus on improving becomes the new measure of intelligence. With their latest models, this seems to be 3d modeling/game making capabilities. Unsurprisingly, because that isn't what the Chinese companies have been focusing on, they appear behind on "frontier" reasoning tests. On top of that, older tests end up being discarded for being saturated. Makes me think that maybe unintentionally, the methodology used for these scores might be biased towards Western companies.

This would also fit with the widely discussed decline in writing quality from Opus and the worse readability of code produced by one of OAI's recent models, despite benchmarks continuing to show progress on all fronts.

Plus, if they're going to update the benchmark so frequently to discard saturated components, is there even a research-related point to keeping it private?

This is really sus. I’d love a more detailed explanation and “take” from AA on what problem they felt needed correcting and why things have shuffled the way they have. And how they can claim their index has relevance if positions can be switched around so easily.

Why are they using a closed source LLM as a judge? Closed source providers can see your traffic and adjust as they see fit.

The current placement of many models simply doesn't makes sense, GLM 5.3 is so much better than Astra at backend tasks, Astra does the most weird nonsensical bugs.

This is what happens when the benchmark team is funded by VC capital.

That seems improbable

My harness consists of multi-level AGENTS.md and SOUL.md containing "grug brain developer" and "asd-ste100" type of speech for clearer communication.

Kimi K3.1 Pro: −9

DeepSeek V4.1 Flash 1017: −13

GLM-5.3: −13

MiMo-V2.6-Pro: −16

Proprietary models

GPT-6 Astra: +4

GPT-6.1 Sol: +3

GPT-6.1 Terra: +3

Claude Opus 5.5: 0

Claude Sonnet 5.5: −2

Claude Fable 5.1: −3

Gemini 4 Argon (high): −3

Muse Wonder Full: −6

Claude Haiku 5.5: −7

Muse Wonder: −9

Gemini 3.8 Flash (high): −10

Grok 4.7 (high): −11

Is there affiliation with OpenAI or coincidence?

One of the new benchmarks is FrontierCode 2.0 by Cognition. AA seems to have accepted Cognition's benchmark scores without questioning, but they should have looked more closely.

On this benchmark, Anthropic's models are run using Claude Code; OpenAI's using Codex, and Gemini is run using... Cognition's own Devin (not Antigravity). Thus its score is much lower than it should be, knocking ~3 points off its AA score. All open-source models are also inexplicably run using Devin, damaging their scores as well.

This may even be intentional, if Cognition has beef with Google, since Antigravity was built using Windsurf's technology. But aside from that, it is to their advantage to make Anthropic/OAI look like leaders in the AI market. They are a company known to lie and exaggerate again and again, so I have trouble giving them benefit of the doubt.

1. Is there a benchmark measuring how much better an LLM performs when you tell it "you are an X"?

2. Does telling an AI agent "you are an AI agent" actually help? Is it the LLM equivalent of reminding a senior developer they're a senior developer, making them feel guilty about being lazy and finally getting them to do their job?

3. What happens when the persona conflicts with the task? If you tell it "you are a PM" and then ask it to write code, does it struggle? Produce worse code? Spawn a sub-agent and tell it they're a senior developer?

[1] https://artificialanalysis.ai/methodology/intelligence-bench...

As long as that's the case, it's no surprise that there is often a perceived difference of quality between benchmarks and real-world use.

I know that running the full composite benchmark set is probably already quite costly, and also that the benchmarks have to be rotated regularly, but it feels like a significant untackled problem.

AAv5.0: 25% Knowledge work: AA-Briefcase v1.1 - 15% AutomationBench-AA - 5% GDP.pdf - 5% (-5%) 25% Coding: FrontierCode 2.0 - 15% (new) Terminal-Bench4.0 - 10% 25% Science: Terminal-Bench Science .1- 15% (new) CritPt v2 (hard subset) - 10% (new version) 25% General: AA-Omniscience v1.1 - 15% HLE-Diamond - 10%

AAv4.3: 30% Agents: AA-Briefcase v1.1 - 15% GDPval-AA v2 - 10% AutomationBench-AA - 5% 20% Coding: Terminal-Bench4.0 - 10% SciCode - 10% 30% General: AA-Omniscience (Accuracy) - 10% AA-Omniscience (Non-hallucination) - 5% GDP.pdf - 10% AA-LCR v1.1 - 5% 20% Scientific Reasoning: HLE - 10% CritPt 10%

tldr: gained Terminal-Bench Science and FrontierCode, dropped SciCode, GDPval-AA and AA-LCR. Lowered weight of GDP.pdf.

Will an LLM topping this bench be the end of humanity? The doomer rhetoric has no place in what should be an objective measure.

They perform pretty badly on the new benchmarks introduced.

The score hit on smaller models is a welcome change especially when compared to models that nearly 10 times larger.

Grok being punished quite a lot is also a very welcome change as the model is frankly not that good.

I would love to see few more points knocked off haiku by introducing steerability/creativity benchmarks.

One thing I’m still surprised / unsure on is haiku 5.5 max besting latest deepseek - but also I think the last haiku was so incredibly bad I still hold some bias whilst using the new one.