One of the new benchmarks is FrontierCode 2.0 by Cognition. AA seems to have accepted Cognition's benchmark scores without questioning, but they should have looked more closely.
On this benchmark, Anthropic's models are run using Claude Code; OpenAI's using Codex, and Gemini is run using... Cognition's own Devin (not Antigravity). Thus its score is much lower than it should be, knocking ~3 points off its AA score. All open-source models are also inexplicably run using Devin, damaging their scores as well.
This may even be intentional, if Cognition has beef with Google, since Antigravity was built using Windsurf's technology. But aside from that, it is to their advantage to make Anthropic/OAI look like leaders in the AI market. They are a company known to lie and exaggerate again and again, so I have trouble giving them benefit of the doubt.
[–] shelfchair 2 days ago
[–] thereitgoes456 2 days ago
It's interesting that the Stirrup scores are somewhat close to the specialized harness scores. Either Stirrup is quite excellent or is it that harness doesn't matter so much after a point?
[–] zurfer 2 days ago
I would think it's the opposite as a more competitive frontier for llms makes the labs shed some margins and enable a broader ecosystem.
[–] thereitgoes456 2 days ago
Additionally: competitive frontier = more commodified = high valuations under more scrutiny.
[1] https://artificialanalysis.ai/evaluations/frontiercode-2-0