points by shelfchair 2 days ago | 0 comments
I don't think this is relevant on AA's FrontierCode score. AA runs FrontierCode 2.0 benchmark using their own Stirrup agent harness [1].
[1] https://artificialanalysis.ai/evaluations/frontiercode-2-0
[1] https://artificialanalysis.ai/evaluations/frontiercode-2-0
It's interesting that the Stirrup scores are somewhat close to the specialized harness scores. Either Stirrup is quite excellent or is it that harness doesn't matter so much after a point?