points by montroser 16 hours ago | 0 comments
Yeah, it is very suspicious. Each time they update their ranking algorithm, the closed models from US labs jump up and any previously high ranking open model jumps down. It happened with MiMo, Kimi, GLM. All were right there amongst the very top, and each time that happened, AA followed quickly with an index update that shunted them dramatically lower.
Check out vals.ai for an alternative. Not sure how independent they really are either, but it's at least good for a cross reference.
[–] vessenes 14 hours ago
This could be another point informing your prior as to how much benchmaxxing and distilling of closed models is being performed by the open weights publishers as well.
[–] parsimo2010 16 hours ago