Hacker News
points by anakaiti 2 days ago | 0 comments
The drop for K3.1 Pro & MiMo V2.6 Pro is quite surprising. Are they benchmaxxed?
[–] adg001 2 days ago
Or, are we witnessing reverse-benchmaxxing? Meaning that first some entities let players optimise for a given benchmark, then they update the measurement methodology so as to retain some players on the top ranks, while dropping others. As former Italian prime minister, and highly controversial public figure, Giulio Andreotti once remarked "Thinking the worst is a sin, but you’re often right."
[–] gobdovan 2 days ago
Did Italy ever had a PM that was not controversial?
[–] adg001 2 days ago
You are neither funny, nor well informed.
[–] gobdovan 1 day ago
Italy has a pretty rich history of controversial PMs. Pre-WW2 Crispi, literally Musolini, recents Berlusconi, Andreotti, Craxi. Don't think I know any other foreign country by how controversial so many of their leaders were.
[–] prodigycorp 2 days ago
Transparently so. MiMo's training dashboard showed the DeepSWE hillclimb in real time.
[–] throwa356262 1 day ago
That's not benchmaxing.

You need a method to show you RL is improving the model that is separate from the training data.

[–] simianwords 2 days ago
I keep seeing that MiMo is in the frontier in terms of performance and cost but I see very few talking about it and the only ones that do talk about it are doing it in context of AA
[–] brainwad 2 days ago
It's also dog slow at AA index tasks compared to other models in its class. Makes me think they optimised for getting on the intelligence/cost frontier at the cost of real world usability.
[–] simianwords 2 days ago
I’d bet on this
[–] wyre 2 days ago
I used MiMo for a couple of days waiting for my Codex sub to reset. I really liked it, its communication style was a breath of fresh air compared to how stale gpt-sol is.