points by Gareth321 2 days ago | 0 comments
Benchmaxing is defined as targeted post-training attempts to optimise for higher benchmarks. This doesn’t change the aggregate intelligence or capability of the model, so it can make a model appear more capable than it is.
[–] ozgung 2 days ago
[–] XenophileJKO 2 days ago
So I assume it depends on what you train.
- Exact problems on benchmark.
- Similar problems to benchmark.
- Broad class of problems or techniques that might apply to benchmark
Each has very different ramifications for how the model performs. The current evidence is that Anthropic and OpenAI is generally operating lower in the list.
One can argue though some models are post-trained heavily for specific benchmarks and less for other datasets. So they are weaker on out of distribution tasks.