Hacker News

points by fallingbananna 3 days ago | 0 comments

I would guess only on the nitpicked benchmarks?

The pelicans place it nicely between Sol and Luna in terms of both pricing and results.

[–] vdfs 3 days ago

pelican is also nitpicked (and probably the most useless)

[–] toxik 3 days ago

Cherry-picked is the term, not nitpicked.

[–] SomeonesAccount 3 days ago

Yes, it may be dumb, but it isn't useless because it is both a) is a benchmark and b) is not picked by openai

[–] vdfs 3 days ago

IMO it's useless because we don't even know what a pelican on a bike is supposed to look like, It's very impressive that that models can make these directly using SVG, but we are way past the "wow" stage and test that can't be gamed and are easy to prove how good the model is in that specific arena are hard to find.

The only thing that pelican have is "models where not trained on this" maybe that true months (or years ago) now it's on every HN post and these AI labs are monitoring HN