points by oh_no 12 hours ago | 0 comments
> this full release warrants a major reassessment.
I really don't think it does, the benchmark numbers are dreadful and the cost is higher than Sol. If you have to use an EU model knock yourself out but there are a ton of models that are a lot smarter for a lot less money.
[–] jacquesm 11 hours ago
[–] killingtime74 5 hours ago
[–] jacquesm 5 hours ago
I run my own benchmarks (this model is - unfortunately too large so that won't be happening here) on data that no outsider has ever seen and this gives sometimes results that are not in line with the order that you would expect if you took the various benchmark sites into account. It is of course entirely possible that our benchmark is not representative but it seems to work quite well as a predictor for which models have value for us and which do not. Recently we looked a model that has been pushed with a significant amount of hype and for the life of me we could not get it to perform even close to the published benchmarks, whereas the models we used for controls were definitely within the margin of error. It's not always equally cut-and-dry.
[–] taneq 9 hours ago