Hacker News
points by awb 3 days ago | 0 comments
> Create a realistic eval suite for your use case using your actual data that you manually label, with easier, more complex, rarer samples, and run each model you are considering against it and decide based on the usual confusion matrix, cost, latency, or whatever metrics matter to you.

I’d imagine there’s only a tiny percentage of users that can or want to do this.

[–] aacid 3 days ago
if you consider every single person that uses some LLM then sure, it is tiny percentage, but I would wage that large percentage don't write technical documentation as GP. large userbase probably don't need THE best solution for their use case and will be comfortable with A solution that is within their budget. so for them simple luna < terra < sol < astra would suffice.

but if you want to use some LLM agent for anything commercial and you want to get the best and at the same time you don't want to overpay the easiest and the surest would be to create your own benchmark for your specific use case.

Hacker News