What's confusing to me in addition to the naming is that there are no published guidelines I can use, as far as I can see. Put yourself in the user's shoes. On what basis should one pick a model for a given task?
I can understand smaller models being dumber and faster. That aligns well with scenarios where you are willing to compromise; e.g. you're classifying content or asking a model to format text into a table.
But when you want "maximum intelligence, cost and time be damned", there's the implication that bigger models are smarter on a linear scale, and every provider's model release naming is currently conflicting with that expectation.
This isn't just a naming problem, but a communication problem. If I knew that Astra was the best at coding but a bit bad at writing, and Sol was almost as smart but better at technical writing, then that's the kind of thing I expect to be documented.
One possible way: A simple matrix with horizontal columns like "writing", "coding", "math", rows are the models, and each box is a number of stars from 1-5.
Instead, this sort of thing becomes lore, shared by word of mouth.
[–] tedsanders 3 days ago
[–] flockonus 3 days ago
[–] tedsanders 3 days ago
But to assess whether my preference on my problem is going to generalize across other people, other programming languages, other documentation types/formats, etc., and then analyze statistical significance is a thornier question that I don't have the data to answer. I'd rather make no recommendation than a weakly informed recommendation.
[–] mrandish 2 days ago
That's what users currently end up having to do on their own. Or they just give up in confusion, which presents in your site analytics as either bounce, buy later or buy less.
Why not have a model comparison site? Visitors choose from a dozen-ish broad categories of work (eg technical documentation, market communications, etc), then pick from a dozen sample prompts across typical tasks in that category, and see the model responses with avg time and approximate cost. Especially helpful would be having a toggle overlay with annotation highlighting relevant differences, then if they have (or create) an account, they can edit the annotations and add their own with a sharing link.
If my job was to actually help my co-workers and friends make better informed choices they feel more confident in, I'd be doing something like that. The extra credit version would help users narrow down to two models and then let them convert to a 'Test Drive' trial - same workflow but from their own prompts, side-by-side A/B response comparisons with annotation and sharing link.
As you said, the answer to "Which model is best for me?" is actually complicated, conditional and inevitably a bit confusing. When that's the reality, the right pattern isn't 'Communicate to Convert', it's 'Build credibility and confidence with maximum clarity and genuine helpfulness'.
[–] tedsanders 2 days ago
People vote on pairs of model responses and they aggregate everyone's votes into a leaderboard.
One thing to be aware of that the distribution of prompts on arena.ai can be pretty different than real-world usage. The people who spend time on that website casting votes tend to be more interested in entertainment than professional productivity. And turn-based voting can punish useful behavior like asking clarifying questions instead of guessing at assumptions.
[–] mrandish 2 days ago
[–] scarab92 3 days ago
It’s overkill most of the time, but frankly all these models are “cheap”, and I’d rather waste compute than brain cycles trying to discern which dials are best for each individual as-hoc query.
[–] selestify 3 days ago
[–] hsuduebc2 3 days ago
[–] junnan 3 days ago
[–] thirtygeo 3 days ago
[–] lazharichir 3 days ago
Create a realistic eval suite for your use case using your actual data that you manually label, with easier, more complex, rarer samples, and run each model you are considering against it and decide based on the usual confusion matrix, cost, latency, or whatever metrics matter to you. There's really no other way to know – and you might be surprised that cheaper models can often perform your task(s) relatively well at a fraction of the cost.
[–] awb 3 days ago
I’d imagine there’s only a tiny percentage of users that can or want to do this.
[–] aacid 3 days ago
but if you want to use some LLM agent for anything commercial and you want to get the best and at the same time you don't want to overpay the easiest and the surest would be to create your own benchmark for your specific use case.
[–] CSMastermind 3 days ago
Every model I've tried is just abysmal at this, for what it's worth.
If any do it well please let me know and I'll give it a spin.
[–] zenoprax 3 days ago
I recently experimented with some image generation models and I was surprised at how much context/prompting is needed to get something that doesn't look Lovecraftian. There was seemingly no downside to adding more and more detail up to the limit. Prose sits between code and images in terms of fuzziness and I wonder if promptmaxxing is the winning strategy for technical documentation.
I've been thinking that fine-tuning a model to write differently shouldn't be that hard (tedious, perhaps) and if I had the resources I would consider doing it myself.
[–] ilaksh 3 days ago
[–] no-name-here 3 days ago
It's high-level (and doesn't break out coding/writing etc), but https://developers.openai.com/api/docs/models says:
> If you're not sure where to start, use GPT-6 Astra, our flagship model for complex reasoning and coding. Choose GPT-6.1 Sol to balance intelligence and cost, or GPT-6 Luna for cost-sensitive, high-volume workloads.
That page also seems to be updated after each new release.
[–] saagarjha 3 days ago
[–] atombender 3 days ago
I do not use agents to write on behalf of myself. I write my won technical write-ups and my own slide presentations. A lot of people don't; and it shows. There's much that agents can do better than humans now, but not everything.
I don't know, to be honest. Technical documentation is such a multidimensional skill, and I also don't know your budget. I'd imagine Terra might have the most readable prose, but Astra might be more likely to catch bugs or mistakes to highlight. If the documentation is about a more esoteric language or situation, Astra might have a greater edge than if it's a common language or situation. If you have a long list of documentation guidelines, that might also give more of an edge to Astra. I wouldn't want to make a recommendation without doing a lot of testing, and I haven't done a lot of testing.