[–] tedsanders 3 days ago
[–] mrandish 2 days ago
That's what users currently end up having to do on their own. Or they just give up in confusion, which presents in your site analytics as either bounce, buy later or buy less.
Why not have a model comparison site? Visitors choose from a dozen-ish broad categories of work (eg technical documentation, market communications, etc), then pick from a dozen sample prompts across typical tasks in that category, and see the model responses with avg time and approximate cost. Especially helpful would be having a toggle overlay with annotation highlighting relevant differences, then if they have (or create) an account, they can edit the annotations and add their own with a sharing link.
If my job was to actually help my co-workers and friends make better informed choices they feel more confident in, I'd be doing something like that. The extra credit version would help users narrow down to two models and then let them convert to a 'Test Drive' trial - same workflow but from their own prompts, side-by-side A/B response comparisons with annotation and sharing link.
As you said, the answer to "Which model is best for me?" is actually complicated, conditional and inevitably a bit confusing. When that's the reality, the right pattern isn't 'Communicate to Convert', it's 'Build credibility and confidence with maximum clarity and genuine helpfulness'.
[–] tedsanders 2 days ago
People vote on pairs of model responses and they aggregate everyone's votes into a leaderboard.
One thing to be aware of that the distribution of prompts on arena.ai can be pretty different than real-world usage. The people who spend time on that website casting votes tend to be more interested in entertainment than professional productivity. And turn-based voting can punish useful behavior like asking clarifying questions instead of guessing at assumptions.
[–] mrandish 2 days ago
[–] scarab92 3 days ago
It’s overkill most of the time, but frankly all these models are “cheap”, and I’d rather waste compute than brain cycles trying to discern which dials are best for each individual as-hoc query.
But to assess whether my preference on my problem is going to generalize across other people, other programming languages, other documentation types/formats, etc., and then analyze statistical significance is a thornier question that I don't have the data to answer. I'd rather make no recommendation than a weakly informed recommendation.