points by bmitc 3 days ago | 0 comments
How are people getting these results? Are they correct?
I use these latest available public models daily in max effort modes, and they barely seem to work and need extreme guidance and as much context and constraints as possible.
[–] robswc 3 days ago
[–] bmitc 2 days ago
> I suppose I could also sit there and spend 9 hours writing out a "perfect spec" but we've already tried that with waterfall development!
And a perfect (or near-perfect) spec is actual source code. And that's where I see myself with LLMs. They are great about generating source code. They are terrible at writing well-specified software, and so they need a lot of context: docs, plans, existing source code, existing templates or examples. And even then, they still need a lot of guidance.
[–] nonethewiser 3 days ago
1. This has not been my experience
2. We dont have the actual frontier models
[–] bmitc 2 days ago
Then what are you doing different if this hasn't been your experience? You're just throwing AI at things you don't know how to do, and it does those things successfully?
[–] tintor 3 days ago
internal unpublished models + large swarm of agents (>2000)
I find that unless the outcome I'm looking for from the model is almost binary and easily verifiable, it is basically faster to do the work myself.
Assumptions made by the models multiply and by the non-deterministic nature (dbf, similar to work by humans) I'll have something end up light years away from where it should be. During a normal workflow, I have to contend with every line I write forcing a very tight feedback loop... I suppose I could also sit there and spend 9 hours writing out a "perfect spec" but we've already tried that with waterfall development!
So, not sure if its the models, rather the class of problems I'm applying them to... but it is a bit ironic they are allegedly able to solve these conjectures but not stop to ask about assumptions.