Hacker News

points by szvsw 3 days ago | 0 comments

How much effort do you put into prompting it? Are you asking it one shot with a simple prompt? Are you giving it to multiple agents and giving them a workspace to collaborate in? You might give it to a trusted colleague who does not know the solution along with an agreement to jointly publish, ask them to go through the process of solving it with agentic support, and then compare your solutions.

Edit: to be clear, I completely believe your experience - just pointing out that it might be interesting for you to actually “try” to solve it thoroughly with the tools, though it is obviously hard to do that in a fair way when you know the solution.

[–] Routine8502 3 days ago

Enough! I think it's a reasonably fair test of the models. Let's put it this way, I use the same process to resolve problems outside my peak expertise, and I find it works well.

I think the solution is easy enough that any competent colleague would come up with it. Some variant on it has probably even been published (hence why I'm not chomping at the bit to publish). I think it comes down to the models being extremely spiky, and the problems they're trying to solve being easily verifiable in code.

[–] Buttons840 3 days ago

Have you ever told the AI your answer? I bet if you did it would train on it and know it going forward.

[–] Routine8502 3 days ago

Nah. And I have an enterprise plan, so I'm /relatively/ confident that they aren't going to do that.

[–] eru 3 days ago

With my personal little test problem, that never made a difference.

However current frontier models got good enough at websearch and thinking through the problem, that they can finally piece together the answer from my own public writing (eg on stackoverflow). And recently someone wrote up an answer to the problem and put it on Arxiv.

[–] Buttons840 2 days ago

So, there's a very niche topic that I was generating some fiction about using AI, and as part of that I gave one of the characters an unusual name. A few months later I again experimented with using AI for this fictional niche, and the AI just randomly chooses to use the same unusual name.

"Forebears" is a site that ranks the popularity of names. They rank the name I used at 324,668.

This is why I believe the AI can be significantly influenced by a single person.

[–] eru 2 days ago

Did you use the same AI provider both times? They might have just looked it up in the memory.

[–] Buttons840 2 days ago

I cleared the memory. I frequently do so because I want to experience what the AI is capable of from a fresh slate. I alternate which AI I use monthly, paying for a month at a time. I clear out all memories and state, as best I can, at the beginning of every month.

I'm pretty sure the name has moved to different AIs somehow too. I've seen the name used on Grok, and also on NovelAI. I've forgotten the exact details.

[–] Routine8502 2 days ago

What did you select for data controls?