Hacker News

points by vintermann 16 hours ago | 0 comments

I tried Argon for a large historical HTR transcription project recently, since I heard it was really good at this. It was really good, but like all top US models I've used it was

* Definitively very good at domain specific knowledge. It can read sloppy 19th century danish gothic handwriting with faded ink just fine. Not perfectly, but probably better than I could have done with a day of struggle for every page.

* Expensive.

* Bad at actually doing what I said. I had told it in the prompt to be careful not to miss text, but I hadn't explicitly told it that some images may have text in double-column layout. So on those images - over half of my 300-page document - it produced an excellent transcription of the first column, and then considered itself done. Want to try again with a more careful prompt? That'll be another $20.

[–] sgc 15 hours ago

For ocr, you have to practice, practice, practice with your prompts on small sample sizes before one-shotting a book. I create a json schema for the output that describes the structure in detail (using ai from my prose descriptions of structure of course). I have specific tough pages I use on openrouter to test every new model I am interested in. Gemini has and does absolutely dominate for this. But flash can be better because it doesn't overthink things.

[–] vintermann 15 hours ago

I did do test images, but apparently not enough. It's not 100% of images it drops the second column on either. On many it gets both, on some it only gets the second column.

I also found that it ignores other instructions sporadically ("Don't use letter variants or ligatures in the output, use the normal Danish/Norwegian alphabet when at all possible", still a few Unicode tall s'es in the output.)

It's good, but it's so frustrating since "worse" models manage to follow such instructions much better.

[–] TheAceOfHearts 11 hours ago

I don't know how useful this reply will be since I haven't tried Argon specifically, but with other models I've found that the best strategy for work that is shaped this way is to have the model process it in chunks and iteratively verify and build up the rules for processing. So you have it process 5 to 10 pages at a time and self-verify it's output before continuing. Every lesson gets propagated forward to the next chunk.

To get the best performance you really need to present your problem as an iterative verification loop.

[–] SecretDreams 15 hours ago

> That'll be another $20.

That seems arguably pretty cheap compared to the human alternatives.

I do get super annoyed at how specific you've got to be with these tools. They are like almost like malicious compliance geniuses lol.

[–] disgruntledphd2 14 hours ago

> They are like almost like malicious compliance geniuses lol.

They're like a monkey's paw or evil genie I think.

You'll get exactly what you ask for, but lots of weirdness you didn't expect.

To be fair to the model makers, deciding what to do with an under-specified prompt is hard, particularly in an RL environment where you can't just ask the user.

For stuff I want to run for a while, I'm defaulting to telling it to ask me the questions upfront to reduce the incidence of these kinds of problems.