It goes way back, I work in AI/Ml (taken over by transformers/LLMs) space since early days of this "AI Summer". We get some nice grad school students interning from top universities. I met one during a trip to the office, from top notch school, introduced me to Triton programming when it first came out. It was early days of LLaMA just coming, when The Bloke was regularly dropping finetunes, I was joking asking him what the hacker stack was for "AI waifus" that the cool kids were using. Lo and behold he drops the name of the SillyTavern project. Which later was really educational in setting up local inference servers and in other ways, may have had to, ahem, delete some browser history.

what are some of the vagaries of how that use case depends on the agent aka harness? i’ve only been focused on sort of how to make models reliable for my math / cs / engineering and arts and crafts work loads, but RP is an important category for some folks. like are the fails of the various models different from more technical yet creative work loads?

eg im working on rigorizing a proof that a fun algorithm of mine is not only the first provably efficient algorithm for a particular problem, but its also the fastest in practice. ive found that for the sort of stuff i need to assume the model will get confused in the out of training distribution stuff from my personal library of ideas, so they will at best be a confused secretary intern mashup