Human writers also use specific constructions for definite (though not always consciously known) reasons, and mapping those correlations onto writing models is only getting done on accident. This is a big part of why LLMs doing pointless negations (it's not X, it's Y) is so jarring: people use that construction in order to preempt an expected misconception, whereas LLMs try to use linear algebra over the learning corpus to generate it in appropriate quantities and places according to RLHF/fine-tuning/etc.

In other words, it isn't just verbiage that has a "preferred" quantity, but also higher-level language features, and the appropriate contexts for those features feel like the sort of thing that will be extraordinarily difficult to replicate without a theory of mind.

edit after coming back and re-reading: the "in other words," and rephrase is because I'm trying to explain something complex enough that I'm not even sure I fully understand it, so I try typing out multiple explanations and leave the ones that make the most sense to me. This might be idiosyncratic to my writing and not a significant feature of training corpuses, but even if it wasn't it's a second example of the sort of thing I'd suspect LLMs to be bad at using appropriately.

I vaguely remember a study that I was able to find when fact-checking this comment before publishing it [1], which found that music needs periods of high predictability to nurture listener expectations for the sake of later subverting those expectations, and the most enjoyable parts of music are those moments where expectations are subverted. This sounds like a similar effect.

1: PDF warning https://www.cell.com/current-biology/pdfExtended/S0960-9822(...

Gemini models are so, so much better in this regard. Their outputs are actually readable.

Interesting, I was thinking the walls of text are better for the watermarks.