TTMark swaps single-token watermarking for adjacent-token pairs, trying to squeeze signal out of low-entropy text.
3 comments
This feels like a fairly direct extension of the Kirchenbauer et al. 2023 LLM watermark, just moving the signal from single tokens into adjacent pairs so you can keep some leverage when the marginal entropy is tiny. The pairwise view is nice, but it also smells a bit like old distribution-preserving steganography work, Hopper, Langford, and von Ahn had most of the relevant shape already back in 2002/2009 (depending which version people remember), where the whole game is hiding in the correlations instead of the alphabet.
I keep wondering how much of the win here is real capacity versus just shifting the detector to where the model already has a strong bigram prior. If the text is low-entropy enough, the detector may end up learning the model’s quirks more than a watermark, which is the same old problem in a different costume ...
Does the overlap between pairs make the detector much more brittle to a single edit or resegmentation, or does the branch-isolating generation somehow keep that from falling apart once the text gets touched?
An edit near one token can knock out two adjacent pairs, so yeah it seems more brittle to single-token changes than the usual one-token watermark, and any resegmentation would be even worse unless the detector is doing its own sliding alignment after the fact.