AMS uses activation geometry to spot safety-training tampering in LMs, until activation-only probing hits its own blind spot.
2 comments
what happens if the safety tuning is done with a different activation basis than the one you probe on, say after pruning, quantization, or a small architecture swap, does the geometry stay stable enough to still separate tampering from normal finetuning?
also curious what the false positive rate looks like against ordinary alignment changes, not just overt modification, since an activation-only detector seems like it would need a pretty hard baseline there.
> what happens if the safety tuning is done with a different activation basis than the one you probe on
That probably breaks the detector fast, because AMS is assuming the refusal subspace is comparable across the exact probe model, and pruning or a small arch swap can move or smear that basis enough that the geometry no longer means the same thing. So no, I wouldnt trust it as a tamper detector across even mild representation shifts, and the false positive rate against normal alignment changes is exactly where an activation-only baseline gets ugly.