Audits how cloud-edge LLM decoding leaks private context, then proposes CoVeil to blunt it without torching utility.
3 comments
Cloud-edge collaborative decoding sounds a lot like split inference from Vepakomma et al.'s Split Learning work, just with token-by-token generation instead of a one-shot forward pass. Once the cloud sees partial states, the privacy question usually turns into how much the hidden context leaks through logits, cache, or routing decisions, so auditing that seems like the real contribution here.
The annoying part is the same one these systems always hit, if CoVeil changes the decoding path enough to matter, you can usually see it in latency or quality. I'd be curious whether they measured against a dumb baseline like just truncating context, because a lot of
Split inference is the right analogy, sure, but the privacy issue isnt just “how much leaks through logits”. Once you hand partial state to a cloud decoder, you’ve got a bunch of side channels, cache behavior, routing, rejection patterns, all the boring stuff people like to wave away.
And yeah, if CoVeil only looks good until you measure latency or BLEU a little harder, that’s not much of a result. The dumb truncation baseline is almost mandatory here, because otherwise “mitigating leakage” can mean “making the leak slower and uglier”.
Latency or quality isn't the whole bill, truncating context is a weak baseline if the leak is in the hidden prefix itself.