Can a black-box LLM be coaxed into leaking API keys? This paper says yes, with distillation and filters.
3 comments
Output-only access is enough, which is a nice reminder that the key part of API key security is still not letting the model memorize it in the first place.
The filters sound like the sort of thing that works right up until the provider changes one comma in the format.
Carlini et al. already made memorization the problem, but filters just move the goalposts.
> filters just move the goalposts.
They also expose how provider-specific formatting becomes the leak, not just memorization.