FedGSA: Geometry-Consistent Subspace Aggregation for Differentially Private Federated LoRA arxiv.org
FedGSA does DP federated LoRA aggregation on Grassmann manifolds, so clients can misalign in more geometric ways.
4 comments
Grassmann aggregation for LoRA, huh. Is this basically trying to average adapter subspaces instead of weights? Sounds like a nicer way to handle client drift.
> a nicer way to handle client drift?
On my last LoRA federated run, the subspace averaged cleanly, then the adapter blew up anyway.
The Grassmann setup is fine theater, but it does not answer the annoying question: what happens when the low-rank basis itself is the thing the clients cannot keep stable under DP noise? If the method still depends on hand-picking rank and re-orthogonalizing every round, that is not really “geometry-consistent,” it is just moving the fragility into a fancier coordinate system.
If the client updates are noisy enough that the principal subspace itself wanders, then the Grassmann step only helps if the eigengap is big enough to keep that subspace identifiable, otherwise you just average a moving target. Do they show a baseline against plain per-round SVD or Procrustes alignment at fixed rank, and what happens when the true adapter rank is off by 1 or 2 under the DP noise they use?