Researchers Propose SAGE Method to Improve Large Language Model Unlearning While Preserving Capabilities

Researchers have developed SAGE (Spectral Activation-GEometry Sanitization), a new post-hoc approach designed to address a fundamental challenge in large language model (LLM) unlearning: the trade-off between removing undesirable knowledge and preserving desired capabilities.

LLM unlearning aims to remove specific knowledge or behaviors from AI models while maintaining their overall functionality. However, existing unlearning methods inherently involve a compromise, where efforts to forget certain information inadvertently damage the model's ability to retain other important capabilities. The research team identified that retention activation bias—a measurable property of how the model processes retained information—can be used to quantify the damage caused by unlearning methods, independent of the specific implementation used.

SAGE operates as a source-agnostic correction tool that can be applied after the primary unlearning process completes, eliminating the need to re-run the original unlearning pipeline. The method collects input data from a small retain proxy, identifies dominant activation patterns, and solves an optimization problem that suppresses update components aligned with high-energy retained directions while preserving the core forgetting mechanism. This approach allows SAGE to be applied to any unlearning method as a final refinement step.

According to the research, SAGE was tested across multiple unlearning methods, model scales, and benchmarks, consistently demonstrating improvements in the retain-forget trade-off. The work highlights post-hoc sanitization of final update vectors as a practical and previously underexplored approach in the machine unlearning field, potentially offering a complementary technique that enhances the effectiveness of existing unlearning strategies.