DRIFT: New Method for Optimizing Training Data in Large Language Models

Researchers have proposed DRIFT (Data Refinement via On-Policy Influence Functions for Supervised Fine-Tuning), a novel approach to optimize training data distribution for Large Language Models (LLMs). The method addresses a critical challenge in machine learning: improving the performance ceiling of models rather than merely preserving performance with reduced data.

The core innovation of DRIFT lies in its use of instance-level data attribution through Influence Functions. Traditional data attribution methods struggle with two key structural problems: a proximity gap caused by off-policy validation targets and bias toward gradient norm. DRIFT overcomes these limitations by utilizing the model's on-policy rollouts as validation targets instead of relying on external reference data. This approach minimizes the parameter proximity gap and better aligns with the fundamental assumptions of influence functions.

The methodology incorporates signed weighting based on trajectory correctness and debiases influence scores against gradient hacking issues. This allows a small set of validation queries to serve as reliable anchors for attributing the entire dataset. Experiments conducted on 7-billion-parameter instruction and reasoning models demonstrate that DRIFT consistently improves performance across both model types, outperforming existing data curation baselines.

The research addresses a fundamental problem in supervised fine-tuning: while previous data curation methods excel at accelerating training under budget constraints, they are less effective at elevating the upper limit of model capabilities. DRIFT shifts the focus from identifying smaller subsets that preserve performance to refining the data distribution toward instances most capable of driving model improvement. The findings suggest that on-policy data attribution represents a promising direction for enhancing LLM training efficiency and capability.