Activation Steering Improves Synthetic Data Generation for Low-Resource Languages

Researchers have developed a novel approach to improve synthetic data generation for low-resource languages using activation steering techniques applied to large language models. The method addresses limitations of current few-shot prompting approaches, which rely on target-language examples and increase computational costs while potentially reducing data diversity through lexical anchoring.

The study introduces two steering strategies: Language Steering, designed to target the linguistic identity of a language, and Quality Steering, which captures text well-formedness by contrasting human-written and backtranslated representations. The researchers evaluated these techniques across four open-source large language models, testing multiple layers and 11 typologically diverse languages. The evaluation focused on generating sentiment and topic classification data, which was then used to finetune smaller classifiers.

Results demonstrate that activation steering applied to early layers consistently improves the diversity of generated synthetic data while often achieving stronger downstream model performance, particularly for low-resource languages. The steering approach was tested in both zero-shot and few-shot prompting settings, with comparisons made against non-steered counterparts to establish the effectiveness of the method. This research suggests that steering techniques offer a more efficient alternative to traditional few-shot prompting for generating high-quality training data in languages with limited existing resources.