Large language model safety alignment typically requires vast amounts of preference data, but current data selection methods often score each pair independently, collapsing directional preference information into scalar quality or diversity scores. This sample-centric view is especially limiting in multi-dataset settings, where shared safety directions coexist with dataset-specific residual risks, according to a paper on arXiv.
To address this, the researchers propose DOG-DPO (Dynamic Optimization in Geometry for Safety Alignment), a training-free data selection framework that treats preference pairs as structured geometric signals. DOG-DPO first represents each preference pair as a direction in model representation space. It then decomposes multi-dataset preference geometry into a global anchor subspace and dataset-specific residual subspaces. Finally, it selects subsets by maximizing diversity-based coverage, encouraging broad, non-redundant coverage of alignment directions before DPO training.
Performance Benchmarks
Across six safety benchmarks and two model backbones, DOG-DPO achieves a strong utility-robustness trade-off using only 11% of the preference pairs. It recovers most of the safety gains of full-data training while remaining entirely teacher-free, training-free, and substantially faster than representative selection baselines.
| Aspect | DOG-DPO | Traditional Selection Baselines |
|---|---|---|
| Data Usage | 11% of preference pairs | Full dataset or varied |
| Training Requirements | Teacher-free, training-free | Often requires teacher model |
| Computational Speed | Substantially faster | Slower due to scoring |
| Safety Alignment | Recovers most gains of full training | Variable |
How DOG-DPO Works Geomtrically
The key innovation lies in preserving the directional information of each preference pair. While existing methods reduce each pair to a single scalar representing quality or diversity, DOG-DPO maintains the full vector direction in the model's representation space. This allows the framework to identify and retain alignment directions that are globally shared across datasets, as well as residual directions unique to specific datasets. The selection process then maximizes coverage of these diverse directions to avoid redundancy.
Implications for Enterprise AI
For enterprises deploying large language models, DOG-DPO offers significant efficiency gains. Reducing data requirements to 11% of original preference pairs can drastically cut data curation and annotation costs. The training-free nature eliminates the need for auxiliary models to guide data selection, simplifying the pipeline. The method is also substantially faster than existing baselines, enabling faster iteration cycles for safety alignment. While the paper focuses on safety alignment, the geometric approach to data selection could have broader applications in any domain where preference pairs are used for model alignment. Enterprise CTOs and AI leaders should consider how such training-free, geometry-based selection methods can improve model safety without heavy computational overhead.
