Topic
preference optimization
Sequential DPO Study Reveals Non-Uniform Forgetting Across Multiple Preference Objectives
A study by Bhandari et al. on sequential Direct Preference Optimization (DPO) finds that later training objectives do not uniformly degrade earlier preferences. Using Llama-3.1-8B-Instruct, the research reveals that forgetting patterns vary from stability to positive transfer depending on objective compatibility and signal strength, offering guidance for multi-objective AI alignment in enterprises.
S-SPPO: Semantic Calibration Boosts LLM Preference Alignment Without Human Data
S-SPPO, a dual-space semantic calibration framework, fixes instability in Self-Play Preference Optimization (SPPO) for large language models. By annealing win targets and enforcing geometric diversity, it achieves superior alignment results on AlpacaEval 2.0 without extra human preferences.
PVminerLLM2 Uses Preference Optimization to Improve Structured Patient Voice Extraction
Researchers introduce PVminerLLM2, an improved set of LLMs for structured extraction of patient voice from unstructured text. The model uses preference optimization with token-level gated stabilization and confusion-aware pair construction to outperform supervised fine-tuning baselines. The code and trained models are publicly available.