Enterprises deploying large language models (LLMs) often need to align them with multiple, sometimes conflicting human preferences — safety, helpfulness, factual accuracy, and brand voice. A common approach is to apply these preferences sequentially using methods like Direct Preference Optimization (DPO). But does later training erase what the model learned earlier? According to a study by Bhandari, Pranav; Fay, Nicolas; Datta, Amitava; Naseem, Usman; and Nasim, Mehwish on arXiv, the answer is not straightforward: forgetting is not uniform.
Four Preference Settings
The researchers studied sequential DPO across four distinct preference settings:
| Setting | Description |
|---|---|
| Distributional conflict | Objectives that compete due to differing data distributions |
| Multi-attribute interaction | Preferences involving multiple attributes that interact |
| Strong safety signal | A high-priority safety objective |
| Compatible response-quality | Objectives that are naturally aligned |
Using the Llama-3.1-8B-Instruct model with LoRA adapters, the team evaluated all objectives after every stage while keeping a fixed base-model reference.
Key Findings on Forgetting
The central finding is that sequential DPO does not produce a single forgetting pattern. According to the study, preference change ranges from "partial degradation to stability, pair-level redistribution, or positive transfer" depending on the relationship between objectives, signal strength, and training order. This means that aggregate metrics can mask heterogeneous changes across individual preference pairs.
To uncover these patterns, the authors used pair-level analysis with length-normalized policy margins. They also applied quartile decomposition, which showed that high-confidence pairs could either degrade or improve depending on the setting. Mechanistic diagnostics revealed that Stage 2 gradients and adapter updates are near-orthogonal to the previous objective across all settings, providing little evidence that direct gradient opposition drives forgetting.
sequential DPO does not produce a single forgetting pattern; preference change ranges from partial degradation to stability, pair-level redistribution, or positive transfer
Implications for Enterprise AI
For enterprise technology leaders, these findings underscore that sequential alignment pipelines must account for objective compatibility and signal strength. Assuming later objectives uniformly overwrite earlier preferences is incorrect. Instead, careful design of the sequence and monitoring of pair-level performance are needed to avoid unintended degradation of critical preferences, such as safety or compliance.
The study recommends that future alignment pipelines should evaluate objective relationships and training order rather than treating all later training as uniformly harmful. This is particularly relevant for organizations that deploy LLMs in regulated industries where multiple behavioral constraints must coexist.