Makeup transfer models power augmented reality (AR) experiences and virtual try-on (VTO) for online makeup shopping, but existing diffusion-based solutions often fail to preserve facial identity and skin tone. According to a recent arxiv.org paper, a new approach called MakeupMirror significantly closes this gap, making production-level VTO more realistic.
The paper, led by researchers including Andreou, Nefeli, Martínez-González, Angel, Sternig, Sabine, Guillaumin, Matthieu, Antonakos, Epameinondas, and Opitz, Michael, introduces several technical innovations over the state-of-the-art baseline Stable-Makeup. These include facial geometry conditioning via ControlNets to maintain facial fidelity, region-specific makeup transfer control for precise application across skin, eyes, and lips, skin tone-based modulation to prevent cross-subject skin tone alteration, and integration of a Levenberg-Marquardt Langevin sampler to speed up inference while preserving quality.
The model was evaluated on three datasets: CPM-Real, Makeup Wild, and the newly collected, more diverse MakeupSelfies. Results show MakeupMirror improves relative facial recognition similarity by +60% and reduces relative skin tone difference by -50% over Stable-Makeup. Inference latency is 0.7 seconds, and an expert acceptance rate of 94% was achieved across core facial identity preservation criteria.
Technical Innovations Overview
MakeupMirror incorporates four key enhancements over Stable-Makeup:
- Facial Geometry Conditioning: ControlNets integrate facial geometry to preserve subject identity.
- Region-Specific Control: Allows precise makeup transfer to separate facial areas (skin, eyes, lips).
- Skin Tone Modulation: Prevents unwanted skin color changes when transferring makeup between different subjects.
- Levenberg-Marquardt Langevin Sampler: Accelerates the diffusion inference process without sacrificing generation quality.
Quantitative Results Comparison
| Metric | Improvement over Stable-Makeup |
|---|---|
| Relative facial recognition similarity | +60% |
| Relative skin tone difference | -50% |
| Inference latency | 0.7s |
| Expert acceptance rate | 94% |
These figures, reported in the paper, demonstrate that MakeupMirror addresses the key bottlenecks that have hindered deployment of diffusion models for real-world VTO applications in e-commerce.
Implications for Enterprise
For enterprise technology leaders in retail and e-commerce, the ability to deliver realistic virtual try-on directly impacts conversion rates and return rates. The paper notes that makeup transfer models "enable fun AR experiences as well as VTO for online makeup shopping." By achieving high-fidelity identity and skin tone preservation with sub-second latency, MakeupMirror brings production-ready performance closer to reality. The use of a custom dataset (MakeupSelfies) also indicates attention to diversity, which is critical for global consumer acceptance.
While the research remains in academic preprint form, the techniques described—especially the integration of ControlNets and efficient sampling—offer a blueprint for engineering teams building VTO systems. The 94% expert acceptance rate suggests that the model's outputs are often indistinguishable from genuine images, a threshold that could satisfy quality standards for commercial deployment.
Looking Ahead
The paper highlights remaining limitations, although specifics are not detailed in the abstract. Future work may focus on further reducing latency or handling more complex makeup styles. For now, MakeupMirror sets a new benchmark for facial attribute preservation in diffusion-based makeup transfer, directly supporting the business need for realistic, real-time virtual try-on.