DeepSeek-AI has released a preview version of its DeepSeek-V4 series, introducing two Mixture-of-Experts (MoE) language models capable of processing context lengths of up to one million tokens. The new models — DeepSeek-V4-Pro and DeepSeek-V4-Flash — incorporate architectural innovations that dramatically reduce computational costs for long-context inference, according to the team's paper published on arXiv.
Models and Parameters
The series includes two variants:
- DeepSeek-V4-Pro: 1.6 trillion total parameters, with 49 billion activated per token.
- DeepSeek-V4-Flash: 284 billion total parameters, with 13 billion activated per token.
Both models support a context length of one million tokens. The larger model, V4-Pro, also has a maximum reasoning effort mode called DeepSeek-V4-Pro-Max, which sets a new state-of-the-art among open models on core tasks, the authors state.
Architectural Innovations
Three key upgrades drive the efficiency improvements:
- Hybrid Attention Architecture: Combines Compressed Sparse Attention (CSA) and Heavily Compressed Attention (HCA) to improve long-context efficiency.
- Manifold-Constrained Hyper-Connections (mHC): Enhances traditional residual connections, improving gradient flow in deep networks.
- Muon Optimizer: Accelerates convergence and improves training stability.
Both models were pre-trained on more than 32 trillion diverse and high-quality tokens, followed by a comprehensive post-training pipeline that further unlocks capabilities.
Efficiency Benchmarks for Million-Token Contexts
| Metric | DeepSeek-V4-Pro vs. DeepSeek-V3.2 |
|---|---|
| Single-token inference FLOPs | 27% (73% reduction) |
| KV cache size | 10% (90% reduction) |
The paper notes that these gains make it routine to support million-token contexts, enabling long-horizon tasks and further test-time scaling. The efficiency improvements come from the hybrid attention mechanisms, which reduce the memory and compute required for very long sequences.
Availability and Open-Source Release
Model checkpoints for both DeepSeek-V4-Pro and DeepSeek-V4-Flash are publicly available at the linked repository in the paper. The open-source release allows enterprise technology teams to deploy and fine-tune the models for custom long-context applications, from document analysis to code generation over millions of tokens.
For enterprise technology decision-makers evaluating AI for large-scale document processing, supply chain analytics, or trade compliance, the ability to process million-token contexts without proportional compute costs could unlock new automation possibilities. The models' open nature also enables on-premise deployment, addressing data sovereignty concerns common in international trade environments.
