Latent Diffusion Models (LDMs) have become the dominant approach for visual synthesis, but their quality-compute trade‑off is largely constrained by the tokenizer's fixed compression ratio. A new research paper published on arXiv proposes a variable‑length tokenization technique that merges tokens to achieve adaptive compression, enabling diffusion transformers to flexibly balance output quality and computational cost.
The Fixed Compression Bottleneck
Traditional tokenizers compress image patches into a fixed number of tokens, forcing the diffusion model to process a constant number of latent representations regardless of image complexity. This rigidity limits the model's ability to allocate compute efficiently — simple regions receive the same token budget as complex ones, wasting resources. Variable‑length tokenizers (VLTs) promise adaptive compression by varying token counts, but conventional VLTs modulate length by truncating ordered token sequences, which makes token semantics depend on token position and breaks representational alignment across different lengths. According to the authors, this cross‑length shift in the latent distribution hinders a single variable‑length diffusion model from operating effectively.
Variable‑Length Tokenization: Promise and Pitfalls
Prior VLT approaches rely on truncating a sorted token sequence to achieve shorter representations. However, because the order of tokens determines their meaning, a token at position 5 in a 10‑token sequence may represent a different visual concept than token at position 5 in a 5‑token sequence. This misalignment forces the diffusion transformer to learn separate latent spaces for each length, defeating the purpose of a unified variable‑length model. The paper identifies this as the core limitation of existing methods.
Merging Tokens for Length Modulation
To overcome this, the researchers propose a novel variable‑length tokenizer that modulates length by merging tokens rather than truncating them. The key insight: encouraging similar tokens to merge enables direct cross‑length representation alignment when the diffusion transformer operates according to the merging pattern. Because merged tokens naturally resolve into components of varying granularity, the model sees a coherent latent space regardless of total token count.
| Approach | Length Modulation | Representation Alignment | Data Dependency During Generation |
|---|---|---|---|
| Conventional VLT | Truncation | Broken (position‑dependent) | Not applicable |
| Proposed merging VLT | Token merging | Maintained via merging pattern | Data‑independent (learnable global merging) |
Learnable Global Merging for Data‑Independent Operation
A critical challenge arises: conventional merging methods are data‑dependent — they analyze the input to decide which tokens to merge. During generation, however, the diffusion process must operate without access to the final image, so the merging pattern must be predetermined. The authors introduce learnable global merging, a data‑independent mechanism that learns fixed merging rules during training. This ensures compatibility with diffusion transformers, because the merging pattern is known and can be used to align representations without requiring input‑specific decisions at inference time.
Results on ImageNet Generation
The researchers evaluated their method on ImageNet 256×256 image generation. When integrated with a diffusion transformer, the merging‑based variable‑length tokenizer achieves a superior gFID‑compute trade‑off compared to prior VLT methods, according to the paper. The code is publicly available (the paper provides a URL for the code repository).
This work demonstrates that careful tokenizer design — specifically, merging instead of truncating — can significantly improve the efficiency of diffusion transformers. For enterprise technology leaders investing in AI infrastructure, such advances could reduce the computational cost of high‑quality visual synthesis, making generative models more practical for large‑scale deployment in applications such as product design, simulation, and automated visual inspection.