Topic
tokenization
Toten Framework Outperforms Statistical Tokenization for Physical Quantities in Brazilian Portuguese Technical Texts
Researchers present Toten, a framework that replaces statistical tokenization with ontology-based classification for physical quantities and technical notation in Brazilian Portuguese. The system leverages external oracles and achieves higher ontological atomicity and numerical reconstruction compared to state-of-the-art baselines.
New Tokenization Method Merges Tokens to Improve Diffusion Transformer Efficiency
A research paper introduces a variable-length tokenizer that merges tokens instead of truncating them, enabling adaptive compression for diffusion transformers. The method, called learnable global merging, addresses representational alignment issues across token lengths and achieves a superior trade-off between image quality (gFID) and computational cost.
G2Rec Framework Structures and Tokenizes User Interests for Generative Recommendation
The G2Rec framework, proposed by researchers, addresses limitations in generative recommendation by unifying holistic graph-based user co-engagement modeling with semantic tokenization. It enables scalable, accurate user interest modeling without requiring ground-truth interests, and has demonstrated superiority through online deployment and experiments on public datasets.
Token Factory: Efficiently Integrating Diverse Signals into Large Recommendation Models
Token Factory is a framework that converts diverse traditional signals into soft tokens for large recommendation models (LRMs), addressing challenges of long prompts, memory footprint, and computational overhead. The approach has been validated in a production-scale environment, promising enhanced performance and efficiency.
FOUNDv2: Unified Quantized Tokenizers Transform User Representation Learning
FOUNDv2, a novel user representation framework, uses quantized tokenizers to transform heterogeneous data into discrete tokens, achieving superior performance and reduced storage costs. The model outperforms task-specific baselines and has been deployed at scale on Alipay, demonstrating practical efficiency.
X-Tokenizer: Semantic Action Tokenizer Boosts Robot Control by 13.5% Over FAST
Researchers propose X-Tokenizer, a new action tokenizer that treats tokenization as semantic interface learning rather than mere compression. Using a lightweight encoder-Semantic Residual Quantization (SRQ)-decoder architecture, it improves multimodal grounding by 13.5% and long-horizon task performance by 8.25 points over existing methods like FAST.