Artificial Intelligence #tokenization#ontology
Toten Framework Outperforms Statistical Tokenization for Physical Quantities in Brazilian Portuguese Technical Texts
Researchers present Toten, a framework that replaces statistical tokenization with ontology-based classification for physical quantities and technical notation in Brazilian Portuguese. The system leverages external oracles and achieves higher ontological atomicity and numerical reconstruction compared to state-of-the-art baselines.
Jul 8, 2026 1 source