Enterprise AI systems depend on high-quality pretrained language models, but for Persian (Farsi) a scarcity of large-scale, well-curated corpora has limited progress. A new paper on arXiv introduces IHUBERT, a Persian pretrained language model that tackles this challenge with a vector-based semantic deduplication pipeline and domain-balanced pretraining.
IHUBERT uses the RoBERTa-base encoder with 125 million parameters, trained from scratch on a 45 GB curated subset of the Sepahr-Danesh collection — approximately 7-8 billion tokens. The preprocessing pipeline includes normalization, exact and near-duplicate removal, anonymization, and critically, vector-database-based semantic deduplication to control distribution balancing across domains and registers.
Vector-Based Semantic Deduplication for Corpus Quality
Redundancy in training data — especially near-duplicate documents — can bias model learning and waste compute. The IHUBERT team applied a multi-stage deduplication strategy. After exact and near-duplicate removal, they employed a vector-database approach: documents were embedded into a vector space and duplicates with high semantic similarity were identified and removed. This ensured the pretraining corpus was both large and diverse, improving generalisation. According to the paper, this allowed for distribution balancing control across domains and registers.
Domain-Balanced Pretraining and Tokenization
To better capture Persian morphology and orthographic variation, the researchers trained a BPE tokenizer with a vocabulary of 139,000 tokens on the full pretraining corpus. In a controlled ablation, BPE produced slightly lower subword fragmentation than WordPiece at the same vocabulary size, validating the tokenization choice. Domain-balanced pretraining ensured that no single text domain dominated, preserving performance across diverse downstream tasks.
Benchmark Performance and Gaps
IHUBERT was evaluated on seven Persian NLU benchmarks covering named entity recognition (NER), sentiment analysis, topic classification, natural language inference (NLI), extractive question answering (QA), and relation extraction. Task-standard metrics were used: entity-level F1, Macro-F1, Exact Match (EM)/F1.
| Benchmark | Task | IHUBERT Score | Notes |
|---|---|---|---|
| PQuAD | Extractive QA | F1 88.3542 | Ranked first |
| ParsiNLU-RC | Extractive QA | F1 49.0987 | Ranked first |
| FarsTail | NLI | Macro-F1 0.8350 | Best result |
| ParsTwiNER | NER | F1 0.8308 | Competitive |
| DigiMag | Topic classification | Macro-F1 0.7953 | Competitive |
| PERLEX | Relation extraction | Macro-F1 0.6684 | Main remaining gap |
IHUBERT achieved its strongest gains on extractive question answering, ranking first on both PQuAD and ParsiNLU-RC. It also attained the best result on FarsTail. On NER and topic classification, it remained competitive (e.g., 0.8308 F1 on ParsTwiNER; 0.7953 Macro-F1 on DigiMag). Relation extraction remains the main remaining gap, with 0.6684 Macro-F1 on PERLEX.
Implications for Enterprise AI
For technology leaders evaluating AI for multilingual or Persian-specific applications, IHUBERT demonstrates that careful data curation — especially semantic deduplication and domain balancing — can yield significant performance improvements without increasing model size. The techniques are transferable to other low-resource languages and to enterprise domains such as trade documentation processing, customs declarations, and logistics communication, where high-quality labeled data is scarce. The 139k BPE tokenizer and open-source release of the model (under CC BY-NC-SA 4.0) offer a strong foundation for building custom Persian NLP pipelines, potentially reducing time and cost in developing AI for trade facilitation and supply chain visibility in Persian-speaking markets.