LiveCodeBench (LCB) has become a widely adopted benchmark for evaluating large language models (LLMs) on code-generation tasks. However, according to a paper published on arxiv.org, LCB remains restricted to Python, leaving open the question of whether LLMs can generalize across the diverse programming languages required in real-world software engineering. To address this, researchers introduced Multi-LCB, a benchmark that evaluates LLMs across 12 programming languages, including Python, while preserving LCB's contamination controls and evaluation protocol.
Benchmarks and Methodology
Multi-LCB transforms Python tasks from the LCB dataset into equivalent tasks in other languages. Because it is fully compatible with the original LCB format, Multi-LCB will automatically track future LCB updates, enabling systematic assessment of cross-language code generation competence. The paper evaluated 24 LLMs for instruction and reasoning on Multi-LCB, uncovering evidence of Python overfitting, language-specific contamination, and substantial disparities in multilingual performance.
Key Findings
The researchers reported that the results establish Multi-LCB as a rigorous new benchmark for multi-programming-language code evaluation, directly addressing LCB's primary limitation and exposing critical gaps in current LLM capabilities. The study found that models that perform well on Python often degrade significantly when asked to generate code in languages such as Java, C++, or JavaScript. Language-specific contamination, where training data for one language leaks into others, also skewed performance.
| Aspect | Details |
|---|---|
| Original Benchmark | LiveCodeBench (Python-only) |
| Extended Benchmark | Multi-LCB (12 languages including Python) |
| Number of Models Evaluated | 24 |
| Key Issues Uncovered | Python overfitting, language-specific contamination, performance disparities |
| Compatibility | Fully compatible with LCB; tracks future updates |
Our results establish Multi-LCB as a rigorous new benchmark for multi-programming-language code evaluation, directly addressing LCB's primary limitation and exposing critical gaps in current LLM capabilities.
Implications for Enterprise AI Adoption
For enterprise technology leaders evaluating LLMs for code generation across multiple programming languages—common in legacy system modernization, polyglot microservices, and full-stack development—the findings suggest that relying solely on Python benchmarks can give a misleading picture of a model's true coding ability. Multi-LCB provides a contamination-aware evaluation that may help procurement teams select models that generalize better across languages, reducing the risk of vendor lock-in or hidden performance cliffs. As organizations increasingly integrate AI-assisted coding into their software supply chains, tools like Multi-LCB can inform more realistic assessments.
The paper is available on arxiv.org, authored by Ivanova; Maria; Zadorozhny; Pavel; Levichev; Rodion; Petrov; Adamenko; Lopatin; Kutalev; Alexey; Babaev; Dmitrii.
