Topic
code generation
Multi-LCB: New Benchmark Evaluates LLMs Across 12 Programming Languages
A new benchmark, Multi-LCB, extends the popular LiveCodeBench to 12 programming languages, revealing LLMs' struggles with multilingual code generation. Evaluation of 24 models uncovered Python overfitting and language-specific contamination.
From Tokens to Regions: CUDA-Sensitive Instruction Tuning for GPU Kernel Generation
A new method called CuSeT (CUDA-Sensitive Instruction Tuning) addresses the struggle of large language models to generate correct CUDA kernels. By combining adaptive token-level masking with region-aware sample reweighting, CuSeT improves functional correctness across multiple model families and scales, achieving competitive performance against frontier CUDA kernel generation models at lower inference cost.
DualGauge: Automated Joint Security-Functionality Benchmarking of Specification-Only Code Generation by LLMs and Coding Agents
Researchers present DualGauge, an automated framework for jointly evaluating correctness and security of code generated by LLMs from natural-language specifications. A benchmark of 307 tasks across three languages shows that even the strongest models achieve under 15% joint security-functionality success, while factors like scale and instruction tuning do not reliably improve outcomes. Three leading agentic coding systems also show no advantage over direct generation.
SPARK Method Activates Latent Security Knowledge in LLMs for Secure Code Generation
SPARK (Security Knowledge Priming and Representation-Guided Knowledge Activation) is a new inference-time method that improves the security of code generated by large language models without requiring retraining. The researchers argue that pretraining data already contains sufficient security material; the bottleneck is activation. Evaluated on 9 open-source and 7 proprietary models, SPARK matches or improves secure code generation baselines while preserving code utility.
Auditing Reward Hackability in Code RL Training Environments Reveals 28.5% Weak Test Suites
A research paper by Rajan on arXiv measures reward hackability in code reinforcement learning (RL) training environments. On a 49-task sample of SWE-bench Verified, 28.5% of tasks have test suites weak enough that a Docker-verified incorrect patch passes them. The study also proposes a hardening procedure using an LLM judge and Docker gate to detect defects.