Topic
data science
MMLongEmbed Benchmark Reveals Limitations in Long-Context Multimodal Embedding Models
MMLongEmbed is the first comprehensive benchmark for evaluating multimodal embedding models (MEMs) in long-context scenarios. It comprises four retrieval tasks covering text, document, and video modalities. The evaluation reveals that current MEMs rely heavily on superficial feature matching and struggle with deep semantic and structural dependencies, with performance degrading systematically based on context length and key information placement.
Research Finds Anomalies in Multivariate Time Series Benchmarks Are Mostly Univariate
A study by researchers Pinet, Cumin, Berlemont, and Vaufreydaz on eight public benchmarks for multivariate time series anomaly detection (MTSAD) finds that labeled anomalies are overwhelmingly univariate—no cross-channel rupture occurs without a univariate deviation. The paper's diagnostic framework and synthetic data experiments show that current benchmarks do not justify cross-channel modeling, as channel-dependent detectors offer no measurable gain over channel-independent ones. The authors call for more structurally diverse evaluation sets.
New Benchmark IRTS-ToolBench Tests LLMs on Irregular Time Series Question Answering
A research paper introduces IRTS-ToolBench, a benchmark of 1,700 questions spanning 10 task types across 13 domains to evaluate large language models (LLMs) and AI agents on irregular time series question answering (TSQA). The benchmark addresses a gap in existing TSQA benchmarks that assume regular sampling, providing standardized inputs and a reproducible evaluation protocol for verifiable agentic data science.