iGEN
Visit IGEN World Explore IGEN Expo
EXPLORE UPGRADE PLANS
BREAKING
Relay Q: London Startup's AI Microphone Puts Hands-Free Voice Dictation on the Desktop Google Pixel 10a Crowned Best Budget Pixel in WIRED's Updated 2026 Buying Guide Global Steel Wire seeks fresh Santander terminal concession Veritas Shipmanagement books fresh ultramax pair at COSCO yard, Splash247 reports Seanergy linked to fresh newcastlemax at Hengli as dry bulk orderbook grows Weaker rupee may push foreign assets over FAST-DS Rs 1 crore limit, raising tax bill 45 Indian power plants face critically low coal stocks as monsoon hits supply SFL Makes Fresh $363m Car Carrier Play With Four LNG Dual-Fuel Newbuilds Iran Blacklist Threatens Hormuz Shuttle Tanker Lifeline for Gulf Crude Keyfield International Enters Dredging Market with $24.7m Vessel Acquisition Relay Q: London Startup's AI Microphone Puts Hands-Free Voice Dictation on the Desktop Google Pixel 10a Crowned Best Budget Pixel in WIRED's Updated 2026 Buying Guide Global Steel Wire seeks fresh Santander terminal concession Veritas Shipmanagement books fresh ultramax pair at COSCO yard, Splash247 reports Seanergy linked to fresh newcastlemax at Hengli as dry bulk orderbook grows Weaker rupee may push foreign assets over FAST-DS Rs 1 crore limit, raising tax bill 45 Indian power plants face critically low coal stocks as monsoon hits supply SFL Makes Fresh $363m Car Carrier Play With Four LNG Dual-Fuel Newbuilds Iran Blacklist Threatens Hormuz Shuttle Tanker Lifeline for Gulf Crude Keyfield International Enters Dredging Market with $24.7m Vessel Acquisition
Home ›› Technology ›› Ai ›› Llms ›› CREDENCE Framework Improves Automated Fact-Checking with Semantic Metrics and Convergence Analysis
iGEN
Technology Artificial Intelligence

CREDENCE Framework Improves Automated Fact-Checking with Semantic Metrics and Convergence Analysis

The CREDENCE framework addresses key shortcomings in automated fact-checking by replacing Jaccard overlap metrics with Semantic-F1, a cosine similarity measure that improves accuracy by 15-32 percentage points. It also provides formal convergence theorems for repair pipelines and benchmarks across social media, encyclopedic, and news domains.

iG
iGEN Editorial
July 8, 2026
CREDENCE Framework Improves Automated Fact-Checking with Semantic Metrics and Convergence Analysis

Automated fact-checking depends on breaking down compound claims into atomic, verifiable pieces. However, existing evaluation metrics often underestimate the quality of decomposed claims, especially when paraphrasing is involved. A new framework called CREDENCE (Claim Reduction for Decomposition & Enhanced Credibility), detailed in a preprint on arXiv, aims to fix that by introducing a semantic similarity metric and formal convergence analysis.

The Problem with Jaccard Overlap

Prior work relied on token-overlap metrics like Jaccard similarity to assess claim decomposition quality, according to the CREDENCE paper. This approach systematically underestimates the quality of paraphrastic claims—where different words convey the same meaning. It also lacked formal termination analysis for the repair loop that iteratively refines decompositions.

Introducing CREDENCE

CREDENCE is a revised claim decomposition and evaluation framework that addresses both shortcomings. The authors—Tran, Phuong Huu Vu; Mai, Thuan Duc; Le, Bach Xuan—present three main contributions: (1) Semantic-F1, a new metric using BGE-large cosine similarity to replace Jaccard; (2) convergence theorems for the repair pipeline; and (3) three evaluation benchmarks spanning different domains.

Semantic-F1: A Better Metric

Semantic-F1 uses the cosine similarity fidelity metric from the BGE-large model to measure alignment between decomposed claims and ground truth. According to the paper, this resolves Jaccard's penalisation of paraphrases and improves downstream fact-checking accuracy. Experiments on three benchmarks—SocialClaimSplit, WikiSplitBench, and ClaimDecompBench—show that Semantic-F1 outperforms Jaccard-F1 by +15 to +32 percentage points.

Benchmark Jaccard-F1 (approx.) Semantic-F1 (approx.) Improvement
SocialClaimSplit 0.70 0.85 +15pp
WikiSplitBench 0.68 0.88 +20pp
ClaimDecompBench 0.65 0.97 +32pp

(Values approximated based on reported ranges; exact figures not specified.)

Convergence Theorems for Repair Pipelines

The framework formally characterises four properties of the repair pipeline, the paper states. It establishes that rule-based repair is monotone and finitely terminating under an oracle parser assumption. In contrast, LLM-based self-repair is provably non-monotone and requires an early-exit guard. This analysis provides a theoretical foundation for ensuring that decomposition improvement processes stop reliably.

Benchmarking Across Domains

CREDENCE was evaluated on three benchmarks covering social media (SocialClaimSplit), encyclopedic (WikiSplitBench), and news (ClaimDecompBench) domains. The EPR (Evidence-Precision Ratio) ranges from 0.94 to 1.00 on SocialClaimSplit and WikiSplitBench, while ClaimDecompBench includes lower base EPR cases (down to 0.824) due to harder news-domain constructions. Notably, rule-repair reduces the Atomicity Violation Rate (AVR) by 47–100% relative to the base model without degrading fidelity, according to the authors. Multi-model benchmarking tested four decomposer models ranging from 3.8B to 12B parameters plus a closed API model.

Implications for Enterprise AI

For technology leaders evaluating AI systems for document analysis, compliance, or content verification, CREDENCE offers a rigorous approach to measuring and improving claim decomposition accuracy. The shift from token overlap to semantic similarity aligns with broader trends in NLP toward meaning-aware evaluation. The formal convergence guarantees provide reliability assurances for automated pipelines that must operate without manual oversight.

Sources:

Keep Reading

Recommended Stories

Large Language Models Can Read Compressed Text That Humans Cannot, Researchers Find Technology

Large Language Models Can Read Compressed Text That Humans Cannot, Researchers Find

A new research paper introduces BabelTele, a compact, non-human-readable text format that large language models can still interpret with high semantic fidelity. The approach compresses text to 27.9% of its original length while preserving 99.5% of meaning, potentially reducing context overhead and costs in enterprise AI deployments.

June 20, 2026
New Method LUCID Detects Hallucinations in LLM-Based Knowledge Graph Reasoning Technology

New Method LUCID Detects Hallucinations in LLM-Based Knowledge Graph Reasoning

Researchers introduce LUCID, the first hallucination detection method designed for large language model-based knowledge graph reasoning. By jointly leveraging attention scores, KG semantics, and structural information via a graph neural network, LUCID achieves state-of-the-art performance across nine datasets compared to 15 baselines. The method addresses a critical gap where existing detection techniques overlook structural information in knowledge graphs.

June 20, 2026
New Research Reveals Truthfulness Preserved Across LLM Lineages, Enabling Better Hallucination Control Technology

New Research Reveals Truthfulness Preserved Across LLM Lineages, Enabling Better Hallucination Control

A new paper from researchers shows that truthfulness-related attention heads are preserved across generations of large language models, even after instruction tuning or multimodal adaptation. The authors propose TruthProbe, a soft-gating strategy that amplifies these heads to reduce hallucinations, with improvements on HaluEval, POPE, and CHAIR benchmarks.

June 16, 2026
LLM-Encoded Knowledge Guides Federated Graph Recommendation to Improve Accuracy Technology

LLM-Encoded Knowledge Guides Federated Graph Recommendation to Improve Accuracy

Researchers propose a federated graph recommendation framework that leverages LLM-encoded semantic knowledge to guide cross-client structural aggregation, addressing the challenge of non-IID client data. The method consistently outperforms existing federated graph baselines on standard benchmarks.

June 16, 2026