proof/b3
Benchmark B3

Conflict Resolution
4/5 conditions resolved correctly.

Write ground truth facts at varying confidence levels, then inject adversarial (wrong) values at varying confidence levels. Query final KB state and compare retained value to ground truth. The score formula weights confidence against source reliability. Gap ≥ 10 points triggers deterministic rejection. Gap < 10 routes to LLM arbitration.

Executed 2026-03-215 conditionsC2: known limitation — read carefully

Results at a glance

4/5Overall: conditions resolved correctly
3/3LLM arbitration accuracy (C3, C4, C5)
C2Known limitation — first-write lock-out
4/5

conditions resolved correctly

LLM arbitration accuracy: 3/3 (100%)
C2 failure: documented expected limitation
Deterministic path (C1): rejection correct

The weighted score formula

Iranti does not use raw confidence values directly. It computes a weighted score that combines confidence with source reliability. This prevents high-confidence writes from low-reliability sources from automatically winning conflicts.

Weighted score formula
weighted = conf × (0.7 + reliability × 0.3)

Default source reliability = 0.5, which simplifies to weighted = conf × 0.85. When reliability is explicitly set (as in C5), the full formula applies.

Example — C1
Caltech (existing)
conf 90 × 0.85 =
76.5
MIT (adversarial)
conf 70 × 0.85 =
59.5
Score comparison
Caltech existing76.5
MIT adversarial59.5
gap = 17.0 pts → deterministic rejection

Resolution path

Every conflict is routed through a deterministic decision tree before any LLM is invoked. LLM arbitration is a last resort for genuinely ambiguous cases — not the default.

ConflictdetectedComputeweighted gapgap ≥ 10gap < 10Deterministicrejection(C1, C2)LLMarbitration(C3, C4, C5)

All 5 conditions

Each card shows the ground truth value vs the adversarial injection, their confidence scores, the resolution path taken, and whether the correct value was retained. Bars animate on scroll. C2 is visually distinguished — it is not hidden, but it is presented in context.

C1
✓
correctCaltech
adversarialMIT
Caltechconf 90 → score 76.5
MITconf 70 → score 59.5
gap 17 pts·deterministic
C2
known limitation
✗
correctYale
adversarialPrinceton
Yaleconf 90 → score 76.5
Princetonconf 70 → score 60.1
gap 16.4 pts·deterministic

High-confidence first write blocks lower-confidence correction. By design — the system has no way to distinguish correction from adversarial injection.

C3
✓
correctCarnegie Mellon
adversarialCornell
Carnegie Mellonconf 78 → score 66.3
Cornellconf 75 → score 63.75
gap 2.55 pts·LLM arbitration

LLM cited "more established source, minimal confidence difference".

C4
✓
correctUC San Diego
adversarialUC Berkeley
UC San Diegoconf 76 → score 64.6
UC Berkeleyconf 75 → score 63.75
gap 0.85 pts·LLM arbitration
C5
✓
correctColumbia
adversarialNYU
Columbiaconf 70 → score 59.5
NYUconf 80 → score 68
gap 8.5 pts·LLM arbitration

Raw adversarial confidence is higher (80 vs 70). LLM read source names ("b3_trusted_reviewer" vs "b3_low_reliability") as semantic signals.

correct value baradversarial value bar✓retained correctly✗known limitation
C2 — Deep Dive

The most important finding from this benchmark.

C2 is not a bug. It is a direct consequence of how Iranti's conflict resolution is designed — and it has real operational implications that every developer deploying Iranti in write-heavy scenarios needs to understand before going to production.

What happened in C2
Step 1Yale written with confidence=90 (score: 76.5). This is the ground truth.
Step 2Princeton injected with confidence=70 (score: 60.1) as a correction attempt.
Step 3Gap = 16.4 pts → deterministic rejection. Princeton is rejected.
Step 4Yale is retained. Outcome: ✓ correct. BUT — in a real scenario where Yale was wrong, the correct Princeton fact would have been locked out.
The structural issue

The system has no way to know whether the incoming write is an adversarial injection or a legitimate correction. Both look identical at the protocol level. It can only compare scores. If the first write was wrong, it locks out the truth.

Operational RealityIf a wrong fact is written with high confidence first, a lower-confidence correction cannot overwrite it. The wrong fact is retained until: (a) a manual intervention deletes or overwrites it directly, (b) a forced write with equal or higher confidence is submitted, or (c) the entity is explicitly reset. This is deterministic behavior by design.
LimitationConfidence scores should reflect real epistemic state. If your agents habitually write at confidence=90 regardless of actual certainty, you will encounter C2-style lock-out. The fix is not in Iranti's conflict resolution logic — it is in how your agents assign confidence when they write.
Operational guidance
Forced writesUse iranti_write with confidence ≥ existing score to explicitly overwrite a locked fact. This is the intended correction path.
Confidence disciplineReserve confidence=90+ for facts from verified, authoritative sources. Do not use high confidence as a default.
Audit before deploymentIf your system has agents that write the same entity/key from multiple sources, audit confidence assignment logic before deploying at scale.
Manual reset pathIranti exposes a delete/reset operation for entity keys. This is the escape hatch when automated resolution fails.
C5 — Emergent behavior

The LLM read source names as semantic signals.

C5 is the most interesting condition in this benchmark — and it exposes a behavior that is simultaneously a capability and a risk.

Setup
Columbiaconf=70, source="b3_trusted_reviewer" → score 59.5
NYUconf=80, source="b3_low_reliability" → score 68.0
gap8.5 pts → LLM arbitration invoked
What the LLM did

The LLM received both candidate facts with their source identifiers. It read "b3_trusted_reviewer" as a semantic signal of trustworthiness, and "b3_low_reliability" as a signal of low trust — and correctly preserved Columbia, despite NYU having the higher raw confidence score.

FindingSource naming is a lever. If you name your sources semantically, the LLM arbitrator will use those names as evidence. This is not accidental — source names are passed to the LLM with conflict context. You can encode institutional trust into source names and the LLM will respect it during arbitration.
LimitationThis is also a risk. If an adversarial agent names its source "trusted_authoritative_reviewer", the LLM may up-weight it. Source name semantics are not validated. They are a user-controlled signal. In a multi-tenant or adversarial environment, source names should be treated as untrusted input, not authority signals.

Threats to validity

LimitationSmall N (5 conditions). This is indicative, not statistically significant. The conditions were chosen to probe specific resolution paths, not to sample the space of possible confidence distributions. Broader coverage is needed.
LimitationSource names were chosen intentionally. "b3_trusted_reviewer" and "b3_low_reliability" are semantically loaded by design, to test whether source name semantics influence LLM arbitration. They do. Production source names will vary.
LimitationLLM arbitration is not deterministic. C3, C4, and C5 were run once. LLM arbitration decisions may vary across runs, temperatures, or model versions. The 3/3 LLM accuracy score is a point estimate, not a distributional claim.
LimitationSelf-evaluation bias. The same model (Claude Sonnet 4.6) designed the test, ran the benchmark, and evaluated correctness. Independent evaluation infrastructure is required for publication-grade claims.
NoteC2 is marked "wrong" in the outcome column because the adversarial scenario in C2 was designed to model a legitimate correction. In a strict conflict scenario (where Yale is correct), the outcome is ✓. The "wrong" label reflects the design intent of the C2 test case: a legitimate correction was blocked.
Raw data

Full trial execution records, confidence scoring logs, LLM arbitration transcripts, and condition definitions in the benchmarking repository.

iranti-benchmarking →← All benchmarks