AUROC breakdown by vertical
Each vertical’s AUROC is computed independently. 12,000 cases distributed ~1,500 per vertical with balanced hallucinated/factual ground-truth labels.
| Vertical | AUROC | Cases | G5 Coverage | Avg MUG Score |
|---|---|---|---|---|
|
Loading probe results…
|
||||
📊 How PROBE works
1. Test Case Generation
12,000 cases across 8 verticals (~1,500 per vertical). Each case has a ground-truth label: hallucinated (false positive) or factual. Cases include edge types: borderline claims, partial verifiability, and multi-claim outputs.
2. MUG 4-Round Debate
Each case runs through MUG (Multi-Agent Uncertainty Grounding): champion
answers, challenger flags claims, champion grounds, scorer rates.
Output: factual_score (0–100) and verdict.
3. G5 External Verification
Flagged claims are routed to SEC EDGAR, HHS/Federal Register, and EUR-Lex adapters for live verification. Results are cached. Lookup mode (live/cached/mocked) is logged per case.
4. AUROC Calculation
AUROC = fraction of correctly ranked pairs. For all pairs (true_hallucinated, factual), AUROC = fraction where score(true_hallucinated) > score(factual). Higher AUROC = better discrimination.
5. HMAC-SHA256 Signing
The result payload (auroc, case_count, run_date)
is signed with HMAC-SHA256 using AUDIT_HMAC_KEY.
Signature is published so you can independently verify the numbers.
6. Rate Limiting
G5 adapters respect rate limits. SEC/HHS/EUR-Lex are never hammered —
lookups are cached or mocked during benchmark runs.
PROBE_MOCK_EXTERNAL=true forces all external lookups to mocked mode.
PROBE result signature
What this score means for you
AUROC = 0.847 means
If you randomly pick one hallucinated output and one factual output from Sturna’s responses, Sturna scores the hallucination lower (more flagged) than the factual output 84.7% of the time. That’s strong discrimination.
Hallucination detection is measurable
Unlike surveys or self-assessments, PROBE is a quantitative benchmark run against ground-truth labels. It’s repeatable. It’s signed. You can compare it across vendors or ask Sturna to re-run after a model update.
Per-vertical scores matter
A composite AUROC can hide weak verticals. The table above shows AUROC broken down by vertical — finance might score 0.91 while healthcare scores 0.78. That’s actionable.
HMAC proof prevents score inflation
The HMAC-SHA256 signature makes it computationally infeasible to alter the results after the run without the key. Sturna can’t retroactively improve the score — it’d break the signature.