V5.1: Formal Validation Metrics
Four-model jury (added Llama) with Krippendorff's alpha validation. Pilot phase for research credibility.
V5.0 demonstrated that jury consensus works, but lacked formal inter-rater reliability (IRR) metrics. For academic rigor and publishability, researchers need Krippendorff's alpha, ICC(2,k), and pairwise agreement statistics.
Four-Model Jury (V5.1 Innovation)
V5.1 adds Llama (open-weights model via OpenRouter) as the fourth juror. This increases methodological diversity beyond proprietary cloud models.
Krippendorff's Alpha
α ≥ 0.80 = Excellent
α ≥ 0.70 = Acceptable for publication
α < 0.70 = Criterion rejected or evidence revised
ECRA Reliability Statement
All V5.1 results include an Evidence-based Credibility & Reliability Attestation (ECRA) statement documenting validation metrics.
| Status | Pilot / Awaiting calibration audit |
| Krippendorff's α Target | ≥ 0.70 per criterion |
V5.1 revealed that IRR metrics are non-negotiable for research credibility. While Llama added diversity, the real innovation was formal validation—making jury agreement measurable and publishable.