V5.2: The Deepseek Experiment
A deliberate test of five-model jury consensus. Why it was attempted, why it failed, and what we learned.
V5.2 attempted to extend the four-model jury (V5.1) by adding Deepseek. The hypothesis was that a fifth diverse model would improve consensus. Testing revealed this assumption was incorrect, and the experiment was rejected. This case study documents why methodological rejection is sometimes the right decision.
Hypothesis
If jury consensus improves with diversity, adding a fifth model (Deepseek) should further reduce jury spread and increase consensus_strong frequency.
Model Selection Rationale
- Deepseek: Open-weights Chinese model with strong reasoning performance
- Diversification goal: Reduce Western model concentration
- Cost consideration: Available via OpenRouter at lower cost
| Metric | V5.1 (4 models) | V5.2 (5 models) | Impact |
|---|---|---|---|
| Avg Jury Spread | 6.2 pts | 8.7 pts ↑ | WORSE |
| Consensus Strong % | 48% | 28% ↓ | WORSE |
| Avg Krippendorff's α | 0.74 | 0.61 ↓ | Below threshold |
V5.2 was rejected because jury spread increased by 40%, consensus strength decreased by 42%, and Krippendorff's α fell below publication threshold (0.61 < 0.70). Deepseek was a systematic outlier on 62% of criteria. Adding a model that worsens consensus is worse than no additional model.
Why This Was the Right Call
Continuing with V5.1 rather than forcing V5.2 into production exemplifies methodological rigor. Not every idea improves the system. Testing and rejection is preferable to adoption of sub-optimal changes.
For Model Selection
- Diversity ≠ Improvement: More models doesn't always improve consensus. Only adopt models that maintain or improve IRR metrics.
- Pilot first: Always test on a representative sample before committing to production.
- Watch for systematic bias: If a new model is consistently higher or lower, investigate root cause before integration.
For Future Methodology Changes
- Define success criteria in advance: What metrics must a new approach beat?
- Test against multiple criteria: Not just one; use spread, consensus_strong, α, and expert judgment.
- Document rejections: Prevent re-testing of failed approaches.
- Accept incrementalism: Sometimes the best progress is staying with a proven system.