Research System Architecture

V4.0: Anchor-Based Heuristic

Single-model scoring with calibration exemplars. The foundation approach.

Objective

V4.0 established the first systematic approach to scoring. The key innovation was using calibration exemplars to anchor the scoring range and improve consistency.

System Architecture
Organizations to ScoreCalibration Exemplars (Injected)MAGA • NAACP • Girl Scouts • Other AnchorsSingle Claude ModelDeterministic Rule EngineScores (0-10)+ Calibration Notes + Reasoning
Core Components

Calibration Exemplars

V4.0 injected reference organizations into the prompt to establish scoring boundaries. For example, the prompt would include:

"Consider these reference points: MAGA (score: 8.5), NAACP (score: 2.0), Girl Scouts (score: 0.5)..."

The rationale: anchoring forces the model to calibrate its scale and reduces drift.

Deterministic Scoring

Single Claude model applying rules consistently. No jury consensus, no multi-model validation. Output includes numerical score plus reasoning.

Limitations & Trade-offs

V4.0 revealed a critical limitation: anchor overcalibration. Organizations tended to cluster near the injected exemplars, creating a 15-20 point ceiling effect. Additionally, single-model scoring has inherent bias risks and no inter-rater validation.

Key Limitations:

  • Anchor overcalibration (~15-20pt clustering around exemplars)
  • Single-model bias (no external validation)
  • No evidence provenance tracking
  • Lack of inter-rater reliability metrics
  • Non-replicable governance (decisions not auditable)
Implementation

V4.0 implementation retained in the codebase for provenance and back-comparison:

  • scorer_v4.py — Original single-model scorer
  • tests/test_scorer_v4.py — Validation tests

Status: Retired from production, maintained for reference only.

← Previous version not applicable

See how V5.0 solved these limitations →