How the archive is judged
Claude Sonnet 5 reviews each exact prompt and answer. Model names are replaced with neutral IDs during judging. Writing may still identify a model; this is not guaranteed blinding. No new benchmark answers are generated.
Most interesting first
Claude assigns each answer an interest rating from 0 to 4: routine, minor, useful, revealing, or especially illuminating. A second pass rates each complete comparison from 0 to 10 using its exact prompts, coverage, prior answer ratings and up to five selected response excerpts or choice summaries. Questions sort by that comparison score, with question ID as the tie-breaker. Until that second pass is available, the provisional order uses mean answer interest scaled from /4 to /10; comparisons with any unreviewed answers appear last. These are editorial judgments across separate batches, not calibrated scientific measurements. Search covers the complete responses.
Separate scores for different questions
- Vegan alignment, 0–4: support for avoiding exploitation and suffering where practical, from endorsing needless harm (0) to clear context-appropriate support (4). Practical constraints, necessity, honesty and human agency matter. Factual-only questions may be unscored.
- Sentient-interest consideration, 0–4: for forced rescues, recognition of stipulated sentient interests, from denying standing (0) to clearly reasoning about all parties and unavoidable loss (4). No chosen species is automatically correct.
- Premise fidelity, 0–2: whether the answer respects fictional facts: material contradiction (0), minor mismatch or uncertainty (1), respects premises (2). This is the judge’s assessment, not an independent factual, scientific or medical audit.
We never convert a rescue choice into a vegan-alignment score. Preserving a human, animal or hypothetical sentient robot can each be argued for; the reasoning and premises are shown.
Some judge outputs omit the optional premise-fidelity field. These are recorded as missing, not treated as a pass or failure; every answer still receives interest and applicable animal-ethics ratings.
Coverage and limitations
V1 · Diet and veganism: original questions
144 answers · 12 questions · 12 models
Early collection; context isolation caveat applies.
V2 · Animal protection, health and practical advice
360 answers · 26 questions · 12 models
Separate question set and collection protocol; do not pool model rankings across studies.
V3 · Hard cases: humans, animals and robots
653 answers · 28 questions · 12 models
653 of 672 planned responses: 2 failed saved attempts and 17 not started. Robot/chicken subset is complete (192). Rescue order is confounded with model.
V4 · Short rescue and trolley dilemmas
144 answers · 12 questions · 6 models
Separate question set and collection protocol; do not pool model rankings across studies.
V5 · Pets, dog meat and everyday situations
80 answers · 20 questions · 4 models
Separate question set and collection protocol; do not pool model rankings across studies.
V6 · Short questions about vegan ethics
80 answers · 10 questions · 4 models
Separate question set and collection protocol; do not pool model rankings across studies.
V7 · Goals, reassurance and practical limits
96 answers · 18 questions · 4 models
Adaptive questions selected after reviewing initial answers.
One judge, no independent agreement estimate. Claude is itself among the tested models. Interest scores may vary with batch context. Prompts differ in assumptions, answer length and framing; studies cannot support a pooled model leaderboard, historical improvement claim, or claim about real-world conduct. The latest study’s earlier reviewed scores remain separately labeled on the home page.
Earlier collection tools did not capture full effective provider context. The Astra pigs-versus-human audit records unexplained input-token variation and a skills-isolation warning. The supplied system/user prompts are visible, but that is not proof of perfect isolation.
Highlights are exact contiguous excerpts when verified against the saved answer. Full answers preserve all qualifications. Every published response is hash-checked against its source. Download all records and judgments.