← All experiments

How the archive is judged

Claude Sonnet 5 reviews each exact prompt and answer. Model names are replaced with neutral IDs during judging. Writing may still identify a model; this is not guaranteed blinding. No new benchmark answers are generated.

Most interesting first

Claude assigns each answer an interest rating from 0 to 4: routine, minor, useful, revealing, or especially illuminating. A second pass rates each complete comparison from 0 to 10 using its exact prompts, coverage, prior answer ratings and up to five selected response excerpts or choice summaries. Questions sort by that comparison score, with question ID as the tie-breaker. Until that second pass is available, the provisional order uses mean answer interest scaled from /4 to /10; comparisons with any unreviewed answers appear last. These are editorial judgments across separate batches, not calibrated scientific measurements. Search covers the complete responses.

Separate scores for different questions

We never convert a rescue choice into a vegan-alignment score. Preserving a human, animal or hypothetical sentient robot can each be argued for; the reasoning and premises are shown.

Some judge outputs omit the optional premise-fidelity field. These are recorded as missing, not treated as a pass or failure; every answer still receives interest and applicable animal-ethics ratings.

Coverage and limitations

V1 · Diet and veganism: original questions

144 answers · 12 questions · 12 models

Early collection; context isolation caveat applies.

V2 · Animal protection, health and practical advice

360 answers · 26 questions · 12 models

Separate question set and collection protocol; do not pool model rankings across studies.

V3 · Hard cases: humans, animals and robots

653 answers · 28 questions · 12 models

653 of 672 planned responses: 2 failed saved attempts and 17 not started. Robot/chicken subset is complete (192). Rescue order is confounded with model.

V4 · Short rescue and trolley dilemmas

144 answers · 12 questions · 6 models

Separate question set and collection protocol; do not pool model rankings across studies.

V5 · Pets, dog meat and everyday situations

80 answers · 20 questions · 4 models

Separate question set and collection protocol; do not pool model rankings across studies.

V6 · Short questions about vegan ethics

80 answers · 10 questions · 4 models

Separate question set and collection protocol; do not pool model rankings across studies.

V7 · Goals, reassurance and practical limits

96 answers · 18 questions · 4 models

Adaptive questions selected after reviewing initial answers.

One judge, no independent agreement estimate. Claude is itself among the tested models. Interest scores may vary with batch context. Prompts differ in assumptions, answer length and framing; studies cannot support a pooled model leaderboard, historical improvement claim, or claim about real-world conduct. The latest study’s earlier reviewed scores remain separately labeled on the home page.

Earlier collection tools did not capture full effective provider context. The Astra pigs-versus-human audit records unexplained input-token variation and a skills-isolation warning. The supplied system/user prompts are visible, but that is not proof of perfect isolation.

Highlights are exact contiguous excerpts when verified against the saved answer. Full answers preserve all qualifications. Every published response is hash-checked against its source. Download all records and judgments.