Research Lab

https://github.com/codecot/skills/tree/main/probes/mafia ↗

A small experimental lab for questions the systems raise — predictions registered before the data, a null baseline for every metric, and nulls reported as intervals rather than as "no effect".

The problem. Building multi-agent systems generates opinions faster than evidence. “Give each agent a persona and they will argue” is the kind of claim that sounds obvious, gets built, and is never measured. Someone has to hold the instruments.

What runs. Controlled experiments with the discipline that makes an answer worth having: predictions written down before the data, a dice-null baseline beside every number, paired arms so each run has a twin, bootstrap intervals, and a null reported as an interval plus a minimum detectable effect. The published study measured how a committee of models degenerates: agents with no private information voted with the crowd 78.9% of the time against a 58.4% chance baseline, while the two roles holding private facts sat at chance. Four prompt-level interventions moved nothing — the best of them −0.9 points at p=0.73. Putting every seat on a frontier model made individual play flawless (acts on private information 100% against 84%; never votes against its own side, 0.0% against 15.1%) and left group conformity where it was: +1.7 points, p=0.62.

A refereed social-deduction game was the instrument, picked because it is the smallest setting where disagreement is scoreable — someone is lying and the referee knows who. The tool is open-source.

Designed next. Arms that add asymmetry rather than hygiene: structured public history, information only some seats hold, and genuinely different payoffs. The prediction is registered already — hygiene will clean hygiene and leave conformity untouched.

The transferable part. Capability fixed the players and left the room exactly as it was. If you want disagreement, engineer asymmetry in the rules; you cannot request it in a prompt.