Worldclass Research Org

What it is

Multi-agent systems are increasingly proposed as autonomous research organizations, yet their quality control is rarely measured. Kacper Saks built this one as a domain-agnostic organization of 39 Claude agent role files plus an orchestrator, in 13 divisions, running a 9-phase lifecycle with 4 human checkpoints. Its structural rule is that nothing self-certifies: every role output must pass an executable, machine-checkable gate and then adversarial sign-off from a paired critic with the authority to kill it. It is built to take research from ideation to a submission-ready package; submission itself stays human-only.

• Tests and make verify
• Citation checks
• Grep gates
• Adversarial sign-off from a paired critic

First run and calibration study

On its first end-to-end run (v0.1.0) the organization examined 5 candidate research directions and abandoned all 5 at the gates; its flagship integrity exhibit was retracted after failing its own verifier. Working quality control, or a machine that rejects everything? v0.2.0 tested this with a pre-registered, blinded study of 20 theses: 10 known-flawed, carrying 15 planted defects across 6 flaw classes, and 10 known-sound reproductions, with labels sealed by SHA-256. Rejections named the correct flaw class and location for 13 of 15 defects with a round-1 auto-fail rule on and 14 of 15 with it off. The sample is small, and one model family both planted and judged the defects.

Erratum v0.2.1: what is now exploratory

On 12 June 2026 an independent second review, by a separate LLM reviewer of the same model family, found a genre confound between the corpus arms (limitation L8): the sound theses were self-identifying reproductions and the flawed theses novel proposals, so a gate could score high specificity by recognizing genre rather than reading content. Erratum v0.2.1 (13 June 2026), documentation-only, downgraded to exploratory the cross-arm specificity results, the sound-thesis kill contrast and the rationale for switching the round-1 rule off. Attribution is measured within the flawed arm and is unaffected. The rule stays off as a design choice, unvalidated on fresh data; a fresh validation must use matched-pair arms.

C1: a program it chose for itself

After the calibration study the organization selected its own next project: it froze a 12-candidate slate in a single commit (11 June 2026) before any judgment, then killed 10 at four non-compensatory gates. The pick, C1, is a planned cross-model benchmark of whether third-party LLM paper-review systems reject flawed research for the right reasons. Each flawed document is to have a sound twin differing only in one planted defect, and the author's own model family is barred from every confirmatory cell. A pre-registration lock attempt on 1 July 2026 halted at the program's own eligibility gate. As of the last logged entry (5 July 2026), C1 is dormant at pre-registration, pending one author action.

• About $6 of external spend so far, all on directional pilots
• $0 of the confirmatory budget spent
• Pre-registration drafted, not frozen
• Evaluator roster selected, not frozen

System B: adversarial pre-review

System B is a separate 7-agent organization (a rubric-internalizer, five specialist reviewers and an area chair) that reviews a paper adversarially against the real, frozen rubric of its target venue before any human submits. It aims for high recall of flaws and can never emit a submit action. Because it shares the author's model family, its accept is necessary but not sufficient; the external venue is the only terminal authority. The calibration-study paper has passed System B's internal pre-review against the REALM @ EMNLP 2026 workshop rubric. That is a same-family verdict, not a venue decision; no external review outcome is recorded.

Reproducibility and openness

The release ships the calibration corpus, all 40 verdicts, the sealed-label hashes and the pre-registration. A Citation Fidelity Protocol gives every citation a typed claim card with a verbatim anchor pinned by SHA-256 to its source; it was added after 3 of 5 v0.1.0 abandonment citations were found misread. Eight limitations (L1–L8) are stated in full. Code is MIT-licensed and documentation CC BY 4.0. The Zenodo DOI covers the v0.2.0 deposit; erratum v0.2.1 is a GitHub tag with no Zenodo deposit.

• make calibration-analyze recomputes the calibration numbers from frozen verdicts
• make verify-citations checks every citation card
• make calibration-prereg-check verifies the pre-registration freeze conditions still hold
• make reproduce reruns the retracted v0.1.0 simulation, which is expected to fail its integrity check