Adversarial collaboration between rival scientists

How Rival Scientists Can Settle Disputes by Designing the Test Together

Adversarial collaboration is a research method in which people who disagree about a scientific question agree in advance on a single test that could prove one of them wrong, then run it together, often with a neutral arbiter. This page is about learning and research methods. It also serves the goal of abundance: reliable knowledge is a resource, and every decade that a scientific dispute stays open is a decade of wasted grants, confused students, and policies built on guesses. Adversarial collaboration turns a slow war of papers into a faster, shared experiment, and the same habit can help ordinary people with disagreements that are problems in living.

The surprising thesis of this page is that the most productive partner for testing an idea is often the person most convinced it is false, because only a rival will insist on the test that could actually break it.

The problem: disputes that never end

In many fields, two camps can argue for decades without either one changing its mind. Each side runs studies designed by like-minded colleagues, defines key terms in its own way, and publishes results that its own camp finds persuasive. The other side replies with a critique, the first side answers, and the cycle repeats.

In a 2022 paper, Cory Clark, Thomas Costello, Gregory Mitchell, and Philip Tetlock argued that open science reforms such as data sharing and preregistration help, but cannot stop researchers from choosing operational definitions that protect their favorite theories. They wrote that scholars too often conduct "performative research to score points" instead of engaging each other's strongest arguments, which "allows contradictory claims to fester unresolved for decades."

This is a form of scarcity. Research money, lab time, and the attention of skilled people are limited. When they are spent on studies that cannot change anyone's mind, the supply of settled, usable knowledge grows more slowly than it could.

How adversarial collaboration works

The method was named and promoted by the psychologist Daniel Kahneman. A 2001 paper by Barbara Mellers, Ralph Hertwig, and Kahneman described the approach as one that "requires both parties to agree on empirical tests for resolving a dispute and to conduct these tests with the help of an arbiter." The paper also offered a suggested protocol.

The basic steps are simple to state and hard to do.

1. The rivals define the disputed claim in words both sides accept.
2. Each side states, before any data exist, what result would count against its own view.
3. The group designs one study or a short series of studies that could produce either outcome.
4. A neutral arbiter, or a team with no stake in the outcome, helps run the work and handle disagreements about procedure.
5. The results are published whatever they show, including the parts that embarrass each side.

Clark describes the second step as committing to "conditions of falsifiability": each party says, in effect, "if we find B then I agree to at least slightly modify my current perspective."

Three real cases

A fight about a famous bias

The 2001 study grew out of a dispute about the "Linda problem," a classic test in which many people judge that a woman is more likely to be "a bank teller and a feminist" than "a bank teller." Hertwig argued that much of this error came from how people read the word "and," and that rewording would make it disappear. Kahneman disagreed for one of the proposed phrases. Mellers acted as arbiter, and each party was allowed one study after the first experiment. The outcome was mixed. Some results matched Hertwig's predictions, and others matched Kahneman's, and the two still offered different interpretations at the end. The authors wrote that the experiments did not resolve every issue, "nor did this miracle occur," but that they "discovered more" than they would have through replies and rejoinders.

Intuition versus bias

In 2009, Kahneman and the decision researcher Gary Klein published a joint paper on a question that had divided their fields: when should a professional's gut feeling be trusted? Kahneman's tradition stressed the flaws of intuition, while Klein's tradition, called naturalistic decision making, studied how experienced professionals decide in real settings, where intuition often works well. Their shared conclusion was practical. Intuition deserves trust when the environment is regular enough to be predictable and the person has had a real chance to learn its patterns, and "subjective experience is not a reliable indicator of judgment accuracy." The subtitle of the paper was "A failure to disagree."

Two theories of consciousness

The largest test so far took on one of the hardest questions in science. The Cogitate Consortium brought together supporters of integrated information theory (IIT) and global neuronal workspace theory (GNWT), plus theory-neutral researchers. Together they preregistered the predictions, the pass or fail criteria, and how results would be interpreted. Neutral teams then measured brain activity in 256 participants using three methods: fMRI (120 people), MEG (102 people), and intracranial recordings (34 people).

Neither theory won. The paper reports that the results "substantially" challenged "key tenets of both theories." A key IIT prediction of sustained synchronization in the back of the brain was not found, and a key GNWT prediction of a burst of activity in the front of the brain when a stimulus disappears was also largely missing.

The project also showed the method's social risks. After early results were presented in 2023, 124 researchers signed an open letter calling IIT "pseudoscience" and arguing that the experiment had tested only "idiosyncratic predictions" rather than the core of the theory. Other scientists called that label inflammatory. The lesson is that agreeing on a test does not guarantee agreement about what the test means, and choosing which predictions to test is itself a contest.

Why it matters for abundance

Reliable knowledge behaves like infrastructure. Once a question is settled well, every later researcher, teacher, and builder can use the answer at almost no added cost. Adversarial collaboration aims to settle more questions with fewer studies, so it can increase the supply of trustworthy findings without raising research budgets.

This matters for the large goals of this wiki. Longevity science, mental health research, economics of basic income, and claims about new technology are all full of camps that talk past each other. A field where rivals routinely design shared tests would waste less money on studies that only preach to the converted. Readers interested in related tools can compare this approach with replication prediction markets, which aim to find weak findings, while adversarial collaboration aims to end live disputes.

It also matters for critical views of authority. A method that forces the most confident experts to state in advance what would prove them wrong gives outsiders a fair standard for judging expert claims, including claims made with official backing.

Using the idea outside the lab

The same structure can help with problems in living. Two people who disagree about a household budget, a workplace process, or a community project can try a small version.

• Write the disputed claim in one sentence both people accept.
• Each person writes what result would change their mind, before looking.
• Agree on a short trial, such as two weeks of tracking spending one way, and a neutral person or a simple record to judge it.
• Share the result even if it favors the other side.

This does not guarantee agreement, but it moves the argument from who is more stubborn to what actually happened.

Limits and open questions

• Slow and costly to set up. Getting rivals to agree on definitions can take months or years, and the Cogitate project needed a large international consortium.
• Reluctance. Clark and colleagues note that researchers are often reluctant to take part, since a public loss can hurt a reputation.
• Partial tests. Rivals may agree only on predictions that are safe or peripheral, so a theory can survive a test that never touched its core, as critics argued in the IIT case.
• Interpretation still differs. In the 2001 study and the consciousness study, both sides kept their own readings of some results.
• Not every dispute is empirical. Questions about values, rights, or definitions of harm cannot be settled by one experiment, although a shared test can still narrow the factual part of the fight.

Discussion questions, essay ideas, and learning related AI prompt ideas

• Explain why a rival might design a stronger test of a theory than the theory's own supporters would.
• Describe what a fair adversarial collaboration on the effects of a basic income program would need to agree on before collecting any data.
• Compare adversarial collaboration with peer review and with replication prediction markets, and discuss which problems each one solves.
• Argue whether funders should require adversarial collaboration before paying for more studies on long-running scientific disputes.
• Discuss what the 2023 pseudoscience letter about integrated information theory reveals about the limits of agreeing on a test in advance.
• Design an adversarial collaboration between people who disagree about whether a common psychiatric label describes a disease or a problem in living, and identify which parts of that dispute data could and could not settle.
• Explore how two people could use the steps on this page to settle a practical disagreement at home or at work.

See also

• Replication prediction markets
• Collaborative research
• Hypercollaboration
• Science reinvented today
• Writing well researched articles
• Mapping collective knowledge
• Artificial consciousness
• The Rosenhan study
• Problems in living