IdeaWazaWiki

Prediction markets for scientific replication

Using Prediction Markets to Find Which Scientific Findings Will Replicate

Prediction markets for scientific replication are small trading games in which researchers buy and sell contracts that pay out if a published study holds up when it is repeated. This page is about learning and research methods, and about abundance: reliable knowledge is scarce, the money to check it is scarcer, and these experiments suggest that scientists already hold much of the knowledge needed to decide which findings deserve trust. The central idea is surprising. Groups of researchers trading with about 100 US dollars each predicted which psychology studies would fail to replicate better than a simple survey of the same people did, before a single replication result was public.[1]

The problem: a shortage of trustworthy findings

A published, peer-reviewed result looks like settled knowledge to a student, a journalist, or a policymaker. Large replication projects show that this appearance often misleads. In 2015 the Open Science Collaboration repeated 100 studies from three psychology journals. Ninety-seven percent of the original studies reported statistically significant results, but only 36 percent of the replications did, and replication effects were about half the size of the originals.[2]

The pattern is wider than psychology. A replication of 18 laboratory experiments from two leading economics journals found a significant effect in the same direction for 11 of them, or 61 percent.[3] A replication of 21 social science experiments published in Nature and Science between 2010 and 2015 found a significant effect in the same direction for 13 of them, or 62 percent, with effects averaging about half the original size.[4] The 2015 market paper cites an estimate that irreproducible preclinical research costs about 28 billion US dollars a year in the United States alone.[1]

Replication is slow and expensive, so only a small share of findings will ever be checked directly. That creates a practical research question: if we cannot repeat everything, how do we decide what to repeat first, and how much should we trust the rest in the meantime?

How a replication market works

The design is simple. For each study that is scheduled for replication, the organizers open a market with a contract that pays one unit if the replication succeeds and nothing if it fails. Traders buy when they think the price is too low and sell when they think it is too high. The final price can be read as the group's estimate of the chance that the study will replicate. A price of 0.80 means the crowd thinks success is likely. A price of 0.20 means the crowd expects failure.

In the 2015 study, invitations went to the email list of the Open Science Framework, and each participant received 100 US dollars to trade with. Before trading opened, participants also answered a survey giving their own probability for each study.[1] That design allowed a direct comparison between individual opinions and the market price that emerged from many people trading against each other.

What the experiments found

Treating a final price above 0.50 as a prediction of success and below 0.50 as a prediction of failure, the markets correctly called 29 of 41 replications, or 71 percent. A simple average of the pre-trading survey correctly called 23 of 40, or 58 percent, which was statistically indistinguishable from a coin flip.[1] The same paper estimated that the hypotheses tested in those psychology studies had a median prior probability of only 9 percent of being true, which helps explain why a single significant result is weak evidence on its own.[1]

Later work pooled four forecasting projects covering 103 published findings in the social and behavioral sciences. Across that larger set, prediction markets determined replication outcomes with 73 percent accuracy, and both market prices and average survey answers were clearly correlated with what actually happened.[5] The authors of the Nature and Science replication project reached a similar conclusion: peer beliefs about replicability were strongly related to real replicability, which suggests the research community could tell in advance which results would hold up.[4]

The original paper drew the policy lesson directly. Its authors argued that prediction markets could give fast information about reproducibility at low cost, and could even help decide which studies to replicate so that limited replication funds go where they matter most.[1]

From markets to machine confidence scores

The idea also moved into government research funding. The US Defense Advanced Research Projects Agency ran a program called SCORE, short for Systematizing Confidence in Open Research and Evidence. Its stated aim was to build automated tools that assign explainable confidence scores to social and behavioral science claims, with reliability equal to or better than the best current human expert methods. The agency lists the program as complete.[6] In that framing, human forecasts such as market prices become the benchmark that software has to match.

This points toward a future in which humans and software agents work together on the question of trust. People supply judgment and stakes. Agents read large literatures and flag claims that look like past failures. Neither replaces the replication itself, which remains the only direct test.

Why it matters for abundance

Most talk about abundance focuses on energy, food, or housing. Reliable knowledge is an input to all of them. A medical researcher, an engineer, or a community group that builds on a false finding wastes time and money, and the waste compounds. Low-cost signals of trust make good knowledge easier to find, the same way search engines made information easier to find.

Replication markets also turn knowledge that already exists, but sits unused in researchers' heads, into a public number. That is a recurring abundance pattern: the scarce resource was never the knowledge itself but a way to pool and share it. The pattern has clear value for fields where false hope is costly, such as claims about aging and longevity treatments, where readers already need tools for judging evidence.

Limits and open questions

  • Accuracy is good, not perfect. A 71 to 73 percent hit rate means roughly one call in four is wrong. A market price is a guide to where to look, not a verdict on any single study.
  • Small samples. The pooled data cover about 100 findings, mostly in psychology, economics, and related social sciences. Results may differ in biology, medicine, or physics.
  • Self-fulfilling risk. If funders use prices to choose what to replicate, low-priced studies get tested and high-priced ones may never be checked, which could hide errors in popular findings.
  • Herding and incentives. Traders may follow famous names or fashionable theories. Researchers may also hesitate to bet publicly against colleagues.
  • Legal limits on real-money markets. Many places restrict real-money prediction markets, which is one reason research markets have used small fixed stakes or points. This page is general education, not legal guidance.
  • Markets do not replace replication. Prices only aggregate what people believe. The repeated experiment is still what settles the question.

How students and researchers can use the idea

Even without a market, the method suggests useful habits. Before trusting a striking result, ask several informed people to write down a probability that it would replicate, then compare their answers. Treat a single significant result with a low prior as provisional. When a study has already been tested in a large replication project, read the replication before citing the original. Groups that run journal clubs or reading circles can practice this by forecasting outcomes of announced replications and keeping score over time, which builds calibrated judgment.

  • Explain why a group of traders could predict replication outcomes better than the average of the same people answering a survey.
  • Describe how a research funder could use market prices to choose which studies to replicate, and what errors that rule might create.
  • Compare human prediction markets with automated confidence scores for scientific claims, and discuss what each does well and poorly.
  • Argue whether reliable knowledge should count as a resource in discussions of abundance and post-scarcity.
  • Design a classroom forecasting game in which students predict the results of announced replication studies and track their accuracy.
  • Discuss how replication forecasts could help readers judge claims about anti-aging and longevity treatments.
  • Explore what happens to scientific careers and incentives if every new paper receives a public replication price.

See also

References

  1. ↑ 1.0 1.1 1.2 1.3 1.4 1.5 Dreber, A., Pfeiffer, T., Almenberg, J., Isaksson, S., Wilson, B., Chen, Y., Nosek, B. A., and Johannesson, M. (2015). Using prediction markets to estimate the reproducibility of scientific research. Proceedings of the National Academy of Sciences 112(50).
  2. ↑ Open Science Collaboration (2015). Estimating the reproducibility of psychological science. Science 349(6251).
  3. ↑ Camerer, C. F., Dreber, A., Forsell, E., et al. (2016). Evaluating replicability of laboratory experiments in economics. Science 351(6280).
  4. ↑ 4.0 4.1 Camerer, C. F., Dreber, A., Holzmeister, F., et al. (2018). Evaluating the replicability of social science experiments in Nature and Science between 2010 and 2015. Nature Human Behaviour 2.
  5. ↑ Gordon, M., Viganola, D., Dreber, A., Johannesson, M., and Pfeiffer, T. (2021). Predicting replicability: Analysis of survey and prediction market data from large-scale forecasting projects. PLOS ONE 16(4).
  6. ↑ Defense Advanced Research Projects Agency. SCORE: Systematizing Confidence in Open Research and Evidence. Program page.
Retrieved from "https://ideawaza.com/index.php?title=Prediction_markets_for_scientific_replication&oldid=113783"