Link rot and the scarcity of reliable knowledge

Why Broken Links Make Reliable Knowledge Scarce and How Durable Citations Restore Abundance

Last checked October 2026.

This page treats link rot as a form of knowledge scarcity and explains how web archiving, content addressing, and permanent storage can keep knowledge abundant. It serves the abundance and post-scarcity focus of this wiki, and it also helps with a common problem in living for students, teachers, researchers, and wiki editors: citing a source that later vanishes. The thesis is simple. Publishing on the web is abundant, but the ability to check a claim against the exact source it rests on is scarce, because that ability depends on someone paying, every year, to keep the path back to the source alive. Durable citations turn that fragile path into a shared asset.

The reader question is concrete. If you cite a web page today in an essay, a wiki article, or a knowledge base, what are the odds that a reader in ten years can see what you saw, and which habits raise those odds?

How much of the web goes missing

A 2024 Pew Research Center study sampled just under one million pages from the Common Crawl archive across the years 2013 to 2023 and tested whether they still loaded. A quarter of those pages were no longer accessible as of October 2023. For pages captured in 2013, the share was 38 percent, compared with 8 percent for pages captured in 2023.

About 23 percent of sampled news pages and 21 percent of sampled government pages contained at least one broken link. Among English Wikipedia pages, 54 percent had at least one reference link that pointed to a page that no longer exists, and 11 percent of all reference links were inaccessible. Pew counted a page as gone only when it returned an error that clearly showed the page or its server no longer existed. Pages whose content had changed were outside the scope of the study, so these figures understate the full problem.

Link rot and reference rot are different failures

Researchers separate two kinds of decay. Link rot means the address no longer returns any content. The second kind means the address still works but no longer shows what the author cited. Zittrain and his coauthors call this reference rot. Klein and colleagues call it content drift and use reference rot as the name for both failures together.

In a 2014 study of three Harvard law journals and all United States Supreme Court opinions, Zittrain, Albert, and Lessig found that more than 70 percent of the URLs in the journals, and 50 percent of the URLs in the opinions, no longer led to the information originally cited. They also showed that a normal success code from a server is weak evidence. Many links that loaded were redirects to a home page, custom error pages, or pages that had been quietly updated.

Science shows the same pattern. Klein and colleagues examined over one million references to web resources drawn from over 3.5 million science, technology, and medicine articles published from 1997 to 2012. They estimated that one in five of these articles suffers from reference rot. Among articles that cite web resources at all, the share rose to seven in ten.

Why this is a scarcity problem

A printed book in many libraries is hard to lose and hard to alter. A web page usually lives in the custody of one host. Zittrain and his coauthors point out that the third party hosting a cited page often has no strong interest in keeping old links working for people who cite them. Tim Berners-Lee made a similar point decades ago: web addresses do not change by themselves; people change them, during redesigns, reorganizations, and shutdowns.

The benefit of a stable link goes to future readers, while the cost falls on a host who may never meet them. Goods with that shape tend to be undersupplied. When a source disappears, the knowledge it held does not vanish from the world all at once, but it becomes expensive again. A reader must search, guess, trust a claim without checking it, or redo the original research. Abundance in this sense means that checking is easy and inexpensive. Link rot pushes checking back toward scarcity.

Three jobs a durable citation must do

A citation that survives has to do three separate jobs, and each tool below focuses mainly on one of them.

• Keep a copy. Web archives store snapshots. The Internet Archive Wayback Machine lets anyone view archived versions of public sites, but it does not collect everything. It skips pages behind passwords or forms, honors robots exclusions and owner requests, and can lose the interactive parts of dynamic pages. Perma.cc, built at Harvard Law School Library with other law libraries, archives a page at the moment of citation and returns a link to that record.
• Prove the copy is the same. Content addressing names data by a cryptographic hash, so any change in the content produces a different identifier. A hash cannot keep a file alive, but it lets a reader confirm that a copy found anywhere matches what the author cited. See How content hashing works for the mechanics.
• Keep more than one keeper. The IPFS documentation states plainly that the network makes content discoverable but does not guarantee that it stays available, and that content can be lost if the one sponsor paying to pin it stops paying. Arweave describes itself as permanent information storage, a claim that rests on its protocol design and economics. Details are covered in Permaweb basics and IPFS and Arweave compared.

When the rescuers need rescuing

Preservation tools can rot too. In 2014, Zittrain and his coauthors noted that WebCite, an early citation archive, had threatened to stop accepting new links unless it received donations, and they warned against relying on any single organization.

The Memento project offers a recent example. Memento defined a standard, RFC 7089, that lets software ask for a page as it existed at a given time, and major web archives support it. Its website, launched in 2009 and run by the Research Library of Los Alamos National Laboratory, was taken down toward the end of 2025 after a managerial decision. A static site run by volunteers brought back its documentation in 2026, but its cross-archive search service and its Robust Links service were discontinued.

Klein and colleagues identified a related trap. Replacing an original link with a link to one archive simply trades one point of failure for another. Their alternative keeps the original address, adds the address of a snapshot, and records the date of citation, so a reader can fall back to any archive that holds a matching copy.

Practical habits for writers, teachers, and wiki editors

• Archive a source when you cite it, not later, so the snapshot matches what you read.
• Keep three things in each important citation: the original address, an archive address, and the date you accessed it.
• For files you publish yourself, such as datasets or reports, publish a hash beside the download so others can verify mirrors.
• Spread copies across independent keepers, such as a public archive, a Git mirror, and a permanent store, rather than trusting one service.
• If you run a site, design addresses you can keep for decades, and redirect old addresses when you reorganize.
• Run periodic link checks and repair or annotate broken references.

Honest limits

• Archiving does not settle what may be republished. Archives can remove material on copyright grounds, and personal data stays sensitive after it is archived. See Publish gates before permanence.
• A matching hash proves that a copy is identical, not that its claims are true.
• Permanent storage and archiving have costs. Someone still pays for disks, bandwidth, and staff.
• Paid access can limit who preserves. Perma.cc is free for academic institutions and courts, while other users need a subscription.
• Dynamic and personalized pages remain hard to capture faithfully.

How durable citations move knowledge toward abundance

Knowledge becomes abundant when it is inexpensive to find, easy to check, and hard to lose. The studies above show that the web falls short on the second and third goals. The design choice that matters most is redundancy of keepers, because every single keeper, however well funded today, can fail. A wiki that already keeps old revisions fits this pattern well, and the Memento project noted that no major wiki had yet adopted its time protocol. That gap is an opening for builders. The permanent decentralized wiki design on this site is one attempt to put these ideas to work.

Discussion questions, essay ideas, and learning related AI prompt ideas

Discussion questions

• Who should pay to keep a cited source available: the author, the publisher, a library, or the reader?
• Is a page that loads but has changed worse than a page that returns an error, and why?
• Which sources in your own work would hurt the most if they disappeared next year?

Essay ideas

• Compare the Pew 2024 findings with the 2014 legal and scientific studies and argue whether reference rot is getting better or worse.
• Explain why stable links behave like an undersupplied public good, using one real example of a reorganized website.
• Evaluate whether content addressing alone can solve reference rot, or whether it needs archives and multiple keepers.

Learning related AI prompt ideas

• Ask an AI system to explain the difference between link rot and reference rot with two everyday examples, then quiz me with five questions.
• Ask an AI system to check the external links in one of my essays and list which ones lack an archived copy.
• Ask an AI system to outline a small research project that measures link rot in the references of one wiki category.

See also

• Post-scarcity economy
• How content hashing works
• Content-addressed wikis
• Permanent decentralized wiki design
• Local-first software
• Community mesh networks
• Open wikis and the permaweb
• Open research