Why municipal open data portals go stale

Municipal open data portals promise machine-readable tables of restaurant inspections, building permits, budget lines, and traffic incidents. A resident checking an inspection score, a reporter building a map, or a software agent fetching a CSV that has not been refreshed since 2019 all depend on those tables staying current. This page is about why many city portals go stale after a splashy launch, which design choices make that failure common, and which practices keep a portal useful. It serves problems in living (people need trustworthy local facts to act), the abundance of reliable civic knowledge (fresh, usable public data instead of broken links and PDF dumps), and research or teaching on this wiki about how civic information systems actually work.

Launch money without maintenance money

Many portals begin as grant-funded or politically timed projects: buy a Socrata, CKAN, or DKAN instance, publish a first batch of datasets, announce transparency. The Sunlight Foundation's open data policy guidelines treat ongoing funding as a separate requirement from the launch itself. Staff (administrative, technical, and legal), software, training, and server maintenance are recurring costs, not one-time line items. Without that budget, the portal ages in place.

A 2021 Urban Data Pioneers report on Tulsa's open data program is blunt about the pattern. Data stewards spent only a fraction of their time on the portal, there was no dedicated open data funding, and no proactive work was being done to populate the catalog. The GIS platform was chosen in part because it could be used at no charge. The same report contrasts that thin model with Boston's Analyze Boston portal, which had a chief data officer, an analytics team, dataset previews, last-update dates, view counts, and multiple download formats. The difference is not ideology. It is staffing and continuous attention.

Catalogs that score well and sit unused

Volume is a poor proxy for value. Austin's Office of the City Auditor, in a July 2023 audit of the city's Open Data Portal, found more than 4,000 assets (datasets, charts, maps, filtered views, and stories). Many were tied to past events or initiatives and were outdated. Forty-two percent of assets had fewer than 100 views; some had one. A utilization target of more than 1,200 annual accesses was met by only 12 percent of assets in 2021 against a 15 percent goal.

A study of roughly 400,000 datasets across 28 portals (Journal of Information Science, 2023) found most datasets underutilized regardless of portal size or platform, and better metadata alone did not clearly drive higher use. Open Knowledge International argued in 2017 that mass publication shifted barriers rather than removing them when data stay split across sites, formats, and access methods.

A JeDEM 2019 case study of New England state portals shows the same pattern. Vermont's portal stayed moderately used; New Hampshire's agency-link approach was inconsistent; Maine's portal went dormant soon after creation and was later taken down. Launch does not guarantee the outcomes in the press release.

Schema drift, broken pipelines, and last-updated dates that lie

Staleness is not only old files. It is also mismatch between the portal and the department system of record. Austin's auditors sampled eleven of the most accessed assets and found that six did not match department source data. Gaps ranged from two missing animal records to hundreds of thousands of missing 311 service requests. Real-time traffic incident reports on the portal showed 85 fatal crashes in a period where the Public Safety Data Warehouse showed 149. Manual uploads invited typos and skipped months; automated feeds were set and then not monitored. Metadata sometimes claimed an update that had not happened. No single person or department owned verification.

That failure mode matters for agents as much as for people. An API that returns yesterday's schema with today's date, or a CSV whose columns silently renamed, breaks maps and notebooks. Link rot is the cousin problem when download URLs die while the catalog row remains. Parallel quality fights appear in hospital price transparency files. Local copies, as discussed under local-first software and data ownership, help when the live portal cannot be trusted.

Update frequency metadata is often missing. A 2024 University of Waterloo thesis measuring open data quality across Waterloo Region municipalities reported an average timeliness score of 0.54, the lowest metric studied, with examples such as a cycling infrastructure dataset created in 2015 and last updated in 2019. Cross-jurisdiction portals amplify the problem: one city refreshes trails in 2024 while a neighbor's layer still dates to 2020, and merged products inherit the weakest link.

PDF and HTML dumps instead of tables

A portal full of PDFs and scraped HTML tables can satisfy a "data published" checkbox while blocking reuse. The Open Definition's open format guidance is explicit: digital is not the same as machine-readable. A PDF that contains a budget table is readable by a person and awkward for a computer; the same table as a spreadsheet or CSV is processable. Bulk download and APIs matter for the same reason. An agent that needs restaurant inspections for a whole city cannot economically OCR hundreds of weekly PDFs, and a journalist on deadline cannot either. Open Knowledge's quality writing stressed processability, timeliness, findability, and completeness as distinct qualities. Prefer CSV, JSON, GeoJSON, or other documented formats; keep a stable schema; publish a changelog when columns change; retire dead sets instead of leaving them labeled "current."

What actually keeps a portal alive

  • Named stewards with time on the clock. Tulsa's stewards existed on paper, but without funded hours the catalog stayed reactive. Austin's audit noted the absence of a chief data or analytics officer and the split between the platform team and departments with no verifier in between. Living portals assign a human owner per priority dataset and a citywide owner for the catalog.
  • Usage feedback as a maintenance signal. View counts, download counts, issue trackers, and request forms tell stewards what to refresh and what to retire. Austin recommended using access metrics to decide what deserves visualizations and what should leave the portal.
  • Update SLAs and matching pipelines. Sunlight's guidelines call for mandating ongoing publication and updates, not a one-time release. Tie each dataset to a refresh schedule, automate from the system of record when possible, and still spot-check that the portal matches the source.
  • Bulk APIs, changelogs, and retirement. Agents and heavy users need stable endpoints and documented schema changes. Dead event datasets should be archived or removed so "last updated" means something. Boston-style previews and format choices lower the cost of discovering whether a file is worth downloading.

Design for consumers, not for checklist scores. If nobody known will use a dataset, publishing it once and never refreshing it creates false confidence. If someone will use it (inspection scores, permits, crash maps, budget actuals), budget the refresh the way the city budgets the underlying service. That is the practical path to more reliable civic knowledge for residents, reporters, researchers, and the agents described in a guide to this wiki for AI agents. Related tooling gaps belong with open-source software that should exist.

  • Why might a portal with 200 well-stewarded datasets outperform one with 4,000 rarely viewed assets?
  • How should a city decide which datasets to retire rather than leave labeled current?
  • Essay idea: compare grant-funded portal launches with libraries or road maintenance as recurring public services.
  • Essay idea: when does a high score on an open data index hide low real-world use?
  • AI prompt idea: Given a portal catalog export with last-updated dates and view counts, rank datasets for refresh, visualization, or retirement.
  • AI prompt idea: Compare a sample CSV from a city portal to a department FOIA extract and list schema and row-count differences.
  • AI prompt idea: Draft an SLA template that ties each priority dataset to a steward, refresh interval, and public changelog.
  • Research idea: measure how often municipal restaurant-inspection CSVs break schema across a year of weekly pulls.

See also

References