Jump to content

Distributing AI model weights with BitTorrent and content addressing

From IdeaWazaWiki
Revision as of 05:26, 16 November 2026 by Waza (talk | contribs) (New article (drafted with AI assistance))
(diff) ← Older revision | Latest revision (diff) | Newer revision → (diff)

Distributing AI model weights with BitTorrent and content addressing is the practice of moving the very large files that make up an AI model through peer-to-peer swarms and hash-named storage instead of a single download server, and of proving that the file that arrives is the one the publisher released. On December 8, 2023, Mistral AI released its Mixtral 8x7B model by posting a link to a torrent with no further explanation; the download was 87 GB.[1] The release showed that a swarm can carry a frontier-scale model, and it also raised the questions this page answers: how such files stay available, and how a downloader knows they are genuine. This page belongs to the Freedom Tech work of this wiki.


Why model files strain ordinary hosting

Hugging Face describes the files in its model and dataset repositories as large and binary, typically gigabytes and in a few cases terabytes each. Storing them directly in Git is impractical, because cloning a repository fetches every past revision, so the Hub keeps small pointer files in Git and the real data in separate storage. Its newer Xet backend splits files into chunks and stores repeated chunks only once.[2] All of that still runs through one company. If the company changes its terms, removes a model, or goes offline, every pointer leads nowhere.

How a swarm moves the bytes

BitTorrent spreads the upload cost across the people downloading. When many people fetch the same file at once, they send pieces to each other, so the original source can serve a very large audience with only a modest increase in load.[3] A magnet link names the content by its hash instead of by a server address, which is why one short link was enough for the Mixtral release.

Version 2 of the protocol suits model files particularly well. Each file gets its own SHA-256 Merkle tree built from 16 KiB blocks, so a client can verify every small block as it arrives, and the specification notes that identical files always produce the same root hash.[3] A model split into several weight shards therefore carries a per-file fingerprint that any mirror can check.

Keeping a model available

The weak point of a swarm is time. A 2010 measurement study of more than 46,000 torrents found that 38 percent became unavailable within their first month, and confirmed that seeders, the peers holding complete copies, strongly affect whether a file can still be fetched.[4] The protocol's own designer observed in 2003 that downloaders frequently stop uploading as soon as their download completes.[5]

Three known fixes address this. Web seeding, specified in BEP 19, lists ordinary HTTP or FTP mirrors inside the torrent so that a plain web server acts as a seed that never leaves.[6] Bundling related files into one swarm helps too; a 2009 study found that even a limited amount of bundling sharply reduces unavailability.[7] Institutional seeding is the third. Academic Torrents describes itself as a community-maintained distributed repository for research datasets and offers BitTorrent infrastructure for public data.[8] How to make running such seeds easy and worthwhile is the subject of Making decentralized nodes easy and worth running.

Content addressing beyond torrents

IPFS names data by a content identifier, or CID, derived from a cryptographic hash, so any change produces a different CID. The IPFS documentation also warns that a CID is not the same as a file's SHA-256 checksum, because IPFS splits content into blocks and adds codec information.[9] A torrent infohash, an IPFS CID, and a plain SHA-256 checksum are therefore three different names for the same weights. A careful publisher lists all three in one manifest, and can store that small manifest permanently, as described in ICP versus Arweave and AO for decentralized platforms.

Proving a download is genuine

A matching hash proves that the bytes match the hash a downloader was given. It does not prove who produced them, and it says nothing about whether loading the file is safe.

The format matters as much as the transport. Pickle, the default format for PyTorch weights, can run arbitrary code when a file is loaded.[10] In early 2024 the security firm JFrog reported roughly 100 model uploads on Hugging Face that ran hidden code when loaded, including about 10 it judged truly malicious, one of which opened a remote shell on the user's machine.[11] Safetensors was built as a format for storing tensors safely, without that code execution path.[12]

Provenance needs signatures. In April 2025 the OpenSSF released version 1.0 of its model signing library, which signs and verifies models of any format and size and supports Sigstore, self-signed certificates, or ordinary key pairs.[13]

A practical recipe

  • Publish weights as safetensors, not pickle files.
  • Write a manifest listing every file with its SHA-256 checksum, size, license, and the torrent infohash or CID.
  • Sign the manifest, and store a copy where it cannot be silently altered.
  • Create a version 2 torrent with at least one web seed, so the swarm survives quiet periods.
  • Before loading, check the signature and every file hash against the manifest.
  • Why might a laboratory choose to release a large model as a torrent instead of on a company hosting service, and what does it give up?
  • How does a per-file Merkle tree in BitTorrent version 2 change what a downloader can verify, and when?
  • What would a fair system look like for keeping seeders online for unpopular but valuable models?
  • How should a downloader decide which publisher identity to trust when a model is signed?
  • What are the differences between checking a hash, verifying a signature, and judging whether a model behaves safely?
  • How could libraries, universities, and archives share the work of keeping important open models available for decades?
  • Which risks remain even when every file is safetensors and every hash matches?

Readings from Wikipedia

See also

References