Jump to content

Why BitTorrent alone cannot replace Hugging Face or GitHub

From IdeaWazaWiki
Revision as of 05:28, 16 November 2026 by Waza (talk | contribs) (New article (drafted with AI assistance))
(diff) ← Older revision | Latest revision (diff) | Newer revision → (diff)

Why BitTorrent alone cannot replace Hugging Face or GitHub comes down to the difference between moving files and running a platform. On 8 December 2023 the company Mistral AI released its Mixtral 8x7B model by posting a single torrent magnet link with no explanation, and anyone with a BitTorrent client could fetch about 87 GB of weights from other downloaders instead of from a company server. A description of the model, its Apache 2.0 license and its benchmark results arrived three days later in a separate blog post.[1][2] The episode shows what peer-to-peer distribution does well and how much of a model hub or code forge lives in layers a torrent does not carry.


What torrents and hash tables do well

BitTorrent splits content into fixed-size pieces. In the original format each piece is identified by a 20-byte SHA-1 hash listed in the torrent's metadata, so a client can reject any corrupted or forged piece from any peer.[3] The version 2 format replaces this with SHA-256 Merkle trees built from 16 KiB blocks, giving every file its own root hash.[4] Because downloaders upload to each other while they download, the original source can serve a very large number of people with only a modest increase in its own load.[4]

The distributed hash table, or DHT, removes the need for a central tracker. It is based on the Kademlia design: every node takes a random ID from the same 160-bit space as torrent infohashes, distances are measured with XOR, and a client finds peers by asking nodes whose IDs sit closest to the infohash it wants. In effect, each peer becomes a tracker.[5] A magnet link only needs that infohash, which is why Mistral could publish a release in one line of text.

For large, unchanging files, this combination is hard to beat. It needs no data center, it verifies every byte, and a popular file gains more sources as more people fetch it.

What a hub adds on top

Hugging Face hosts models, datasets and Spaces as Git repositories, with a custom storage system called Xet for chunk-level deduplication of very large files.[6] Each model repository renders its README.md as a model card. Its YAML metadata records the license, training datasets, base model, task and evaluation results, and the Hub uses those fields so people can filter and search across models.[7] Authors can gate a model so that users must request access first, and they can revoke that access later.[8] Every repository also has discussions and pull requests, stored as special Git references such as refs/pr/42.[9] GitHub offers the same social layer for code: issues, pull requests, reviews and permissions.

The missing layers

Set against those features, a torrent swarm lacks several things.

  • Discovery and search. A DHT answers one question: who has the content with this exact infohash? It offers no keyword search, so finding a model by task or license needs a separate index.[5]
  • Versions. A torrent describes fixed content, so any change produces a new infohash. A draft extension, BEP 46, lets a publisher's public key point to the latest infohash through signed DHT entries, but that is a pointer, not a history with branches and diffs.[10]
  • Identity and permissions. A swarm has no accounts. Anyone holding the infohash can join, and nothing in the protocol grants, gates or revokes access.
  • Collaboration. Issues, reviews and discussions are outside the protocol entirely.
  • Trust in the source. A hash proves that bytes match the torrent, not that the torrent came from who it claims. With Mixtral, trust came from the fact that Mistral posted the link on its own account.[2]
  • Persistence. A 2010 study of 46,227 torrents found that more than 38 percent lost all their seeders within the first month, most of those within five days. The authors concluded that unavailability cannot be fixed by tuning BitTorrent's algorithms and must instead be solved by giving people reasons to keep seeding.[11]

Web seeds, defined in BEP 19, let an ordinary HTTP server act as a seed that is always available.[12] They help a great deal, but they bring back a server that someone has to pay for and keep online.

What this means for a decentralized hub

The practical lesson is to treat BitTorrent as the bulk data layer and build the other layers on purpose. Release records and model cards need a permanent, addressable home. Registries, names and permissions need shared logic that many parties can trust. Code review needs a collaboration protocol, and seeding needs incentives. Each of those pieces is the subject of a companion article in this series, and together they describe how a decentralized model hub could match what a central platform offers.

Anyone publishing open weights by torrent today can already close part of the gap. Ship the model card and license inside the torrent, publish the infohash from a channel people already trust, add a web seed, and keep at least one seeder running for the life of the release.

  • Which features of Hugging Face or GitHub would people miss first if only a torrent of the files remained, and why?
  • How does a content hash prove integrity without proving authorship, and what extra step closes that gap?
  • What incentives could keep rare model weights seeded for decades instead of days?
  • Compare a torrent infohash, a Git commit hash and a signed pointer such as BEP 46, and explain what each one can and cannot express.
  • How could keyword search over models work without a central server, and what would it cost in speed or spam resistance?
  • Why did a release with no explanation still spread quickly, and what risks would the same method carry for a less trusted publisher?
  • Design a minimal release package for open weights that works well when shared only by torrent.
  • What would a fair split of duties look like between a torrent swarm, a permanent storage network and a federated forge?

Readings from Wikipedia

See also

References

  1. ↑ Willison, Simon (11 December 2023). "Mixtral of experts". Simon Willison's Weblog. https://simonwillison.net/2023/Dec/11/mixtral-of-experts/. 
  2. ↑ 2.0 2.1 McKay, Chris (11 December 2023). "Mistral Unveils Mixtral 8x7B: A Leading Open SMoE Model". Maginative. https://www.maginative.com/article/mistral-unveils-mixtral-8x7b-a-leading-open-smoe-model/. 
  3. ↑ "BEP 3: The BitTorrent Protocol Specification". BitTorrent.org. https://www.bittorrent.org/beps/bep_0003.html. 
  4. ↑ 4.0 4.1 "BEP 52: The BitTorrent Protocol Specification v2". BitTorrent.org. https://www.bittorrent.org/beps/bep_0052.html. 
  5. ↑ 5.0 5.1 "BEP 5: DHT Protocol". BitTorrent.org. https://www.bittorrent.org/beps/bep_0005.html. 
  6. ↑ "Repositories". Hugging Face. https://huggingface.co/docs/hub/repositories. 
  7. ↑ "Model Cards". Hugging Face. https://huggingface.co/docs/hub/model-cards. 
  8. ↑ "Gated models". Hugging Face. https://huggingface.co/docs/hub/models-gated. 
  9. ↑ "Pull requests and Discussions". Hugging Face. https://huggingface.co/docs/hub/repositories-pull-requests-discussions. 
  10. ↑ "BEP 46: Updating Torrents Via DHT Mutable Items". BitTorrent.org. https://www.bittorrent.org/beps/bep_0046.html. 
  11. ↑ Kaune, Sebastian (2010). "Unraveling BitTorrent's File Unavailability: Measurements and Analysis". 2010 IEEE Tenth International Conference on Peer-to-Peer Computing (P2P). doi:10.1109/P2P.2010.5569991. 
  12. ↑ "BEP 19: WebSeed, HTTP/FTP Seeding (GetRight style)". BitTorrent.org. https://www.bittorrent.org/beps/bep_0019.html.