Distributed computing

Distributed computing is a field of computer science concerned with computational systems whose components operate on multiple networked computers and coordinate their actions by communicating with one another. Distributed computing can make it possible to divide computation, storage, data processing, or services among many machines rather than requiring everything to happen on a single computer.

This topic page is for organizing the development of distributed computing related content on this wiki.

Distributed computing is used in scientific research, Internet services, cloud computing, databases, artificial intelligence, peer-to-peer networks, telecommunications, financial systems, large-scale data processing, and many other areas. Distributed systems can range from a few computers on a local network to systems involving thousands or millions of devices spread around the world.

Please, feel free to improve upon what you see; your contributions will be greatly appreciated.

Major concepts

A distributed system consists of multiple computational components that work together toward one or more shared goals. Individual computers in the system are often called nodes.

Important topics include:

Distributed system Computer network Peer-to-peer network Distributed database Distributed file system Cloud computing Edge computing Parallel computing

Fault tolerance Replication Consensus Concurrency Load balancing Distributed algorithm Decentralization

Distributed systems face challenges that do not arise in the same way on a single computer. Messages can be delayed or lost. Computers can fail independently. Two nodes may temporarily have different versions of the same information. A network connection can fail while the computers on both sides continue operating.

As a result, distributed computing involves both dividing work and coordinating the pieces reliably.

Distribution and parallelism

Distributed computing and parallel computing overlap, but they are not identical.

Parallel computing generally focuses on performing multiple computational operations at the same time. These operations may occur within one computer or across many computers.

Distributed computing emphasizes multiple independent computers communicating through a network. A distributed system may perform work in parallel, but it must also deal with network delays, independent failures, synchronization, and communication among machines.

A large scientific simulation, for example, may divide a calculation among hundreds of computers. A distributed web application may instead use separate computers for databases, application servers, caching, storage, and other services.

Coordination and consensus

Computers in distributed systems frequently need to coordinate their actions.

They may need to determine:

• Which node should perform a task.
• Which version of data is current.
• Whether a transaction has completed.
• How to recover after a failure.
• How duplicated information should be synchronized.
• In what order events occurred.
• Whether enough participants agree to accept a decision.

Consensus refers broadly to methods that allow distributed participants to agree on shared information or decisions despite failures or conflicting information.

Different systems solve coordination problems in different ways. Some depend on trusted central servers. Others elect temporary leaders, use voting or quorum systems, or employ algorithms designed to continue functioning even when some participants fail.

Reliability and fault tolerance

Distribution can improve reliability by preventing one machine from becoming the only place where an important service or dataset exists.

For example, a file can be replicated across several computers. If one computer fails, another copy may remain available.

However, redundancy alone does not guarantee reliability. Designers must decide how copies are updated, how failures are detected, and what happens when machines reconnect after temporarily losing communication.

Fault tolerance is therefore an important area of distributed computing. A fault-tolerant system is designed to continue providing some or all of its intended functions when parts of the system fail.

Learning about failure is particularly important because real distributed systems must operate under imperfect conditions rather than assuming that networks and computers will always work correctly.

Applications

Distributed computing supports many technologies and research activities.

Scientific computing Weather modeling Astronomy research Artificial intelligence Search engines Distributed databases Web applications

Cloud services Peer-to-peer systems Content distribution Financial networks Telecommunications Large-scale simulations Collaborative computing

Some research projects also use volunteer computing, where people contribute unused computing resources to scientific projects. Other systems distribute work across large data centers or clusters specifically designed for high-performance computation.

Learning projects and resources

Learning materials and learning projects can be used by multiple departments. Please cooperate with other departments that use the same learning resource.

This wiki has adopted the "learning by doing" model for education. Lessons should center on learning activities for participants on this wiki.

Also, select a descriptive name for each learning project. Learning projects can be listed in order according to Meta:Naming conventions.

• Distributed computing
• Collaborative research on this wiki
• Distributed P2P search engine
• Decentralizing wiki technology
• Peer to peer wiki system
• Miscellaneous papers
• Search this wiki for distributed computing

Learning by doing

Distributed computing can be learned effectively through small experiments that demonstrate communication, coordination, failure, and recovery.

Possible projects include:

• Create two programs that communicate over a network.
• Divide a computational problem among several computers and combine the results.
• Build a small peer-to-peer messaging or file-sharing system.
• Create a distributed key-value store.
• Simulate unreliable network connections and observe how a program responds.
• Replicate a dataset across several nodes and experiment with conflicting updates.
• Build a simple load-balancing system that distributes requests among multiple servers.
• Compare the performance of a calculation running on one machine with the same calculation divided among several machines.
• Design a distributed search engine and identify how crawling, indexing, searching, and data synchronization could be divided among nodes.
• Develop a prototype for a distributed or peer-to-peer wiki and identify which components require consensus or replication.

Students can also study existing open-source distributed systems and create diagrams showing how their components communicate.

Research questions

Distributed computing remains an active area of research because improvements in scale often create new problems.

Research topics can include distributed consensus, distributed artificial intelligence, distributed databases, security, privacy, network efficiency, energy use, fault recovery, decentralized systems, distributed storage, and coordination among autonomous software agents.

Researchers may also investigate whether a particular system is truly distributed in practice. A service may use thousands of computers while still depending on one organization for control. Another system may distribute ownership and operation among independent participants.

Technical architecture and organizational decentralization are related questions, but they should be examined separately.

Discussion questions, essay ideas, and learning related AI prompt ideas

• What problems become more difficult when a program operates across several computers instead of one?
• What is the difference between distributed computing and parallel computing?
• How can a distributed system continue operating when one of its nodes fails?
• When is replication useful, and what new problems can replication create?
• Compare centralized, distributed, and decentralized computing architectures.
• How does network latency affect distributed applications?
• What types of scientific research benefit most from distributed computing?
• Design a distributed research platform. Which tasks should be performed locally, and which should be coordinated across the network?
• Investigate a major distributed system failure and explain what can be learned from it.
• Ask an AI system to design a simple distributed computing architecture, then evaluate whether its assumptions about networking and failure are reasonable.
• Ask an AI system to explain a distributed consensus algorithm, then verify the explanation using technical documentation or academic references.
• Ask an AI system to compare peer-to-peer computing with cloud computing, then independently check its claims about reliability, scalability, and control.

Readings

• About distributed computing

Readings from Wikipedia

• Distributed computing
• Distributed computing system
• Distributed algorithm
• Parallel computing
• Consensus
• Fault tolerance
• Distributed database
• Distributed file system
• Peer-to-peer
• Volunteer computing

External links

• German Distributed Computing Wiki (with a bit of English), with information for English visitors

See also

Distributed technology Distributed systems Parallel computing Peer-to-peer network Distributed database Distributed file system Cloud computing

Computer networking Decentralization Fault tolerance Distributed artificial intelligence P2P MediaWiki Computer science Collaborative computing