Partial network partitions are a peculiar type of network fault that disrupts communication between some but not all nodes in a computer cluster. And for what has recently been found to be a surprisingly catastrophic source of computer system failures, partial network partitions have not been studied comprehensively by computer scientists or network administrators.
The good news is a team of researchers at the Cheriton School of Computer Science has not only examined the problem in detail across many diverse computer systems, but the team has also proposed a “Nifty” solution to fix it.
To understand the scope and impact of this systems problem, recent master’s graduate Mohammed Alfatafta along with current master’s student Basil Alkhatib, PhD student Ahmed Alquraan, and their supervisor Professor Samer Al-Kiswany comprehensively examined 51 partial network partitioning failures across 12 popular cloud-based computer systems. Their data sources ranged from publicly accessible issue-tracking systems to blog posts to discussion on developer forums.
A cluster is a set of connected computers that work together in a way that makes them appear to a user as a single system.
If a partial network partition occurs, groups of servers within a cluster are disconnected, while others are not. In this example, the computers comprising Group 1 can communicate with those of Group 3, and those of Group 2 can communicate with those of Group 3. But the Group 1 computers cannot communicate with the Group 2 computers. This partial partition leads to Group 1 seeing Group 2 as down and Group 2 seeing Group 1 as down. This often leads to both catastrophic and lasting impacts.
Even for researchers who delve into the intricacies of computer systems and networking, the results were unexpected. The team found that 75 percent of the failures they studied have a catastrophic impact that manifested as data loss, system unavailability, data corruption, stale and dirty reads, and data unavailability. They also discovered that 84 percent of the failures are silent — they don’t send users error or warning messages. The study further revealed that more than two-thirds (69 percent) of failures require three or fewer events to manifest, and that 24 percent of the failures were permanent, so even after fixing the partial partition the failure persists.
“These failures can result in shutting down a banking system for hours, losing your data or photos, search engines being buggy, some messages or emails getting lost or the loss of data from your database,” said Professor Al-Kiswany.
“Partial partitioning is a catastrophic failure that is easy to manifest and is deterministic, which means if a sequence of events happens, the failure is going to occur,” Professor Al-Kiswany continued. “We found that the partition in only one node was responsible for the manifestation of all failures, which is scary because even misconfiguring one firewall in a single node leads to a catastrophic failure.”
To fix the problem, the team of researchers developed a novel software-based approach, which they dubbed Nifty — an abbreviation for network partitioning fault-tolerance layer — to prevent these system failures. Nifty is both simple and transparent.
To help ensure wide adoption by data centres, Nifty had three goals — for it to be system agnostic, in other words, that it can be deployed in any distributed system, for it to not require any changes to existing computer systems, and for it to impart negligible overhead.
“Our findings motivated us to build Nifty, a transparent communication layer that masks partial network partitions,” said Mohammed Alfatafta, the recent master’s graduate who led the project. “Nifty builds an overlay between nodes to detour signals around partial partitions. Our prototype evaluation with six popular systems shows that Nifty overcomes the shortcomings of current fault tolerance approaches and effectively masks partial partitions while imposing negligible overhead.”
While Nifty keeps the cluster connected, it may increase the load on bridge nodes, leading to lower system performance. “System designers who use Nifty can optimize the data or process placement or use a flow-control mechanism to reduce the load of bridge nodes,” Professor Al-Kiswany said. “To facilitate system-specific optimization, we’ve designed Nifty with an API to identify bridge nodes.”
To learn more about the research on which this feature is based, please see Mohammed Alfatafta’s virtual presentation at OSDI '20, the 14th USENIX Symposium on Operating Systems Design and Implementation.
Toward a Generic Fault Tolerance Technique for Partial Network Partitioning
Please also see Mohammed Alfatafta, Basil Alkhatib, Ahmed Alquraan, Samer Al-Kiswany, “Toward a Generic Fault Tolerance Technique for Partial Network Partitioning,” 14th USENIX Symposium on Operating Systems Design and Implementation, 2020.
Nifty source code
Nifty’s source code is freely available at https://github.com/UWASL/NIFTY.
Usenix Artifact Evaluation Committee
At submission time, the authors of a study can choose the criteria by which their contribution, or “artifact” as it is called, will be evaluated. The criteria correspond to three separate badges that can be awarded to a paper. This study received all three Usenix Artifact Evaluation Committee badges —essential criteria that establish the availability, functionality and real-world impact of the research.
