BaseFS

Basically Available, Soft-State, Eventually Consistent File Sytstem for P2P Cloud Management

Created by Marc Aymerich /
glic3rinu.github.io/basefs/presentation

Cloud Management

Client-Server Architecture

Availability

Scalability

HA Client-Server Architecture

Increased complexity

Traditional Cloud is
Centralized

Community Cloud is
Peer-to-peer

How can we do better
P2P management?

Consistency or Availability ...

... You must choose wisely

Eventual Consistency

• Available under partition:
       No need for stable quorums
• High performance:
       Progress without coordination
• Geographic and administrative scalability
       No latency bottleneck, no trust between nodes
• Weakly consistent communication channel:
       Gossip-style network protocol, NO APIs
• Nodes don't need to trust each other: Byzantine
       failures on SC systems are practically untractable

BaseFS

Log Entries

Log Entries Properties

Log Blocks

View

• Conflict resolved composition of the log
• Self-certified filesystem with write permissions
• Proof-of-authority:
  1. Higher hierarchy key branch
  2. If equal, more contributors branch
  3. If equal, higher root hash branch

View Example

Synchronization Protocol

• Log disemination
  • after partition
  • large files
  • bootsrap joining nodes
• Unseen-biased randomized node selection
  $$p_i = t_i/\sum_{j=1}^{1,n} t_j$$
• Efficient divergence detection with Merkle trees

File System API

• Natural way of doing configuration in UNIX
• Implemented with FUSE (Filesystem in Userspace)
• Watchers: react to events, similar to inotify

Example:
Replace CONFINE controller and node software with BaseFS

Installation

Bootstrap

# Create new key
$ basefs genkey

# Createw new filesystem
$ basefs bootstrap confine -i <ip>

# Mount
$ mkdir ~/confine
$ basefs mount confine ~/confine

Build Data model and handlers

Model CONFINE system with directories and files, considering that permissions are hierarchical.

/users/user1/info
/users/user1/auth_keys/key1
/groups/a/members/user1
/groups/a/nodes/node1
/groups/a/slices/slice1/node1

Write handlers that execute on changes,
~/confine/handlers/confine-node.sh creates CONFINE slivers

Distribute

# Get the log from another machine
$ basefs get confine <ip>[:<port>]
$ mkdir ~/confine
$ basefs mount confine ~/confine

# CONFINE nodes should define the handler for creating VMs
$ basefs mount confine ~/confine \
    --handler ~/confine/handlers/confine-node.sh

Evaluation

Parametrization

Param: Max Gossiped Blocks

Param: Sync Protocol Interval

Param: Docker Cluster Size

Network Evaluation

Effect of Latency

Effect of Bandwidth Limitation

Effect of Packet Loss

Community-Lab

Community-Lab: Convergence

Com-Lab: Traffic distribution

File System IO Evaluation

File System Read Performance

File System Write Performance

Plans for world domination:
As it is

• Community cloud configuration management
• Distributed version control system
• Distributed Dropbox-like applications
• Self-updatable documents (encyclopedia, discography)

Plans for world domination:
Better IO performance

Plans for world domination:
Support for larger files

• Mutable P2P file-sharing
• Generalized filesystem

Summary

1. Strong consistency imposses strong constrains
2. We have a lot to gain if we can relax consistency: scalability, availability and simplicity
3. Cryptographic hashes are awesome!
   • CvRDT Specialized Merkle tree is a powerfull data structure with useful guarantees
   • Merkle trees allow for efficient synchronization
4. Gossip protocols are efficient ways for group membership and state disemination in large weakly consistent systems
5. Prototyping distributed filesystems is easy and fun with Python and FUSE

THE END

- Read the paper
- Try it out
- Source code & documentation