Today, most apps are cloud-only. Data must travel halfway around the world to a remote data center just to arrive at a device in the same room. These apps become entirely unusable if their data connection is slow or a Wi-Fi router breaks. Cloud-only apps have made it difficult for the "deskless workforce" to get their work done. People require database access on mobile devices, where crucial on-the-ground operations are happening, but their cloud-only applications do not function 100% of the time. Operations halt while waiting for a mobile phone to reconnect. It isn't a great user experience; it costs businesses money and can even put people in life-threatening situations when real-time data is needed for quick decision-making.
Despite these problems, most applications today are cloud-only. Why? If you want to send your friend a picture of a cat, you can use AirDrop, which sends the data directly between the two devices, peer-to-peer. So why don't all apps send other kinds of data directly, peer-to-peer, between devices? Why isn't the cloud optional?
The answer is that it's simply easier to build cloud-only applications. Over the past few decades, investors have funneled billions of dollars into cloud-only databases and tools. Because of this, developers don't have to think about TCP/IP networking, database partitioning, or on-disk compression whenever they want to update a field in a database table. For years, this existing toolkit of cloud-only tools has made cloud-only applications the fastest way to deploy collaborative software.
Peer-to-peer is not new, but it is still hard to use
There are scalable peer-to-peer protocols for data that doesn't change, such as AirDropping cat pictures or torrenting movies and music – BitTorrent accounts for 27% of all upstream Internet traffic1. But, developers need the reliability of a database when data changes at the speed of Slack and requires the precision of an airplane safety checklist. Unfortunately, few engineers in the world can build a peer-to-peer database. It's a new software engineering paradigm that needs more education, tool development, and user experience improvements. Because of this, we haven't had a cloud-optional, peer-to-peer database. Until now.
Ideally, a peer-to-peer database needs to be:
- User-friendly. Developers are users too! Instead of sending data to a remote server, the application needs to write data to its local database first in the form of changes, then listen for changes from other devices, and recombine them on the fly. Ditto provides an API on top of these details so developers can focus on their business logic instead of synchronization logic.
- Cloud-optional. Devices can go into dead zones, routers can crash, or cloud services can go down. All devices must see the same query results given the same set of changes, even if the changes arrive in a different order. Ditto's Conflict-Free Replicated Data Types (CRDTs) provide a consistent view of the data for every device. These data structures are still very much a new topic in computer science research.
- Partitioned. Mesh networks can generate a tremendous amount of data that can overwhelm small devices if each node aggressively tries to sync every piece of data. Ditto provides a Big Peer and a Small Peer SDK with different replication strategies. The Small Peer is selfish, which means it only synchronizes data it explicitly requests, giving developers complete control over storage and bandwidth usage. The Big Peer is greedy, which means it synchronizes as much as possible.
- Ad-hoc. Devices may join and leave the mesh at any time. Despite this churn, all devices should still see and have input on the same data: the same "source of truth." Routing messages between appropriate nodes becomes a mathematical challenge as the mesh network topology changes over time. Ditto's novel delta state CRDT is great for ad-hoc mesh networks because they are flexible and do not rely on having the full history of a database table to write or read the latest value.
- Forward-Compatible. Since devices update at different times, they need to account for incoming data with different schema. For example, if a device is offline and therefore outdated, it should still be able to read new data and sync. Ditto is causally consistent, providing a reliable order of changes that can be inspected, including incorporating metadata about schema changes over time.
A shared set of tools
These challenges are straightforward to explain but bring up a sizable, entangled web of discussions. There are many different ways to design and implement caching, data synchronization, querying, and distributed architecture.
It is a lot of work for development teams to create a reliable peer-to-peer database that syncs data in a partially connected mesh. Organizations either can't hire the talent to complete the job or don't see the financial benefit of investing in building it themselves. It is more efficient to build a database that many different apps use so that each individual organization doesn't need to invest in building and maintaining their own cloud-optional database.
With Ditto, developers have a complete end-to-end cloud-optional solution that combines the best of cloud software with the best of peer-to-peer software. On the surface, the shift from cloud-only to cloud-optional seems subtle. But this is a more fundamental paradigm shift and provides tremendous business opportunities. Ditto is setting the standard as one of the first CRDT-based peer-to-peer databases that turn the existing plethora of mobile devices into a cloud-optional, edge sync platform.
If you are a developer and want to make your cloud optional, get started today for free. We're hiring, so look at our available listings and see if you think Ditto might be a good fit for you.
Footnotes
The Global Internet Phenomena Report, Sandvine; 2019. ↩