Why did we attach a database to our peer to peer sync solution?
We actually don't get this question a lot, but ocassionally prospective customers will ask: "Why does Ditto have a database as a primary part of the sync system?"
The short answer:
it's easier to use as a database and if it was just network code is way too difficult to understand".
The long answer:
There are several reasons why we haven't seen a larger proliferation of p2p apps in the last decade. With traditional client server architectures, application design patterns are relatively straightforward. Your apps post some data Almost all the data, querying, and logic is housed in one central location in the backend database. However, peer to peer apps completely challenge the centralization by taking distributed systems to their extremes.
Here are just some of the reason why peer to peer networking code is so difficult to code for.
- Each peer in the distributed system will leave and join the mesh unpredictably. Apps can go into dead-zones or even crash. These are all scenarios that can halt an app from working.
- Data almost always needs to be heavily partitioned. Mesh networks can generate a gigantic amount of data that can overwhelm bandwidth and storage constraints on devices if each node is aggressively trying to sync every piece of data.
- Peers joining the system have no central source of truth. Some apps may leave the mesh with some relevant data for another peer attempting to query at the same time.
- As more devices enter the network, routing messages between appropriate nodes becomes a large mathematical challenge.
These challenges are very straight forward to explain but bring up a gigantic quagmire of discussions around proposed solutions. We don't think that this is for the faint of heart and opens up a lot of questions surrounding caching, data synchronization, querying, and distributed architecture. If an enterprise company is to build a mobile application using WiFi or Bluetooth, they are certainly going to spend a tremendous amount of time whiteboarding the "correct" architecture instead of deploying code to the customer.
So at Ditto we realized that these complexities were too hard to fathom for the vast majority of software engineers. There are several open source frameworks that exist for peer to peer mesh networking. However, they only provide a low level access around how devices communicate rudimentary data with each other. We already have evidence that companies have taken great investments to create a generic framework around open source solutions. However, the difficulties of creating a sensible platform that syncs data in a partially connected mesh becomes so intense that developers back away and begin to create bespoke solutions. What is truly sad is that these efforts are typically abandoned over time.
Designing Ditto, we abstract away all the networking that once developers needed to handle themselves. What we found was that vast amount of developers just wanted to work with data not networking code. While there are a myriad of ways to interact with Bluetooth or WiFi, working with a database is extremely straightforward. Developers only need to learn how to insert, remove, update, and delete data. Ditto will handle all the connecting, disconnecting, reconnecting and network security seamlessly underneath the abstraction layer.