P2P Topology in offline local network : consensus & dynamic number of nodes

This site is best viewed in a modern browser with JavaScript enabled.

P2P Topology in offline local network : consensus & dynamic number of nodes

Anthony

Hello Bernardo,

I would like to first thank you for your work in the development of this tool. It is small yet seems truly powerful and conveniant for embedded applications.

I am currently investigating its use in a scenario where a set of embedded linux devices would be connected in a local enterprise network without internet, sharing a decentralized DB. Over time, devices would join then permanently leave the network, with a minimum of 1 device present at any given time.

Ideally, all devices would be embedded with the same default program, without manual configurations to distinguish between primaries/secondaries. So far, P2P topology seems to fit best.

These devices can already discover eachother on the network in an accurate manner and have almost real-time knowledge of the number and adress of the other nodes on the network. As such, they are able to close and re-open their litesync connection with new parameters if needed (total_primary_nodes, connect adresses, ..) dynamically based on the devices leaving/joining the network.

Also, (this is key) devices that are offline would be software-restrained to not perform any operation on their DB until they are back online. As such, this is a scenario where DB availability for offline devices is not really the concern. However, synchronization when online is important and should be based on latest commits.

A common scenario :
1. There is first only 1 device on the network and some records are added to its DB
2. Then 7 additonnal devices join and some records are added/updated/deleted
3. Later, 4 devices leave the network permanently (including the first device)
And so on ... with potentially full rotation of devices joining then permanently leaving the network.

The desired behaviour would be that devices joining a network (for the first time or not) would always have their DB updated with the latest up-to-date data from the devices already present in the network, independently from the ratio of new devices joining versus already existing online devices. If we take as example the transition from step 1 to 2 in the example above :

. One device on the network as a primary node (total_primary_nodes=1) , a table is created and some data is added ( DB > Table A : row 1, row 2 are added)
. Then 7 devices are added at the same time on the network as primary nodes (on these devices : DB> Empty). LiteSync connection with DB is automatically closed then re-opened on all devices with new URI connection string with 8 total_primary_nodes with adequate connection adresses.
. New primary nodes synchronize with the first node and all 8 devices have DB > Table A : row 1, row 2
...

If we narrow my question down, I guess it would be : how is consensus managed by LiteSync in P2P topology ? Would table 'A' be lost because the majority of nodes joining do not have Table A ? Or is there tracking of the order of the commits, which would enable the desired outcome ?

If not, an alternative (I guess) would be to temporarily add new nodes as secondaries and wait for them to be synchronized before setting them up as primaries. My second general question would be : Does LiteSync allow the changing of node types on existing DBs ? Or even more generally : is it possible to close then re-open a connection with different URI parameters on the same DB ?

I didn't yet have the chance to try LiteSync (will do very soon), and wanted to ask in case you already have a quick answer

Thanks in advance !

Anthony

Bernardo

Hello Anthony,

That is an interesting use case

I can see that you understood very well how to use LiteSync

I suspect that the above scenario may work, as long as the new devices when reaching the network do not have LiteSync enabled. Because LiteSync could connect first to other nodes, and then having more nodes than the total_primary_nodes can cause some problem on the consensus protocol.

In the example you posted above, having just a single node with content and the other new 7 do not have (or have older content) the nodes will elect a new leader and they consider which node has newer content. So that node would be the leader and distribute its content to the other nodes, exactly as desired.

Also, LiteSync allow nodes to write to the local database when off-line, so even if other nodes have local changes, these will also be distributed between the nodes.

Changing the node type is not fully supported, and not needed in your use case.

Please inform if you have any problem on your tests

Happy testing! =)

Anthony

Bernardo In the example you posted above, having just a single node with content and the other new 7 do not have (or have older content) the nodes will elect a new leader and they consider which node has newer content.

That's the piece of information I was unsure of, thanks for the answer

Indeed, when new devices join the network they won't already be connected to their DB with LiteSync (and if they were connected before, they would have closed the connection we they left the network) : When devices join the network, a hand-shake mechanism would have them all (devices joining + those already present) first be disconnected from their DB, before they all re-connect with the updated total_primary_nodes value. The robustness of this mechanism, to avoid race conditions, would indeed be key.

It is indeed a not very common use case, but for which LiteSync could do the job efficiently. I guess you could think of it as a project for a 'dynamic decentralized data synchronization system'. Devices need to aquire location-specific data (location being associated to a network) and share it among themselves, in an automated setting where any device can come and go, and manual configuration of a 'fixed' primary (or set of primaries to avoid a single point-of-failure) would preferrably be avoided.

I'll gladly share the progress here

Bernardo

An important information:

The consensus protocol only progress when the majority of the nodes are on-line.

In the above example, if total_primary_nodes is 8 but then 4 of them disconnect, there will be no more synchronization until the number of connected nodes reaches 5.

As I understood, your app on all devices would then close the databases and reopen them with a new value for total_primary_nodes (4 on this case)

I suspect it will work fine, unless there is some dark corner case there

To be honest I never saw someone with a use case like yours, of dynamically changing these values. Our automated tests do not cover this. So I am curious to know how the code will handle this.

Bernardo

Cool!

Generally consensus is not a simple thing. It requires the majority of nodes to be participating on it otherwise "split brain" can occur (the formation of separate groups having independent progress)

So you must make sure that individual sub-groups will not occur. Paying attention to the moment when nodes leave the network. The critical point is when decreasing the value of total_primary_nodes

What if the network on the enterprise was partitioned and then the nodes are not seeing the others, but they are on-line in a separate group? If both groups decrease the value of total_primary_nodes, then "split brain" will occur

Bernardo

One simple solution to this is to have a node that must be always on-line, and set it as primary node. Then all the others are configured as secondary nodes, and so they can come and leave freely

Anthony

Thanks for the info ! I might go with a star topology later, depending on these potential issues you mention.

Regarding my experiement with P2P, I've been able to do some tests, and would love to get your feedback on this one issue :

When a new node joins the network and establishes a connection for the first time, LiteSync nicely sets its node_id to 0 and goes through a discovery process to negotiate a unique ID. Works like a charm!

However, when this new node disconnects and then reconnects, it seems to keep its old node_id instead of going through the discovery process again. Here's a quick run-through:

. Two nodes are active on the network with IDs 1 and 2.
. A third node joins the network and doesn't immediately discover the other two, so it gets node_id 1 with total_primary_nodes=1.
. Then it discovers the other nodes, closes its connection, and reopens it with the correct parameters and total_primary_nodes=3. But, it retains its initial node_id as 1.

From my observation, the discovery phase only happens when the .db connection is opened for the first time. I believe the state parameters are stored in dedicated LiteSync tables and are not reset upon reconnection.

I thought the new node should go through the discovery process again upon reconnection and get a new node_id. Is this the intended behavior or is there a way to reset the node state manually?

Looking forward to hearing your thoughts.

Best,
Anthony

Bernardo

The node id must be unique per node. It cannot be modified, to avoid conflicts.

In the example above, you have 2 separate networks. This is exactly what should be avoided.

The third node should not have its db opened with total_primary_nodes=1

And the db connection from the other 2 nodes must be closed before the connection from the third is opened. Now the 3 must open the db with total_primary_nodes=3

A start topology is way easier and simple to manage.