Long Pulling Issue with Large DB

Hong

Hi, I'm in basehead Team.
We have many corporate uses with Databases that are 2-8 GB.
From initial testing with our biggest database, the pull process takes over 5 minutes for the single database.
And 5mins for 2GB is not acceptable to them.
Is there a way to fast the pulling with Large DB?
Thanks

Bernardo

Hello,

Do you understand that this download of the db only happens on the very first time a secondary node is opened, right?

On subsequent openings of the app, the db will already be there and it will download only the changes made while it was off-line.

Anyway, I took a look at the code and indeed it can be made faster.

Currently the transfer of the db is made in chunks. The main reason is to let the primary node respond to other requests at the same time, so it does not fully dedicates to the transfer.

But it is using chunks of only 10 pages (40KB ). Increasing this number can make the transfer faster and still be backwards compatible to existing installations.

A dedicated worker thread for each transfer could be even better, but I am not sure if it can be made backwards compatible.

Anyway, using 1 or 10 MB per chunk can increase the performance. I will make some tests and include the changes on a new release.

Thank you for pointing this out!

Bernardo

I now perceived that for big databases, the main cause of slow transfer (for secondary nodes) is that the primary node creates a copy of the db locally and removes the log tables. These tables become big with time, so removing them and doing a VACUUM decreases the size of the transfer. This is very good for smaller databases, in which the log tables consume more than 50% of the db size (it is used only on primary nodes).

But the copy of a big database (many GBs) is slow.

Maybe for big dbs it would be better to transfer the whole thing, and then delete these tables on the receiver node.

The downside is that the secondary node will have a bigger database, so probably we will have to make this optional because some users may prefer a slower download but smaller db.

BaseHead

I'm gonna jump in here also....
"Do you understand that this download of the db only happens on the very first time a secondary node is opened, right?
Yep we understand this but 5 minutes it's brutally long so our users are complaining to us....haha
so we are looking at any way to speed this up.

We thought about RAR'ing the DB and transmitting it with our own protocol and unpacking it on the secondary node for the first connection. Is something like that possible and still re-link to the primary after?

Any other ideas you can think of will be appreciated immensely as we are rolling out this feature soon
thx Bernardo!
Steve
-basehead

Bernardo

I was able to make it faster

Download the binaries of the new release or pull the source and build

Note that both primary and secondary nodes must be using this new version (2.5.0) for the download to work properly

Please inform if it is working as expected

Hong

Hi.
I have another request. We currently have multiple database files on our server. The server can share multiple databases simultaneously.
The problem is that clients have no way to know which databases are being shared.
So, we introduced a separate TCP framework in our project.
I know that LiteSync already implements the TCP/UDP protocol.
Do you plan to add this convenience feature to your framework in the future?
Thank you.

BaseHead

Awesome about the Speed Increase Bernardo!
We will test that out today. ;-)