Synametrics Technologies

Hi,

I would like to have a additional remote backup on my VPS Server. I installed and setup syncrify. Basically it is working. If I'm starting backup of a folder with e.g. 20 files each 5GB filesize the sync starts with 5 files backap simultaneously. This is very good because in this case the speed of myupload which is around 40 MBit is used in total.

If I have to stop backup process for whatever reason and start it later again it will start syncronizing file by file (for every file it resumes where it stopped before). This behaviour is much slower because top speed for syncing a single file goes up around 10 MBit and not more. 

This seems to be a bug or if not I don't see the reason for the behaviour It would be nice if allways several files are synced simultaneously or there should be an option for this. 

Regards,

Tim

Tim,

This is not a bug. I'll explain below.

When you start backing up files the first time, Syncrify does not have to match blocks, no MD5 signatures are calculated. Therefore, it utilizes the entire available bandwidth to transfer the files.

Once a copy of a previously backed up file exists on the destination, Syncrify will try to figure out the difference between the two files. There are two scenarios:

1. A file is modified towards the end - this happens when a previous backup was terminated creating a partial file. In this case, it only has to generate one MD5 signature on both client and server's end.
2. A file is modified from anywhere in the middle - this is a more common scenario and requires block matching besides simple MD5 calculation on the entire file.

In either case, you will not see any network activity until Syncrify figures out the delta.

Moreover, matching blocks is a CPU and I/O intensive process. Syncrify is designed to use a single thread if it finds files larger than 50MB in a folder, which puts less load on the machine.

If you have a fast network between your client and server, you may want to consider using Simple Copy. In this case, Syncrify will transfer the entire 5GB file in its entirety without spending any time in matching MD5 signatures and you will see a spike in network bandwidth with less CPU.

Hello,

thank you for the detailed answer. In my case option one is the case.

If I understand the explanation right it would be possible to MD5 the next file when the first is MD5ed (I think at the moment the process is really file by file - first MD5, then transfer, then next file MD5, then transfer).

There would be two options to speed this process up in my opinion without inceasing CPU load:

1. If a sync was stopped there are 5 files which are not complete. Instead of syncing this 5 files in sequence (MD5 + transfer) without parallelization it would be possible to MD5 one, transfer the data after that and start 4 new files in this time. That will not increase CPU load and save a lot of time in case of lager files and enough upload bandwith which is not used for a single file. After the first inclomplete file is MD5ed and transfered the next inclomplete can start. But in all the time bandwith is used with new files without incresing CPU load.

2. MD5 one file and when it's done start with transfer and MD5 the next one in prarallel. Not as goos as solution one but better.

The third option would be to give the user the choice to do MD5 in a parallel way but if syncrify is not designed for that this is maybe a bigger change. In my case the server has the power to do that.

That means for me it is not the problem that I see no network activity. In my case 40 MBit Upload is only used with several files. One file will only be around 10MBit. Which means it is much slower if only one file is transfered. On the other hand it makes not much sense to transfer 4GB again if the file Size is 5GB. So it realy depends on where the transfer per file was stopped per file. A general setting is a 50% chance to get an improvement or a disadvantage.