| | |

An excellent alternative to ‘rsync’ for backups to cloud

I have been using NextCloud for a long while now. It is a smart solution, and although I do not like its agent (I’ve had many problems with it around Hebrew file names on multiple types of operating systems, which never matured enough for a full bug description and details) – for an easy access to data, and ability to contain an additional backup copy (not to mention all the extra capabilities – check it out, if you are not familiar with it yet) – it is an excellent solution.

I’ve been using a rather simple and “primitive” method of backing up, using a common set of tools. I’ve been using ‘rsync’ over WebDav (implemented using ‘davfs2‘ Linux FUSE drivers), bit it did not work well. It’s not that ‘davfs2’ is horrible – for its limited purposes – it is good enough, it’s just that its performance, combined with how NextCloud works, and combined with how ‘rsync’ works – results in a low performance and bad behaviour.

Years ago, I was able to sync my data to NextCloud, and then keep it in-sync using rsync. It was a long and tiresome process, but it worked – since most changes were minor at each daily iteration. I encountered major problems recently, when my entire NextCloud has been modified and moved to a different platform (more modern Linux, container-based deployment and so on), and the easiest solution, or so I thought, was to re-sync everything.

I was proven wrong. Or at least – wrong while using the tools I’ve been using in the past. Performing ‘rsync’ was extremely slow, unreliable and ineffective – especially with relationship with ‘davfs2’ filesystem. I’ve had to look for a different solution, and I did look for a different FUSE filesystem implementations for WebDav. This has resulted badly, meaning that there is no real solution for that. So, with little other options, I was looking for rsync interface on NextCloud, or similar tool.

This is where I hit a gold mine. The tool rclone was providing everything I wanted, and much more. Not only does it work well with NextCloud (as ‘webdav’ client), meaning – extremely fast and reliably (including network and error recovery), but it also provides a wrapper interface for CloudFlare free account 100MB upload limit. Take a look at an example of the configuration file I use (names are modified), placed in the default location ~/.config/rclone/rclone.conf:

type = webdav
url = https://mydomain.com/remote.php/webdav/
vendor = nextcloud
user = myuser
pass = SecretHashed password

type = chunker
remote = nextcloud:/Documents
chunk_size = 90Mi

In my “Documents” directory, I have files larger than 90MB. However – by the nature of rclone’s WebDav agent, there is no file split, so using the ‘chunker’ overlay allows splitting the files correctly, and it works wonderfully.

An example of the code which performs sync at an incredible speed is below:

# Clone a directory with smaller files, so no need for chunker wrapper
/usr/bin/rclone sync --multi-thread-streams 8 --log-file /tmp/myfiles.log --log-level INFO /home/myuser/Dropbox/myfiles nextcloud:/myfiles
# Clone a directory with huge files, and transfer it through CloudFlare's file size limitation of 100MB
# The "chunker" wrapper "/" points at nextcloud:/Documents , as can be seen above
/usr/bin/rclone sync --multi-thread-streams 8 --log-file /tmp/Documents.log --log-level INFO /home/myuser/Documents chunker:/

Do try it out. rclone exists for both Windows, Linux and Mac.

Similar Posts

Leave a Reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.