My initial thoughts were that rsync would be perfect for this, but it turned out that as time went by - the time taken just kept increasing from less than an hour to more than 4 hours.

When I converted my homelab to proxmox and also to their backup solution the requirements was different - no longer was it few massive files, but instead a lot of “smaller” files that needed to be transferred. But I had learned to live with the speed it took since it was simple and it worked. This is what I used previously to transfer my backups to when I was using VEEAM - it was not great and transferring my weekly full backup took hours - even though I have a 1Gbit internet connection. These directories can be on the same system, different systems, so its very flexible and works great. Rsync is a program that can synchronize directories. Simple is good and for a first time copy scp will probably be okay’is - but as a solution its not good, since it is not a sync protocol - and to make it into a sync protocol you would need to create scripts to compare destination with source and then only copy the delta. Simply copy the files via scp on a schedule.So instead of having a coupld of HUGE files I need to transfer to the “cloud” - I have many thousands - in fact at this moment there are 97k files that is my backup - all these needs to be synced to. The new solution stores a lot of files in a not so ideal directory structure in my opinion. With my old backup solution VEEAM - a couple of HUGE files was generated that contained all the delta updates of the backups. Unfortunately I do not have the budget to use their ZFS solution, since that would most definately be the speediest solution to backing up my backups because I am also running my filesystem on ZFS, which would have allowed me to just replicate snapshots which is very fast since it is just streams of data and not files that gets copied. With my old backup solution I backed up my “backup” to the cloud to my provider rsync which simplied is just a ssh connection where they have enabled certain programs to run and the underlying storage they use is ZFS. I recently started using proxmox in my homelab instead of using VMWare ESXi - this lead me to Proxmox Backup Server since it allows delta backups like my previous backup solution for ESXi.

Rsync to backblaze b2