You might not even like rsync. Yeah it’s old. Yeah it’s slow. But if you’re working with Linux you’re going to need to know it.

In this video I walk through my favorite everyday flags for rsync.

Support the channel:
https://patreon.com/VeronicaExplains
https://ko-fi.com/VeronicaExplains
https://thestopbits.bandcamp.com/

Here’s a companion blog post, where I cover a bit more detail: https://vkc.sh/everyday-rsync

Also, @BreadOnPenguins made an awesome rsync video and you should check it out: https://www.youtube.com/watch?v=eifQI5uD6VQ

Lastly, I left out all of the ssh setup stuff because I made a video about that and the blog post goes into a smidge more detail. If you want to see a video covering the basics of using SSH, I made one a few years ago and it’s still pretty good: https://www.youtube.com/watch?v=3FKsdbjzBcc

Chapters:
1:18 Invoking rsync
4:05 The --delete flag for rsync
5:30 Compression flag: -z
6:02 Using tmux and rsync together
6:30 but Veronica… why not use (insert shiny object here)

  • state_electrician@discuss.tchncs.de
    link
    fedilink
    English
    arrow-up
    24
    arrow-down
    2
    ·
    9 hours ago

    Why videos? I feel like an old man yelling at clouds every time something that sounds interesting is presented in a fucking video. Videos are so damn awful. They take time, I need audio and I can’t copy&paste. Why have they become the default for things that should’ve been a blog post?

    • czardestructo@lemmy.world
      link
      fedilink
      English
      arrow-up
      5
      ·
      4 hours ago

      Thank you for putting into words what ive subconsciously been thinking for years. Every search result prioritizes videos at the top and I’m still annoyed every time. Or even worst I have to hunt through a 10 minute video for the 30 seconds of info I needed. Stoohhhhpppp internet of new! Make it good again!

    • kchr@lemmy.sdf.org
      link
      fedilink
      English
      arrow-up
      3
      arrow-down
      1
      ·
      6 hours ago

      Hear hear. Knowledge should be communicated in an easily shareable way that can also be archived as easily, in contrast to a video requiring hundreds of MB:s.

  • 1984@lemmy.today
    link
    fedilink
    English
    arrow-up
    5
    ·
    edit-2
    8 hours ago

    I never thought of it as slow. More like very reliable. I dont need my data to move fast, I need it to be copied with 100% reliability.

  • clif@lemmy.world
    link
    fedilink
    English
    arrow-up
    15
    ·
    24 hours ago

    I’ll never not upvote Veronica Explains. Excellent creator and excellent info on everything I’ve seen.

  • atk007@lemmy.world
    link
    fedilink
    English
    arrow-up
    23
    ·
    1 day ago

    Rsnapshot. It uses rsync, but provides snapshot management and multiple backup versioning.

  • Appoxo@lemmy.dbzer0.com
    link
    fedilink
    English
    arrow-up
    4
    ·
    1 day ago

    Veeam for image/block based backups of Windows, Linux and VMs.
    syncthing for syncing smaller files across devices.

    Thank you very much.

    • Encrypt-Keeper@lemmy.world
      link
      fedilink
      English
      arrow-up
      8
      ·
      1 day ago

      I’m not super familiar with Syncthing, but judging by the name I’d say Syncthing is not at all meant for backups.

    • conartistpanda@lemmy.world
      link
      fedilink
      English
      arrow-up
      6
      ·
      1 day ago

      Syncthing is technically to synchronize data across different devices in real time (which I do with my phone), but I also use it to transfer data weekly via wi-fi to my old 2013 laptop with a 500GB HDD and Linux Mint (I only boot it to transfer data, and even then I pause the transfers to this device when its done transferring stuff) so I can have larger data backups that wouldn’t fit in my phone, since LocalSend is unreliable for large amounts of data while Synchting can resume the transfer if anything goes wrong. On top of that Syncthing also works in Windows and Android out of the box.

    • HereIAm@lemmy.world
      link
      fedilink
      English
      arrow-up
      8
      arrow-down
      1
      ·
      1 day ago

      Compared to something multi threaded, yes. But there are obviously a number of bottlenecks that might diminish the gains of a multi threaded program.

    • okamiueru@lemmy.world
      link
      fedilink
      English
      arrow-up
      9
      ·
      1 day ago

      That part threw me off. Last time i used it, I did incremental backups of a 500 gig disk once a week or so, and it took 20 seconds max.

  • NuXCOM_90Percent@lemmy.zip
    link
    fedilink
    English
    arrow-up
    71
    arrow-down
    1
    ·
    2 days ago

    I would generally argue that rsync is not a backup solution. But it is one of the best transfer/archiving solutions.

    Yes, it is INCREDIBLY powerful and is often 90% of what people actually want/need. But to be an actual backup solution you still need infrastructure around that. Bare minimum is a crontab. But if you are actually backing something up (not just copying it to a local directory) then you need some logging/retry logic on top of that.

    At which point you are building your own borg, as it were. Which, to be clear, is a great thing to do. But… backups are incredibly important and it is very much important to understand what a backup actually needs to be.

    • tal@olio.cafe
      link
      fedilink
      English
      arrow-up
      24
      ·
      edit-2
      2 days ago

      I would generally argue that rsync is not a backup solution.

      Yeah, if you want to use rsync specifically for backups, you’re probably better-off using something like rdiff-backup, which makes use of rsync to generate backups and store them efficiently, and drive it from something like backupninja, which will run the task periodically and notify you if it fails.

      rsync: one-way synchronization

      unison: bidirectional synchronization

      git: synchronization of text files with good interactive merging.

      rdiff-backup: rsync-based backups. I used to use this and moved to restic, as the backupninja target for rdiff-backup has kind of fallen into disrepair.

      That doesn’t mean “don’t use rsync”. I mean, rsync’s a fine tool. It’s just…not really a backup program on its own.

      • koala@programming.dev
        link
        fedilink
        English
        arrow-up
        1
        ·
        1 day ago

        Beware rdiff-backup. It certainly does turn rsync (not a backup program) into a backup program.

        However, I used rdiff-backup in the past and it can be a bit problematic. If I remember correctly, every “snapshot” you keep in rdiff-backup uses as many inodes as the thing you are backing up. (Because every “file” in the snapshot is either a file or a hard link to an identical version of that file in another snapshot.) So this can be a problem if you store many snapshots of many files.

        But it does make rsync a backup solution; a snapshot or a redundant copy is very useful, but it’s not a backup.

        (OTOH, rsync is still wonderful for large transfers.)

        • tal@olio.cafe
          link
          fedilink
          English
          arrow-up
          3
          ·
          1 day ago

          Because every “file” in the snapshot is either a file or a hard link to an identical version of that file in another snapshot.) So this can be a problem if you store many snapshots of many files.

          I think that you may be thinking of rsnapshot rather than rdiff-backup which has that behavior; both use rsync.

          But I’m not sure why you’d be concerned about this behavior.

          Are you worried about inode exhaustion on the destination filesystem?

          • koala@programming.dev
            link
            fedilink
            English
            arrow-up
            3
            ·
            24 hours ago

            Huh, I think you’re right.

            Before discovering ZFS, my previous backup solution was rdiff-backup. I have memories of it being problematic for me, but I may be wrong in my remembering of why it caused problems.

      • melfie@lemy.lol
        link
        fedilink
        English
        arrow-up
        8
        arrow-down
        2
        ·
        2 days ago

        Having a synced copy elsewhere is not an adequate backup and snapshots are pretty important. I recently had RAM go bad and my most recent backups had corrupt data, but having previous snapshots saved the day.

        • melfie@lemy.lol
          link
          fedilink
          English
          arrow-up
          2
          ·
          23 hours ago

          Don’t understand the downvotes. This is the type of lesson people have learned from losing data and no sense in learning it the hard way yourself.

          • tomenzgg@midwest.social
            link
            fedilink
            English
            arrow-up
            1
            ·
            8 hours ago

            How would you pin down something like this? If it happened to me, I expect I just wouldn’t understand what’s going on.

            • melfie@lemy.lol
              link
              fedilink
              English
              arrow-up
              0
              ·
              edit-2
              3 hours ago

              I originally thought it was one of my drives in my RAID1 array that was failing, but I noticed copying data was yielding btrfs corruption errors on both drives that could not be fixed with a scrub and I was also getting btrfs corruption errors on the root volume as well. I figured it would be quite an odd coincidence if my main SSD and 2 hard disks all went bad and I happened upon an article talking about how corrupt data can also occur if the RAM is bad. I also ran SMART tests and everything came back with a clean bill of health. So, I installed and booted into Memtester86+ and it immediately started showing errors on the single 16Gi stick I was using. I happened to have a spare stick that was a different brand, and that one passed the memory test with flying colors. After that, all the corruption errors went away and everything has been working perfectly ever since.

              I will also say that legacy file systems like ext4 with no checksums wouldn’t even complain about corrupt data. I originally had ext4 on my main drive and at one point thought my OS install went bad, so I reinstalled with btrfs on top of LUKS and saw I was getting corruption errors on the main drive at that point, so it occurred to me that 3 different drives could not have possibly had a hardware failure and something else must be going on. I was also previously using ext4 and mdadm for my RAID1 and migrated it to btrfs a while back. I was previously noticing as far back as a year ago that certain installers, etc. that previously worked no longer worked, which happened infrequently and didn’t really register with me as a potential hardware problem at the time, but I think the RAM was actually progressively going bad for quite a while. btrfs with regular scrubs would’ve made it abundantly clear much sooner that I had files getting corrupted and that something was wrong.

              So, I’m quite convinced at this point that RAID is not a backup, even with the abilities of btrfs to self-heal, and simply copying data elsewhere is not a backup, because something like bad RAM in both cases can destroy data during the copying process, whereas older snapshots in the cloud will survive such a hardware failure. Older data backed up that wasn’t coped with faulty RAM may be fine as well, but you’re taking a chance that a recent update may overwrite good data with bad data. I was previously using Rclone for most backups while testing Restic with daily, weekly, and monthly snapshots for a small subset of important data the last few months. After finding some data that was only recoverable in a previous Restic snapshot, I’ve since switched to using Restic exclusively for anything important enough for cloud backups. I was mainly concerned about the space requirements of keeping historical snapshots, and I’m still working on tweaking retention policies and taking separate snapshots of different directories with different retention policies according risk tolerance for each directory I’m backing up. For some things, I think even btrfs local snapshots would suffice with the understanding that it’s to reduce recovery time, but isn’t really a backup . However, any irreplaceable data really needs monthly Restic snapshots in the cloud. I suppose if don’t have something like btrfs scrubs to alert you that you have a problem, even snapshots from months ago may have an unnoticed problem.

    • non_burglar@lemmy.world
      link
      fedilink
      English
      arrow-up
      1
      arrow-down
      1
      ·
      2 days ago

      I use rsync and a pruning script in crontab on my NFS mounts. I’ve tested it numerous times breaking containers and restoring them from backup. It works great for me at home because I don’t need anything older than 4 monthly, 4 weekly, and 7 daily backups.

      However, in my job I prefer something like bacula. The extra features and granularity of restore options makes a world of difference when someone calls because they deleted prod files.

      • tal@olio.cafe
        link
        fedilink
        English
        arrow-up
        1
        ·
        edit-2
        1 day ago

        I don’t know if there’s a term for them, but Bacula (and I think AMANDA might fall into this camp, but I haven’t looked at it in ages) are oriented more towards…“institutional” backup. Like, there’s a dedicated backup server, maybe dedicated offline media like tapes, the backup server needs to drive the backup, etc).

        There are some things that rsnapshot, rdiff-backup, duplicity, and so forth won’t do.

        • At least some of them (rdiff-backup, for one) won’t dedup files with different names. If a file is unchanged, it won’t use extra storage, but it won’t identify different identical files at different locations. This usually isn’t all that important for a single host, other than maybe if you rename files, but if you’re backing up many different hosts, as in an institutional setting, they likely files in common. They aren’t intended to back up multiple hosts to a single, shared repository.

        • Pull-only. I think that it might be possible to run some of the above three in “pull” mode, where the backup server connects and gets the backup, but where they don’t have the ability to write to the backup server. This may be desirable if you’re concerned about a host being compromised, but not the backup server, since it means that an attacker can’t go dick with your backups. Think of those cybercriminals who encrypt data at a company and wipe other copies and then demand a ransom for an unlock key. But the “institutional” backup systems are going to be aimed at having the backup server drive all this, and have the backup server have access to log into the individual hosts and pull the backups over.

        • Dedup for non-identical files. Note that restic can do this. While files might not be identical, they might share some common elements, and one might want to try to take advantage of that in backup storage.

        • rdiff-backup and rsnapshot don’t do encryption (though duplicity does). If one intends to use storage not under one’s physical control (e.g. “cloud backup”), this might be a concern.

        • No “full” backups. Some backup programs follow a scheme where one periodically does a backup that stores a full copy of the data, and then stores “incremental” backups from the last full backup. All rsnapshot, rdiff-backup, and duplicity are always-incremental, and are aimed at storing their backups on a single destination filesystem. A split between “full” and “incremental” is probably something you want if you’re using, say, tape storage and having backups that span multiple tapes, since it controls how many pieces of media you have to dig up to perform a restore.

        • I don’t know how Bacula or AMANDA handle it, if at all, but if you have a DBMS like PostgreSQL or MySQL or the like, it may be constantly receiving writes. This means that you can’t get an atomic snapshot of the database, which is critical if you want to be reliably backing up the storage. I don’t know what the convention is here, but I’d guess either using filesystem-level atomic snapshot support (e.g. btrfs) or requiring the backup system to be aware of the DBMS and instructing it to suspend modification while it does the backup. rsnapshot, rdiff-backup, and duplicity aren’t going to do anything like that.

        I’d agree that using the more-heavyweight, “institutional” backup programs can make sense for some use cases, like if you’re backing up many workstations or something.

  • mesa@piefed.socialOP
    link
    fedilink
    English
    arrow-up
    51
    ·
    2 days ago

    Ive personally used rsync for backups for about…15 years or so? Its worked out great. An awesome video going over all the basics and what you can do with it.

    • Eager Eagle@lemmy.world
      link
      fedilink
      English
      arrow-up
      7
      arrow-down
      1
      ·
      edit-2
      2 days ago

      It works fine if all you need is transfer, my issue with it it’s just not efficient. If you want a “time travel” feature, your only option is to duplicate data. Differential backups, compression, and encryption for off-site ones is where other tools shine.

      • suicidaleggroll@lemmy.world
        link
        fedilink
        English
        arrow-up
        1
        ·
        edit-2
        1 day ago

        If you want a “time travel” feature, your only option is to duplicate data.

        Not true. Look at the --link-dest flag. Encryption, sure, rsync can’t do that, but incremental backups work fine and compression is better handled at the filesystem level anyway IMO.

        • Eager Eagle@lemmy.world
          link
          fedilink
          English
          arrow-up
          1
          ·
          edit-2
          24 hours ago

          Isn’t that creating hardlinks between source and dest? Hard links only work on the same drive. And I’m not sure how that gives you “time travel”, as in, browsing snapshots or file states at the different times you ran rsync.

          Edit: ah the hard link is between dest and the link-dest argument, makes more sense.

          I wouldn’t bundle fs and backup compression in the same bucket, because they have vastly different reqs. Backup compression doesn’t need to be optimized for fast decompression.

          • BCsven@lemmy.ca
            link
            fedilink
            English
            arrow-up
            1
            ·
            22 hours ago

            Snapper and BTRFS. Its only adjusts changes in data, so time travel is just pointing to what blocks changed and when, and not building a duplicate of the entire file or filesystem. A snapshot is instant, and new block changes belong to the current default.

      • bandwidthcrisis@lemmy.world
        link
        fedilink
        English
        arrow-up
        3
        ·
        1 day ago

        I have it add a backup suffix based on the date. It moves changed and deleted files to another directory adding the date to the filename.

        It can also do hard-link copied so that you can have multiple full directory trees to avoid all that duplication.

        No file deltas or compression, but it does mean that you can access the backups directly.

        • koala@programming.dev
          link
          fedilink
          English
          arrow-up
          2
          ·
          1 day ago

          Thanks! I was not aware of these options, along with what other poster mentioned about --link-dest. These do turn rsync into a backup program, which is something the root article should explain!

          (Both are limited in some aspects to other backup software, but they might still be a simpler but effective solution. And sometimes simple is best!)

      • NekuSoul@lemmy.nekusoul.de
        link
        fedilink
        English
        arrow-up
        1
        ·
        edit-2
        2 days ago

        Agree. It’s neat for file transfers and simple one-shot backups, but if you’re looking for a proper backup solution then other tools/services have advanced virtually every aspect of backups so much it pretty much always makes sense to use one of those instead.

    • confusedpuppy@lemmy.dbzer0.com
      link
      fedilink
      English
      arrow-up
      3
      ·
      edit-2
      2 days ago

      I use rsync for many of the reasons covered in the video. It’s widely available and has a long history. To me that feels important because it’s had time to become stable and reliable. Using Linux is a hobby for me so my needs are quite low. It’s nice to have a tool that just works.

      I use it for all my backups and moving my backups to off network locations as well as file/folder transfers on my own network.

      I even made my own tool (https://codeberg.org/taters/rTransfer) to simplify all my rsync commands into readable files because rsync commands can get quite long and overwhelming. It’s especially useful chaining multiple rsync commands together to run under a single command.

      I’ve tried other backup and syncing programs and I’ve had bad experiences with all of them. Other backup programs have failed to restore my system. Syncing programs constantly stop working and I got tired of always troubleshooting. Rsync when set up properly has given me a lot less headaches.

      • sugar_in_your_tea@sh.itjust.works
        link
        fedilink
        English
        arrow-up
        3
        ·
        1 day ago

        That would only matter if it’s lots of small files, right? And after the initial sync, you’d have very few files, no?

        Rsync is designed for incremental syncs, which is exactly what you want in a backup solution. If your multithreaded alternative doesn’t do a diff, rsync will win on larger data sets that don’t have rapid changes.

  • vext01@lemmy.sdf.org
    link
    fedilink
    English
    arrow-up
    4
    ·
    1 day ago

    I used to use rsnapshot, which is a thin wrapper around rsync to make it incremental, but moved to restic and never looked back. Much easier and encrypted by default.

  • Mio@feddit.nu
    link
    fedilink
    English
    arrow-up
    5
    arrow-down
    1
    ·
    edit-2
    1 day ago

    I think the there are better alternatives for backup like kopia and restic. Even seafile. Want protection against ransomware, storage compression, encryption, versioning, sync upon write and block deduplication.

    • Toribor@corndog.social
      link
      fedilink
      English
      arrow-up
      1
      arrow-down
      1
      ·
      1 day ago

      This exactly. I’d use rsync to sync a directory to a location to then be backed up by kopia, but I wouldn’t use rsync exclusively for backups.