Slow Backups

[janstadt]

Im fairly new to restic and the docs are great, but overwhelming. I was hoping to get some direct help on why my scheduled backups are taking so long (2+ days). I know that the first backup will generally be slow but i was under the impression that the subsequent ones would use the parent backup to speed the process up. I am backing up roughly 6tb via the rest-server to a local machine on a gig network.

Config

{
  "modno":  6,
  "version":  3,
  "instance":  "seed",
  "repos":  [
    {
      "id":  "Seed",
      "uri":  "[uri]",
      "password":  "[pwd]",
      "prunePolicy":  {
        "schedule":  {
          "clock":  "CLOCK_LAST_RUN_TIME"
        },
        "maxUnusedPercent":  10
      },
      "checkPolicy":  {
        "schedule":  {
          "clock":  "CLOCK_LAST_RUN_TIME"
        },
        "readDataSubsetPercent":  0
      },
      "autoUnlock":  true,
      "commandPrefix":  {
        "ioNice":  "IO_BEST_EFFORT_LOW"
      }
    }
  ],
  "plans":  [
    {
      "id":  "Seed",
      "repo":  "Seed",
      "paths":  [
        "/userdata"
      ],
      "excludes":  [
        "/userdata/recordings"
      ],
      "schedule":  {
        "cron":  "0 2 * * 1,3,6",
        "clock":  "CLOCK_LOCAL"
      },
      "retention":  {
        "policyTimeBucketed":  {
          "hourly":  24,
          "daily":  7,
          "weekly":  4,
          "monthly":  3
        }
      }
    }
  ],
  "auth":  {
    "users":  [
      {
        "name":  "[name]",
        "passwordBcrypt":  "[encrypted pwd]"
      }
    ]
  }
}

Any help would be greatly appreciated.

[garethgeorge]

Main thing that jumps out to me is that you’re using IO_BEST_EFFORT_LOW, this can be a good idea in the long run since it leaves resources more open for real time apps but could also be starving backrest of disk IO on your initial backup if other services are running. Try setting the IO priority to default or high.

Are you backing up an HDD?

[garethgeorge]

Hmm, that's not great performance. Are your run times consistent or is that spiky?

Also curious if you're using docker or a bare-metal install? The docker overlay filesystem can cause performance issues / I've noticed that it can cause extra scanning in some cases where restic should avoid it.

[janstadt]

Consistent runtimes. Running the docker version of backrest. How would i check for extra scanning?

[janstadt]

After updating to the defaults, its still real slow. machine -> gig net -> machine running rest server -> 8tb disk. I was doing it monthly since it took forever and i thought it was due to doing it monthly that was causing that, but now i set it to every other day and its still taking a very long time. Clearly something is misconfigured on my end. @garethgeorge What other information might you need to determine if its the host config or possibly the destination that might cause the slowdown?

[garethgeorge]

Hey -- the first thing that may be worth trying is running backrest without docker (to eliminate the possibility that something in the VFS is preventing restic from properly skipping files that haven't had metadata changes). This will also give better IO performance by removing the filesystem virtualization, but this probably doesn't matter too much in your case since the HDDs are almost certainly the bottleneck.

If you are sticking with docker, how often does your container restart? Is it possible that there are issues e.g. with how you mount XDG_CACHE_HOME ?

Finally, re: your backup dataset -- do you have large numbers of small files? How well does it perform with restic cli?

It's definitely unexpected that each scan is taking ~48h. I do notice in backrest docker that performance can be a bit spiky though so it's likely that this issue isn't unique to you -- just particularly bad with the size of your dataset. E.g. I have a 1TB dataset that I can see actually experiences quite large spikes in latency

I'll try running some experiments with baremetal backrest on my NAS as well to see if it makes a difference with what i'm observing there. Not sure if it's possibly the same bug you're seeing.

[janstadt]

Im running on unraid and my XDG_CACHE_HOME is mapped to /mnt/user/cache which is a "fused" filesystem as opposed to a direct file system. It still ultimately uses the cache drive but i wonder if the hybrid fused filesystem is slowing it down a bit or something? I can remount it directly to the cache to avoid that and see if that'll help. To your question about small files, yeah lots of photos, but there are plenty of larger multi gig files as well. I can experiment with running backrest natively but i fear unraid will be the limiting factor.