Borg: [question] Guidance on max_segment_size

[russelldavis]

I'm using borg to back up to a local repo which I then sync to s3 using rclone (I think this approach is fairly common, I've seen it mentioned in various places).

I discovered the append_only config, which is crucial to avoid compaction on every run[1], which would cause too much data to change when syncing with rclone. However, I'd like to periodically run borg prune (with append_only disabled) to keep the size small.

With the default max_segment_size of 500MB, it seems that compacting will still be somewhat inefficient with data. For a worst-case example, if I delete a 50MB file which gets pruned (and nothing else changes), a new 500MB segment will be created and uploaded to s3. A lower segment size would of course reduce the amount of inefficiency (in terms of data uploading).

I'm planning to experiment with a smaller segment size (starting with 50MB), but just wanted to sanity check the approach here, since I haven't seen this discussed much. What kind of performance downsides should I expect by reducing max_segment_size? Does the size of the repo make a difference? (Mine is relatively small at 3GB.)

I saw that the max size was much smaller (5MB) in an older version of borg, but I'm sure some other internals changed along with that.

Thanks.

[1] I see this will no longer be necessary as of version 1.2, which is great.

[ThomasWaldmann]

attic and very old borg used 64kiB target block size and 5MiB segment size.

because these small blocks led to a high management overhead and potentially huge ram consumption, target block size was increased to 2MiB. IIRC a bit later, segment size was optimized for optimum throughput on locally connected (or other high bw, low latency) backup disks. Besides that optimization, 5MiB segment file size of course was a bit small anyway if one considers 2MiB blocks in there.

Noone is holding you back from going down for the max_segment_size. Guess everything bigger than 20MiB should be ok. If you change, only future segments will obey the new size target.

With smaller segments and fast low latency disks, performance will decrease a bit because segments have to be closed and new ones opened/created more often.

BTW, another change in 1.2 is that it will write the manifest into a separate segment file. This has the advantage that at next backup time, when the newer manifest supersedes the previous one, it won't poke a logical small hole into a much bigger segment file, but will just completely make that small separate previous manifest-only segment file superseded and logically empty (and thus, it can be just deleted - without the need to shuffle other stuff from a non-compact segment into a new compact segment).

[russelldavis]

That's helpful, thanks! I may create a guide at some point on using local borg backups + rclone, so this will be useful.