Borg: simple archive copy between repos

[stevegt]

Hi All,

I know this has been hashed over before in e.g. #1697, #1421, and #895, but I'm looking for a simple archive copy method that isn't lossy and doesn't involve building the filesystem tree on disk in an intermediate extract. What I'm looking for is the moral equivalent of this:

borg extract --stdout repo1:arc1 | borg create repo2:arc1 -

This of course just creates a single file in repo2:arc1 named 'stdin', so won't work for filesystem trees (but works great for disk images).

The borg export-tar command comes close, but it's lossy (xattrs) and there's no borg import-tar yet anyway.

Is there something I'm missing? Is there any consensus yet on how this should work, and is anyone actively working on code for it?

I have to say that #1697 was looking pretty good while it lasted... ;-)

[ThomasWaldmann]

There is no active work on this.

Not sure if export-tar (to improve) and import-tar (todo) could be easily made to transport all the data/metadata. Maybe a custom, more simple format would be better for that purpose.

Alternatively, borg recreate with target in other repo.

But I guess most of this is already covered in existing tickets...

[enkore]

I have to say that #1697 was looking pretty good while it lasted... ;-)

I think some people misunderstood what that ticket was about, since it wasn't about copying archives and would not have provided that functionality. #895 is the ticket about replication (i.e. making exact copies (replicas) of an archive in another repository), although the end of comment #0 describes something else that wouldn't work out.

[enkore]

The tar format used is extensible and you can easily make up your own extensions (key-value pairs attached to a file). In the other tickets about it I pointed to some existing standards / conventions regarding some metadata (xattrs and such).

[stevegt]

An advantage of tar transfer would be interoperability with other systems. It sounds like #895 discusses a more low-level approach, dealing directly with borg's internal file formats.

If it makes sense, we might consider #3249 (this one) to be about tar transfer, and #895 to be about something that operates at a lower level and is likely to be more efficient (but not likely interoperable). If there's consensus on this, I can rename #3249 accordingly.

[stevegt]

A note for lurkers: Right now borg uses the python tarfile module. This module supports both reading and writing tar files, so implementing import-tar should not be too hard. The downside of the tarfile module is that afaict right now it doesn't support xattrs (which could be why export-tar is lossy). So we either replace the tarfile module or live with the lossiness until upstream gets enhanced. I think I'm leaning towards living with the lossiness, at least in the short term.

https://github.com/borgbackup/borg/blob/4a58310433f21857fc124eed4bafde2956bac46a/src/borg/archiver.py#L756

(Note to self: If you used rsync -aHS... to do backups and installs in the past, then you've already lost any extended attributes you may have had in the first place. And all of the machines anyone has ever installed with systemimager use the -aHS... flags you contributed in that 2003 patch. You needed to include -X, which rsync didn't support until 2008: https://github.com/finley/SystemImager/issues/6#issue-271077874)

[enkore]

Re. transfer I don't quite see what a ticket is needed for - exporting more metadata in tars has separate tickets, and there is a ticket for import-tar. So, just add "tar transfer" as a note to the import-tar ticket, so that there'll be an example in the docs?

[stevegt]

Didn't see those before -- for reference, @enkore is talking about #2233 and #2234, I think. The current status of those isn't clear though -- one is closed. There's an import proof of concept working and is targeted for 1.2? Do I have that right?

176 was also about tar import/export, but is also closed.

[enkore]

Right

176 is closed because it became separate import and export tickets.

The tar format used is extensible and you can easily make up your own extensions (key-value pairs attached to a file). In the other tickets about it I pointed to some existing standards / conventions regarding some metadata (xattrs and such).

These would be #2521 (xattr, ACL) and #2562 (sparse). There is no ticket for BSD flags yet.

[alejandro-perez]

I've done the following with apparent success.

borg mount REPO1::ARCHIVE /tmp/mountpoint
borg create --timestamp $ARCHIVE_TS REPO2::ARCHIVE /tmp/mountpoint

Sure it won't be the fastest, but is it correct in terms of preservation of metadata?

[ThomasWaldmann]

@alejandro-perez it depends what you have in the archive and what you need to be preserved.

IIRC, fuse will not reflect posix ACLs currently. It might work at some time in the future when the fuse / llfuse (and kernel / glibc?) stuff is updated everywhere AND borg gets some new code that uses the fuse ACL support, but this stuff is not widespread yet and the code in borg is not there yet.

IIRC, xattrs (extended attributes) will work.

user/group NAMES I guess will only be correct if the local uid/gid to name mapping is the same as on the machine which was backed up. that can easily go wrong and is a widespread problem, not borg specific.

Numerical user/group IDs (uid/gid) should be correct. mode should be correct.

bsdflags: not sure about these, maybe they don't work within fuse? but maybe a lot of people can live without them being preserved.

directory names, file names and file contents should be fine.

[alejandro-perez]

@ThomasWaldmann Thanks, since I always use --numeric-owner option, and I do not use ACLs, I should not have problems. I acknowledge this is not perfect, but at least gets the job in some cases, and wrote it since I found no similar solution in the repository issues.

In any case, a proper way of doing this would be appreciated.

[alecmev]

The lack of a proper way to clone repositories is what has tipped the scales towards Duplicacy for me, despite not being a fan of their licensing, and liking some other aspects of Borg.

I'm planning on having three repositories, and one of them is bound to be replaced at some point (e.g. an external HDD falling off a shelf). @alejandro-perez's procedure can be scripted to replicate all archives, of course, but, as pointed out, not everything is preserved, and, at a glance, this seems extremely inefficient, especially if the source repository is a remote in another country, potentially with a metered egress.

[ThomasWaldmann]

Closing this as more-or-less duplicate of #3867 (which also was closed).

We have other tickets about export and import of archives (as tar or other).

[akvadrako]

Where is the current ticket about export and import of archives? I would like to move archives around between repos and the only way I can see to do it is via mount and making a new backup.

[enkore]

4337