Borg: Error on init a new repo on a smb share

[csteinforth]

Have you checked borgbackup docs, FAQ, and open Github issues?

Yes

Is this a BUG / ISSUE report or a QUESTION?

BUG

System information. For client/server mode post info for both machines.

Your borg version (borg -V).

borg 1.1.13

Operating system (distribution) and version.

Linux EOS 5.7.8-arch1-1 #1 SMP PREEMPT Thu, 09 Jul 2020 16:34:01 +0000 x86_64 GNU/Linux
(Endeavour OS)

Hardware / network configuration, and filesystems used.

Local: BTRFS
Remote: NAS with XFS

How much data is handled by borg?

Nothing on this command.

Full borg commandline that lead to the problem (leave away excludes and passwords)

borg init --encryption=repokey-blake2 /mnt/cifs/share/Borg/EOSHome

Describe the problem you're observing.

Creating a new repo on a smb share is not possible any more. I am quite sure that it worked before.

Can you reproduce the problem? If so, describe how. If not, describe troubleshooting steps you took before opening the issue.

The issue is reproducible, command line above brings the same message.

Include any warning/errors/backtraces from the system logs

Local Exception
Traceback (most recent call last):
  File "/usr/lib/python3.8/site-packages/borg/archiver.py", line 4565, in main
    exit_code = archiver.run(args)
  File "/usr/lib/python3.8/site-packages/borg/archiver.py", line 4497, in run
    return set_ec(func(args))
  File "/usr/lib/python3.8/site-packages/borg/archiver.py", line 176, in wrapper
    return method(self, args, repository=repository, **kwargs)
  File "/usr/lib/python3.8/site-packages/borg/archiver.py", line 290, in do_init
    key = key_creator(repository, args)
  File "/usr/lib/python3.8/site-packages/borg/crypto/key.py", line 110, in key_creator
    return key.create(repository, args)
  File "/usr/lib/python3.8/site-packages/borg/crypto/key.py", line 690, in create
    key.save(target, passphrase)
  File "/usr/lib/python3.8/site-packages/borg/crypto/key.py", line 800, in save
    target.save_key(key_data)
  File "/usr/lib/python3.8/site-packages/borg/repository.py", line 328, in save_key
    self.save_config(self.path, self.config)
  File "/usr/lib/python3.8/site-packages/borg/repository.py", line 313, in save_config
    config.write(fd)
  File "/usr/lib/python3.8/site-packages/borg/platform/base.py", line 176, in __exit__
    os.replace(self.tmppath, self.path)
FileExistsError: [Errno 17] File exists: '/mnt/cifs/share/Borg/EOSHomeCS/config.tmp' -> '/mnt/cifs/share/Borg/EOSHome/config'

Platform: Linux EOS 5.7.8-arch1-1 #1 SMP PREEMPT Thu, 09 Jul 2020 16:34:01 +0000 x86_64
Linux: Unknown Linux  
Borg: 1.1.13  Python: CPython 3.8.3 msgpack: 0.5.6
PID: 85163  CWD: /mnt/cifs/share
sys.argv: ['/usr/bin/borg', 'init', '--encryption=repokey-blake2', '/mnt/cifs/share/Borg/EOSHome']
SSH_ORIGINAL_COMMAND: None

[infectormp]

I think this is not related to borg, see https://github.com/borgbackup/borg/issues/5190

[csteinforth]

I just checked in Manjaro with

~ >>> borg -V                                                                  
borg 1.1.11

and

~ >>> uname -a                                                                 
Linux manjaro 5.4.44-1-MANJARO #1 SMP PREEMPT Wed Jun 3 14:48:07 UTC 2020 x86_64 GNU/Linux

that it works. Downgrading borg to 1.1.11 or switching to an LTS kernel in Endeavour OS does not solve the issue. Is there anything missing in python install?

Thanks in advance,

Christoph

[ThomasWaldmann]

borg does nothing differently for smb (or nfs or any filesystem).

so if something works on a local and proven fs like ext4 and does not work for smb (or nfs or some fuse fs or some cloud fs, or ...), it likely means there is something broken / behaving unexpectedly in that filesystem.

[ThomasWaldmann]

in this special case (see traceback), renaming the new config file over the existing config file (to atomically replace it) does not work.

renaming a file over an existing file is expected behaviour on posix (e.g. linux) filesystems.

[ThomasWaldmann]

https://docs.python.org/3/library/os.html#os.replace

[ThomasWaldmann]

@csteinforth can you try this:

• create 2 more new files there (both not empty and with different content)
• on the shell try: mv /mnt/cifs/share/Borg/EOSHomeCS/testfile.tmp /mnt/cifs/share/Borg/EOSHome/testfile

does it work as expected?

[csteinforth]

[csteinforth@EOS ~]$ echo "Hallo" > /mnt/cifs/share/BorgBackup/EOSHomeCS/testfile.tmp
[csteinforth@EOS ~]$ echo "Hallo hallo" > /mnt/cifs/share/BorgBackup/EOSHomeCS/testfile
[csteinforth@EOS ~]$ mv /mnt/cifs/share/BorgBackup/EOSHomeCS/testfile.tmp /mnt/cifs/share/BorgBackup/EOSHomeCS/testfile
mv: das Verschieben von '/mnt/cifs/share/BorgBackup/EOSHomeCS/testfile.tmp' nach '/mnt/cifs/share/BorgBackup/EOSHomeCS/testfile' ist nicht möglich: Die Datei existiert bereits

So I get the message that the file already exists. May this be some kind of permission problem?

[ThomasWaldmann]

try: mv /mnt/cifs/share/Borg/EOSHomeCS/testfile.tmp /mnt/cifs/share/Borg/EOSHome/doesnotexist

[csteinforth]

mv /mnt/cifs/share/BorgBackup/EOSHomeCS/testfile.tmp /mnt/cifs/share/BorgBackup/EOSHomeCS/doesnotexist

works!

[ThomasWaldmann]

OK, so that means you don't have a permissions issue, but the fs rejects to rename over an existing file (which is unexpected).

Guess you need to find out why CIFS does that and whether there is a way to change that behaviour.

[csteinforth]

Ok, thanks for your answers! I will have a look into this. There must be a difference in my Manjaro install (it works) and my Endeavour OS install (doesn't work).

[ThomasWaldmann]

I'm closing this for now, does not look like a borg issue.

[OkanEsen]

Weird, this issue started to pop up a few days ago and I can't figure out what the underlying issue is. The weird part is that I haven't changed anything and the setup was working fine for months.

There must be an issue with CIFS but I haven't been able to trace the issue yet.

I'll try to dig deeper an report back whenever I find a solution.

The error I get is:

Traceback (most recent call last):
  File "/usr/lib64/python3.8/site-packages/borg/archiver.py", line 4565, in main
    exit_code = archiver.run(args)
  File "/usr/lib64/python3.8/site-packages/borg/archiver.py", line 4497, in run
    return set_ec(func(args))
  File "/usr/lib64/python3.8/site-packages/borg/archiver.py", line 176, in wrapper
    return method(self, args, repository=repository, **kwargs)
  File "/usr/lib64/python3.8/site-packages/borg/archiver.py", line 1640, in do_prune
    manifest.write()
  File "/usr/lib64/python3.8/site-packages/borg/helpers.py", line 454, in write
    self.repository.put(self.MANIFEST_ID, self.key.encrypt(data))
  File "/usr/lib64/python3.8/site-packages/borg/crypto/key.py", line 370, in encrypt
    self.nonce_manager.ensure_reservation(num_aes_blocks(len(data)))
  File "/usr/lib64/python3.8/site-packages/borg/crypto/nonces.py", line 85, in ensure_reservation
    self.commit_repo_nonce_reservation(reservation_end, repo_free_nonce)
  File "/usr/lib64/python3.8/site-packages/borg/crypto/nonces.py", line 48, in commit_repo_nonce_reservation
    self.repository.commit_nonce_reservation(next_unreserved, start_nonce)
  File "/usr/lib64/python3.8/site-packages/borg/repository.py", line 354, in commit_nonce_reservation
    fd.write(bin_to_hex(next_unreserved.to_bytes(8, byteorder='big')))
  File "/usr/lib64/python3.8/site-packages/borg/platform/base.py", line 176, in __exit__
    os.replace(self.tmppath, self.path)
FileExistsError: [Errno 17] File exists: '/nas/backup/nonce.tmp' -> '/nas/backup/nonce'

System info:

Platform: Linux 5.7.8-200.fc32.x86_64 #1 SMP Thu Jul 9 14:34:51 UTC 2020 x86_64
Linux: Unknown Linux
Borg: 1.1.13  Python: CPython 3.8.3 msgpack: 0.5.6
Operating System: Fedora 32 (Workstation Edition)

[csteinforth]

@OkanEsen I was able to solve it somehow. On my NAS (quite old :wink: ) there is an option to enable SMB2. I chose this, and changed vers=1.0 to vers=2.0 in my mount options. After that it was working again.

[OkanEsen]

Huh, weird, that was indeed the solution, thanks @csteinforth! Kinda curios why it was working before but that's for another day. :)

In my case, I used vers=default which, according to man mount.cifs 8:

default - Tries to negotiate the highest SMB2+ version supported by both the client and server.

The sec value, if set to ntlm, should be also updated accordingly (sec=ntlmssp), since ntlm didn't work on SMB2+ for me.

[csteinforth]

@OkanEsen vers=default does not work for me, I have to stick to vers=2.0. But the really strange thing is that it worked before. I did a fresh install of Endeavour OS two days ago and it did not work with vers=1.0, and I can confirm the same on the current live iso.

@ThomasWaldmann keep up the good work on borg and thanks for pushing me into the correct direction 👍

[kalsan]

For those who cannot use vers 2.0, here's some more background: Looks like the problem occurs in more recent kernels.

References:
https://unix.stackexchange.com/questions/599281/cifs-mount-is-returning-errors-when-operating-on-remote-files-file-exists-inte
https://serverfault.com/questions/1025734/cifs-automounts-suddenly-stopped-working/1025993#1025993

[kalsan]

Update: Under Debian 10, installing linux-image-5.4.0-0.bpo.2-amd64 from buster-backports solves the problem.

[ThomasWaldmann]

Can you also please add the kernel version that has the problem?

[kalsan]

The non-working kernel version was 4.19.0 (Debian Buster default).