Borg: Error: Failed to create/acquire the lock

[Troyciv]

I accidentally disconnected my storage device during backup. I got the following error:

Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/borg/repository.py", line 72, in __del__
    self.close()
  File "/usr/lib/python3/dist-packages/borg/repository.py", line 190, in close
    self.io.close()
  File "/usr/lib/python3/dist-packages/borg/repository.py", line 586, in close
    self.close_segment()
  File "/usr/lib/python3/dist-packages/borg/repository.py", line 822, in close_segment
    sync_dir(os.path.dirname(self._write_fd.name))
  File "/usr/lib/python3/dist-packages/borg/platform.py", line 8, in sync_dir
    fd = os.open(path, os.O_RDONLY)
FileNotFoundError: [Errno 2] No such file or directory: '/media/user/Backup/Backup/data/1'
Local Exception.
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/borg/archiver.py", line 85, in wrapper
    return method(self, args, repository=repository, **kwargs)
  File "/usr/lib/python3/dist-packages/borg/archiver.py", line 249, in do_create
    create_inner(archive, cache)
  File "/usr/lib/python3/dist-packages/borg/archiver.py", line 223, in create_inner
    read_special=args.read_special, dry_run=dry_run)
  File "/usr/lib/python3/dist-packages/borg/archiver.py", line 303, in _process
    read_special=read_special, dry_run=dry_run)
  File "/usr/lib/python3/dist-packages/borg/archiver.py", line 303, in _process
    read_special=read_special, dry_run=dry_run)
  File "/usr/lib/python3/dist-packages/borg/archiver.py", line 303, in _process
    read_special=read_special, dry_run=dry_run)
  File "/usr/lib/python3/dist-packages/borg/archiver.py", line 303, in _process
    read_special=read_special, dry_run=dry_run)
  File "/usr/lib/python3/dist-packages/borg/archiver.py", line 277, in _process
    status = archive.process_file(path, st, cache, self.ignore_inode)
  File "/usr/lib/python3/dist-packages/borg/archive.py", line 683, in process_file
    chunks.append(cache.add_chunk(self.key.id_hash(chunk), chunk, self.stats))
  File "/usr/lib/python3/dist-packages/borg/cache.py", line 371, in add_chunk
    self.repository.put(id, data, wait=False)
  File "/usr/lib/python3/dist-packages/borg/repository.py", line 535, in put
    segment, offset = self.io.write_put(id, data)
  File "/usr/lib/python3/dist-packages/borg/repository.py", line 784, in write_put
    fd = self.get_write_fd(raise_full=raise_full)
  File "/usr/lib/python3/dist-packages/borg/repository.py", line 662, in get_write_fd
    self.close_segment()
  File "/usr/lib/python3/dist-packages/borg/repository.py", line 816, in close_segment
    os.fsync(self._write_fd.fileno())
OSError: [Errno 5] Input/output error

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/borg/archiver.py", line 1684, in main
    exit_code = archiver.run(args)
  File "/usr/lib/python3/dist-packages/borg/archiver.py", line 1621, in run
    return args.func(args)
  File "/usr/lib/python3/dist-packages/borg/archiver.py", line 85, in wrapper
    return method(self, args, repository=repository, **kwargs)
  File "/usr/lib/python3/dist-packages/borg/repository.py", line 94, in __exit__
    self.close()
  File "/usr/lib/python3/dist-packages/borg/repository.py", line 190, in close
    self.io.close()
  File "/usr/lib/python3/dist-packages/borg/repository.py", line 586, in close
    self.close_segment()
  File "/usr/lib/python3/dist-packages/borg/repository.py", line 816, in close_segment
    os.fsync(self._write_fd.fileno())
OSError: [Errno 5] Input/output error

Platform: Linux ThinkPad 4.4.0-96-generic #119-Ubuntu SMP Tue Sep 12 14:59:54 UTC 2017 x86_64 x86_64
Linux: Ubuntu 16.04 xenial
Borg: 1.0.7  Python: CPython 3.5.2
PID: 16056  CWD: /home/user/.local/share/Anki2/User 1/collection.media
sys.argv: ['/usr/bin/borg', 'create', '--info', '--stats', '--progress', '--show-rc', '--compression', 'lz4', '/media/user/Backup/Backup/::{hostname}-{now:%Y-%m-%d}', '/home', '--exclude', '/home/*/.cache', '--exclude', '/home/*/.local/share/Trash', '--exclude', '/home/*/.thumbnails', '--exclude', '/home/*/VirtualBox VMs', '--exclude', '/home/user/Dropbox', '--exclude', '/home/user/Steam', '--exclude', '/home/user/.steam', '--exclude', '/home/user/.www.kiwix.org']
SSH_ORIGINAL_COMMAND: None

terminating with error status, rc 2
Repository /media/user/Backup/Backup does not exist.
terminating with error status, rc 2

Now when I try to run borg again, I get the following:

Failed to create/acquire the lock /media/user/Backup/Backup/lock.exclusive (timeout).
terminating with error status, rc 2

[dimejo]

Make sure that nothing is accessing the repo and then remove the lock.

[Troyciv]

perfect, thank you

[FabioPedretti]

BTW, if you upgrade to borg 1.1 there should be no more need to remove the lock, this error shouldn't happen any more. See BORG_HOSTNAME_IS_UNIQUE here.

[ThomasWaldmann]

Guess this is solved.

[obelixA]

this is still happening with version 1.1.10
borg list /path/to/backup
results in
Failed to create/acquire the lock /path/to/backup/lock.exclusive (timeout)

any hints how to solve this? my backup server (everything locally, source and destination) hangs for the last three months making no backup anymore. this is horrible, because customer data is affected...

[ThomasWaldmann]

If your backup hangs for 3 months without you noticing, you should think about better monitoring of it.

I guesst the least you could do is always send an email with the backup log and result status, then you'll notice issues almost immediately. More advanced is using some monitoring tool that monitors all sorts of things (including borg), stuff like icinga/nagios and others.

A repo can be locked due to a parallel borg process having the lock (that is intended).

Or due to a stale lock which can happen if a borg process experiences a sudden death (sigkill, segfault, hardware issues, power loss, etc.) and the lock can't be removed by the borg process which created it.
In such a case, please read the docs of borg break-lock.

When running all locally, borg should be able to detect and remove such stale locks automatically in many cases.

There are some edge cases though and there are existing PRs for improving the locking system, but "usually, it works".

If it doesn't and fails frequently for your, maybe try to find out what's unusual on your system.

[obelixA]

monitoring - yes.
but the borg issue is _always_
today I removed the locks manually, then the cron job starts the next backup... and bord hangs at the same place. starting the "borg list" results in "failed to..."
there is nothing special about the system. It's an ubuntu with nearly nothing running but backup for customer data (fortunately this is the test server to get used to borg backup - but this should get the productive server asap)
any ideas what else could lock the borg backup?
I can promise: no sigkill, segfault, hardware issue, power loss or other unforeseen event...

[obelixA]

additional info: if no job is running, the list command works. this is strange...

[ThomasWaldmann]

Are you mixing up ownership/permissions by running borg as different user against same repo?

[obelixA]

nope - user root starts the cron job and I am root while trying to check the repo

[jdchristensen]

The error messages in the first message are input/output errors, which indicate something more fundamentally wrong than a locking problem. Maybe you should give ls -l output for some of the repo dirs. E.g. if you cd into the repo, you could do ls -ld, ls -l, and ls -l data and post the output.

Then, you need to give the exact commands you run and the exact error messages.

[obelixA]

the first message in this thread isn't mine ;-)
I only wanted to say, that the locking problem is not solved.
If a backup job is running (with root) I cannot list the backups. If the job is done, the listing is ok...

[ThomasWaldmann]

I am locking this issue now. There is no point in appending to 3yo closed tickets.

If you think there is an issue in borg, then open a fresh ticket and provide much more information (and also try to reproduce it yourself on other system(s) first).