Borg: extract always fails with "No space left on device"

[keepitsimpleengineer]

A 201 GB backup for simple Windows 10 system from a USB drive extract always fails with No space left on device.

All on local system
Source partition:797.6 GB free (USB3)
Cache partition:66.6 GB free (SSD)
Destination partition: 976.6 GB free (HDD)
System memory: 15.6 GB
borg (borgbackup) 1.1.4-1
encryption none
compression zlib,5
python 3.6.4-1

OSError: [Errno 28] No space left on device

Output with trace back at https://pastebin.com/arH7Pjbu

Suggestion to post this as an issue at https://bbs.archlinux.org/viewtopic.php?id=233815

Workaround was to add a few hundred GB to ~/.cache/borg

[enkore]

This is a bug because the exception crashing borg is item-related and should not crash borg. See excerpt below.

run/media/ljohnson/WIN_10b/System Volume Information/tracking.log: write: [Errno 28] No space left on device
Local Exception
Traceback (most recent call last):
  File "/usr/lib/python3.6/site-packages/borg/archive.py", line 607, in extract_item
    fd.truncate(pos)
OSError: [Errno 28] No space left on device

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/lib/python3.6/site-packages/borg/archive.py", line 609, in extract_item
    self.restore_attrs(path, item, fd=fd.fileno())
  File "/usr/lib/python3.6/site-packages/borg/archive.py", line 166, in __exit__
    raise BackupOSError(self.op, exc_val) from exc_val
borg.archive.BackupOSError: truncate_and_attrs: [Errno 28] No space left on device

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3.6/site-packages/borg/archiver.py", line 4175, in main
    exit_code = archiver.run(args)
  File "/usr/lib/python3.6/site-packages/borg/archiver.py", line 4107, in run
    return set_ec(func(args))
  File "/usr/lib/python3.6/site-packages/borg/archiver.py", line 150, in wrapper
    return method(self, args, repository=repository, **kwargs)
  File "/usr/lib/python3.6/site-packages/borg/archiver.py", line 163, in wrapper
    return method(self, args, repository=repository, manifest=manifest, key=key, archive=archive, **kwargs)
  File "/usr/lib/python3.6/site-packages/borg/archiver.py", line 750, in do_extract
    stripped_components=strip_components, original_path=orig_path, pi=pi)
  File "/usr/lib/python3.6/site-packages/borg/archive.py", line 609, in extract_item
    self.restore_attrs(path, item, fd=fd.fileno())
OSError: [Errno 28] No space left on device

[ThomasWaldmann]

copied from archlinux site, post by dom0:

It is possible that the root cause is a bug too, since the files in
"System Volume Information" may possess some strange properties
not handled correctly by Borg. Try to --exclude them and see if that helps.

I would see that as the root cause of this issue. And as long as borg does not get special support for windows (and windows specialities like this) I would not consider this as a bug. You can also get weird effects on linux if you backup/restore to /proc, /sys or so (and everybody knows that and does not backup that). I don't know enough details about that "System Volume Information" - does it even make sense to backup that?

Yes, a per item problem should not crash borg. OTOH, "no space left" sounds pretty game over usually.

[Anakonda]

There is no point in backuping System Volume Information. It might even be harmfull as system restore points, mount points and volume snapshots via volume shadow copy are stored there.

[ThomasWaldmann]

I checked out 1.1.4 and had a look, but fail to see the error in per-item extract exception handling.

archiver.py:751 should deal with it, no?

@enkore do you have more details / insights?

[enkore]

The TB looks kinda funny, because

605                    with backup_io('truncate_and_attrs'):
606                        pos = item_chunks_size = fd.tell()
607                        fd.truncate(pos)                                 <- errno 28
608                        fd.flush()
609                        self.restore_attrs(path, item, fd=fd.fileno())   <- errno 28 *again*, in the same call tree! how!?

Traceback (most recent call last):
File "/usr/lib/python3.6/site-packages/borg/archive.py", line 607, in extract_item
fd.truncate(pos)
OSError: [Errno 28] No space left on device

The above exception was the direct cause of the following exception:

propagated through backup_io, okay ...

During handling (i.e. inside an "except:" clause) of the above exception, another exception occurred:

Traceback (most recent call last):
File "/usr/lib/python3.6/site-packages/borg/archiver.py", line 4175, in main
exit_code = archiver.run(args)
File "/usr/lib/python3.6/site-packages/borg/archiver.py", line 4107, in run
return set_ec(func(args))
File "/usr/lib/python3.6/site-packages/borg/archiver.py", line 150, in wrapper
return method(self, args, repository=repository, *kwargs)
File "/usr/lib/python3.6/site-packages/borg/archiver.py", line 163, in wrapper
return method(self, args, repository=repository, manifest=manifest, key=key, archive=archive, *kwargs)
File "/usr/lib/python3.6/site-packages/borg/archiver.py", line 750, in do_extract
stripped_components=strip_components, original_path=orig_path, pi=pi)
File "/usr/lib/python3.6/site-packages/borg/archive.py", line 609, in extract_item
self.restore_attrs(path, item, fd=fd.fileno())
OSError: [Errno 28] No space left on device

Notice how archiver.py:750 is not reachable while handling an exception. The traceback doesn't really make any sense to me; it might be missing some important detail (because of nested context managers?).

[enkore]

IOW I definitely think there is a bug here, because this shouldn't lead to a crash. Just isn't clear to me where the bug is hiding, exactly.

[ThomasWaldmann]

I just put a manual raise there, no problem with exception handling:

with backup_io('truncate_and_attrs'):
    pos = item_chunks_size = fd.tell()
    raise OSError(errno.ENOSPC, "disk full", item.path)  # does not abort borg, rc == 1
    fd.truncate(pos)
    fd.flush()
    self.restore_attrs(path, item, fd=fd.fileno())

[ThomasWaldmann]

Same for write:

with backup_io('write'):
    if sparse and self.zeros.startswith(data):
        # all-zero chunk: create a hole in a sparse file
        fd.seek(len(data), 1)
    else:
        fd.write(data)
    raise OSError(errno.ENOSPC, "disk full", item.path)  # does not abort borg, rc == 1

[ThomasWaldmann]

I could not reproduce a problem.

Guess we need some way to reproduce this, that does not involve windows (or files from windows that should not be backed up / restored anyway).

[enkore]

The traceback is from Linux though and suggests a control flow issue - in theory it should be possible to spot by looking at the code.

[ThomasWaldmann]

Somehow related: #3808

[ThomasWaldmann]

@keepitsimpleengineer can you reproduce this with a borg version that has the fix for #3808, e.g.:

• a borg release >= 1.1.6
• borg > 1.0.12 (not released yet)
• borg 1.0.x with #3835 patch applied
• borg current 1.0-maint branch code (git checkout)

[keepitsimpleengineer]

Unable, borge deprecated as a result of system change.

On Tue, Jul 3, 2018 at 9:58 AM, TW [e-mail] wrote:

@keepitsimpleengineer https://github.com/keepitsimpleengineer can you
reproduce this with a borg version that has the fix for #3808
https://github.com/borgbackup/borg/issues/3808 (e.g. 1.1.6+)?

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/borgbackup/borg/issues/3574#issuecomment-402225089,
or mute the thread
https://github.com/notifications/unsubscribe-auth/ARqNWiGn6NDF1WJnhMm7BHriGIj0TAeEks5uC6KagaJpZM4RqT0c
.

--

--

lj, the keepitsimpleengineer …my web
site…http://keepitsimpleengineering.com/
http://keepitsimpleengineering.com/

――――――――――――――――――

――――――――――――――――――

[ThomasWaldmann]

I am closing this as it might be fixed (see above) or it might be some effect triggered by restoring stuff that is neither intended to be backed up nor restored.

If someone is still able to reproduce this under practically valid conditions with a current version of borg, please open a new issue for it.