borg under windows 10's linux subsystem

[ThomasWaldmann]

try it, test it, document outcome.

[robaato]

First impression:
borg 1.0.8 from bundled binary borg-linux64 -> starts fine

mkdir aaa
./borg-linux64 init aaa
./borg-linux64 create --info --progress aaa::bbb /usr/
./borg-linux64 list aaa::bbb
./borg-linux64 check aaa
./borg-linux64 extract aaa::bbb
diff -rN --brief /usr usr
// no result except for few broken links, which are already broken on source

all above run fine. Looks promising. Will try to backup and restore something from Windows partitions, above is all inside linux subsystem.

[ThomasWaldmann]

@robaato thanks for the update. btw, you don't need to create the repo dir first, borg init does that, too.

[ThomasWaldmann]

My first (install via pip from pypi package) and second impression (install from git repo) is also that it works.

Installation is like on (native) Ubuntu (Trusty, 14.04). Only thing I noticed not working is the FUSE support (borg mount) as it requires the fuse kernel module, which is not there.

Now running the tests...

[ThomasWaldmann]

23 test fails:

• create_test_files: os.mknod -> Function not implemented (ENOSYS) (with root it works, so just a bad errno?) - solved by #1963.
• get_acl (from linux platform module) returns dict without b'acl_access' key (no ACL support?)
• fuse tests fail (no fuse kernel support) - likely can be avoided by omitting llfuse package.

The failing ones are somehow expected, all others work.

[ThomasWaldmann]

Docs updated, see PR #1962.

[BluCodeGH]

I have run borg 1.0.9 under the windows 10 subsystem as an administrator and while using sudo. I am backing up files from my (windows) home folder to another drive. The only errors I am seeing are [Errno 13] Permission Denied, and [Errno 5] Input/output error. Error 13 occurs on some files and folders, such as AppData/Local/ElevatedDiagnostics or Appdata/Local/Google/Chrome/User Data/Default/Current Tabs, and error 5 occures on files called LOCK, in various folders.

[enkore]

That's nice to hear.

Locking errors on Windows are to be expected unless VSS is used, although EIO / I/O error is kind of a dramatic error code for that.

[BluCodeGH]

Is there a way to get borg on lxss working with VSS?

[jdchristensen]

And any idea why some files (like ElevatedDiagnostics and Current Tabs) get Errno 13, while the LOCK files get Errno 5? The LOCK files are probably not important, but it would be good if the other files could be backed up.

[PythonNut]

@BluCodeGH

Is there a way to get borg on lxss working with VSS?

I just gave this a shot, and as far as I can tell, the answer is no, since WSL doesn't support NTFS Junctions, which is how VSS snapshots are "mounted", which is unfortunate.

[Clete2]

I am running under Windows 10 subsystem backing up 1.5TB of data without any errors. borg check on the remote host is also reporting that the integrity is good. Is there anything else I should check to ensure that the backup is complete and accurate?

FYI -- I am backing up video game downloads from the following game downloading services: Steam, Blizzard (formerly Battle.net), Origin, and some other misc games without downloading services.

[ThomasWaldmann]

@Clete2 borg check (in full mode, not just in --repository-only mode) being ok is quite a good sign already.

a more practical test is to actually extract all data again and compare it to the original data.
or extract --dry-run if you do not have the storage space.

[mnpenner]

Assuming Borg works under WSL/Win10Bash, how would we get it to start automatically on Windows startup?

[enkore]

Perhaps this helps: https://superuser.com/a/1112022/107877

[ThomasWaldmann]

Well, maybe a matter of taste, but I find anything major starting automatically on startup (or shutdown) rather annoying.

[mnpenner]

@ThomasWaldmann I do too, but we're talking about backup here. That has to run continuously, or we're vulnerable, no?

Thanks @enkore I'll take a look.

[sdellenb]

😃 Backup works surprisingly well with WSL, though there's not much data yet in the user's Windows folders.
I've configured the Windows Task Scheduler to run bash (C:\Windows\System32\bash.exe) with a wrapper script (arguments -c "~/bin/borgmatic_wrapper.sh") daily at 18:00, with the option "Run task as soon as posible after a schedule start is missed".
I'll see how the setup holds up in the future.

Note: To get borgbackup 1.0.10, I had to install it from an alternative PPA. I also installed borgmatic for easier configuration and my wrapper script collects the borgmatic output and mails it with msmtp (on Linux machines, I let cron handle this).

[rbn920]

I used the linux64 binary today and seems to work just fine.

[theChaosCoder]

I just wanted to report that you can now mount network shares under WSL with the new Windows 10 Fall Creators Update.
Just tested it and it's working great. No more cygwin needed.

$ sudo mount -t drvfs '\\server\share' /mnt/share

See also https://blogs.msdn.microsoft.com/wsl/2017/04/18/file-system-improvements-to-the-windows-subsystem-for-linux/

[theChaosCoder]

I need to correct myself:

it worked with my small repo. But I get often this error:

Local Exception
Traceback (most recent call last):
  File "borg/archiver.py", line 4024, in main
  File "borg/archiver.py", line 3952, in run
  File "borg/archiver.py", line 148, in wrapper
  File "borg/archiver.py", line 528, in do_create
  File "borg/archiver.py", line 494, in create_inner
  File "borg/archive.py", line 446, in save
  File "borg/archive.py", line 250, in flush
  File "borg/archive.py", line 266, in write_chunk
  File "borg/cache.py", line 872, in add_chunk
  File "borg/crypto/key.py", line 362, in encrypt
  File "borg/crypto/nonces.py", line 85, in ensure_reservation
  File "borg/crypto/nonces.py", line 48, in commit_repo_nonce_reservation
  File "borg/repository.py", line 309, in commit_nonce_reservation
  File "borg/platform/base.py", line 170, in __exit__
  File "borg/platform/base.py", line 138, in close
  File "borg/platform/base.py", line 58, in sync_dir
OSError: [Errno 5] Input/output error

Platform: Linux Chaos-PC 4.4.0-43-Microsoft #1-Microsoft Wed Dec 31 14:42:53 PST 2014 x86_64 x86_64
Linux: debian stretch/sid
Borg: 1.1.0  Python: CPython 3.5.4
PID: 47  CWD: /mnt/borgBackup

Even borg init triggers OSError: [Errno 5] Input/output error.

I remember having similar problems with borg 1.0.10 in cygwin, but 1.0.9 was ok. I will investigate further.

[ThomasWaldmann]

Often or always? Maybe syncing a dir is not supported on windows?

[theChaosCoder]

always
borg create this error randomly (large repo), and this error is produced always at the end. (tested also with small repos)

[theChaosCoder]

Is there an easy way to test sync?

[ThomasWaldmann]

import errno
import os
import sys

def sync_dir(path):
    fd = os.open(path, os.O_RDONLY)
    try:
        os.fsync(fd)
    except OSError as os_error:
        # Some network filesystems don't support this and fail with EINVAL.
        # Other error codes (e.g. EIO) shouldn't be silenced.
        if os_error.errno != errno.EINVAL:
            raise
    finally:
        os.close(fd)

if __name__ == '__main__':
    path = sys.argv[1]
    print("Syncing dir %s" % path)
    sync_dir(path)

Usage:

python3 syncdir.py somedirectory

[theChaosCoder]

Syncing dir /mnt/borgBackup/
Traceback (most recent call last):
  File "stest.py", line 20, in <module>
    sync_dir(path)
  File "stest.py", line 8, in sync_dir
    os.fsync(fd)
OSError: [Errno 5] Input/output error

/mnt/borgBackup/ fails <-- mounted via the new DrvFs (smb share)
/mnt/d/ is OK! /mnt/d/ == D:\

[ThomasWaldmann]

so, the "linux filesystem" syncs dirs ok (in WSL), but the mounted drvfs always fails with the dir sync?

[theChaosCoder]

exactly.

[theChaosCoder]

Also a mounted usb stick works too

sudo mount -t drvfs D: /mnt/Ddrive/ <-- sync OK
sudo mount -t drvfs F: /mnt/usbstick/ <-- sync OK
sudo mount -t drvfs '\\synology\Backups' /mnt/borgBackup/ <-- sync Fail (also fails if mounted in Win as a Network Drive z: )

//edit
Dir sync via cygwin works with the samba share.

[ThomasWaldmann]

\synology\Backups is what? smbfs via drvfs?

Guess that's a bug in one of these filesystems.
They should either support dir sync or say "invalid" (EINVAL), but not "I/O error" (EIO).

[theChaosCoder]

yep it is smbfs via drvfs
I also thinks it's a bug or just not implemented...

[ThomasWaldmann]

Can you report this to microsoft?
We can't suppress EIO, it could be a serious error.
Feel free to give them syncdir.py.

[enkore]

FlushFileBuffers does not support directory handles, therefore I'd guess that the kernel doesn't contain that functionality - this looks like a bad mapping of error codes (ERROR_INVALID_HANDLE => EIO instead of EINVAL or EBADF).

[bj0]

just tried with a freshly created repo, and get:

Local Exception
Traceback (most recent call last):
File "borg/archiver.py", line 4024, in main
File "borg/archiver.py", line 3952, in run
File "borg/archiver.py", line 148, in wrapper
File "borg/archiver.py", line 528, in do_create
File "borg/archiver.py", line 492, in create_inner
File "borg/archiver.py", line 593, in _process
File "borg/archiver.py", line 593, in _process
File "borg/archiver.py", line 593, in _process
File "borg/archiver.py", line 593, in _process
File "borg/archiver.py", line 593, in _process
File "borg/archiver.py", line 593, in _process
File "borg/archiver.py", line 593, in _process
File "borg/archiver.py", line 571, in _process
File "borg/archive.py", line 991, in process_file
File "borg/archive.py", line 919, in chunk_file
File "borg/archive.py", line 911, in chunk_processor
File "borg/cache.py", line 874, in add_chunk
File "borg/repository.py", line 1054, in put
File "borg/repository.py", line 1424, in write_put
File "src/borg/platform/linux.pyx", line 248, in borg.platform.linux.SyncFile.write
File "src/borg/platform/linux.pyx", line 222, in borg.platform.linux._sync_file_range
OSError: [Errno 38] Function not implemented

Platform: Linux 4.4.0-43-Microsoft #1-Microsoft Wed Dec 31 14:42:53 PST 2014 x86_64 x86_64
Linux: debian stretch/sid
Borg: 1.1.0 Python: CPython 3.5.4

[ThomasWaldmann]

@bj0 looks like they didn't implement this yet (see fcntl.h):

    int sync_file_range(int fd, int64_t offset, int64_t nbytes, unsigned int flags)

[henfri]

Hello,

I am trying this under ubuntu (in windows 10). when creating a backup, I get:
Local Exception.

Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/borg/archiver.py", line 2168, in main
    exit_code = archiver.run(args)
  File "/usr/lib/python3/dist-packages/borg/archiver.py", line 2104, in run
    return set_ec(func(args))
  File "/usr/lib/python3/dist-packages/borg/archiver.py", line 101, in wrapper
    kwargs['manifest'], kwargs['key'] = Manifest.load(repository, compatibility)
  File "/usr/lib/python3/dist-packages/borg/helpers.py", line 253, in load
    data = key.decrypt(None, cdata)
  File "/usr/lib/python3/dist-packages/borg/key.py", line 281, in decrypt
    data = self.compressor.decompress(self.dec_cipher.decrypt(data[41:]))
  File "borg/compress.pyx", line 210, in borg.compress.Compressor.decompress (borg/compress.c:4825)
ValueError: ('No decompressor for this data found: %r.', b'\x03\x00')

I have installed python3-lz4
Borg is installed via apt.

What is missing?

Greetings,
Hendrik

[henfri]

With this release it works:
https://github.com/borgbackup/borg/releases

[ThomasWaldmann]

0300 is zstd compressed data, you need at least borg 1.1.4 for that.

[henfri]

Thanks. In fact I was aware of the fact that zstd needs 1.1.4, but I was not using zstd for this job.
But in this archive on the Server I have jobs that did use zstd.

[ThomasWaldmann]

The problem is that the Manifest (contains, among other stuff, the list of all archives) is compressed with zstd. BTW, this is not specific to WSL, so it is misplaced in this ticket.

[henfri]

I understand that -now.
I was not aware yesterday though.

Greetings,
Hendrik

[henfri]

Hello again,

unfortunately, I get many 'connection closed by remote host' messages. I have not yet been able to complete a backup. The connection is usually lost after around one hour (but varies).
After that, the lock is still on the Server and I need to break it manually.

Are there any options to make borg do a retry?

Greetings,
Hendrik

[ThomasWaldmann]

@henfri no. but you rather should fix the root cause anyway. maybe ssh or router disconnecting, or other connection issue?

About the server lock: did you check whether the server process ("borg serve") has terminated?

[henfri]

Hello Thomas,

thanks for your reply.
The Server has been quite stable. I did a full extract test recently from my linux client. Of course I cannot exclude that this has changed since..

Regarding the server process:
I am not sure. I thought that the server process is started via ssh by the client?
Below you find a ps of when I started the backup and now (the backup has ended with connection closed again).

At start of backup:

  PID COMMAND                          STARTED     ELAPSED     TIME COMMAND
henfri@openmediavault:~$ ps -eo pid,comm,lstart,etime,time,args | grep borg
22362 sshd            Sat Mar 10 15:56:53 2018       22:04 00:00:00 sshd: borg [priv]
22365 sshd            Sat Mar 10 15:56:53 2018       22:04 00:00:35 sshd: borg@notty
22368 borg            Sat Mar 10 15:56:54 2018       22:03 00:00:01 borg serve --restrict-to-path /srv/_dev_disk_by-id_ata-ST5000DM000-1FK178_W4J1K7R1
22369 borg            Sat Mar 10 15:56:55 2018       22:02 00:00:19 borg serve --restrict-to-path /srv/_dev_disk_by-id_ata-ST5000DM000-1FK178_W4J1K7R1
11089 sshd            Sat Mar 10 13:17:47 2018    03:01:10 00:00:00 sshd: borg [priv]
11091 sshd            Sat Mar 10 13:17:47 2018    03:01:10 00:13:43 sshd: borg@notty
11092 borg            Sat Mar 10 13:17:47 2018    03:01:10 00:00:01 borg serve --restrict-to-path /srv/_dev_disk_by-id_ata-ST5000DM000-1FK178_W4J1K7R1
11093 borg            Sat Mar 10 13:17:49 2018    03:01:08 00:04:53 borg serve --restrict-to-path /srv/_dev_disk_by-id_ata-ST5000DM000-1FK178_W4J1K7R1

Now:

  PID COMMAND                          STARTED     ELAPSED     TIME COMMAND
henfri@openmediavault:~$ ps -eo pid,comm,lstart,etime,time,args | grep borg
22362 sshd            Sat Mar 10 15:56:53 2018    01:39:23 00:00:00 sshd: borg [priv]
22365 sshd            Sat Mar 10 15:56:53 2018    01:39:23 00:00:35 sshd: borg@notty
22368 borg            Sat Mar 10 15:56:54 2018    01:39:22 00:00:01 borg serve --restrict-to-path /srv/_dev_disk_by-id_ata-ST5000DM000-1FK178_W4J1K7R1
22369 borg            Sat Mar 10 15:56:55 2018    01:39:21 00:00:19 borg serve --restrict-to-path /srv/_dev_disk_by-id_ata-ST5000DM000-1FK178_W4J1K7R1
24109 sshd            Sat Mar 10 16:19:22 2018    01:16:54 00:00:00 sshd: borg [priv]
24111 sshd            Sat Mar 10 16:19:22 2018    01:16:54 00:07:43 sshd: borg@notty
24112 borg            Sat Mar 10 16:19:22 2018    01:16:54 00:00:01 borg serve --restrict-to-path /srv/_dev_disk_by-id_ata-ST5000DM000-1FK178_W4J1K7R1
24113 borg            Sat Mar 10 16:19:24 2018    01:16:52 00:02:41 borg serve --restrict-to-path /srv/_dev_disk_by-id_ata-ST5000DM000-1FK178_W4J1K7R1

The PID
22362 sshd
22365 sshd
22368 borg
22369 borg

have kept running, while
11089 sshd
11091 sshd
11092 borg
11093 borg

have stopped and are seemingly replaced by
24109 sshd
24111 sshd
24112 borg
24113 borg

[ThomasWaldmann]

Please try that again and:

• first terminate all borg serve processes
• then run precisely one backup (borg create)
• as you did, ps while the backup is running, after the client terminated and after the borg serve terminated

As long as borg serve is running (even after connection has broke down), it is expected that there might be a lock on the repo. No bug here.

But I guess after some timeout borg serve would notice that the connection has broken down and would terminate itself and also remove the repo lock. Only if it does not do that, we can consider that a bug (please file a separate bug report then with infos how to reproduce).

[henfri]

Hello Thomas,

thanks. Before I could do this, the backup did run through. I have changed nothing (TM). The only difference was, that I did run a ping with the parameter -i 0.2 to the server in parallel. I would not expect this to have an effect though.

Now, I have tried to do what you wrote above. Following findings:
1) before the backup, borg does not run on the server
2) during the backup I see these processes:

31617 sshd            Sun Mar 11 09:08:36 2018       00:10 00:00:00 sshd: borg [priv]
31622 sshd            Sun Mar 11 09:08:36 2018       00:10 00:00:00 sshd: borg@notty
31623 borg            Sun Mar 11 09:08:37 2018       00:09 00:00:01 borg serve --restrict-to-path /srv/Shortended
31626 borg            Sun Mar 11 09:08:38 2018       00:08 00:00:06 borg serve --restrict-to-path /srv/Shortened

3) The backup finishes successfully. Certainly the fact that there is little to backup helps here.... no prove that anything is fixed now.

Off topic: I have a question on this:

But I guess after some timeout borg serve would notice that the connection has broken down and would terminate itself and also remove the repo lock.

Borg is run via ssh. If the connection breaks down, wouldn't borg be stopped in a way that it cannot remove the lock anymore?

Totally OT:
Do you plan to get rid of the lock in future and work with some mechanism (e.g. DB) that allows parallel access to a repository? Of course, one can setup one repository for each client, but then one cannot profit of the deduplication across clients.

Greetings,
Hendrik

[ThomasWaldmann]

We try to use context managers everywhere, so even if an exception happens, stuff (like locks) gets cleaned up / teared down in a orderly manner.

Long term it would be nice to get rid of the locking, but don't hold your breath, there are no specific plans for the near future.

[TheJP]

I got it to work by installing from git (tag 1.1.5) and commenting out this line https://github.com/borgbackup/borg/blob/master/src/borg/platform/__init__.py#L24 (from .linux import SyncFile)

I had the same problem as @bj0. See the referenced issue for some more details.

[ThomasWaldmann]

@TheJP nice trick to get it working. it syncs a bit less then (not on write, only on close/sync), but if that makes it work...

[TheJP]

That's how I understood it yes. But I rather have it working a bit inefficient than not at all. It did run successfully on a couple hundred GB and some 100'000 files by now.

Great project btw :)

[ThomasWaldmann]

Not sure it is less efficient. It is just not as quickly making sure write-data is actually on the media.
As more is cached and sync is happening later, writes might be a bit more bursty.

[ThomasWaldmann]

@TheJP do you know how one can detect WSL (opposed to normal Linux)? Hmm, we could just try this call...

[TheJP]

I didn't know a way, but it looks like you could check for "Microsoft" in /proc/version (https://github.com/Microsoft/WSL/issues/423).
A bit hacky, but not problematic in this case I think:

• A false positive would lead to the usage of the old SyncFile functionality on an actual linux system. No problem and not likely that another Linux distribution would have "Microsoft" in its /proc/version.
• A false negative would produce the same result as the current Borg version. (An Exception in my case.)

[intdel]

@ThomasWaldmann You can detect it through the kernel name:
Linux Computername 4.4.0-17134-Microsoft #1-Microsoft Tue Apr 10 18:04:00 PST 2018 x86_64 x86_64 x86_64 GNU/Linux

EDIT: I just saw @TheJP just said the same thing.

[henfri]

@TheJP

I got it to work by installing from git (tag 1.1.5) and commenting out this line https://github.com/borgbackup/borg/blob/master/src/borg/platform/__init__.py#L24 (from .linux import SyncFile)

How did you install from git?
when running pip install -e .
I get this error:

    writing entry points to src/borgbackup.egg-info/entry_points.txt
    /usr/lib/python3.5/distutils/dist.py:261: UserWarning: Unknown distribution option: 'python_requires'
      warnings.warn(msg)
    warning: manifest_maker: standard file '-c' not found

    reading manifest template 'MANIFEST.in'
    writing manifest file 'src/borgbackup.egg-info/SOURCES.txt'
    running build_ext
    building 'borg.platform.linux' extension
    x86_64-linux-gnu-gcc -pthread -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -g -fstack-protector-strong -Wformat -Werror=format-security -Wdate-time -D_FORTIFY_SOURCE=2 -fPIC -I/usr/include/python3.5m -c src/borg/platform/linux.c -o build/temp.linux-x86_64-3.5/src/borg/platform/linux.o
    src/borg/platform/linux.c:585:21: fatal error: sys/acl.h: No such file or directory
    compilation terminated.
    error: command 'x86_64-linux-gnu-gcc' failed with exit status 1

    ----------------------------------------

I am sure I can fix this, but then I will get the next error, so I wonder, whether I am missing something bigger.

Greetings,
Hendrik

[henfri]

sudo -H apt-get install libacl1-dev libssl-dev

fixed it for me.

[laurensdijkstra]

I also get

  File "src/borg/platform/linux.pyx", line 255, in borg.platform.linux.SyncFile.write
  File "src/borg/platform/linux.pyx", line 229, in borg.platform.linux._sync_file_range
OSError: [Errno 38] Function not implemented

Reading the comments here I understand it is due to a function not being implemented on the WSL side. The workaround could be commenting out a line from the src. How does one go about achieving that goal?

And does disabling this feature mean that my VM image will be copied over in its entirety every time a small change is made to it?

[ThomasWaldmann]

Maybe some WSL user should report that to microsoft. That might be a pain now, but long term easier than having to do a source based install and patching the files.

Making it not sync file ranges regularly will likely lead to a different caching behaviour, so data might stay a bit longer in memory before they get written to disk. So while it might impact performance and data safety, there is no different behaviour other than that.

[laurensdijkstra]

It has already been reported to MS 2 years ago in https://github.com/Microsoft/WSL/issues/645. @TheJP referenced this issue before. Doesn't seem like a lot is happening to fix it.

[ks07]

So I've just been experimenting with borg in WSL and ran into the sync_file_range bug - but thankfully it appears this only affects repos hosted on Windows, and doesn't trigger if the repo is hosted remotely on Linux. It also seems to only trigger if the backup source is sufficiently large (presumably the sync is only triggered once enough data has been written?) . Am I right in thinking this? Or is this bug likely to be triggered even if the remote repo is Linux based?

[stephendt]

Just chiming in to say that I appear to be affected by this bug as well. Unfortunately I'm using Storage Spaces formatted as NTFS as my repo. I am on 1.1.5. Will check back to see if a fix arrives.

[ThomasWaldmann]

the problem with sync_file_range is worked around by #4317.

i'ld appreciate it if somebody who uses WSL could practically test it (i currently do not have a WSL installation, so it is completely untested by me for that platform).

basically this is automating what some of you did by commenting out the SyncFile import.

[ThomasWaldmann]

the WSL broken sync_file_range workaround is now in master and 1.1-maint (and in 1.1.9).

btw, the workaround is to use the same code as is also used on BSD and macOS (and as was used when people commented out the one import in the platform module).

[ThomasWaldmann]

would be good if some people add their experiences with borg 1.1.9 under WSL below.

[brummelte]

@LocutusOfBorg https://launchpad.net/~costamagnagianfranco/+archive/ubuntu/borgbackup is still at 1.1.8. I will test it when the new version is available.

[LocutusOfBorg]

done.

[brummelte]

Thanks, I've tested 4 backup repositories now. All of them work. Previously only the small ones sometimes worked with WSL.

Small before (cygwin):

borgbackup version 1.1.7
------------------------------------------------------------------------------
Archive name: Jan-Jan-2019-02-20_07-47-45
Archive fingerprint: 5a165138db025188fcf7b4b46394d4d6cecd1404fbcac6d6ba80e2a4d7411c32
Time (start): Wed, 2019-02-20 07:47:56
Time (end):   Wed, 2019-02-20 07:48:19
Duration: 22.88 seconds
Number of files: 9824
Utilization of max. archive size: 0%
------------------------------------------------------------------------------
                       Original size      Compressed size    Deduplicated size
This archive:                4.27 GB              3.90 GB            143.29 kB
All archives:                4.69 TB              4.26 TB              6.04 GB

                       Unique chunks         Total chunks
Chunk index:                   47251             12761242
------------------------------------------------------------------------------

Small after (WSL):

borgbackup version 1.1.9
Creating archive at …
------------------------------------------------------------------------------
Archive name: Jan-jan-2019-02-20_09-56-46
Archive fingerprint: fbb89aaeb6e8480504152bae92538c25ada9631beb5ffb9ab12191ac6178dfde
Time (start): Wed, 2019-02-20 09:56:46
Time (end):   Wed, 2019-02-20 09:57:05
Duration: 18.69 seconds
Number of files: 9899
Utilization of max. archive size: 0%
------------------------------------------------------------------------------
                       Original size      Compressed size    Deduplicated size
This archive:                4.27 GB              3.90 GB            193.30 kB
All archives:                4.70 TB              4.26 TB              6.04 GB

                       Unique chunks         Total chunks
Chunk index:                   47299             12784076
------------------------------------------------------------------------------

Big before (cygwin):

borgbackup version 1.1.7
------------------------------------------------------------------------------
Archive name: Jan-Jan-2019-02-04_08-55-23
Archive fingerprint: 962bcaeeb95d2cd612a06af6f7c3b8bd1fbdf685d9bf5981f623eadf77c868d6
Time (start): Mon, 2019-02-04 08:55:26
Time (end):   Mon, 2019-02-04 09:45:00
Duration: 49 minutes 33.90 seconds
Number of files: 1312181
Utilization of max. archive size: 1%
------------------------------------------------------------------------------
                       Original size      Compressed size    Deduplicated size
This archive:               70.28 GB             38.48 GB              1.83 GB
All archives:                9.83 TB              4.55 TB            109.95 GB

                       Unique chunks         Total chunks
Chunk index:                 1392093            151273982
------------------------------------------------------------------------------

Big after (WSL):

borgbackup version 1.1.9
Creating archive at …
------------------------------------------------------------------------------
Archive name: Jan-jan-2019-02-20_15-34-06
Archive fingerprint: 06ff40c8d51554618b7661543f005eb6e1750e772a7996d75319213f370b29b2
Time (start): Wed, 2019-02-20 15:34:09
Time (end):   Wed, 2019-02-20 16:23:09
Duration: 49 minutes 0.40 seconds
Number of files: 1019883
Utilization of max. archive size: 0%
------------------------------------------------------------------------------
                       Original size      Compressed size    Deduplicated size
This archive:               58.48 GB             28.39 GB            133.29 MB
All archives:                9.94 TB              4.61 TB            111.96 GB

                       Unique chunks         Total chunks
Chunk index:                 1426888            153218526
------------------------------------------------------------------------------

Speed seems to be about the same.

[jumper444]

Presumable the sync_file_range line/command was(is) in the program for a reason. I concur that it throws a WSL error since not implemented and that people are bypassing the error (now in experimental build) by commenting out the line. This seems rather casual without an explanation as to why that line is there in the first place and what effects may occur by removing it. Can anyone elaborate?

[ThomasWaldmann]

@jumper444 since borg 1.1.9 that code is not executed on WSL any more, no need to modify the borg code.

this does not affect data safety, the sync file range was just to get a less "bursty" behaviour (OS caching lots of write data and then writing it all at once to disk) and have the writes better spread over time.

[laurensdijkstra]

The binary borg-linux64, version 1.1.9, that I downloaded from https://github.com/borgbackup/borg/releases still has the same issue. Am I missing some environment variables that make borg recognize it's WSL?

[ThomasWaldmann]

@laurensdijkstra https://github.com/borgbackup/borg/blob/1.1.9/src/borg/platform/linux.pyx#L243

This is the code for that.

Can you manually check whether you see Microsoft in /proc/version (use the same user as when running borg).

[theChaosCoder]

Output in "ubuntu 18.04" WSL

cat /proc/version
Linux version 4.4.0-17763-Microsoft ([email protected]) (gcc version 5.4.0 (GCC) ) #379-Microsoft Wed Mar 06 19:16:00 PST 2019

[ThomasWaldmann]

OK, so the 1.1.9 code should work. Can somebody debug this on WSL? I don't have a WSL installation.

[laurensdijkstra]

I'm sorry, I copy pasted the command from earlier, when I was using the borg command. Replacing with borg-linux64 makes it work :)

[ThomasWaldmann]

@laurensdijkstra oh. so you are saying the 1.1.9 code for WSL works for you?

[theChaosCoder]

I just tested a borg(1.1.9) init + create on local hdd, no errors in WSL.

[laurensdijkstra]

Yes!