Borg: Create standalone Windows installer

[billyc]

On Windows, borg requires a working cygwin install with lots of dependencies. It is difficult to build all the pieces correctly (such as msgpack-python), and the end product feels pretty foreign to casual users who just want a working backup program. Worse, by the time you've got python, cython, gcc, and all the build tools set up correctly, it takes up more than 800MB of disk space. Many of the cygwin packages are only required to _build_ borg, not to run it.

I created a one-off standalone Windows installer that encapsulates all the pieces needed to run borg: a self-contained cygwin, python, and all dependencies.

UPDATED: unofficial installers are now here:

• https://github.com/billyc/borg-releases
• GPG key fingerprint: EC2B 7E69 BDA3 F260 8396 3E41 40ED 1F77 9784 BBF0

This is merely a STOP-GAP, until someone creates a more appropriate Windows build and installer.
(see issue #105 for some earlier discussion about this topic)

[ThomasWaldmann]

Hint: maybe Vagrant plus a modern.ie VM could be used to automate. See our Vagrantfile about how we run test and build platforms for other OSes. There are even some bounties related to Windows, Vagrant and automated testing.

[billyc]

I pushed the NSIS installer script to a new repo at https://github.com/billyc/borgwin - eventually i hope we can merge the installer into the main borg build infrastructure (of which I'm still clueless)

For this installer to actually become a real thing, I think the following steps need to be automated (just getting these ideas out of my head and onto paper) (discussion welcome of course!)

• Create a development cygwin install on a standard Windows platform (or I guess, one for 32bit and one for 64bit?)
• Add in all the build dependencies: cython, gcc, g++, msgpack-python, pkg-config, binutils, etc etc
• build and test borg from source/git
• Create a separate minimal cygwin "target" without all the build tools, to save space
• Copy the built packages - msgpack-python, borg, etc
• I stripped out a ton of superfluous files from the cygwin install to make it a lot smaller: I did that manually but easy to script that
• Zip up the target cygwin install and copy to the NSIS installer folder, and run NSIS.
• Voila! Profit.

[billyc]

@ThomasWaldmann I'll take a look at the Vagrant stuff. I've never used Vagrant before, it will be an uphill climb for me. =)

[billyc]

@ThomasWaldmann when you get a moment, could you please point me to docs describing how to run the test suite under Cygwin. I've gotten everything set up according to the development doc online, and can run tox, but two issues are snagging me:

• I don't have fakeroot. There is no fakeroot on Cygwin, right? So do I just run tox from my virtualenv directly or is there some other step I need to take?
• Running tox directly, the test suite hangs after about one minute, in the step which generates test datasets. I get a few lines of random characters on the console, and then it just freezes.

[billyc]

More info -- when running tox, it hangs at the following point:

Billy@PSRC3826 ~/borg
$ tox
GLOB sdist-make: /home/Billy/borg/setup.py
py32 create: /home/Billy/borg/.tox/py32
ERROR: InterpreterNotFound: python3.2
py33 create: /home/Billy/borg/.tox/py33
ERROR: InterpreterNotFound: python3.3
py34 create: /home/Billy/borg/.tox/py34
py34 installdeps: -rrequirements.d/development.txt, attic
py34 inst: /home/Billy/borg/.tox/dist/borgbackup-0.28.3.dev49+ngd1377e5.zip
py34 installed: Attic==0.16,borgbackup==0.28.3.dev49+ngd1377e5,cov-core==1.15.0,coverage==3.7.1,Cython==0.23.4,mock==1.3.0,msgpack-python==0.4.6,pbr==1.8.1,pluggy==0.3.1,py==1.4.30,pytest==2.8.3,pytest-benchmark==3.0.0,pytest-cov==1.8.1,six==1.10.0,tox==2.2.1,virtualenv==13.1.2,wheel==0.24.0
py34 runtests: PYTHONHASHSEED='1074826412'
py34 runtests: commands[0] | py.test --cov=borg --cov-config=../.coveragerc --benchmark-skip --pyargs borg.testsuite
========================================================== test session starts ===========================================================
platform cygwin -- Python 3.4.3, pytest-2.8.3, py-1.4.30, pluggy-0.3.1
benchmark: 3.0.0 (defaults: timer=time.perf_counter disable_gc=False min_rounds=5 min_time=5.00us max_time=1.00s calibration_precision=10 warmup=False warmup_iterations=100000)
rootdir: /home/Billy/borg, inifile: setup.cfg
plugins: benchmark-3.0.0, cov-1.8.1
collected 319 items

py34/lib/python3.4/site-packages/borg/testsuite/archive.py .......
py34/lib/python3.4/site-packages/borg/testsuite/archiver.py .sFFFF....F.........ss.....s..X..FFF....ssssssssssssssssssssssssssssssssssssss......FFFF.

[billyc]

FYI while this is under development, I've updated the "unofficial" windows installer to 0.29.0

Pretty sure you need to uninstall the previous version before this will install correctly. I'll fix that for the next one.

Installer link:
https://goo.gl/KDxA2y

[jungle-boogie]

Hi @billyc,

Perhaps you can have borg integrated with http://babun.github.io

[ghost]

I'd suspect that there's not much point of distributing a Windows installer unless it has an accompanying frontend.

It took me 5 minutes to get Borg working in Cygwin from the Borg documentation; if a user can't accomplish that, then they're going to generally struggle with Borg too unless they have a frontend to simplify the use of Borg for them.

[digidream]

I strongly disagree with @Orbixx . I tried the Cygwin installation and I ended up with cca 900 MB Cygwin folder with msgpack-python not built (thus emulated painfully slow in Python). Pruning Cygwin packages afterwards is possible, but ineffective slow trial and error process.

I tried to install Windows binary whl file for msgpack, but it failed. There is no msgpack-python for Cygwin. Building from source would probably require installing MSVS 2015 or good knowledge of mingw/cygwin.

I am happily using borg in Linux, but the amount of knowledge to prepare Windows build is above my capabilities. Maybe @billyc could help how to build msgpack or, please, update your binary distribution to 1.0.0. Thanks.

[ThomasWaldmann]

@digidream http://borgbackup.readthedocs.org/en/latest/installation.html#cygwin note that we changed gcc-core package to gcc-g++ package - maybe that was the reason why it didn't compile msgpack for you.

[billyc]

@digidream yes ensuring gcc-g++ is included in your cygwin install should allow msgpack to build correctly. At least, that was what broke it for me.

I will be updating the one-off windows installer -- so is that a "two-off" installer? =) -- but I'm swamped right now and haven't had a chance.

[digidream]

Thanks @billyc & @ThomasWaldmann ! The gcc-g++ was really the problem!

BTW, I started a small repo digidream/borg-cygwin with Windows batch file for unattended automatic CygWin build.

[billyc]

Note, I've updated my unofficial Windows installer to the 1.0.0 release. This isn't tested beyond "it works for me", and if you already have the 0.29 installed you will probably have to uninstall that first.

Download here--
https://drive.google.com/file/d/0B_DctUyCak1zOXpqWGUwOFp1Tnc/view?usp=sharing

But it's a start =)

[digidream]

My automated build script creates two - build and install - CygWin installations, to get rid of unnecessary build packages in the final one (basically an idea of @billyc). Currently I am at 126 MB (about a half of the above 1.0.0 package). Maybe time to merge it with NSIS installer?

[billyc]

Great, I'll take a look -- sounds much better than my hack!

[billyc]

@digidream I just sent you a pull request which integrates the NSIS installer step with your cygwin build.

[billyc]

Also updated link to smaller installer - https://drive.google.com/file/d/0B_DctUyCak1zbV9mUlNVMFhxY3c/view?usp=sharing

Installer is now 41 MB -- and expands to 126 MB. Much better, thanks @digidream !

[enkore]

What's the state of Borg under Windows? Do e.g. permissions / user ACLs work (stored / restored correctly)?

[ThomasWaldmann]

stuff reflected in a "posix way" by cygwin likely works, but windows acls aren't reflected that way iirc.

[infectormp]

May be we soon can run borg on windows without CygWin :)
The Ubuntu Userspace for Windows Developers - http://blog.dustinkirkland.com/2016/03/ubuntu-on-windows.html

[billyc]

Updated for release 1.0.2 --
https://drive.google.com/file/d/0B_DctUyCak1zMHdwMzMzcFFDd1E/view?usp=sharing

[ClipperS87]

@billyc could you please kindly update your installer to the "latest" version, too?

[RonnyPfannschmidt]

@billyc do you have a script for making it, it might make sense to put it into a appevoyer deploy task to upload them to the github downloads automatically for each tag/release

[billyc]

@ClipperS87 the installer is already updated for "latest" which is currently 1.0.2. Did I miss something?

@RonnyPfannschmidt The script for building and creating the installer is at https://github.com/billyc/borg-cygwin -- it's not perfect, but it's a good starting point if someone has time to integrate it with the existing Vagrant build infrastructure. I don't have any time to work on this for several weeks though :-(

[ClipperS87]

The release is not the „latest“:

Especially the DIFF

http://borgbackup.readthedocs.org/en/latest/usage.html#borg-diff

and list format type unique_chunks

http://borgbackup.readthedocs.org/en/latest/usage.html#borg-list

are the feature I am looking for….

Von: Billy Charlton [mailto:[email protected]]
Gesendet: Mittwoch, 27. April 2016 19:56
An: borgbackup/borg
Cc: Alexander; Mention
Betreff: Re: [borgbackup/borg] Create standalone Windows installer (#440)

@ClipperS87 https://github.com/ClipperS87 the installer is already updated for "latest" which is currently 1.0.2. Did I miss something?

@RonnyPfannschmidt https://github.com/RonnyPfannschmidt The script for building and creating the installer is at https://github.com/billyc/borg-cygwin -- it's not perfect, but it's a good starting point if someone has time to integrate it with the existing Vagrant build infrastructure. I don't have any time to work on this for several weeks though :-(

—
You are receiving this because you were mentioned.
Reply to this email directly or https://github.com/borgbackup/borg/issues/440#issuecomment-215170975 view it on GitHub https://github.com/notifications/beacon/AO4g13FY8RXbRGbLrHOzupGtuqII63lxks5p76MqgaJpZM4Gm9Dm.gif

[enkore]

The release is not the „latest“:
diff... unique_chunks...

You mean "unstable" or "master". Latest is 1.0.2. (This might seem like nit-picking, but with for a backup software we'll want to be clear which versions are supported for production use, and which aren't)

NOT RELEASED DEVELOPMENT VERSIONS HAVE UNKNOWN COMPATIBILITY PROPERTIES.

That being said, you might be interested in the Windows branch. @Anakonda has posted standalone ZIPs.

[calebpaine]

Anyone successfully used the binary from billyc?

if I give it anything other than a local relative path, it fails:

C:\Users\paine>borg init C:\borg\
/usr/bin/bash: -c: line 0: unexpected EOF while looking for matching `"'
/usr/bin/bash: -c: line 1: syntax error: unexpected end of file

C:\Users\paine>borg init C:\borg
Exception ignored in: <bound method RemoteRepository.__del__ of <RemoteRepository ssh://C/~/borg>>
Traceback (most recent call last):
  File "/usr/lib/python3.4/site-packages/borg/remote.py", line 181, in __del__
    assert False, "cleanup happened in Repository.__del__"
AssertionError: cleanup happened in Repository.__del__
Local Exception.
Traceback (most recent call last):
  File "/usr/lib/python3.4/site-packages/borg/archiver.py", line 1498, in main
    exit_code = archiver.run(args)
  File "/usr/lib/python3.4/site-packages/borg/archiver.py", line 1452, in run
    return args.func(args)
  File "/usr/lib/python3.4/site-packages/borg/archiver.py", line 68, in wrapper
    repository = RemoteRepository(location, create=create, lock_wait=self.lock_wait, lock=lock, args=args)
  File "/usr/lib/python3.4/site-packages/borg/remote.py", line 167, in __init__
    version = self.call('negotiate', RPC_PROTOCOL_VERSION)
  File "/usr/lib/python3.4/site-packages/borg/remote.py", line 230, in call
    for resp in self.call_many(cmd, [args], **kw):
  File "/usr/lib/python3.4/site-packages/borg/remote.py", line 276, in call_many
    raise Exception('FD exception occurred')
Exception: FD exception occurred

Platform: CYGWIN_NT-10.0 europa 2.5.0(0.297/5/3) 2016-04-11 09:58 x86_64
Borg: 1.0.2  Python: CPython 3.4.3
PID: 5488  CWD: /cygdrive/c/Users/paine
sys.argv: ['/bin/borg', 'init', 'C:borg']
SSH_ORIGINAL_COMMAND: None

(edited by enkore: code tags)

[calebpaine]

Sorry if I sounded a little pretentious with that last comment, was getting a little frustrated.

Relative paths seems to work as expected, but anything with an absolute path, or a remote repo fail with the message above.

Working local example:

> C:\Users\paine>borg init ./borgbackup
> Enter new passphrase:
> Enter same passphrase again:
> Do you want your passphrase to be displayed for verification? [yN]:
> 
> C:\Users\paine>borg create -v --stats ./borgbackup::testing ./Documents
> ------------------------------------------------------------------------------
> Archive name: testing
> Archive fingerprint: a8b49ac2d2a9e4962cfae1f12f8094b83b30b179cdf26f433fbc2216eafdbff6
> Time (start): Sat, 2016-05-28 14:45:25
> Time (end):   Sat, 2016-05-28 14:46:08
> Duration: 43.28 seconds
> Number of files: 1782
> ------------------------------------------------------------------------------
>                        Original size      Compressed size    Deduplicated size
> This archive:                2.39 GB              2.39 GB              2.39 GB
> All archives:                2.39 GB              2.39 GB              2.39 GB
> 
>                        Unique chunks         Total chunks
> Chunk index:                    2658                 2722
> ------------------------------------------------------------------------------
> 

This is on Windows 10 Pro using billyc's binary from the top comment, not sure what other info you might need.

[digidream]

You must use a cygwin style of paths for borg arguments:

borg init /cygdrive/c/borg

[calebpaine]

I feel like a noob, completely overlooked that and I use cygwin at work for some things. Still not working remote, but I can use it over a mapped share drive now using the driver letter. Combined with a simple batch file and scheduled tasks, and I'm pretty happy with how it works.

Hopefully in a month or two when the linux/ubuntu integration for Windows 10 gets released mainstream we can skip some of this silliness.

[nj]

@calebpaine As @digidream states, you need to run it inside /cygdrive/c.

However, if you're using borg.bat it should work as expected - like the action in Scheduled Tasks:

1. _Program/script:_ "C:\Program Files\Borg\borg.bat"
2. _Add arguments (optional):_ create ... ssh://your-host/path Users Windows (just the arguments you normally add, whereas Users and Windows are the folders in the root directory)
3. _Start in (optional):_ C:\ (or which drive your data is placed on)

@billyc FYI 1.0.3 was released on May 20th 😎

[billyc]

Updated to 1.0.3
link here:
https://drive.google.com/open?id=0B_DctUyCak1zQXI0anhFeXNnZ2s

[ckasdf]

I installed NSIS, downloaded the installer from https://github.com/billyc/borg-cygwin, ran install.bat, then "Borg Backup Setup v1.0.0.exe."

Opened the cygwin terminal, typed "borg" and "borgbackup" but both commands come back as "command not found."

I cd'd to the borg installation, tried running "./borg.bat" and was told there's no such file as "/bin/borg" so that definitely indicates I've not done something right. How do I get the executable into /bin?

When installing Borg, the system put it in c:\program files\borg; should it have been installed elsewhere, perhaps within the cygwin environment? Looking at the original install.bat, I see it has "pip install borgbackup" which would indicate I wouldn't need to run the borg setup exe, right? Yet I'm wondering if the pip tool(s) installed properly, since running pip myself tells me it's not a valid command either.

[enkore]

iirc his installer includes cygwin and you'd have to use the "borg shell" (or something) installed along with it

[ckasdf]

What do you mean by borg shell? I went ahead and ran the secondary "setup" exe which installed cygwin to c:\windows\system32.

When I go into /cygdrive/c/Windows/System32/Borg-installer/bin then run "./borg" it tells me:
-bash: ./bin/borg: /usr/bin/python3: bad interpreter: No such file or directory

If I go up one directory (out of bin) and run "./borg.bat" it actually works (shows the "--help" output), but this seems like things aren't installed completely properly. When installing these things, are there specific directories I should be shooting for?

Okay, I figured it out - part of my problem was I had a pre-existing cygwin install, which didn't have access to all the same stuff as the embedded cygwin @billyc provided.

Using the embedded cygwin has things working as expected now. Thanks for your help, @enkore!

One last thing; how easy/difficult would it be to get borgmatic running within this project now?
https://github.com/witten/borgmatic

[billyc]

@ckasdf I'm a bit confused what you're trying to do but I'll try and help.

If you just want to _use_ Borg, you don't need to build the installer or install NSIS. You can install the prebuilt binary and then you can run borg.bat from any Windows command prompt. The prebuilt installer also adds a cygwin-based "Borg Shell" which is a Unix-flavored command terminal. There should be an icon to run that on your desktop after install. You don't need to use that shell if the borg.bat script is working for you.

But -- It sounds like you're attempting to _build_ the installer. If that's what you want to do, then after the installer is built, run Borg Backup Installer 1.0.3.exe, and it will install Borg to wherever you want -- I think the default is C:\Program Files\Borg but it shouldn't matter where it goes. It definitely does NOT need to be installed in C:\Windows\System32.

(Note also I just pushed an update to the installer script which builds the latest release, borg 1.0.3 at http://github.com/billyc/borg-cygwin

Hope this helps. -billy

[billyc]

ah looks like you figured things out while i was typing my response. =)

[ckasdf]

@billyc - I sure did :) But I thank you for your efforts! Linux is my primary OS, but I've got friends/family who have been needing a backup solution, and the one I've used for them isn't perfect. I recently discovered borg for myself, and have been using it for a few weeks - love it! I'm also new to cygwin, so I'm new to how it's set up and such hah.

Not sure how the system decided to install things in sys32, but now that I know the process, I'll be able to install it in an appropriate directory on others' PCs.

So, two questions that remains:

• When I just ran a backup against my home directory (/cygdrive/c/Users/ckasdf) it's telling me "Device or resource busy" and referencing Google Chrome data, and other in-use files. I know Windows has Volume Shadow Copy to mitigate this issue; can cygwin + borg use this?
• What about using borgmatic via cygwin? :D

[enkore]

@1: You can use the VSS command line utility (vssadmin) to create a shadow copy and mount that and then you should be able to run borg against it. Correct support for VSS has been debated about for the Borg for Windows port in #983.

@2: May work with Python for Windows? Probably have to adjust the path to borg one way or another...

---

Linux is my primary OS, but I've got friends/family who have been needing a backup solution, and the one I've used for them isn't perfect. I recently discovered borg for myself, and have been using it for a few weeks - love it! I'm also new to cygwin, so I'm new to how it's set up and such hah.

Without attempting to make a promise or guarantee or whatever I _personally_ think that we _might_ see a Borg for Windows alpha with 1.1. The port is naturally a native Windows application and we aim to support NTFS features.

With cygwin you'll be missing most FS-level stuff, notably, ACLs and therefore access control. So with cygwin it's only useful to backup (non-shared) user data ATM. Which is of course already quite handy ;)

[ckasdf]

@1 - I'll check into that, thanks!
@2 - I'll research that and see what would be involved. Thanks for pointing me in a good starting direction.

Re: ACL - yeah, no worries, the two people I have in mind are single users of their machines, so permissions, etc aren't major important right now.

One more question to add, and I'm not sure if the answer is possible.

In borg, it's possible to mount a backup with:
borg mount repo::bkName mountPoint

I tried that, and as I suspected it would, it freaked out, throwing a bunch of python errors.
I think that command relies on fusermount - is it possible to get that working in a cygwin environment?

[billyc]

Hmm, this looks to me like a conflicting Python executable is in your system PATH. The borg installer includes its own copy of Python. It appends the borg install directory to the end of your system path, but if you already have python installed, it's probably grabbing the wrong one when you run borg.

These types of problems happen all the time on Windows when there are multiple copies of Python installed.

If you put the borg install folder at the beginning of your PATH, I bet Borg will start working... but then you'll probably start having trouble with other Python programs.

I'm not sure the best way to solve this until a real port of Borg for Windows is completed. There might be a way for my installer to insist it calls the correct Python, too -- I'll look into that when I get a chance.

From: Mark Penner [mailto:[email protected]]
Sent: Thursday, June 30, 2016 7:40 PM
To: borgbackup/borg
Cc: Billy Charlton; Mention
Subject: Re: [borgbackup/borg] Create standalone Windows installer (#440)

Thank you for your efforts @billychttps://github.com/billyc, but your Windows installer didn't quite work.

I get

$ "/C/Program Files/Borg/bin/borg"

bash: /C/Program Files/Borg/bin/borg: /usr/bin/python3: bad interpreter: Permission denied

Which I discovered is just Python being dumb, but...

$ python "/C/Program Files/Borg/bin/borg"

Traceback (most recent call last):

File "C:/Program Files/Borg/bin/borg", line 5, in

from pkg_resources import load_entry_point

File "C:\Program Files (x86)\Python 3.5\lib\site-packages\pkg_resources__init__.py", line 3084, in

@_call_aside

File "C:\Program Files (x86)\Python 3.5\lib\site-packages\pkg_resources__init__.py", line 3070, in _call_aside

f(*args, **kwargs)

File "C:\Program Files (x86)\Python 3.5\lib\site-packages\pkg_resources__init__.py", line 3097, in _initialize_master_working_set

working_set = WorkingSet._build_master()

File "C:\Program Files (x86)\Python 3.5\lib\site-packages\pkg_resources__init__.py", line 651, in _build_master

ws.require(__requires__)

File "C:\Program Files (x86)\Python 3.5\lib\site-packages\pkg_resources__init__.py", line 952, in require

needed = self.resolve(parse_requirements(requirements))

File "C:\Program Files (x86)\Python 3.5\lib\site-packages\pkg_resources__init__.py", line 839, in resolve

raise DistributionNotFound(req, requirers)

pkg_resources.DistributionNotFound: The 'borgbackup==1.0.3' distribution was not found and is required by the application

That one I don't know how to fix.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHubhttps://github.com/borgbackup/borg/issues/440#issuecomment-229840607, or mute the threadhttps://github.com/notifications/unsubscribe/ABXIRjxAYAZtq5rB6WwBthZkGdu2kiKcks5qRH38gaJpZM4Gm9Dm.

---

NOTICE OF PUBLIC DISCLOSURE: This e-mail account is public domain. Any correspondence from or to this e-mail account may be a public record. Accordingly, this e-mail, in whole or in part, may be subject to disclosure pursuant to RCW 42.56, regardless of any claim of confidentiality or privilege asserted by an external party.

[PlasmaPower]

Maybe a batch file to set the path before starting borg, carrying over arguments?

[mnpenner]

@billyc Sorry, I figured it out and then deleted my post :smile: It doesn't work under Git Bash but it seems to run fine with cmd.exe as long as I use /cygdrive/c/ instead of C:\ for all my file paths.

Another thing I noticed is that --progress keeps writing new lines but the demo video shows the progress option updating the same line. Not a big deal, but it isn't as pretty :stuck_out_tongue_winking_eye:

[billyc]

Updated unofficial installer to 1.0.6
Download here

Does anybody still use this? Can I stop doing these?

[kdar]

@billyc I used your borg-cygwin project as a basis for my ckbackup project, which I greatly appreciate. It seems that you provided the tools for someone to install any new version of borg on windows, so I'm not sure it's required to keep making installers for it (that's if you want to stop doing it).

[DanielSzoska]

@billyc Thanks for updating the installer. I used the old version (1.0.3) two weeks ago for installing borgbackup on my parents computer. I will update it to 1.0.6 with your installer asap.

[billyc]

@kdar oh it's easy to keep making them; and some people don't want to build it themselves. I just wanted to make sure that it was still useful.

Sounds like it is, for now! =)

[mnpenner]

@billyc I'm using the unofficial installer. I probably wouldn't use borg at all without it; too much of a pain in the butt to set up myself, so thank you for releasing it!

[memiks]

really thank you for this package ;)

[billyc]

Updated unofficial installer to 1.0.7
Download here

[ThomasWaldmann]

@billyc Do you use gpg? You could sign the binary and publish binary+signature (and also your gpg key fingerprint elsewhere) to avoid anybody tampers with it.

[mrhut10]

just a brief question for a newbie, would this version (downloading now) be portable and not interfere with my cygwin already installed?

thank you in advanced. looking forward to playing with (currently a tarsnap cygwin user and do love his work)

[enkore]

I created a one-off standalone Windows installer that encapsulates all the pieces needed to run borg: a self-contained cygwin, python, and all dependencies.

Shouldn't interfere at all with another cygwin install.

[Anachron]

Google drive is the one of the worst options possible for distributing binaries.

[ThomasWaldmann]

Hint: if everybody GPG-signed stuff, distribution channel would not matter (at least not for security).

[billyc]

Moved 1.0.7 installer to GitHub and signed it using GPG:

• borg-installer-1.0.7.exe
• borg-installer-1.0.7.exe.asc (signature)
• Key fingerprint: EC2B 7E69 BDA3 F260 8396 3E41 40ED 1F77 9784 BBF0

[skabanov]

Hello. Check please my comment and patch at https://github.com/borgbackup/borg/issues/1268 i think it will be useful to patch cygwin for your package.

[billyc]

@skabanov it's beyond my capabilities to patch cygwin.

[billyc]

Unofficial 1.0.8 installer created.

• borg-installer-1.0.8.exe
• borg-installer-1.0.8.exe.asc (signature)
• Key fingerprint: EC2B 7E69 BDA3 F260 8396 3E41 40ED 1F77 9784 BBF0

[llimz]

Many thanks for your package.

[jumper444]

BilliC,
Thanks for the package as others have already mentioned.
Would you be able to roll a 1.0.9, especially since there is a security fix vs 1.0.8?

[MrBrax]

Well i just can't get this to work. borg.bat or in cygwin, i get the same error (init a repo via ssh):

Local Exception.
Traceback (most recent call last):
  File "/usr/lib/python3.4/site-packages/borg/archiver.py", line 1928, in main
    exit_code = archiver.run(args)
  File "/usr/lib/python3.4/site-packages/borg/archiver.py", line 1873, in run
    return args.func(args)
  File "/usr/lib/python3.4/site-packages/borg/archiver.py", line 74, in wrapper
    lock_wait=self.lock_wait, lock=lock, append_only=append_only, args=args)
  File "/usr/lib/python3.4/site-packages/borg/remote.py", line 200, in __init__
    version = self.call('negotiate', RPC_PROTOCOL_VERSION)
  File "/usr/lib/python3.4/site-packages/borg/remote.py", line 288, in call
    for resp in self.call_many(cmd, [args], **kw):
  File "/usr/lib/python3.4/site-packages/borg/remote.py", line 340, in call_many
    raise Exception('FD exception occurred')
Exception: FD exception occurred

[enkore]

Can you give a few more details, eg. which operating system you are using?

[MrBrax]

@enkore wow that was a fast reply! Windows Server 2008 R2 for the source, and FreeNAS 9.10 on the backup side.

I can connect via ssh in the borg cygwin terminal, so the connection itself should work. I don't have to install anything on the FreeNAS side, do i? Or have i gotten this all wrong?

[enkore]

You'll need to have borg installed on the FreeNAS side (either through package manager or by static binary, although I'm not sure the FreeBSD binaries work on FreeNAS 9).

If that's indeed the cause of this error / traceback then that's quite curious, because on nix/BSD the error is reported in vastly different ways to Borg and handled correctly by Borg.

[MrBrax]

@enkore so i have to have it installed on both ends? Well that explains it then. Getting it installed will probably be a pain though, i don't even know what package manager it uses, nor do i use ssh on it.

[enkore]

Yes. Alternatively you can use eg. NFS or SMB to mount a directory locally and access it like a local repository. This is typically a bit slower, but also has less memory requirements on the server. I believe someone tested SMB already (although that was probably on Linux). NFS on Windows always had extremely poor performance for me. FreeNAS being a NAS should support SMB out of the box.

If FreeNAS supports FreeBSD ports, then it should be possible/not too hard: http://www.freshports.org/archivers/py-borgbackup/

[MrBrax]

@enkore So do i have to run all the commands on FreeNAS now instead of on Windows? Because then i can't access the files, i don't want to expose it to the world. Borg binary works on FreeNAS it seems, and i created a test repo.

I was just hoping borg would access the server via SSH and just copy everything to it.

[nj]

@enkore @MrBrax There's a discussion about it here (someone tries installing it) - so it should basically be possible:
https://forums.freenas.org/index.php?threads/borg-backup.49221/

Alternative, it can be installed through Docker.

@MrBrax It's recommended to have borg installed on the remote (hence your FreeNAS). Alternatively if you aren't able to, it's still possible through a mount - but would not be as fast and optimal, as with borg installed both places:
http://borgbackup.readthedocs.io/en/stable/quickstart.html#remote-repositories

[enkore]

When Borg is installed on the NAS you can access repositories on the NAS transparently, eg.

windows/Cygwin$ borg create user@my-nas:/data/some/repository::Some_Archive /my/files

You don't need to run commands manually on the NAS in this case.

---

If you don't have Borg on the NAS then you would create a SMB share or similar (eg. sshfs, NFS) on the NAS, which allows your computer to access the directory with the Borg repository on the NAS. Eg.

# SMB share is mounted on drive T:\
windows/Cygwin$ borg create /cygdrives/t/my/repo::Some_Archive /my/files
# (I think it's /cygdrives but you'd have to check it yourself, I don't use Cygwin)

[MrBrax]

But why would borg be needed on the server? Isn't it just used as storage? That would be my first assumption.

@nj sheesh that sounds like a pain, and i've barely heard about docker before.

@enkore that still gives me the same error. i have to call borg on freenas by using /usr/local/bin/borg though, is that it?

[enkore]

Yes, if borg on the server side is not in the $PATH by default, you'd need to pass the full path to the binary as the --remote-path argument (see help), eg.

windows/Cygwin$ borg create --remote-path /usr/local/bin/borg ...

---

But why would borg be needed on the server? Isn't it just used as storage? That would be my first assumption.

It is only used for storage. Borg uses a custom protocol over SSH because that allows it to use very little bandwidth in many instances, which can be a big advantage with narrow-band internet, but isn't as important on local networks like in your case.

[MrBrax]

@enkore wow it's never happy. still giving the same crappy error. Not even supplying a reason for it.

[enkore]

Then there might be some other issue. Like I said above - normally remote errors are handled by other code and result in clear error messages, not strange things like this message (which I have never observed).

Perhaps it's an entirely different Cygwin-specific problem... ? Did someone else using Borg on Cygwin see this error?

[MrBrax]

yeah i'm out of ideas. i'll keep using rclone and replace the backups every week until this is fixed then somehow..

error "originates" with this definition, so something's not working right with cygwin here:
self.x_fds = [self.stdin_fd, self.stdout_fd, self.stderr_fd]

[MrBrax]

Wow i actually got it to work! borg.bat AND remote path solved it.

Not being able to make bash scripts to automate backups easily is gonna be a pain though.

edit

And now it's working in cygwin? What the hell???

edit

Okay the error was that i had put it as ssh://address:/path, i removed the colon and now it works!!

[ThomasWaldmann]

@MrBrax from 1.0.10rc1 (not released yet) changelog:

borg serve: fix transmission data loss of pipe writes, #1268
This affects only the cygwin platform (not Linux, *BSD, OS X).

See #1268.

Thus, I'd strongly advise against using remote repos with cygwin unless you can run a fixed borg code.
Working with locally mounted filesystems for the repository is not affected by that.

[MrBrax]

@ThomasWaldmann Good to know, i won't decommission the old backup tool yet then.

[enkore]

Ah right. Using the current Cygwin nightly also fixes it (they fixed it as well + we got that workaround TW made), although I think that billyc's installer ships it's own Cygwin which would not have the cygwin bugfix.

[MrBrax]

It's also using 1.0.8, and 1.0.9 implemented some new security thing, so it whines about that too. This needs an update :)

[ThomasWaldmann]

@enkore you say "Using the current Cygwin nightly also fixes it" - did you practically test that?

If so, please comment on and close #2075.

[enkore]

From the ML linked in #2075:
https://cygwin.com/snapshots/

2017-01-19 21:18:12 UTC diff: https://cygwin.com/snapshots/x86/cygwin-diffs-20170114-20170119

Which seems to have the pipe write fix (or just the changelog entry?).

[ThomasWaldmann]

that doesn't answer my question.

[billyc]

Unofficial 1.0.9 installer created.

• borg-installer-1.0.9.exe
• borg-installer-1.0.9.exe.asc (signature)
• Key fingerprint: EC2B 7E69 BDA3 F260 8396 3E41 40ED 1F77 9784 BBF0

[billyc]

and just now catching up on the comment thread: is the most recent Cygwin broken? I didn't apply any patches, I merely rebuilt borg using the latest standard Cygwin installer.

[ThomasWaldmann]

@billyc AFAIK only the nightly build has the fix, not the normal release.

[MrBrax]

Data is getting transferred though, how would i know if it's corrupted like that?

[ThomasWaldmann]

@mrbrax I usually noticed the issues when running the RemoteArchiver tests - when 2 borg processes do client/server as it is done in practice when using remote (ssh:) repositories.

[enkore]

borg check

[ThomasWaldmann]

1.0.10rc1 is released (with cygwin remote fix).

[billyc]

Unofficial 1.0.9 installer created.

• borg-installer-1.0.9.exe
• borg-installer-1.0.9.exe.asc (signature)
• Key fingerprint: EC2B 7E69 BDA3 F260 8396 3E41 40ED 1F77 9784 BBF0

[billyc]

Unofficial 1.0.10 installer created.

• borg-installer-1.0.10.exe
• borg-installer-1.0.10.exe.asc (signature)
• Key fingerprint: EC2B 7E69 BDA3 F260 8396 3E41 40ED 1F77 9784 BBF0

[Anachron]

nice! what are the changes? @billyc

[billyc]

@Anachron I have no idea -- this is an automated build which runs when the maintainer announces a new point release. Check the release notes.

• https://github.com/borgbackup/borg/releases/tag/1.0.10

[theChaosCoder]

I just updated my cygwin build (also teststed the build from billyc) and since 1.0.10 I get an error msg with smb paths:

borg list //synology/Backups/My-PC/Borg

usage: borg list [-h] [--critical] [--error] [--warning] [--info] [--debug]
[--lock-wait N] [--show-rc] [--no-files-cache] [--umask M]
[--remote-path PATH] [--short] [--list-format LISTFORMAT]
[-P PREFIX]
[REPOSITORY_OR_ARCHIVE]

borg list: error: argument REPOSITORY_OR_ARCHIVE: Invalid location format: "//synology/Backups/My-PC/Borg"

No problems with older builds.

[enkore]

Can you please open a separate issue?

(In the mean time, a workaround could be to mount \\synology\Backups as a network drive and use that)

[amitron72]

@billyc , Hi, just how mature is the windows installation? thank you

[billyc]

It is not mature at all. It's an unofficial installer, barely tested. It does work for me, though.

---

From: amitron72 [e-mail]
Sent: Monday, March 6, 2017 4:53:01 AM
To: borgbackup/borg
Cc: Billy Charlton; Mention
Subject: Re: [borgbackup/borg] Create standalone Windows installer (#440)

@billychttps://github.com/billyc , Hi, just how mature is the windows installation? thank you

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHubhttps://github.com/borgbackup/borg/issues/440#issuecomment-284388022, or mute the threadhttps://github.com/notifications/unsubscribe-auth/ABXIRgr6YAXHbZwH660Aq6tmBRDR2O45ks5rjAGtgaJpZM4Gm9Dm.

NOTICE OF PUBLIC DISCLOSURE: This e-mail account is public domain. Any correspondence from or to this e-mail account may be a public record. Accordingly, this e-mail, in whole or in part, may be subject to disclosure pursuant to RCW 42.56, regardless of any claim of confidentiality or privilege asserted by an external party.

[billyc]

Unofficial 1.0.11 installer created.

• borg-installer-1.0.11.exe
• borg-installer-1.0.11.exe.asc (signature)
• Key fingerprint: EC2B 7E69 BDA3 F260 8396 3E41 40ED 1F77 9784 BBF0

[JonasOlson]

Having it as a package for the package manager Chocolatey would be great too.

[ojaksch]

Having it as a package for the package manager Chocolatey would be great too.

Yes, that would be nice. Please let us know when do you start creating a .nupkg ;)

[ojaksch]

While you're cooking it I prepared a working template for Chocolatey. Needs some beautifying but seems to work fine.

borgbackup.1.0.11.nupkg.zip

[ojaksch]

...online: https://chocolatey.org/packages/borgbackup/
Happy backup'in!

[JonasOlson]

@ojaksch: I was just going to say that I have a lot to learn before I could make such a package myself, and now you have something going already… actually, something finished now, in the middle of me typing this comment!

That's great!

[ojaksch]

@JonasOlson You're welcome. If you're interested in creating your own chocolatey packages, I would be glad if you've learned your lessions there some day and adopt this package then.

[monoflash]

In the end, official support for windows is planned?

[jumper444]

The ongoing likely solution appears to be either:
1) a (automated?) rolled-up cgywin package which is what was going on above here, but since it isn't automated the person had to keep doing it and that gets old and I guess they stopped (no blame intended; it's a job)

2) Run Windows 10 and use the (new) Windows Subsystem for Linux:
https://docs.microsoft.com/en-us/windows/wsl/about
which should (i presume) let anyone just use/install following the regular Linux based instructions already written for Borg. I think this is the way to go going forward since it doesn't really require a special "windows" trick (other than Microsoft making Linux run which means THEY are responsible for it, not somebody else random out in the world, like a person who has to keep a hacked cygwin install running).

[jumper444]

If somebody tried it with WSL and got it working it would be valuable if those instructions (to the extent they might differ from the 'regular' linux instructions) could be added to the docs.

[jumper444]

The FAQ for Windows Subsystem for Linux explains the reason for them creating it and it exactly addresses issues like this. The extra work and issues to shoehorn Borg over to windows leaves that build or track orphaned (or understaffed; or always behind, etc). So from FAQ:

=====
Why would I use, for example, Ruby on Linux instead of on Windows?

Some cross-platform tools were built assuming that the environment in which they run behaves like Linux. For example, some tools assume that they are able to access very long file paths or that specific files/folders exist. This often causes problems on Windows which often behaves differently from Linux.

Many languages like Ruby and node are often ported to, and run great, on Windows. However, not all of the Ruby Gem or node/NPM library owners port their libraries to support Windows, and many have Linux-specific dependencies. This can often result in systems built using such tools and libraries suffering from build and sometimes runtime errors or unwanted behaviors on Windows.

These are just some of issues that caused many people to ask Microsoft to improve Windows’ command-line tools and what drove us to partner with Canonical to enable native Bash and Linux command-line tools to run on Windows.