Streisand: How to setup from Bash on Windows (Ubuntu) (Windows Subsystem for Linux, WSL)

[artkpv]

Expected behavior:

From WSL setup is done on Azure.

Actual Behavior:

... 
 [WARNING]: Found both group and host with same name: localhost

[DEPRECATION WARNING]: 'include' for playbook includes. You should use 'import_playbook' instead. This feature will be removed in version 2.8. Deprecation warnings can
 be disabled by setting deprecation_warnings=False in ansible.cfg.
[DEPRECATION WARNING]: The use of 'include' for tasks has been deprecated. Use 'import_tasks' for static inclusions or 'include_tasks' for dynamic inclusions. This
feature will be removed in a future release. Deprecation warnings can be disabled by setting deprecation_warnings=False in ansible.cfg.
[DEPRECATION WARNING]: include is kept for backwards compatibility but usage is discouraged. The module documentation details page may explain more about this
rationale.. This feature will be removed in a future release. Deprecation warnings can be disabled by setting deprecation_warnings=False in ansible.cfg.
... 
# region chosen

... 

TASK [azure-security-group : Create Azure resource group] **************************************************************************************************************
fatal: [localhost]: FAILED! => {"changed": false, "failed": true, "msg": "Do you have azure>=2.0.0 installed? Try `pip install 'azure>=2.0.0' --upgrade`- No module named azure.mgmt.network.models"}
        to retry, use: --limit @/home/wild/streisand/playbooks/azure.retry

PLAY RECAP *************************************************************************************************************************************************************
localhost                  : ok=3    changed=0    unreachable=0    failed=1

Steps to Reproduce:

1. As per https://github.com/StreisandEffect/streisand/blob/master/documentation/AZURE.md
2. Chosing only LP2T and OpenVPN
3. other by default

Additional Details:

Log output from Ansible or other relevant services (link to Gist for longer output):

https://gist.github.com/w1ld/9272ea17ef5c1fbdf4155f10fb8df41f

Target Cloud Provider:

Azure

Operating System of target host:

Linux?

Operating System of client:

Windows 10. Bash on Windows.

Version of Ansible, using ansible --version :

ansible 2.4.0.0

Output from git rev-parse HEAD in your Streisand directory :

66f27ec3befe21ba9e24ab492e536317cfd54e58

UP.
Azure CLI is up to date.

[cpu]

@w1ld Running Streisand from Windows isn't something we officially support. I recommend that if you aren't comfortable working through encountered problems on your own that you try running Streisand from Linux (in a VM or on another cloud instance).

Did you verify that you have the azure pip module installed per the error output?

Do you have azure>=2.0.0 installed?

What does pip show azure return to you?

[artkpv]

@cpu thank you for your attention! Yeah, probably to be done on Virtual Machine though WSL looks possble.

pip show azure
Name: azure
Version: 2.0.0
Summary: Microsoft Azure Client Libraries for Python
Home-page: https://github.com/Azure/azure-sdk-for-python
Author: Microsoft Corporation
Author-email: [email protected]
License: MIT License
Location: /usr/local/lib/python2.7/dist-packages
Requires: azure-servicemanagement-legacy, azure-servicebus, azure-graphrbac, azure-storage, azure-mgmt, azure-datalake-store, azure-servicefabric, azure-keyvault, azure-batch

[cpu]

@w1ld No problem. I think @nopdotcom has been experimenting with WSL and Docker to try and improve Windows support but its very preliminary work and my expectation is that Streisand will fail in interesting ways until more kinks are worked out.

Interesting that the Azure pip module is installed but yet it still complains about a missing submodule. Can you try: python -c "import azure.mgmt.network.models"?

I've never used the Azure provisioning and also have zero experience with WSL - I'll likely have to drop off of helping debug this but perhaps someone else will have ideas.

[artkpv]

@cpu
It couldn't find the module:

python -c "import azure.mgmt.network.models"
Traceback (most recent call last):
  File "<string>", line 1, in <module>
ImportError: No module named azure.mgmt.network.models

$ python --version
Python 2.7.12
$ pip --version
pip 9.0.1 from /usr/local/lib/python2.7/dist-packages (python 2.7)

[alimakki]

@w1ld, I found these docs (Install with apt-get for Bash on Ubuntu on Windows) on Microsoft's support site, perhaps that can help resolve your missing module library?

[nopdotcom]

Here's a script for Ubuntu that should pull all the dependencies. It needs to be run as root, or you can run the commands one-by-one with sudo.

Side note: This script was intended for disposable VMs, so it throws all kinds of Python stuff into /usr/local; there ought to be a way of doing this without local root.

I did deploy to EC2 successfully yesterday from Bash on Ubuntu on Windows, but I have not tested the above script on a clean install yet.

[tiliarou]

Which windows built do you have ? Linux dependencies were supposedly better with newest build (after Creator 1703).

[artkpv]

Thanks all for help! I'm actually lost with what to install when.

Solved python -c "import azure.mgmt.network.models". But i stuck with this now:

TASK [azure-security-group : Create Azure resource group] ***********************************************************
fatal: [localhost]: FAILED! => {"changed": false, "failed": true, "msg": "Do you have azure>=2.0.0 installed? Try `pip install 'azure>=2.0.0' --upgrade`- No module named containerservice"}

# ... 

$ python -c "import  containerservice"
Traceback (most recent call last):
  File "<string>", line 1, in <module>
ImportError: No module named containerservice

UPDATE
But then I run sudo -H pip install 'azure>=2.0.0' --upgrade (w/o -H it shows the warning) (and it updates something), it fails with the previous one " No module named azure.mgmt.network.models":

$ python -c "import azure.mgmt.network.models"
Traceback (most recent call last):
  File "<string>", line 1, in <module>
ImportError: No module named azure.mgmt.network.models

How to install dependencies?

@tiliarou 1703 version installed (10.0.15063 Build 15063). (Waiting for Fall Creators Update on 17th of Oct.)

@nopdotcom
Run all commands there with sudo -H. Showed this warn if w/o -H:

The directory '/home/wild/.cache/pip/http' or its parent directory is not owned by the current user and the cache has been disabled. Please check the permissions and owner of that directory. If executing pip with sudo, you may want sudo's -H flag.

[tiliarou]

Win10 1709 was released yesterday, it contains changes to WSL like:

"We're excited about all the WSL team has accomplished leading up to the Fall Creators Update. Here's the list of updates we'll discuss in this post:
WSL is no longer a Beta feature
Install Linux distros via the Windows Store
WSL now runs multiple Linux distros
WSL no longer requires developer mode
WSL comes to Windows Server & Microsoft Azure VM's!
WSL now supports USB/serial comms
Miscellaneous fixes and improvements
Deprecating the name "Bash on Windows""

Sources:
https://blogs.msdn.microsoft.com/commandline/2017/07/28/windows-subsystem-for-linux-out-of-beta/
https://github.com/microsoft/bashonwindows

[nopdotcom]

@artkpv @tiliarou: I'm going to move one machine to 1709. But the previous version was working for me...except I've never tested Azure. Which might be the problem. Could you do a deploy to another cloud provider to see if Azure is the difference?

The sudo -H stuff I'd like to get rid of in favor of a Python virtualenv for all the non-binary dependencies.

[artkpv]

Does it work for Azure at all? I've tried on fresh installation at Ubuntu (VMWare) following steps at https://github.com/StreisandEffect/streisand/ and it failed with the same

fatal: [localhost]: FAILED! => {"changed": false, "failed": true, "msg": "Do you have azure>=2.0.0 installed? Try `pip install 'azure>=2.0.0' --upgrade`- No module named azure.mgmt.network.models"}

[nopdotcom]

It hasn't been tested on Azure for a while, probably because the core contributors don't really have Azure accounts. I have one somewhere, but the Azure cost structure is not the greatest for Streisand nodes. I'll see if I can get to this in the next day or two.

I tried handling the dependencies with a Python virtualenv; that may help you on WSL. See https://github.com/StreisandEffect/streisand/issues/1007#issuecomment-338214964 and for a lightly-tested script, https://github.com/nopdotcom/streisand/blob/builder-scripts/util/venv-dependencies.sh

[nopdotcom]

If you have time, please try a freshly revised venv-dependencies.sh which I pushed just now.

[artkpv]

@nopdotcom Run it in WSL and soursed the env. but it failed with:

fatal: [localhost]: FAILED! => {"changed": false, "failed": true, "msg": "Do you have azure>=2.0.0 installed? Try `pip install 'azure>=2.0.0' --upgrade`- No module named dns.version"}

And under Ubuntu on VMWare it fails with No module named azure.mgmt.network.models 😞

Update.
I have the subscription disabled. So I guess hence the issue. Sorry about that. Wasn't clear at first

[artkpv]

No, still the issue. Tried with another active subscription. Ubuntu on VMWare fails with "No module named dns.version" as above.

[alimakki]

@artkpv, I noticed that the Azure dependencies have changed for Ansible 2.4, could you try the following pip install ansible[azure] and see if that resolves your issue?

[artkpv]

@alimakki No, doesn't work. Under venv-dependencies.sh it fails with "no .. dns.version" and without it the "no .. azure.mgmt.network.models" error.

[alimakki]

I've played around this a bit more, and managed to get it working somehow, without virtualenv

pip install ansible --user # note the --user argument
pip install boto boto3 ansible[azure] packaging dopy==0.3.5 "apache-libcloud>=1.5.0" linode-python pyrax # no user tag here
export PATH=$PATH:~/.local/bin

then ./streisand.sh

[artkpv]

@alimakki Tried, fails with "azure.mgmt.network.models" 😞 in WSL. Something incorrect at my side, probably. Checking it.

[alimakki]

Perhaps the dependencies need the —user flag as well

pip install boto boto3 ansible[azure] packaging dopy==0.3.5 "apache-libcloud>=1.5.0" ...—user

[nopdotcom]

I've updated https://github.com/nopdotcom/streisand/blob/builder-scripts/util/venv-dependencies.sh to install the up-to-date Azure dependencies in the virtualenv. If it doesn't work, I could really use a log; this may bite other people if I don't figure it out.

[artkpv]

Ok, now it works. I've installed the Creators Fall Update (Build 16299). Then i've reinstalled WSL (lxrun.exe /uninstall and lxrun.exe /install). And then did as per the guide at the front page (installing python-pip, ansible, etc.). A configuration at Azure was created (now i'm figuring out how to connect, it fails with l2tp actually). I don't know what was the reason of this ticket issues. Probably anyone who reproduce this should update to 16299 build also (still unclear it didn't work on newly installed Ubuntu at VMWare). Thanks all for help!