streisand error during AWS setup

[Dreamer1886]

Expected behavior:

Locating streisand.html and streisand-firewall-information

Actual Behavior:

getting error message during installation

PLAY [Provision the EC2 Server] ****************

TASK [Gathering Facts] *******************
ok: [localhost]

TASK [Set the AWS Region fact] ***************
ok: [localhost]

TASK [Set the AWS VPC ID fact] ***************
skipping: [localhost]

TASK [Set the AWS VPC Subnet ID fact] **************
skipping: [localhost]

TASK [ec2-security-group : Create the EC2 security group] ******
fatal: [localhost]: FAILED! => {"changed": false, "failed": true, "msg": "boto3 required for this module"}
to retry, use: --limit @/Users/DR/Documents/streisand/playbooks/amazon.retry

PLAY RECAP ***********************
localhost : ok=2 changed=0 unreachable=0 failed=1

Steps to Reproduce:

1.

[cpu]

@Dreamer1886 Please answer all of the questions from the new issue template:

Operating System of client:

Version of Ansible, using ansible --version :

Output from git rev-parse HEAD in your Streisand directory :

Have you installed boto3? What does running pip show boto3 reveal? Are you provisioning from Windows using WSL?

[Dreamer1886]

MacOS
ansible 2.4
output from git rev-parse HEAD: fc0cca533af5efdb69b73c900cb31398067c94d4

I've installed boto3, but pip show boto3says command not found

[cpu]

Hi again @Dreamer1886,

I've installed boto3, but pip show boto3says command not found

How did you install boto3? This is why Streisand is failing for you with the Amazon provisioning mode.

Is it possible you have both Python3 and Python2 installed? If so it may be that pip2 install boto3 is required.

[Dreamer1886]

I believe I have actually downloaded python3 at some point, but I thought I deleted it. Is there a way to check if it's deleted? The commands I've run show that I have python2.7

[Dreamer1886]

I used this sudo pip install boto

[cpu]

I used this sudo pip install boto

Ahh! I think there might be a mistake in our docs. Can you try running sudo pip install boto3 (note the 3).

[Dreamer1886]

I've run pip2 install boto3 since sudo pip install boto3 says command not found

I got this
Successfully installed boto3-1.4.7 botocore-1.7.32 docutils-0.14 futures-3.1.1 jmespath-0.9.3 python-dateutil-2.6.1 s3transfer-0.1.11 six-1.11.0

pip show boto3 still says command not found though

[cpu]

since sudo pip install boto3 says command not found

Does pip install boto3 work? (no sudo)

[Dreamer1886]

no

[cpu]

@Dreamer1886 Unfortunately I'm not sure what to suggest to you in this case. It seems your system's Python/pip are in a strange state. This isn't a problem with Streisand that we can fix with any work in this repository.

[nopdotcom]

Are you using Homebrew python? I think the /usr/bin/python OS X ships may have issues. I'm not sure anybody tests with the OS Python, but I suppose I could start.

Are you familiar with Python virtualenv? You can create an isolated Python environment that you can install stuff into; it won't affect the rest of your system. If you're interested in trying, you can set it up with:

pip install virtualenv
virtualenv ~/my-env

Each time you use the virtualenv, you need to enter into it with this command:

source ~/my-env/bin/activate

Your prompt should change to something like: (my-env)red-herring:nop-streisand $ to indicate you're using the my-env virtualenv.

From there, it's easier to make sure your system packages aren't stomping on things, and you shouldn't need root.

pip install ansible
pip install boto boto3 msrest msrestazure azure==2.0.0rc5 packaging dopy==0.3.5 "apache-libcloud>=1.5.0" linode-python pyrax 

This is pretty new, so it would be a help to me if you tried it.

[Dreamer1886]

@nopdotcom. Thanks! I'll try it out and let you know if it works

[Dreamer1886]

ok apparently my AWS server had not completed setup yet when I was trying, but now I'm dealing with another problem during installation

Streisand will now set up your server. This process usually takes around ten minutes. Press Enter to begin setup...
:

PLAY [Provision the EC2 Server] ****************

TASK [Gathering Facts] *******************
ok: [localhost]

TASK [Set the AWS Region fact] ***************
ok: [localhost]

TASK [Set the AWS VPC ID fact] ***************
skipping: [localhost]

TASK [Set the AWS VPC Subnet ID fact] **************
skipping: [localhost]

TASK [ec2-security-group : Create the EC2 security group] ******
changed: [localhost]

TASK [ec2-security-group : Pause for fifteen seconds to ensure the EC2 security group has been created] *
Pausing for 15 seconds
(ctrl+C then 'C' = continue early, ctrl+C then 'A' = abort)
ok: [localhost]

TASK [ec2-security-group : Open necessary ports across supported services in the EC2 security group] *
changed: [localhost]

TASK [ec2-security-group : Open the L2TP/IPsec ports in the EC2 security group] *
changed: [localhost]

TASK [ec2-security-group : Open the OpenConnect ports in the EC2 security group] *
changed: [localhost]

TASK [ec2-security-group : Open the OpenVPN ports in the EC2 security group] *
changed: [localhost]

TASK [ec2-security-group : Open the stunnel port in the EC2 security group] **
changed: [localhost]

TASK [ec2-security-group : Open the Shadowsocks ports in the EC2 security group] *
changed: [localhost]

TASK [ec2-security-group : Open the Tor ports in the EC2 security group] *
changed: [localhost]

TASK [ec2-security-group : Open the WireGuard ports in the EC2 security group] *
changed: [localhost]

TASK [genesis-amazon : set_fact] ***************
ok: [localhost]

TASK [genesis-amazon : Get the ~/.ssh/id_rsa.pub contents] *******
ok: [localhost]

TASK [genesis-amazon : Remove the 'streisand' SSH key from Amazon if it already exists. This is to prevent problems if two people with two different keys are sharing the same AWS account.] *
fatal: [localhost]: FAILED! => {"changed": false, "failed": true, "msg": "boto required for this module"}
to retry, use: --limit @/Users/alykhairy/Documents/streisand/playbooks/amazon.retry

PLAY RECAP ***********************
localhost : ok=14 changed=9 unreachable=0 failed=1

[nopdotcom]

That's strange. While you're inside the virtualenv, could you run this for me?

pip list | grep boto

[Dreamer1886]

pip install virtualenv is not found for some reason

[Dreamer1886]

@nopdotcom I'm sorry, but I'm not an advanced user and won't be able to experiment with this, I just need to know why I run into this error during installation and if there's a way to get around this.

Remove the 'streisand' SSH key from Amazon if it already exists. This is to prevent problems if two people with two different keys are sharing the same AWS account.]

[nopdotcom]

I'm not the best writer. If something I wrote confuses you, it's more than likely my fault.

I want people to be able to set up Streisand, and I want to automate as much of that is reasonable.

Which version of the OS are you running? I can pull up a VM and see what's up, and I can probably give you an exact transcript of what to write. (I've been meaning to write this anyway.)

[Dreamer1886]

@nopdotcom that's ok, really appreciate your help. I'm on macOS 10.12

[nopdotcom]

[still waiting for vm]

Anyway, it's still complaining at the lack of a boto module, so something bad is happening in your Python path, since you did install it--hopefully in the right place? A virtualenv eliminates those variables.

Other alternatives: I have a Docker build getting ready to be checked in. If you're a Docker person, this is easy.

Another: Amazon Lightsail is free for the first month, and is reasonably user-friendly. I have a slightly outdated but complete walkthrough for setting up a Streisand server from an Lightsail Ubuntu 16.04 instance.

If you have a Google Cloud account, you can use the free Google Cloud Shell to install Streisand dependencies and deploy. (You can deploy to any service, not just Google.) Each Cloud Shell session gets a fresh OS (but you keep $HOME). I am not officially supporting the cloud shell yet, but soon.

[nopdotcom]

I'll make a transcript of the whole business on 10.12 and post it as a gist.

[nopdotcom]

I brought up a bare 10.12 machine, installed Homebrew, and did this:

    3  brew install python
   11  brew unlink python && brew link python
   12  pip install virtualenv
   13  virtualenv ~/streisand-deps
   15  source ~/streisand-deps/bin/activate
   16  pip install ansible
   17  pip install boto boto3 msrest msrestazure azure==2.0.0rc5 packaging dopy==0.3.5 "apache-libcloud>=1.5.0" linode-python pyrax
   18  git clone https://github.com/StreisandEffect/streisand.git
   19  cd streisand
   20  ssh-keygen
   21  ./streisand
   22  history >myhistory

(I have deleted the lines I screwed up, but I retained the original line numbering so you can tell how much I screwed up.)

[Dreamer1886]

I get a command not found with line 13

[nopdotcom]

Something is strange in your Python setup. Was MacPorts installed at some point?

Could you run these commands for me?

type -f python python2 pip pip2 virtualenv
echo $PATH
python -V
python2 -V
pip -V

[Dreamer1886]

type -f python python2 pip pip2 virtualenv
python is /usr/bin/python
python2 is /usr/local/bin/python2
pip is hashed (/usr/local/bin/pip)
pip2 is /usr/local/bin/pip2
-bash: type: virtualenv: not found
XX-MacBook-Air:~ XX$ echo $PATH
/Users/XX/anaconda2/bin:/Users/XX/anaconda/bin:/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin:/opt/X11/bin
XX-MacBook-Air:~ XX$ python -V
Python 2.7.10
XX-MacBook-Air:~ XX$ python2 -V
Python 2.7.14
XX-MacBook-Air:~ XX$ pip -V
pip 9.0.1 from /usr/local/lib/python2.7/site-packages (python 2.7)
XXMacBook-Air:~ XX$

[nopdotcom]

Let me make a puzzled frownie face at your computer: 😦

ls -ln /usr/local/bin/python* /usr/local/bin/pip*
pip show virtualenv

If pip show virtualenv fails, try pip2 install virtualenv and pip2 show virtualenv.

[Dreamer1886]

pip show virtualenv returned this

Name: virtualenv
Version: 15.1.0
Summary: Virtual Python Environment builder
Home-page: https://virtualenv.pypa.io/
Author: Jannis Leidel, Carl Meyer and Brian Rosner
Author-email: [email protected]
License: MIT
Location: /Users/XX/Library/Python/2.7/lib/python/site-packages
Requires:
You have new mail in /var/mail/XX

[Dreamer1886]

lrwxr-xr-x 1 501 80 31 21 Oct 20:23 /usr/local/bin/pip -> ../Cellar/python/2.7.14/bin/pip
lrwxr-xr-x 1 501 80 32 21 Oct 20:23 /usr/local/bin/pip2 -> ../Cellar/python/2.7.14/bin/pip2
lrwxr-xr-x 1 501 80 34 21 Oct 20:23 /usr/local/bin/pip2.7 -> ../Cellar/python/2.7.14/bin/pip2.7
lrwxr-xr-x 1 501 80 35 21 Oct 20:23 /usr/local/bin/python2 -> ../Cellar/python/2.7.14/bin/python2
lrwxr-xr-x 1 501 80 42 21 Oct 20:23 /usr/local/bin/python2-config -> ../Cellar/python/2.7.14/bin/python2-config
lrwxr-xr-x 1 501 80 37 21 Oct 20:23 /usr/local/bin/python2.7 -> ../Cellar/python/2.7.14/bin/python2.7
lrwxr-xr-x 1 501 80 44 21 Oct 20:23 /usr/local/bin/python2.7-config -> ../Cellar/python/2.7.14/bin/python2.7-config
lrwxr-xr-x 1 501 80 36 21 Oct 20:23 /usr/local/bin/pythonw2 -> ../Cellar/python/2.7.14/bin/pythonw2
lrwxr-xr-x 1 501 80 38 21 Oct 20:23 /usr/local/bin/pythonw2.7 -> ../Cellar/python/2.7.14/bin/pythonw2.7
XX-MacBook-Air:~ XX$ pip show virtualenv
Name: virtualenv
Version: 15.1.0
Summary: Virtual Python Environment builder
Home-page: https://virtualenv.pypa.io/
Author: Jannis Leidel, Carl Meyer and Brian Rosner
Author-email: [email protected]
License: MIT
Location: /Users/XX/Library/Python/2.7/lib/python/site-packages
Requires:

[nopdotcom]

What's puzzling me is that the pip that comes with Homebrew will definitely install stuff all in /usr/local, including putting binaries in /usr/local/bin.

It's possible something from one of those Anaconda installs is messing things up, although I'm not sure how.

To list all the environment variables that might be affecting Python, try:

env | sort | grep ^PY

This is empty on my computer.

I'm running out of reasonable things to debug if this doesn't turn up much. We can always do the install from Docker, or from a cloud machine.

[Dreamer1886]

yea I'm thinking it might be anaconda too, which I can just delete.
env | sort | grep ^PY is not returning anything

[nopdotcom]

Although it's not merged, you might want to check out https://github.com/nopdotcom/streisand/blob/7d8cb345a31f3ab59c0267a0872e30d9d69558b1/util/venv-dependencies.sh

[nopdotcom]

That script been revised slightly to keep up with Azure dependencies.

[nopdotcom]

I'm going to close this for now. @Dreamer1886, reopen this issue or open another issue if you're still running into problems. (Finding out how things break is important to us!)