St2: Proxy settings don’t work for new pack install method

[sidkrishna]

Pls refer to issue: https://github.com/StackStorm/st2docs/issues/352

"st2 pack install openstack" (openstack pack taken as example), does not work event after giving "git config --global http.proxy http://user:[email protected]:port" as suggested in the documentation. If "git clone" for the repo is done manually on the command prompt it works, but fails via "packs.download" action.

"st2 search" works with modifications suggested in Modify: /opt/stackstorm/st2/lib/python2.7/site-packages/st2common/services/packs.py (st2docs/issues/352)

brocade@BWC2:~$ st2 pack search openstack
+-----------+--------------------------------------------+---------+------------------+
| name      | description                                | version | author           |
+-----------+--------------------------------------------+---------+------------------+
| libcloud  | st2 content pack containing libcloud       | 0.1.0   | StackStorm, Inc. |
|           | integrations                               |         |                  |
| openstack | OpenStack integration pack                 | 0.1.0   | st2dev           |
+-----------+--------------------------------------------+---------+------------------+

But pack install for "openstack" pack fails with error " unable to access 'https://github.com/StackStorm-Exchange/stackstorm-openstack/': Could not resolve host: github.com".

brocade@BWC2:~$ st2 pack install openstack

        [  failed   ] download pack

id: 585a450ae79fa05ed986edc9
action.ref: packs.install
parameters:
  packs:
  - openstack
status: failed
error: Traceback (most recent call last):
  File "/opt/stackstorm/st2/local/lib/python2.7/site-packages/st2common/runners/python_action_wrapper.py", line 242, in <module>
    obj.run()
  File "/opt/stackstorm/st2/local/lib/python2.7/site-packages/st2common/runners/python_action_wrapper.py", line 154, in run
    output = action.run(**self._parameters)
  File "/opt/stackstorm/packs/packs/actions/pack_mgmt/download.py", line 75, in run
    verifyssl=verifyssl, ref=pack_version)
  File "/opt/stackstorm/packs/packs/actions/pack_mgmt/download.py", line 102, in _clone_repo
    repo = Repo.clone_from(repo_url, temp_dir)
  File "/opt/stackstorm/st2/lib/python2.7/site-packages/git/repo/base.py", line 925, in clone_from
    return cls._clone(git, url, to_path, GitCmdObjectDB, progress, **kwargs)
  File "/opt/stackstorm/st2/lib/python2.7/site-packages/git/repo/base.py", line 880, in _clone
    finalize_process(proc, stderr=stderr)
  File "/opt/stackstorm/st2/lib/python2.7/site-packages/git/util.py", line 336, in finalize_process
    proc.wait(**kwargs)
  File "/opt/stackstorm/st2/lib/python2.7/site-packages/git/cmd.py", line 290, in wait
    raise GitCommandError(self.args, status, errstr)
git.exc.GitCommandError: Cmd('git') failed due to: exit code(128)
  cmdline: git clone -v https://github.com/StackStorm-Exchange/stackstorm-openstack /root/bd373ecc09ed3463a0b7849cc13d7dc4
  stderr: 'Cloning into '/root/bd373ecc09ed3463a0b7849cc13d7dc4'...
fatal: unable to access 'https://github.com/StackStorm-Exchange/stackstorm-openstack/': Could not resolve host: github.com

After adding proxy details in the git config, able to to manually go a "git clone":

brocade@BWC2:~$ git config --global http.proxy http://10.254.2.5:3128
brocade@BWC2:~$ git clone https://github.com/StackStorm-Exchange/stackstorm-openstack
Cloning into 'stackstorm-openstack'...
remote: Counting objects: 823, done.
remote: Total 823 (delta 0), reused 0 (delta 0), pack-reused 823
Receiving objects: 100% (823/823), 136.74 KiB | 0 bytes/s, done.
Resolving deltas: 100% (649/649), done.
Checking connectivity... done.

But "st2 pack install.." still fails:

brocade@BWC2:~$ st2 pack install openstack

        [  failed   ] download pack

id: 585a46b4e79fa05ed986edcc
action.ref: packs.install
parameters:
  packs:
  - openstack
status: failed
error: Traceback (most recent call last):
  File "/opt/stackstorm/st2/local/lib/python2.7/site-packages/st2common/runners/python_action_wrapper.py", line 242, in <module>
    obj.run()
  File "/opt/stackstorm/st2/local/lib/python2.7/site-packages/st2common/runners/python_action_wrapper.py", line 154, in run
    output = action.run(**self._parameters)
  File "/opt/stackstorm/packs/packs/actions/pack_mgmt/download.py", line 75, in run
    verifyssl=verifyssl, ref=pack_version)
  File "/opt/stackstorm/packs/packs/actions/pack_mgmt/download.py", line 102, in _clone_repo
    repo = Repo.clone_from(repo_url, temp_dir)
  File "/opt/stackstorm/st2/lib/python2.7/site-packages/git/repo/base.py", line 925, in clone_from
    return cls._clone(git, url, to_path, GitCmdObjectDB, progress, **kwargs)
  File "/opt/stackstorm/st2/lib/python2.7/site-packages/git/repo/base.py", line 880, in _clone
    finalize_process(proc, stderr=stderr)
  File "/opt/stackstorm/st2/lib/python2.7/site-packages/git/util.py", line 336, in finalize_process
    proc.wait(**kwargs)
  File "/opt/stackstorm/st2/lib/python2.7/site-packages/git/cmd.py", line 290, in wait
    raise GitCommandError(self.args, status, errstr)
git.exc.GitCommandError: Cmd('git') failed due to: exit code(128)
  cmdline: git clone -v https://github.com/StackStorm-Exchange/stackstorm-openstack /root/bd373ecc09ed3463a0b7849cc13d7dc4
  stderr: 'Cloning into '/root/bd373ecc09ed3463a0b7849cc13d7dc4'...
fatal: unable to access 'https://github.com/StackStorm-Exchange/stackstorm-openstack/': Could not resolve host: github.com
'

traceback: None
failed_on: download pack
start_timestamp: 2016-12-21T09:09:08.163215Z
end_timestamp: 2016-12-21T09:09:35.743867Z
+--------------------------+----------------------+---------------+----------------+-------------------------------+
| id                       | status               | task          | action         | start_timestamp               |
+--------------------------+----------------------+---------------+----------------+-------------------------------+
| 585a46b4e79fa05de3e3ccea | failed (27s elapsed) | download pack | packs.download | Wed, 21 Dec 2016 09:09:08 UTC |
+--------------------------+----------------------+---------------+----------------+-------------------------------+

[LindsayHill]

For those who come looking for this, here's a workaround until it's fixed:

Assuming git proxy has been configured:

cd $HOME
git clone https://github.com/StackStorm-Exchange/stackstorm-servicenow.git
st2 pack install file:///$PWD/stackstorm-servicenow

[ClarksonCJ]

I had an additional issue after I had used the workaround suggested by @LindsayHill. Where pip cannot download the dependancies for a virtual environment. I received this error "Exception: Failed to install requirement "six>=1.9.0,<2.0": Collecting six<2.0,>=1.9.0", and also noticed there was no log in my CNTLM trace. I had to modify the st2common/util/virtualenvs.py file and add the proxy environment variables to the get_env_for_subprocess_command() method e.g.
env['HTTP_PROXY'] = "<proxy_url>" env['HTTPS_PROXY'] = "<proxy_url>"

[martypare]

Thank you ClarksonCJ that fixed it for me as I was also stuck installing dependencies for the openstack pack. Works like a charm now.

[ClarksonCJ]

@martypare glad I could help, I am working on a PR to issue to @armab et al. with a more generic fix so hopefully in the near future there will be no need to hack the virtualenv file like this ;-)

[Mierdin]

@sidkrishna: Did you try exporting HTTP_PROXY and HTTPS_PROXY env vars (as well as no_proxy)? I did some testing and exporting these environment variables seems to make both work.
EDIT: I was wrong; I was running this in development, which runs as a different user than a "real" install

It's true, st2common/services/packs.py will allow st2 pack search to work, but it won't fix st2 pack install. Conversely, adding a proxy to global .gitconfig will allow pack downloads to work, but not pack search. This is because pack search uses the requests library to look at our pack index, and the pack downloader uses GitPython to clone git repos. Each of those solutions are specific only to their component.

Fortunately, it appears that both GitPython and requests make use of the HTTP_PROXY and HTTPS_PROXY env vars when defined, so it seems like that's the missing link here. It looks like this is just a problem with the docs, but feel free to confirm this on your own

[ClarksonCJ]

@Mierdin I had both the uppercase and lowercase versions of all proxy env vars set, in addition to the git and pip config amendments. The only way I could get the product working on the corporate proxy was to make the adjustments suggested previously.

requests does indeed consume the environment variables, but it appears that they were not being copied to the virtualenv when it is created, which is where my local change to st2common/util/virtualenv.py came in. I "think" (not researched it fully yet) that the virtualenv's for the packs are being created from a parent virtualenv which is not copying the environment config from the host (I need to confirm this).

[Mierdin]

@ClarksonCJ I should have mentioned, I did not run into your specific issue. I am trying a few things to a) verify my theory, and b) try to replicate your issue. Thanks for clarifying

[Mierdin]

So I managed to get a fresh VM spun up with the latest packages, and it looks like the problem is that some of the code runs as st2 user, which in a normal installation has login disabled, so setting environment variables the usual way isn't possible. Currently brainstorming a solution, will update this thread if we come up with something.

[Mierdin]

All - please note https://github.com/StackStorm/st2docs/pull/388, that contains a more suitable workaround for st2 pack install that doesn't require any Python changes and also allows pack dependencies to install as well.

I would like to keep this issue open, as we are going to be pursuing a more permanent solution to this problem in a future release.