Salt: file.managed with proxy broken in 2015.5

[porterjamesj]

Hey all, I have a state that looks something like this:

unzip:
  pkg.installed

http_proxy:
  environ.setenv:
    - value: 'http://proxy-url'

https_proxy:
  environ.setenv:
    - value: 'https://proxy-url'

/usr/local/bin/consul_dir:
  archive.extracted:
    - source: https://dl.bintray.com/mitchellh/consul/0.5.0_linux_amd64.zip
    - archive_format: zip
    - source_hash: sha256=161f2a8803e31550bd92a00e95a3a517aa949714c19d3124c46e56cfdc97b088
    - require:
        - environ: http_proxy
        - environ: https_proxy
        - pkg: unzip

where proxy-url is a standin for the actual url of my http proxy. With a 2014.7 minion the archive extracted fine, with a 2015.5 minion, I get:

----------
          ID: /usr/local/bin/consul_dir
    Function: archive.extracted
      Result: False
     Comment: Unable to manage file: [Errno 111] Connection refused
     Started: 17:58:15.121916
    Duration: 420.282 ms
     Changes:
----------

Debugging with salt-call, this appears to be happening here, it looks like the new utils.http doesn't use the http_proxy and https_proxy environment variables, whereas whatever was responsible for the http calls in 2014.7 did.

[jfindlay]

@porterjamesj, thanks for the report.

[garo]

Noticed the same problem after spending way too many hours wondering why my proxy settings doesn't take any effect.

[gboddin]

Same issue here, it's a blocker when you're working in a proxy controlled environment.

[luitzifa]

Same issue with file.managed, but only with https

[jfindlay]

I've changed the title of this issue because archive.extracted uses file.managed and the problem has been confirmed with file.managed itself.

[The-Loeki]

It used to work if you set requests_lib: True in the minion conf, however in Beryllium this seems broken, and Tornado HTTP client won't pick up on it either unless it's switched over to curl_httpclient.

[The-Loeki]

Related to #8177

[nlm]

It would be nice that salt honors properly all the protocol_proxy environment variables in salt-minion with all the backends it uses:

• http_proxy
• https_proxy
• ftp_proxy
  and possibly other protocols supported by salt.

It seems that most of this can be fixed in salt.fileclient or salt.utils.{http,s3,...}

[JensRantil]

Workaround:

download-my-file:
  cmd.run:
    - name: "wget -q -O /srv/my-file.tgz http://somedomain/my-file.tgz"
    - creates: /srv/my-file.tgz
    - env:
      - http_proxy: "my-proxy.domain.com"

Clearly, this doesn't add any hashsum, so it's less secure.

[chbiel]

Is there any progress on this issue?

[The-Loeki]

See #29322; the new (default) Tornado backend should be able to use PycURL, which is proxy-capable.

[quasiben]

What about doing the following above any states:

http_proxy:
  environ.setenv:
    - value: "my-proxy.domain.com"

?

[The-Loeki]

AFAIK the Tornado backend _only_ listens to config options

https://docs.saltstack.com/en/latest/ref/configuration/minion.html#proxy-host

[quasiben]

@The-Loeki looking through that I noticed that the minion doesn't support https proxying. Is that correct?

[The-Loeki]

No, the idea is you specify in the config a backend.
Of that you have three choices:

• requests; this backend _should_ listen to the env vars we all know (ftp/http(s)_proxy, IIRC you need https_proxy set for HTTPS to be proxied)
• urrlib2: No clue, haven't used this one
• The default, Tornado uses config options proxy_* to determine proxy usage, seemingly irrespective of proto.

Both backends need PyCuRL IIRC to do the proxying.

[maltris]

As of 2017.7 the problem mentioned by OP still exists.

[stale[bot]]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

If this issue is closed prematurely, please leave a comment and we will gladly reopen the issue.

[joachimtingvold]

This is still an issue, and thus, should be re-opened.

It seems the only way to make Tornardo use a proxy, is to change the Salt Minion configuration file, and restart the Salt Minion. You cannot change it "on the fly" (i.e. for just one state, or similar).

What I've tried;

• Set http_proxy and https_proxy environment variables for the root user.
• Use environ.setenv to set the same two variables, with or without update_minion. Does not matter what order, or if I require the state before doing file.managed or archive.extracted.

None of them seems to work, even in 2019.2.0.

[joachimtingvold]

@jfindlay , not sure if stalebot re-opens the issue if a new comment is added, so pinging you to be sure (i.e. to manually re-open it).

I'm aware that the connected issue (#8177) technically has been fixed. However, it's only been fixed in such a way that you still need to configure the Salt Minion.

My opinion is that this should be fixed so that a) Tornardo respects the http_proxy/https_proxy environment variables, or b) a flag that you can pass to file.managed (and all of the modules that uses file.managed, to set the proxy just for that one state.

Useful if you have IPv6-only minions that should not use a proxy, unless for a specific state (i.e. very specific external sources that only has IPv4).

[peter-fe]

@jfindlay, @tingvold

Same issues here with managed.file & archive.extracted behind a proxy, it does indeed work if we set that proxy_host and proxy_port settings on the minion.

But actually we have lots of different environments, some of them have proxies, some not, some have different ones.
We currently control these settings by salt formulas which are setting up the common http_proxy, https_proxy, no_proxy environment variables.

Having this additional/odd piece of configuration needed for the minions in environments with proxies really feels wrong. Especially as it is only possible to set them via file and the minion has to get restarted to respect them (so not even creating them on the fly on hosts with proxies through salt states is possible).

Why going into this completely strange direction of ignoring the existing env and widely spread standards in handling these settings?