Stack: Many --nix command do not forward the https?_proxy environment variables to the inner command

Created on 5 Oct 2016  Â·  41Comments  Â·  Source: commercialhaskell/stack

General summary/comments (optional)

opening an issue because i'm not sure if i'm doing something stupid and this is my fault or not.

Running on centos 6

It seems like the globalManager isn't respecting the http_proxy and https_proxy env variables. I'm not completely sure though since I'm a n00b with this.

I'm trying to build yi, and when I run stack setup it seems unable to fetch defaultStackSetupYaml and the request times out in getSetupInfo.

I hackily added (copy pasted from http-client then added the managerSetProxy call)

globalManager :: IORef Manager
globalManager = unsafePerformIO $ do                                                                                                                                                                                                                                                                                                                                                                                                                                     
    context <- NC.initConnectionContext                                                                                                                                                                                                                          
    let settings = managerSetProxy (proxyEnvironment Nothing) $ mkManagerSettingsContext (Just context) def Nothing                                                                                                                                              
    manager <- newManager settings                                                                                                                                                                                                                               
    newIORef manager                                                                                                                                                                                                                                         {-# NOINLINE globalManager #-}

to src/main/Main.hs, rebuilt stack, and successfully fetched the yaml file with 

 do {req <- parseRequest defaultStackSetupYaml; man <- getGlobalManager; httpLbs req man >>= print}

after including Network.HTTP.Client.TLS Data.IORef System.IO.Unsafe Data.Default.Class in ghci and in the executable in stack.cabal. (fetching the defaultStackSetupYaml does not work in ghci without my modifications to globalManager.)
Any help would be appreciated!

Steps to reproduce

Run stack setup (my stack is currently on ghc 7.10 and am trying to update to 8) in yi's homedir.

Expected

The request to respect the proxy and go through.

Actual

stack setup -v
Version 1.2.1, Git revision a11c2c2c43f779c6fe7086a2e4c8d974305609a0 (dirty) (4178 commits) x86_64 hpack-0.14.1
2016-10-04 19:07:17.670482: [debug] Checking for project config at: /home/justinchiu/haskell/yi/stack.yaml
@(Stack/Config.hs:819:9)
2016-10-04 19:07:17.670714: [debug] Loading project config file stack.yaml
@(Stack/Config.hs:837:13)
2016-10-04 19:07:17.674091: [debug] Trying to decode /home/justinchiu/.stack/build-plan-cache/x86_64-linux/nightly-2016-07-31.cache
@(Data/Store/VersionTagged.hs:68:5)
2016-10-04 19:07:17.682562: [debug] Success decoding /home/justinchiu/.stack/build-plan-cache/x86_64-linux/nightly-2016-07-31.cache
@(Data/Store/VersionTagged.hs:72:13)
2016-10-04 19:07:17.683085: [debug] PATH is: Just "redacted"
@(Stack/Nix.hs:130:6)
2016-10-04 19:07:17.683215: [debug] Using a nix-shell environment from file: /home/justinchiu/haskell/yi/shell.nix
@(Stack/Nix.hs:131:6)
2016-10-04 19:07:17.683704: [debug] Run process: /home/justinchiu/.nix-profile/bin/nix-shell --pure /home/justinchiu/haskell/yi/shell.nix --arg ghc "with (import <nixpkgs> {}); haskell.compiler.ghc801" --run "'/home/justinchiu/.local/bin/stack' $STACK_IN_NIX_EXTRA_ARGS '--internal-re-exec-version=1.2.1' 'setup' '-v'"
@(Stack/Exec.hs:65:5)
Version 1.2.1, Git revision a11c2c2c43f779c6fe7086a2e4c8d974305609a0 (dirty) (4178 commits) x86_64 hpack-0.14.1
2016-10-04 19:07:19.494480: [debug] Checking for project config at: /home/justinchiu/haskell/yi/stack.yaml
@(Stack/Config.hs:819:9)
2016-10-04 19:07:19.494742: [debug] Loading project config file stack.yaml
@(Stack/Config.hs:837:13)
2016-10-04 19:07:19.507568: [debug] Trying to decode /home/justinchiu/.stack/build-plan-cache/x86_64-linux/nightly-2016-07-31.cache
@(Data/Store/VersionTagged.hs:68:5)
2016-10-04 19:07:19.513657: [debug] Success decoding /home/justinchiu/.stack/build-plan-cache/x86_64-linux/nightly-2016-07-31.cache
@(Data/Store/VersionTagged.hs:72:13)
2016-10-04 19:07:19.518049: [debug] Getting system compiler version
@(Stack/Setup.hs:354:17)
2016-10-04 19:07:19.518702: [debug] Run process: /nix/store/95s500qrn3h2ipvkkcxbn9q41gpjpbgh-ghc-7.10.3/bin/ghc --info
@(System/Process/Read.hs:306:3)
2016-10-04 19:07:19.582440: [debug] Process finished in 63ms: /nix/store/95s500qrn3h2ipvkkcxbn9q41gpjpbgh-ghc-7.10.3/bin/ghc --info
@(System/Process/Read.hs:306:3)
2016-10-04 19:07:19.585246: [debug] Run process: /sbin/ldconfig -p
@(System/Process/Read.hs:306:3)
2016-10-04 19:07:19.587780: [debug] Process finished in 2ms: /sbin/ldconfig -p
@(System/Process/Read.hs:306:3)
2016-10-04 19:07:19.588171: [debug] Run process: /nix/store/45qrn064f92kfxnjg99z39zxda671h6f-gcc-wrapper-5.4.0/bin/gcc -v
@(System/Process/Read.hs:306:3)
2016-10-04 19:07:19.603318: [debug] Process finished in 14ms: /nix/store/45qrn064f92kfxnjg99z39zxda671h6f-gcc-wrapper-5.4.0/bin/gcc -v
@(System/Process/Read.hs:306:3)
2016-10-04 19:07:19.603563: [debug] Found shared library libtinfo.so.5 in 'ldconfig -p' output
@(Stack/Setup.hs:523:29)
2016-10-04 19:07:19.603984: [debug] Did not find shared library libtinfo.so.6
@(Stack/Setup.hs:537:38)
2016-10-04 19:07:19.604088: [debug] Did not find shared library libncursesw.so.6
@(Stack/Setup.hs:537:38)
2016-10-04 19:07:19.604222: [debug] Did not find shared library libgmp.so.10
@(Stack/Setup.hs:537:38)
2016-10-04 19:07:19.604305: [debug] Found shared library libgmp.so.3 in 'ldconfig -p' output
@(Stack/Setup.hs:523:29)
2016-10-04 19:07:19.604365: [debug] Using gmp4 GHC build
@(Stack/Setup.hs:569:9)
HttpExceptionRequest Request {
  host                 = "raw.githubusercontent.com"
  port                 = 443
  secure               = True
  requestHeaders       = []
  path                 = "/fpco/stackage-content/master/stack/stack-setup-2.yaml"
  queryString          = ""
  method               = "GET"
  proxy                = Nothing
  rawBody              = False
  redirectCount        = 10
  responseTimeout      = ResponseTimeoutDefault
  requestVersion       = HTTP/1.1
}
 ConnectionTimeout

Stack version

``` $ stack --version Version 1.2.1, Git revision a11c2c2c43f779c6fe7086a2e4c8d974305609a0 (dirty) (4178 commits) x86_64 hpack-0.14.1

Method of installation

git clone stack...
stack install .
``````

further investigation required bug
Source
picture justinchiu
❤2

Most helpful comment

The Stack guide documents the solution I believe:

https://docs.haskellstack.org/en/stable/nix_integration/#the-nix-shell

See http://nixos.org/nix/manual/#options-1 for more about what a "pure" Nix shell means.

By default stack --nix spawns a pure Nix shell, for better reproducibility. A pure shell throws away the parent environment, including the PATH, so most system resources will be invisible inside the shell (think of it as a sandbox). This is not unlike Docker.

If you want to pass through some environment variables, then a simple solution is to stack --nix --no-nix-pure.

BTW, stack setup --nix won't do what you expect / won't work. No equivalent of stack setup is required when --nix is passed. @YPares please add a message to that effect to stack setup already.

picture mboes on 10 Jan 2017
👍2 ❤1

All 41 comments

I have the same issue.
The solution from this issue #1673 helped me.

VPashkov picture VPashkov on 10 Oct 2016

Thanks for the detailed report!

For reference, stack is using the global manager from http-client-tls:

-- | Evil global manager, to make life easier for the common use case
globalManager :: IORef Manager
globalManager = unsafePerformIO $ do
    -- We may decide in the future to just have a global
    -- ConnectionContext and use it directly in tlsManagerSettings, at
    -- which point this can again be a simple (newManager
    -- tlsManagerSettings >>= newIORef). See:
    -- https://github.com/snoyberg/http-client/pull/227.
    context <- NC.initConnectionContext
    let settings = mkManagerSettingsContext (Just context) def Nothing
    manager <- newManager settings
    newIORef manager
{-# NOINLINE globalManager #-}

This is only different from your code in that you've added managerSetProxy (proxyEnvironment Nothing). Based on my read of https://www.stackage.org/haddock/lts-7.3/http-client-0.4.31.1/Network-HTTP-Client.html#v:proxyEnvironment this would be a no-op, but perhaps it isn't?

Pinging @snoyberg since he is familiar with this stuff.

mgsloan picture mgsloan on 11 Oct 2016
👍1

Just to clarify, the following is _not_ problematic behavior:

proxy                = Nothing

It means that the Request doesn't override the value in the Manager.

Can you do the following:

  • Confirm which environment variable you have set for proxying
  • Try setting something like export http_proxy=http://localhost:1234 (or some other unused port) and see if it causes an exception about not being able to connect to that port
snoyberg picture snoyberg on 11 Oct 2016

I'm on Windows 7 and proxy support still doesn't seem to work with the latest version of Stack:

$ stack upgrade
Updating package index Hackage (mirrored at https://github.com/commercialhaskell/all-cabal-hashes.git) ...
Fetching package index ...
Fetched package index.
Populating index cache ...
Populated index cache.
Already at latest version, no upgrade required

$ stack --version
Version 1.2.0 i386 hpack-0.14.0

My setup for proxy settrings is:

$ env|grep proxy
http_proxy=http://redacted:8080
https_proxy=http://redacted:8080

If I then try to install anything, it fails to download the build plan because it doesn't seem to use the right proxy settings:

$ stack install --resolver=lts-7.8 Agda
Run from outside a project, using implicit global project config
Downloading lts-7.8 build plan ...
HttpExceptionRequest Request {
  host                 = "raw.githubusercontent.com"
  port                 = 443
  secure               = True
  requestHeaders       = []
  path                 = "/fpco/lts-haskell/master//lts-7.8.yaml"
  queryString          = ""
  method               = "GET"
  proxy                = Nothing
  rawBody              = False
  redirectCount        = 10
  responseTimeout      = ResponseTimeoutDefault
  requestVersion       = HTTP/1.1
}
 (ProxyConnectException "raw.githubusercontent.com" 443 (Status {statusCode = 403, statusMessage = "Forbidden"}))

If using a different HTTP client that honors the proxy settings, I can download the build plan just fine:

$ wget  https://raw.githubusercontent.com/fpco/lts-haskell/master//lts-7.8.yaml
--2016-11-17 10:53:05--  https://raw.githubusercontent.com/fpco/lts-haskell/master//lts-7.8.yaml
Resolving redacted (redacted)... 10.redacted
Connecting to redacted (redacted)|10.redacted|:8080... connected.
Proxy request sent, awaiting response... 200 OK
Length: 4836248 (4.6M) [text/plain]
Saving to: ‘lts-7.8.yaml’

lts-7.8.yaml               100%[=======================================>]   4.61M  5.76MB/s   in 0.8s

2016-11-17 10:53:06 (5.76 MB/s) - ‘lts-7.8.yaml’ saved [4836248/4836248]
gergoerdi picture gergoerdi on 17 Nov 2016

Whoops, dropped the ball on this:

$ stack --version
Version 1.2.1, Git revision 3541d38207ff7d50cb49a0d26ab0b4f69a8f9c04 (4307 commits) x86_64 hpack-0.14.1
$ env | grep proxy
http_proxy=fwdproxy:8080
ftp_proxy=fwdproxy:8080
https_proxy=fwdproxy:8080
$ stack setup
HttpExceptionRequest Request {
  host                 = "raw.githubusercontent.com"
  port                 = 443
  secure               = True
  requestHeaders       = []
  path                 = "/fpco/stackage-content/master/stack/stack-setup-2.yaml"
  queryString          = ""
  method               = "GET"
  proxy                = Nothing
  rawBody              = False
  redirectCount        = 10
  responseTimeout      = ResponseTimeoutDefault
  requestVersion       = HTTP/1.1
}
 ConnectionTimeout
$ export http_proxy=http://localhost:1111; export https_proxy=http://localhost:1111
$ env | grep proxy
http_proxy=http://localhost:1111
ftp_proxy=fwdproxy:8080
https_proxy=http://localhost:1111
$ stack setup
HttpExceptionRequest Request {
  host                 = "raw.githubusercontent.com"
  port                 = 443
  secure               = True
  requestHeaders       = []
  path                 = "/fpco/stackage-content/master/stack/stack-setup-2.yaml"
  queryString          = ""
  method               = "GET"
  proxy                = Nothing
  rawBody              = False
  redirectCount        = 10
  responseTimeout      = ResponseTimeoutDefault
  requestVersion       = HTTP/1.1
}
 ConnectionTimeout

but not quite sure what to make of this. For more context, trying to run stack setup to pull ghc 8 so i can build yi

justinchiu picture justinchiu on 17 Nov 2016

Exactly same error here with stack v1.3.2 on NixOS with proxy settings (environment variables in place). When trying same on a computer without needing to deal with proxy - all works as expected.

$ stack install --nix ghc-mod hlint stylish-haskell hindent pandoc
[1 of 2] Compiling Main             ( /home/kuznero/.stack/setup-exe-src/setup-mPHDZzAJ.hs, /home/kuznero/.stack/setup-exe-src/setup-mPHDZzAJ.o )
[2 of 2] Compiling StackSetupShim   ( /home/kuznero/.stack/setup-exe-src/setup-shim-mPHDZzAJ.hs, /home/kuznero/.stack/setup-exe-src/setup-shim-mPHDZzAJ.o )
Linking /home/kuznero/.stack/setup-exe-cache/x86_64-linux-nix/tmp-Cabal-simple_mPHDZzAJ_1.24.0.0_ghc-8.0.1 ...
Progress: 4/129HttpExceptionRequest Request {
  host                 = "s3.amazonaws.com"
  port                 = 443
  secure               = True
  requestHeaders       = []
  path                 = "/hackage.fpcomplete.com/package/SHA-1.6.4.2.tar.gz"
  queryString          = ""
  method               = "GET"
  proxy                = Nothing
  rawBody              = False
  redirectCount        = 10
  responseTimeout      = ResponseTimeoutDefault
  requestVersion       = HTTP/1.1
}
 (ConnectionFailure getAddrInfo: does not exist (Name or service not known))

HttpExceptionRequest Request {
  host                 = "s3.amazonaws.com"
  port                 = 443
  secure               = True
  requestHeaders       = []
  path                 = "/hackage.fpcomplete.com/package/Cabal-1.24.2.0.tar.gz"
  queryString          = ""
  method               = "GET"
  proxy                = Nothing
  rawBody              = False
  redirectCount        = 10
  responseTimeout      = ResponseTimeoutDefault
  requestVersion       = HTTP/1.1
}
 (ConnectionFailure getAddrInfo: does not exist (Name or service not known))
...
kuznero picture kuznero on 4 Jan 2017

What value are you putting in your environment variable? While the exception generated by network isn't particularly helpful, it seems to indicate that the DNS name it's looking up cannot be found.

I've put together a branch + simple test case to help out with this if you're able to run it, on the http-client repo. It's the better-getAddrInfo-exception branch (https://github.com/snoyberg/http-client/tree/better-getAddrInfo-exception). If you check it out, you can run:

$ stack build http-client
$ stack ghc attic/sanity.hs

Then running sanity.hs with appropriate environment variables _should_ give the same error you're seeing above, but now with slightly more information.

snoyberg picture snoyberg on 4 Jan 2017

I will be able to try your repo test case tomorrow when I will get back to work.
With regards to environment variable values, I've tried these:

http_proxy=http://127.0.0.1:3128/
https_proxy=http://127.0.0.1:3128/
HTTP_PROXY=http://127.0.0.1:3128/
HTTPS_PROXY=http://127.0.0.1:3128/

I am using CNTLM for proxying through our domain level proxy.

When this didn't work, I've tried using Windows proxy directly without CNTLM with following environment variable values:

http_proxy=http://username:password@proxyhost:proxyport/
https_proxy=http://username:password@proxyhost:proxyport/
HTTP_PROXY=http://username:password@proxyhost:proxyport/
HTTPS_PROXY=http://username:password@proxyhost:proxyport/
kuznero picture kuznero on 4 Jan 2017

In reply to @snoyberg for running test case in better-getAddrInfo-exception branch:

kuznero@soul:~/Projects/GitHub/http-client (branch: better-getAddrInfo-exception)
$ stack build --nix --install-ghc http-client
these paths will be fetched (70.20 MiB download, 987.94 MiB unpacked):
  /nix/store/75p1400r0bg58ap13smyf75i9vy6bblc-ghc-7.10.3-doc
  /nix/store/f0pdkzhsdf305m52gvf0vk31f2dliky1-ghc-7.10.3
fetching path ‘/nix/store/75p1400r0bg58ap13smyf75i9vy6bblc-ghc-7.10.3-doc’...

*** Downloading ‘https://cache.nixos.org/nar/0xdncpl3nm53b8ccyv71k2j1lnn8anai66rj4cnj9zp9agck8pza.nar.xz’ (signed by ‘cache.nixos.org-1’) to ‘/nix/store/75p1400r0bg58ap13smyf75i9vy6bblc-ghc-7.10.3-doc’...
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 8776k  100 8776k    0     0   572k      0  0:00:15  0:00:15 --:--:--  571k

fetching path ‘/nix/store/f0pdkzhsdf305m52gvf0vk31f2dliky1-ghc-7.10.3’...

*** Downloading ‘https://cache.nixos.org/nar/03lkjx76jz71ys1ardh4ga1hsw75rqwknd0n15khwgh2gxif1dpp.nar.xz’ (signed by ‘cache.nixos.org-1’) to ‘/nix/store/f0pdkzhsdf305m52gvf0vk31f2dliky1-ghc-7.10.3’...
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 61.6M  100 61.6M    0     0   517k      0  0:02:01  0:02:01 --:--:--  532k

[1 of 2] Compiling Main             ( /home/kuznero/.stack/setup-exe-src/setup-mPHDZzAJ.hs, /home/kuznero/.stack/setup-exe-src/setup-mPHDZzAJ.o )
[2 of 2] Compiling StackSetupShim   ( /home/kuznero/.stack/setup-exe-src/setup-shim-mPHDZzAJ.hs, /home/kuznero/.stack/setup-exe-src/setup-shim-mPHDZzAJ.o )
Linking /home/kuznero/.stack/setup-exe-cache/x86_64-linux-nix/tmp-Cabal-simple_mPHDZzAJ_1.22.5.0_ghc-7.10.3 ...
Progress: 4/25HttpExceptionRequest Request {
  host                 = "s3.amazonaws.com"
  port                 = 443
  secure               = True
  requestHeaders       = []
  path                 = "/hackage.fpcomplete.com/package/mtl-2.2.1.tar.gz"
  queryString          = ""
  method               = "GET"
  proxy                = Nothing
  rawBody              = False
  redirectCount        = 10
  responseTimeout      = ResponseTimeoutDefault
  requestVersion       = HTTP/1.1
}
 (ConnectionFailure getAddrInfo: does not exist (Name or service not known))

HttpExceptionRequest Request {
  host                 = "s3.amazonaws.com"
  port                 = 443
  secure               = True
  requestHeaders       = []
  path                 = "/hackage.fpcomplete.com/package/base64-bytestring-1.0.0.1.tar.gz"
  queryString          = ""
  method               = "GET"
  proxy                = Nothing
  rawBody              = False
  redirectCount        = 10
  responseTimeout      = ResponseTimeoutDefault
  requestVersion       = HTTP/1.1
}
 (ConnectionFailure getAddrInfo: does not exist (Name or service not known))

HttpExceptionRequest Request {
  host                 = "s3.amazonaws.com"
  port                 = 443
  secure               = True
  requestHeaders       = []
  path                 = "/hackage.fpcomplete.com/package/data-default-class-0.0.1.tar.gz"
  queryString          = ""
  method               = "GET"
  proxy                = Nothing
  rawBody              = False
  redirectCount        = 10
  responseTimeout      = ResponseTimeoutDefault
  requestVersion       = HTTP/1.1
}
 (ConnectionFailure getAddrInfo: does not exist (Name or service not known))

HttpExceptionRequest Request {
  host                 = "s3.amazonaws.com"
  port                 = 443
  secure               = True
  requestHeaders       = []
  path                 = "/hackage.fpcomplete.com/package/network-2.6.2.1.tar.gz"
  queryString          = ""
  method               = "GET"
  proxy                = Nothing
  rawBody              = False
  redirectCount        = 10
  responseTimeout      = ResponseTimeoutDefault
  requestVersion       = HTTP/1.1
}
 (ConnectionFailure getAddrInfo: does not exist (Name or service not known))

And since that fails I cannot get to running stack ghc --nix attic/sanity.hs:

attic/sanity.hs:1:8:
    Could not find module `Network.HTTP.Client'
    Use -v to see a list of the files searched for.

Please let me know if you think I can do something to finally run sanity.hs.

kuznero picture kuznero on 5 Jan 2017

Can you compile the executable on a non proxied machine and then run on the
proxied machine? I'd like to get to the point where this is being reproed
against just http-client so we can simplify debugging.

On Thu, Jan 5, 2017, 10:51 AM Roman Kuznetsov notifications@github.com
wrote:

In reply to @snoyberg https://github.com/snoyberg for running test case
in better-getAddrInfo-exception branch:

kuznero@soul:~/Projects/GitHub/http-client (branch: better-getAddrInfo-exception)
$ stack build --nix --install-ghc http-client
these paths will be fetched (70.20 MiB download, 987.94 MiB unpacked):
/nix/store/75p1400r0bg58ap13smyf75i9vy6bblc-ghc-7.10.3-doc
/nix/store/f0pdkzhsdf305m52gvf0vk31f2dliky1-ghc-7.10.3
fetching path ‘/nix/store/75p1400r0bg58ap13smyf75i9vy6bblc-ghc-7.10.3-doc’...

* Downloading ‘https://cache.nixos.org/nar/0xdncpl3nm53b8ccyv71k2j1lnn8anai66rj4cnj9zp9agck8pza.nar.xz’ (signed by ‘cache.nixos.org-1’) to ‘/nix/store/75p1400r0bg58ap13smyf75i9vy6bblc-ghc-7.10.3-doc’...
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 8776k 100 8776k 0 0 572k 0 0:00:15 0:00:15 --:--:-- 571k

fetching path ‘/nix/store/f0pdkzhsdf305m52gvf0vk31f2dliky1-ghc-7.10.3’...

* Downloading ‘https://cache.nixos.org/nar/03lkjx76jz71ys1ardh4ga1hsw75rqwknd0n15khwgh2gxif1dpp.nar.xz’ (signed by ‘cache.nixos.org-1’) to ‘/nix/store/f0pdkzhsdf305m52gvf0vk31f2dliky1-ghc-7.10.3’...
% Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
100 61.6M 100 61.6M 0 0 517k 0 0:02:01 0:02:01 --:--:-- 532k

[1 of 2] Compiling Main ( /home/kuznero/.stack/setup-exe-src/setup-mPHDZzAJ.hs, /home/kuznero/.stack/setup-exe-src/setup-mPHDZzAJ.o )
[2 of 2] Compiling StackSetupShim ( /home/kuznero/.stack/setup-exe-src/setup-shim-mPHDZzAJ.hs, /home/kuznero/.stack/setup-exe-src/setup-shim-mPHDZzAJ.o )
Linking /home/kuznero/.stack/setup-exe-cache/x86_64-linux-nix/tmp-Cabal-simple_mPHDZzAJ_1.22.5.0_ghc-7.10.3 ...
Progress: 4/25HttpExceptionRequest Request {
host = "s3.amazonaws.com"
port = 443
secure = True
requestHeaders = []
path = "/hackage.fpcomplete.com/package/mtl-2.2.1.tar.gz"
queryString = ""
method = "GET"
proxy = Nothing
rawBody = False
redirectCount = 10
responseTimeout = ResponseTimeoutDefault
requestVersion = HTTP/1.1
}
(ConnectionFailure getAddrInfo: does not exist (Name or service not known))

HttpExceptionRequest Request {
host = "s3.amazonaws.com"
port = 443
secure = True
requestHeaders = []
path = "/hackage.fpcomplete.com/package/base64-bytestring-1.0.0.1.tar.gz"
queryString = ""
method = "GET"
proxy = Nothing
rawBody = False
redirectCount = 10
responseTimeout = ResponseTimeoutDefault
requestVersion = HTTP/1.1
}
(ConnectionFailure getAddrInfo: does not exist (Name or service not known))

HttpExceptionRequest Request {
host = "s3.amazonaws.com"
port = 443
secure = True
requestHeaders = []
path = "/hackage.fpcomplete.com/package/data-default-class-0.0.1.tar.gz"
queryString = ""
method = "GET"
proxy = Nothing
rawBody = False
redirectCount = 10
responseTimeout = ResponseTimeoutDefault
requestVersion = HTTP/1.1
}
(ConnectionFailure getAddrInfo: does not exist (Name or service not known))

HttpExceptionRequest Request {
host = "s3.amazonaws.com"
port = 443
secure = True
requestHeaders = []
path = "/hackage.fpcomplete.com/package/network-2.6.2.1.tar.gz"
queryString = ""
method = "GET"
proxy = Nothing
rawBody = False
redirectCount = 10
responseTimeout = ResponseTimeoutDefault
requestVersion = HTTP/1.1
}
(ConnectionFailure getAddrInfo: does not exist (Name or service not known))

And since that fails I cannot get to running stack ghc --nix
attic/sanity.hs:

attic/sanity.hs:1:8:
Could not find module `Network.HTTP.Client'
Use -v to see a list of the files searched for.

Please let me know if you think I can do something to finally run
sanity.hs.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/commercialhaskell/stack/issues/2672#issuecomment-270593182,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AADBB9V8oZ3VamQxv-ulH7JQJUnfyPxLks5rPK7zgaJpZM4KOW1o
.

snoyberg picture snoyberg on 5 Jan 2017

Now I am having a NixOS specific error as it seems when running on proxied computer (bits built on non-proxied computer):

$ attic/sanity 
attic/sanity: error while loading shared libraries: libgmp.so.10: cannot open shared object file: No such file or directory

Though I have gmp package installed system wide. How can I overcome this?

I have tried running with nix-shell -p gmp, nix-shell -p gmp4 and nix-shell -p gmp5. Didn't help.

kuznero picture kuznero on 5 Jan 2017

@snoyberg finally I managed to run it, and it just works:

kuznero@soul:~/Projects/GitHub/http-client (branch: better-getAddrInfo-exception!)
$ nix-build --no-out-link "<nixpkgs>" -A gmp
/nix/store/50s6fp36mb6akz5l87x0pw4ppa66jvzq-gmp-6.1.1

kuznero@soul:~/Projects/GitHub/http-client (branch: better-getAddrInfo-exception!)
$ echo $LD_LIBRARY_PATH
/run/opengl-driver/lib:/run/opengl-driver-32/lib

kuznero@soul:~/Projects/GitHub/http-client (branch: better-getAddrInfo-exception!)
$ LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/nix/store/50s6fp36mb6akz5l87x0pw4ppa66jvzq-gmp-6.1.1/lib

kuznero@soul:~/Projects/GitHub/http-client (branch: better-getAddrInfo-exception!)
$ echo $LD_LIBRARY_PATH                                                                     
/run/opengl-driver/lib:/run/opengl-driver-32/lib:/nix/store/50s6fp36mb6akz5l87x0pw4ppa66jvzq-gmp-6.1.1/lib

kuznero@soul:~/Projects/GitHub/http-client (branch: better-getAddrInfo-exception!)
$ attic/sanity                                                                              
Response {responseStatus = Status {statusCode = 200, statusMessage = "OK"}, responseVersion = HTTP/1.1, responseHeaders = [("Via","1.1 W89505"),("Connection","Keep-Alive"),("Proxy-Connection","Keep-Alive"),("Content-Length","31"),("Date","Thu, 05 Jan 2017 09:50:30 GMT"),("Content-Type","application/json"),("Server","nginx"),("Access-Control-Allow-Origin","*"),("Access-Control-Allow-Credentials","true")], responseBody = "{\n  \"origin\": \"212.93.55.21\"\n}\n", responseCookieJar = CJ {expose = []}, responseClose' = ResponseClose}

So, that does not show any error, but I cannot still run stack install --nix hindent for example. What else can be wrong then?

kuznero picture kuznero on 5 Jan 2017

Hmm, interesting. If you just run something like stack unpack mtl without using --nix, what happens?

snoyberg picture snoyberg on 5 Jan 2017

It just works :)

kuznero@soul:~/Projects/GitHub
$ stack unpack mtl
mtl-2.2.1: download
Unpacked mtl-2.2.1 to /home/kuznero/Projects/GitHub/mtl-2.2.1/

kuznero@soul:~/Projects/GitHub
$ ls
mtl-2.2.1

I guess this explains why stack setup --nix works fine and able to download everything it needs.

kuznero picture kuznero on 5 Jan 2017

So in other words, this only occurs when run from _inside_ the nix invocation? I'm woefully ignorant of what goes on inside the Nix world, I can't even fathom what would be triggering this. Can you somehow run the sanity exe inside a matching Nix env and see if that triggers something? I'm guessing this might be an environment variable encoding issue, but that's a huge guess out of left field.

snoyberg picture snoyberg on 5 Jan 2017

:) Environment variables are defined in a rather conventional way. And again, on non-proxied NixOS computer it works fine. While I was investigating this issue I read on one of the blogs related to something like ruby or something like that, that after library is getting a redirect reply it stops applying proxy settings. Can that be the case with http-client?

kuznero picture kuznero on 5 Jan 2017

That shouldn't be the case, but you can test it by modifying the sanity exe to use a URL that should be getting a redirect.

snoyberg picture snoyberg on 5 Jan 2017

I have tried querying http://s3.amazonaws.com/hackage.fpcomplete.com/package/base-orphans-0.5.4.tar.gz (not https) and it works. I will try changing sanity.hs to use tls and try with https version of that url.

But on that note, is it possible to instruct stack to not to use https?

kuznero picture kuznero on 5 Jan 2017

You can modify the package-indices settings in stack.yaml or your global config to give it a different base URL, that should do it, though there are a number of other URLs you might need to change too.

snoyberg picture snoyberg on 5 Jan 2017

I have tried that before, but it seems to have affect on stack setup flow only...

kuznero picture kuznero on 5 Jan 2017

So clearly there's something going on inside http-client that's the problem, and almost certainly unrelated to Stack itself. And it's also almost certainly the case that it only happens on Nix machines. The only way I can see this getting resolved is some reproducible case against http-client itself.

snoyberg picture snoyberg on 5 Jan 2017

Will it still make sense for me to try to use tls manager and run test through https?

kuznero picture kuznero on 5 Jan 2017

Sure, that's fine, but at least naively it doesn't look like TLS has nothing to do with this.

snoyberg picture snoyberg on 5 Jan 2017

Interesting, running sanity and sanity-tls over http and https respectively just works fine.

kuznero picture kuznero on 5 Jan 2017

@snoyberg can you maybe point in certain direction on where should I start digging with this issue? Maybe involve people who's more savvy ion NixOS? As of now with NixOS+Proxy it seems that it is simply impossible to use stack unfortunately.

kuznero picture kuznero on 5 Jan 2017

I had a comment earlier about running the compiled sanity executable from inside the nix environment, I'd imagine via something like stack --nix exec ./sanity. Were you able to try that?

snoyberg picture snoyberg on 5 Jan 2017

I have not tried doing so through stack before, trying running sanity directly results in this:

kuznero@soul:~/Projects/GitHub/http-client (branch: better-getAddrInfo-exception!)
$ LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/nix/store/50s6fp36mb6akz5l87x0pw4ppa66jvzq-gmp-6.1.1/lib

kuznero@soul:~/Projects/GitHub/http-client (branch: better-getAddrInfo-exception!)
[127] $ ~/Downloads/sanity
Response {responseStatus = Status {statusCode = 200, statusMessage = "OK"}, responseVersion = HTTP/1.1, responseHeaders = [("Via","1.1 W89505"),("Connection","Keep-Alive"),("Proxy-Connection","Keep-Alive"),("Content-Length","4996"),("Date","Thu, 05 Jan 2017 10:14:04 GMT"),("Age","12594"),("Content-Type","binary/octet-stream"),("ETag","\"e590aa3f2de078eba409723eceed153a\""),("Server","AmazonS3"),("Accept-Ranges","bytes"),("x-amz-id-2","TemE4sHjIKxQXUN6Nt2cNXJ1EysxdU7pgXsiCHJusdtzESyBd0OfhKd+VLCLVtpVYfm1dBQ77lk="),("x-amz-request-id","074C8C99B326EBEE"),("Last-Modified","Fri, 19 Jun 2015 20:38:51 GMT")], responseBody = "\US\139...

Then I tried running it through stack as you suggested:

kuznero@soul:~/Projects/GitHub/http-client (branch: better-getAddrInfo-exception!)
$ stack exec --nix ~/Downloads/sanity
/home/kuznero/Downloads/sanity: error while loading shared libraries: libgmp.so.10: cannot open shared object file: No such file or directory

And trying to add --extra-lib-dirs to stack exec does not seem to help (not sure if I do it correctly though):

kuznero@soul:~/Projects/GitHub/http-client (branch: better-getAddrInfo-exception!)
$ stack exec --nix --extra-lib-dirs=/nix/store/50s6fp36mb6akz5l87x0pw4ppa66jvzq-gmp-6.1.1/lib ~/Downloads/sanity                                                                                             
/home/kuznero/Downloads/sanity: error while loading shared libraries: libgmp.so.10: cannot open shared object file: No such file or directory

@snoyberg sorry for throwing in seemingly unrelated problems, but I would really like to help reproducing and fixing it if possible.

kuznero picture kuznero on 5 Jan 2017

Alright, I got a mini Nix setup going. What does stack --nix exec env have to say? In particular, I'm interested in stack --nix exec env | grep -i ^http. For the record, on my machine:

$ stack exec --nix env | grep -i ^http
$ stack exec env | grep -i ^http
HTTPS_PROXY=http://127.0.0.1:3128
HTTP_PROXY=http://127.0.0.1:3128
http_proxy=http://127.0.0.1:3128
https_proxy=http://127.0.0.1:3128

In other words, the env vars aren't being passed in to nix.

snoyberg picture snoyberg on 5 Jan 2017

Interesting... Who controls what environment to pass when stack is running with --nix option? I can also see that stack exec --nix env | grep -i proxy produces nothing on my box. And even running this does not help:

http_proxy=http://127.0.0.1:3128 stack exec --nix env | grep -i proxy

I am puzzled about where to look next. Does stack support additional NixOS options when --nix is used?

Another thing to remember here is that stack setup --nix is also then running with --nix option, but it seems that it has access to all the necessary environment variables.

kuznero picture kuznero on 5 Jan 2017

I'm not sure about any of this, it's beyond my expertise. But I think it's safe to say that we've identified the real issue:

__Many --nix command do not forward the https?_proxy environment variables to the inner command__

I'm going to update the issue title, and then perhaps someone with more Nix expertise can comment on the right way to address this.

snoyberg picture snoyberg on 5 Jan 2017

Ping @YPares

borsboom picture borsboom on 5 Jan 2017
👍1

FWIW the original block I had in installing is gone and everything works fine for me now after pulling.

justinchiu picture justinchiu on 5 Jan 2017

The Stack guide documents the solution I believe:

https://docs.haskellstack.org/en/stable/nix_integration/#the-nix-shell

See http://nixos.org/nix/manual/#options-1 for more about what a "pure" Nix shell means.

By default stack --nix spawns a pure Nix shell, for better reproducibility. A pure shell throws away the parent environment, including the PATH, so most system resources will be invisible inside the shell (think of it as a sandbox). This is not unlike Docker.

If you want to pass through some environment variables, then a simple solution is to stack --nix --no-nix-pure.

BTW, stack setup --nix won't do what you expect / won't work. No equivalent of stack setup is required when --nix is passed. @YPares please add a message to that effect to stack setup already.

mboes picture mboes on 10 Jan 2017
👍2 ❤1

No equivalent of stack setup is required when --nix is passed. @YPares please add a message

OK my bad. It seems this has been resolved in the latest Stack. stack setup does what you expect: prefetch and unpack the appropriate GHC from Nixpkgs (instead of using the official upstream bindists, which won't work with Nix). In any case, this is tangential to the ticket.

mboes picture mboes on 10 Jan 2017

That had solved my problem! :) Thanks @mboes!

kuznero picture kuznero on 10 Jan 2017

@mboes does it make sense to make some exceptions to the environment variable rule, including the proxy environment variables?

@borsboom would the same exception logic make sense for Docker as well?

snoyberg picture snoyberg on 10 Jan 2017

@snoyberg I think so. @YPares in fact it might be nice to add a field in the stack.yaml where the user can whitelist arbitrary environment variables.

mboes picture mboes on 10 Jan 2017

@mboes I'm not sure you can ask nix-shell to forward some env vars. I think you cannot
So this mean we should recreate those variables in the generated nix code, but this wouldn't work in the case where the user gives a custom shell.nix.

YPares picture YPares on 10 Jan 2017

You co​uld do it by passing a special --args to the Nix shell containing
the env vars you want to set with the values from the host environment. It
would be a tad finicky I agree.

mboes picture mboes on 10 Jan 2017

I modified the nix README to make it clearer that the pure shell changes the env vars: see commit 05acb2c

YPares picture YPares on 10 Jan 2017
👍1

Great addition, thanks.

mboes picture mboes on 10 Jan 2017
Was this page helpful?
0 / 5 - 0 ratings

Related issues

symbiont-joseph-kachmar picture symbiont-joseph-kachmar  Â·  3Comments
s5k6 picture s5k6  Â·  3Comments
mgsloan picture mgsloan  Â·  3Comments
Cosmius picture Cosmius  Â·  3Comments
silky picture silky  Â·  3Comments