Stack: Better support for bad connection

[flip111]

When i travel by train i can get free wifi in the train, but the connection is really slow. But i still would like to do some coding with GHCI. Normally i just launch stack ghci, all the dependencies were already loaded in at home so it should just start right away. Instead i see this

» stack ghci
Selected mirror https://s3.amazonaws.com/hackage.fpcomplete.com/                                 
Downloading timestamp                                                                            
Downloading snapshot                                                                             
Downloading mirrors                                                                              
Cannot update index (no local copy)                                                              
Downloading index                                                                                
Updating package index Hackage (mirrored at https://s3.amazonaws.com/hackage.fpcomplete.com/) ...

After a 20 minutes or so, this gets added recv: resource vanished (Connection reset by peer) and the program abort. And i can't code in the train this way.

[flip111]

This one also appears some time

SomeRemoteError (HttpExceptionRequest Request {
  host                 = "s3.amazonaws.com"
  port                 = 443
  secure               = True
  requestHeaders       = [("Accept-Encoding",""),("Cache-Control","no-transform")]
  path                 = "/hackage.fpcomplete.com/snapshot.json"
  queryString          = ""
  method               = "GET"
  proxy                = Nothing
  rawBody              = False
  redirectCount        = 10
  responseTimeout      = ResponseTimeoutDefault
  requestVersion       = HTTP/1.1
}
 ConnectionTimeout)

[mgsloan]

Hmm, that's strange! It should only do that if the hackage info is missing, or if you need to build a package version that it doesn't know about. So, not sure what could cause this.

[flip111]

If there are some debug options (like debug log output) i could try those next time i run into the problem.

[domenkozar]

I'm getting this on travis:

[Blaisorblade]

@domenkozar That might be relevant, but there stack is downloading a build plan it didn't know about—which is always the case unless you cache ~/.stack. Then, Travis network is rather flaky, so their docs recommend travis_retry as a wrapper to network-related commands. Not sure how to separate the downloads from the rest. (Maybe the output is still wrong, but I'm not sure why).

@flip111

If there are some debug options (like debug log output) i could try those next time i run into the problem.

Stack has the -v option.

[QuRyu]

same here. I guess part of the reason is I'm in mainland China, but even download from GitHub can run pretty fast for me, usually 200~300 kb/s
so I was NEVER able to finishing downloading the package index so never tried stack, though I earnestly wanted to
is there any way to download the package manually? tried to copy the index downloaded by cabal into stack but stack says it could not decode the package

[Blaisorblade]

@Quentin-L You could try configuring package-indices in ~/.stack/config.yaml to avoid the Amazon S3 mirror—I guess S3 — docs are at https://docs.haskellstack.org/en/stable/yaml_configuration/#package-indices.

Sadly I don't know the ideal config to use—at some point I had this:

package-indices:
- name: HackageOrig
  download-prefix: https://hackage.haskell.org/package/
  http: https://hackage.haskell.org/00-index.tar.gz

which should be no worse than cabal-install was.

[Blaisorblade]

@mgsloan There are enough users with problems with S3 that we should have docs on how to avoid it (did I say this already)? Is my previous comment still good nowadays?

package-indices:
- name: HackageOrig
  download-prefix: https://hackage.haskell.org/package/
  http: https://hackage.haskell.org/00-index.tar.gz

[QuRyu]

@Blaisorblade Thanks! It only took me four minutes to download the package!! Finally I can enjoy Stack LOL

[QuRyu]

@Blaisorblade Is there anyway to work around the ghc downloading? I installed ghc 8.0.2 through home-brew and linked it to stack, but a project I forked uses ghc 8.0.1 so stack suggests me to use stack setup. Downloading ghc is another hell that I have never conquered.

[Blaisorblade]

I am not sure there... There are ways to use a custom set of GHC download URLs... And bindists from the GHC website should work. The comment below has an example (see the setup-info section of config.yaml). Sorry these are very incomplete instructions, but I'm on my phone.

https://github.com/commercialhaskell/stack/issues/416#issuecomment-292966247

[QuRyu]

@Blaisorblade After some unsuccessful attempt, I finally found the easiest way is to download ghc distribution manually, copy it to stack directories, and tell stack to compile it.

[hengxin]

@QuRyu Could you please show me more detailed instructions? Specifically, where to download ghc? What are the stack directories? How to tell stack to compile this ghc distribution? Great thanks.

[QuRyu]

@hengxin I was on macOS so I used brew to install stack. I am under the impression that by typing "stack ghc" you would get instructions on how to install ghc, since stack does not come bundled with ghc.
I later switched to a Stackage mirror hosted on TUNA[1] and was able to finish downloading.

[1] https://zhuanlan.zhihu.com/p/25005809

[Blaisorblade]

I’m not on a computer right now (so can’t give good instructions), but you don’t need to compile GHC, ~you just follow the instructions to install a bindist (which involve iirc configure and make but still don’t compile anything).~

EDIT: see below.

[Blaisorblade]

Correction: you need to place a GHC bindist tarball into the right subdirectory of ~/.stack/programs.
For instance, to install GHC 8.0.2 on a 64-bit Linux system if you didn’t have that GHC version setup with stack, you’ll need to place the correct GHC bindist tarball in ~/.stack/programs/x86_64-linux/ghc-8.0.2.tar.xz, then run stack setup to get it installed.

(Correct files are available at https://github.com/commercialhaskell/ghc/releases, not sure about a good mirror for China).

[hengxin]

@QuRyu @Blaisorblade Thanks. It works now.

[Blaisorblade]

@hengxin Would you mind adding details on what worked for you? My instructions were vague (couldn’t test much), it might help others 😉

[hengxin]

@Blaisorblade I am using Ubuntu. stack setup did not work for me (I got the ConnectionTimeout info).

Following the instructions given by @QuRyu , I have added

package-indices:
  - name: Tsinghua
    download-prefix: http://mirrors.tuna.tsinghua.edu.cn/hackage/package/
    http: http://mirrors.tuna.tsinghua.edu.cn/hackage/00-index.tar.gz

to ~/.stack/config.yaml. (Notice that https://... did not work for me.)

Now, stack setup, stack build, and stack exec work as expected.

Following @Blaisorblade , I have checked the ~/.stack/programs directory, and found the following files in ~/.stack/programs/x86_64-linux:

• ghc-8.0.2 (dir)
• ghc-8.0.2.installed
• ghc-8.0.2.tar.xz

[Blaisorblade]

@hengxin Thanks! I’m confused that this helped for downloading GHC in stack setup, but I see why using such a mirror will help downloading hackage packages. Anyway, hope this helps others in a similar situation!

[QuRyu]

@hengxin glad it helps.
@Blaisorblade The mirror also includes GHC and Stackage snapshot.

[Blaisorblade]

@QuRyu Ah, but one also needs extra configuration:

setup-info: "http://mirrors.tuna.tsinghua.edu.cn/stackage/stack-setup.yaml"
urls:
  latest-snapshot: http://mirrors.tuna.tsinghua.edu.cn/stackage/snapshots.json
  lts-build-plans: http://mirrors.tuna.tsinghua.edu.cn/stackage/lts-haskell/
  nightly-build-plans: http://mirrors.tuna.tsinghua.edu.cn/stackage/stackage-nightly/

I can’t read your link but the snippet seems clear. I didn’t know Stack accepted configuration for mirrors.

For info: https://groups.google.com/forum/m/#!topic/haskell-stack/LHG9DSrz8k8 has details, and https://github.com/ndmitchell/offline-stack shows how to use Stack offline (after downloading enough material), which probably helps with the original issue.

I’m tempted to now mark this whole issue as duplicate, though I’m not sure which issues discuss the remaining annoyances.

[Blaisorblade]

OK, closing as duplicate of #2982 which has progress on using Stack offline.

[samravshanov]

@Blaisorblade, what about using stackage instead?

package-indices:
- name: Stackage
  download-prefix: https://www.stackage.org/lts-9.14/package/
  http: https://www.stackage.org/lts-9.14/00-index.tar.gz

[Blaisorblade]

@samandarr I’m afraid I’m not sure what’s your question? Stack uses stackage by default, that config looks like it fixes a particular snapshot, and that is... not good I guess?

[samravshanov]

@Blaisorblade I had same sort of issue above. Then I used the configuration suggested by you and it changed my life :D

package-indices:
- name: HackageOrig
  download-prefix: https://hackage.haskell.org/package/
  http: https://hackage.haskell.org/00-index.tar.gz

I think it changes pkg mirror from S3 to hackage.haskell.org ? I had really bad connection with S3.
Now everything is fine. What I was referring to maybe it is better idea to use stackage LTS as a mirror ?

[samravshanov]

@Blaisorblade I highly appreciate your help. It (your configuration) just works now. Could you tell me magic behind it :D? Does Stack indeed uses stackage by default? In my case, it seemed as its downloading from https://s3.amazonaws.com/hackage.fpcomplete.com which was really really slow.

[Blaisorblade]

@samandarr Sorry, let me clarify — Stack downloads (a copy of) Hackage, by default from a mirror on S3. If hackage.haskell.org is faster for you, things will still work.

Stack then uses Stackage elsewhere to pick sets of compatible package versions — if my stack.yaml uses lts-8.0 as resolver, it’ll get by default those package versions. I’m not sure which host is used to download the build plans by default?

But I fear such a stack.yaml might break if you use stackage.org with a specific LTS in package-indices — not sure, because I’ve never seen that config — you’re free to use it, but we don’t support it. I wonder if those URLs are provided for those who use stackage with cabal-install instead of Stack (which I wouldn’t recommend)?

Worse, if I include a stack.yaml with my app (as I should), and you use it, I pick the LTS version I use, so it’s hard for you to stick to a single LTS.

EDIT: TL;DR. Using hackage.haskell.org as you’re doing makes sense, using stackage.org as you did looks... surprising and I’m not sure how much it works.

[jelmd]

Hmmm - I put the following into ${STACK_ROOT}/config.yaml

package-indices:
- name: HackageOrig
  download-prefix: https://hackage.haskell.org/package/
  http: https://hackage.haskell.org/00-index.tar.gz

But stack setup still does not work. It stops with:

Preparing to install GHC to an isolated location.
This will not interfere with any system-level installation.
Preparing to download ghc-8.0.2 ...HttpExceptionRequest Request {
  host                 = "github.com"
  port                 = 443
  secure               = True
  requestHeaders       = []
  path                 = "/commercialhaskell/ghc/releases/download/ghc-8.0.2-release/ghc-8.0.2-x86_64-deb7-linux.tar.xz"
  queryString          = ""
  method               = "GET"
  proxy                = Nothing
  rawBody              = False
  redirectCount        = 10
  responseTimeout      = ResponseTimeoutDefault
  requestVersion       = HTTP/1.1
}
 (InternalException (HostCannotConnect "github-production-release-asset-2e65be.s3.amazonaws.com" [Network.Socket.connect: <socket: 13>: does not exist (Connection refused)]))

Using s3.amazonaws.com is a really, really bad idea!

[Blaisorblade]

FWIW, your log shows a timeout downloading GHC from GitHub, and GitHub’s using S3 — it’s not just GitHub. So, blocking S3 seems a really bad idea; nevertheless, I agree that Stack (and maybe GitHub?) should ideally support using mirrors. And stack does! So somebody should follow the example by Tsinghua.edu.cn (link below) and set up a mirror.

The GitHub URL comes from stack-setup-2.yaml, and customizing that requires using the setup-info config key (as used above). I suppose the config in https://zhuanlan.zhihu.com/p/25005809 might avoid this issue (I don’t read Chinese, but the config’s in English and was discussed above).

[xiaopong]

What's the status of this bad connection handling issue? This is a really important issue to resolve. This makes me avoid using stack whenever possible. I'm in Shanghai, so the default S3 repository is simply not accessible. I have http proxy to an "external" machine, but stack can't handle http proxy :( Sure, there's the Tsinghua mirror, it works find when stars are aligned, but a lot of times, I just get the same connection timeout issue.

Another issue with stack is, it does not know that a package is already downloaded. It downloads a package, runs into some issue unpacking it, and exits. You re-run the command, it will stubbornly tries to download again. And that will take forever, as you keep running into connection timeout. I've been stuck with this issue for two days now, trying to upgrade to ghc-8.4.3.

[flip111]

I like to use stack -- online with a bad connection --

Using it offline is not a good solution because then it doesn't check for newer versions and i have to remember to update it all the time. Fallback to offline version when connection is down is acceptable though.

Please reopen because i don't think offline is the solution for this problem.

[mihaimaruseac]

One of the reasons for pantry is to support multiple mirrors and offline support.