Stack: Optionally go to hackage.haskell.org instead of hackage.fpcomplete.com for source .gz files.

[philipcraig]

General summary/comments (optional)

Allow stack to use hackage.haskell.org instead of hackage.fpcomplete.com for source files.

The reason is that hackage.haskell.org doesn't mirror / gateway the link into an amazon s3 link, which is banned from big company firewalls. hackage,fpcomplete.com does.

Steps to reproduce

For example:

stack install clock

Wanted

Have an option such that stack downloads clock from https://hackage.haskell.org/package/clock-0.7.2/clock-0.7.2.tar.gz

Actual

Downloads clock from https://hackage.fpcomplete.com/package/clock-0.7.2.tar.gz which turns into an s3 request from amazon, which is banned by big companies.

Stack version

$ stack --version
Version 1.1.2, Git revision c6dac65e3174dea79df54ce6d56f3e98bc060ecc (3647 commits) x86_64 hpack-0.14.0

### Method of installation

* Official binary, downloaded from stackage.org

[Blaisorblade]

The docs is unclear enough I had to check the sources and I'm _still_ not sure, but we might be supporting this already. Based on https://github.com/commercialhaskell/stack/blob/master/doc/yaml_configuration.md#package-indices, can you adding to your config in ~/.stack/config.yaml the following?

package-indices:
- name: Hackage
  download-prefix: https://hackage.haskell.org/package/
  git: https://github.com/commercialhaskell/all-cabal-hashes.git

If that works, this is a documentation bug.

[philipcraig]

Hi,

Thanks for the info, but no joy yet.

I've tried adding that in three places:

STACK_ROOT/config.yaml
STACK_ROOT/global-project/stack.yaml
project level stack.yaml

In each, it seems to make no difference to behaviour, so, e.g.

"stack install clock"

or adding a clock dependency results in a failed trip to s3, as before.

[Blaisorblade]

Can you post a complete log of stack install -v clock?

[philipcraig]

C:\hsdev\foo>stack install -v clock
Version 1.1.2, Git revision c6dac65e3174dea79df54ce6d56f3e98bc060ecc (3647 commits) x86_64 hpack-0.14.0
2016-08-18 20:09:26.459270: [debug] Checking for project config at: C:\hsdev\foo\stack.yaml @(stack_9kewhubnl5WIl89fhd1ea2:Stack.Config src/Stack\Config.hs:811:9)
2016-08-18 20:09:26.461270: [debug] Loading project config file stack.yaml @(stack_9kewhubnl5WIl89fhd1ea2:Stack.Config src/Stack\Config.hs:829:13)
2016-08-18 20:09:26.465270: [debug] Trying to decode C:\Stack\build-plan-cache\x86_64-windows\lts-6.12.cache @(stack_9kewhubnl5WIl89fhd1ea2:Data.Binary.VersionTagged src/Data\Binary\VersionTagged.hs:55:5)
2016-08-18 20:09:26.474271: [debug] Success decoding C:\Stack\build-plan-cache\x86_64-windows\lts-6.12.cache @(stack_9kewhubnl5WIl89fhd1ea2:Data.Binary.VersionTagged src/Data\Binary\VersionTagged.hs:64:13)
2016-08-18 20:09:26.475271: [debug] Getting system compiler version @(stack_9kewhubnl5WIl89fhd1ea2:Stack.Setup src/Stack\Setup.hs:341:17)
2016-08-18 20:09:26.477272: [debug] Run process: ghc --info @(stack_9kewhubnl5WIl89fhd1ea2:System.Process.Read src/System\Process\Read.hs:283:3)
2016-08-18 20:09:26.669291: [debug] Asking GHC for its version @(stack_9kewhubnl5WIl89fhd1ea2:Stack.Setup.Installed src/Stack\Setup\Installed.hs:94:13)
2016-08-18 20:09:26.670291: [debug] Run process: ghc --numeric-version @(stack_9kewhubnl5WIl89fhd1ea2:System.Process.Read src/System\Process\Read.hs:283:3)
2016-08-18 20:09:26.786302: [debug] Getting Cabal package version @(stack_9kewhubnl5WIl89fhd1ea2:Stack.GhcPkg src/Stack\GhcPkg.hs:165:5)
2016-08-18 20:09:26.787303: [debug] Run process: ghc-pkg --no-user-package-db field --simple-output Cabal version @(stack_9kewhubnl5WIl89fhd1ea2:System.Process.Read src/System\Process\Read.hs:283:3)
2016-08-18 20:09:26.985322: [debug] Resolving package entries @(stack_9kewhubnl5WIl89fhd1ea2:Stack.Setup src/Stack\Setup.hs:221:5)
2016-08-18 20:09:26.987323: [debug] Getting global package database location @(stack_9kewhubnl5WIl89fhd1ea2:Stack.GhcPkg src/Stack\GhcPkg.hs:48:5)
2016-08-18 20:09:26.987323: [debug] Run process: ghc-pkg --no-user-package-db list --global @(stack_9kewhubnl5WIl89fhd1ea2:System.Process.Read src/System\Process\Read.hs:283:3)
2016-08-18 20:09:27.137338: [debug] Run process: ghc-pkg --global --no-user-package-db dump --expand-pkgroot @(stack_9kewhubnl5WIl89fhd1ea2:System.Process.Read src/System\Process\Read.hs:283:3)
2016-08-18 20:09:27.345358: [debug] Ignoring package haskeline due to wanting version 0.7.2.3 instead of 0.7.2.1 @(stack_9kewhubnl5WIl89fhd1ea2:Stack.Build.Installed src/Stack\Build\Installed.hs:189:5)
2016-08-18 20:09:27.346358: [debug] Ignoring package Cabal due to wanting version 1.22.8.0 instead of 1.22.5.0 @(stack_9kewhubnl5WIl89fhd1ea2:Stack.Build.Installed src/Stack\Build\Installed.hs:189:5)
2016-08-18 20:09:27.349359: [debug] Run process: ghc-pkg --user --no-user-package-db --package-db C:\Stack\snapshots\2bcf57b6\pkgdb dump --expand-pkgroot @(stack_9kewhubnl5WIl89fhd1ea2:System.Process.Read src/System\Process\Read.hs:283:3)
2016-08-18 20:09:27.494373: [debug] Run process: ghc-pkg --user --no-user-package-db --package-db C:\hsdev\foo\.stack-work\install\cf5a45a3\pkgdb dump --expand-pkgroot @(stack_9kewhubnl5WIl89fhd1ea2:System.Process.Read src/System\Process\Read.hs:283:3)
2016-08-18 20:09:27.686392: [debug] Trying to decode C:\Stack\indices\Hackage\00-index.cache @(stack_9kewhubnl5WIl89fhd1ea2:Data.Binary.VersionTagged src/Data\Binary\VersionTagged.hs:55:5)
2016-08-18 20:09:27.880412: [debug] Success decoding C:\Stack\indices\Hackage\00-index.cache @(stack_9kewhubnl5WIl89fhd1ea2:Data.Binary.VersionTagged src/Data\Binary\VersionTagged.hs:64:13)
2016-08-18 20:09:27.898414: [debug] Getting global package database location @(stack_9kewhubnl5WIl89fhd1ea2:Stack.GhcPkg src/Stack\GhcPkg.hs:48:5)
2016-08-18 20:09:27.899414: [debug] Run process: ghc-pkg --no-user-package-db list --global @(stack_9kewhubnl5WIl89fhd1ea2:System.Process.Read src/System\Process\Read.hs:283:3)
2016-08-18 20:09:28.039428: [debug] Precompiled cache input = ["--dependency=base=base-4.8.2.0-14035a44a8b95c6832da6dae1420f59e"] @(stack_9kewhubnl5WIl89fhd1ea2:Stack.Build.Cache src/Stack\Build\Cache.hs:261:13)
2016-08-18 20:09:28.043428: [debug] Downloading /hackage.fpcomplete.com/package/clock-0.7.2.tar.gz @(stack_9kewhubnl5WIl89fhd1ea2:Network.HTTP.Download.Verified src/Network\HTTP\Download\Verified.hs:231:9) ProxyConnectException "s3.amazonaws.com" 443 (Right (StatusCodeException (Status {statusCode = 403, statusMessage = "Forbidden"}) [] (CJ {expose = []})))

[Blaisorblade]

Thanks! I fixed formatting using ```.

[Blaisorblade]

Thanks. I could reproduce this, and it basically seems that beyond changing the setting, you'll need to remove C:\Stack\build-plan-cache\x86_64-windows\lts-6.12.cache, since it caches even the URL to use!

To be safer, I'd also rename the package index by changing the snippet to:

package-indices:
- name: HackageOrig
  download-prefix: https://hackage.haskell.org/package/
  git: https://github.com/commercialhaskell/all-cabal-hashes.git

You want this only in STACK_ROOT/config.yaml (that is for you, IIUC, C:\stack\config.yaml).

[Blaisorblade]

EDIT: that's not enough it seems... I'll try more carefully.

[Blaisorblade]

So I think I managed. The config I needed was the following:

package-indices:
- name: Hackage
  download-prefix: https://hackage.haskell.org/package/
  http: https://hackage.haskell.org/00-index.tar.gz

The problem is that the all-cabal-hashes repository already contains two resolved URLs, listing both mirrors (from Amazon and from Hackage).

EDIT: the evidence I have is this log entry:

2016-08-18 22:09:30.351111: [debug] Downloading /package/clock-0.7.2.tar.gz @(stack_JAcs0EuAGbK5rtqRdoxQaL:Network.HTTP.Download.Verified src/Network/HTTP/Download/Verified.hs:231:9)

which prints the correct path (hostnames aren't included, see below).

Also, sorry @philipcraig for fooling myself so often, I hope I didn't again.

Technical details

For instance

$ git ls-tree current-hackage clock/0.7.2/clock.json
100644 blob 8d43ab2cc7a4cd4c7cccde2f00386390d9d5a13f    clock/0.7.2/clock.json
$ git cat-file -p 8d43ab2cc7a4cd4c7cccde2f00386390d9d5a13f
{"package-hashes":{"MD5":"3bb91ede556cde92a64c635da68699cd","Skein512_512":"98cdeff213b26ec5ad28249558df25446e1811bc421bf1ce9589035a6754ff57a03863cd7792b15d78ea70a85b6bae977009c2f4616a9dcc576550705db8a6cd","SHA1":"9ab44e86f04cd76eaf9cb0acea2223c1361ba931","SHA512":"cf44dfca6c0825a003018c68266ef0fe503a9b147aaf784435f6297b365c6d18fc71b63b77a23ee2a3298ace638a35de0c6118e361d43b418a45d4fdd24d57cf","SHA256":"886601978898d3a91412fef895e864576a7125d661e1f8abc49a2a08840e691f"},"package-locations":["https://hackage.haskell.org/package/clock-0.7.2/clock-0.7.2.tar.gz","https://s3.amazonaws.com/hackage.fpcomplete.com/package/clock-0.7.2.tar.gz"],"package-size":7465}

Of the URLs, stack insists using https://s3.amazonaws.com/hackage.fpcomplete.com/package/clock-0.7.2.tar.gz. The log says Downloading /hackage.fpcomplete.com/package/clock-0.7.2.tar.gz, but that's because there's no good way to print the URLs we parse (see #2431 and #2267).

[philipcraig]

It works! Thanks so much.

I'm not quite good to go yet though -- even with this config, if I want stack to setup ghc via stack setup, it still goes to s3 to download ghc.

If I give it a ghc that I already have, it's fine. The test is, I was able to stack install clock.

I can use --stack-setup-yaml to give it a customised stack-setup.yaml file, which doesn't point to s3 even for the setup ghc files.

However, in general use, I then get bitten by #2248. And I can't do a stack upgrade --git, as the stack upgrade process itself gets hit by the same issue.

[Blaisorblade]

Is there a way of persuading stack setup to get ghc from [...]

It seems yes, but I'm not sure how it exactly works (ahem, still new here). As long as that's a standard GHC bindist.

Looking at stack setup --help I see a --ghc-bindist which is made for you, but I'm not 100% sure how to use it. It mentions --ghc-variant (a global option, see stack --help). Maybe there are docs explaining this better?

Something is explained here, in "Support for custom GHC binary distributions". Maybe that helps?
https://github.com/commercialhaskell/stack/blob/master/ChangeLog.md#0150

(Alternatively, one could --stack-setup-yaml with the URL for a variant of https://raw.githubusercontent.com/fpco/stackage-content/master/stack/stack-setup-2.yaml, using the right URL, but that might be more annoying to do).

[Blaisorblade]

BTW, I consider even the part of the "solution" that worked as a hack.

[philipcraig]

Thanks. What would you consider not to be a hack? Removing the need to delete the cache file, or something else?

Also note that I still have #2248 in real-world use, and that stack upgrade --git when using hackage via the above settings also doesn't work due to, er, #2248.

Net result: can't yet use stack.

[Blaisorblade]

Also note that I still have #2248 in real-world use, and that stack upgrade --git when using hackage via the above settings also doesn't work due to, er, #2248.

Ah thanks, I've just seen that error too. I assume that would also happen if you use stack build from a git checkout, with a binary stack. It looks like you'd need a (64bit) stack binary. I have a Win10 64bit VM, but no clue about code signing.

Thanks. What would you consider not to be a hack? Removing the need to delete the cache file, or something else?

That might have been a shot in the dark—I'm not yet sure of the entire logic, and not sure of which steps are needed.

A problem is that, IIUC, you can't use git: https://github.com/commercialhaskell/all-cabal-hashes.git at all.
Hence, you download the index not incrementally but from scratch each time (like cabal does, so it's not _so_ bad).

Why? You can't use that repo because it embeds actual URLs (an example is above in https://github.com/commercialhaskell/stack/issues/2509#issuecomment-240844481)—though it contains multiple URLs for each package, _both_ Hackage and FPComplete. And for some reason it picks the FPComplete one, apparently always. One could maybe allow picking between them at least. But that's like a denormalized database, because it hardcodes a list of two mirrors. And that makes me unhappy.

Looking at the spec for that repo (https://github.com/commercialhaskell/commercialhaskell/wiki/Git-backed-Hackage-index-signing-and-distribution), It seems we'd want to restrict it to relative URLs.

[philipcraig]

You are right on all points.

[Blaisorblade]

So here's an unofficial Stack binary, for Win64, from current master (d5b8c6ef6740e280e73604a4e04d1d58c5b14413), built on a MS Windows 10 VM. I suppose this is not up to code, and it's not signed, but it should avoid #2248 and get you going. You might want to do a stack upgrade --git to be safe.

https://www.dropbox.com/s/ti108qrvx1w83vb/stack-d5b8c6ef6740e280e73604a4e04d1d58c5b14413-win64.zip?dl=0

Note: I'll take this down soon.

Version output:

$ /c/Users/IEUser/Documents/stack/.stack-work/install/a604a5e3/bin/stack --version
Version 1.1.3, Git revision d5b8c6ef6740e280e73604a4e04d1d58c5b14413 (dirty) (4014 commits) x86_64 hpack-0.14.0

@sjakobi should we have nightlies instead? (I'm also happy with more common releases).

[sjakobi]

should we have nightlies instead? (I'm also happy with more common releases).

Nightly builds might be a good way to get more beta testers who could help us test new features and detect regressions.

@borsboom, your thoughts on this as stack's release manager?!

[philipcraig]

Thanks @Blaisorblade . That worked.

[Blaisorblade]

Regarding custom GHC URLs, we do have decent docs:
https://docs.haskellstack.org/en/latest/yaml_configuration/#ghc-variant
https://docs.haskellstack.org/en/latest/yaml_configuration/#setup-info

[philipcraig]

So, when could we expect nightlies? @borsboom

cheers.

[Blaisorblade]

FTR: @snoyberg pointed out we could fallback to Hackage when Stackage downloads fail, without changes in the repos.

[borsboom]

Nightlies would be nice; it's a matter of putting in the effort to get them set up (and decide which platforms they should be built for). A separate issue would make sense.