Stack: Find system ghc via binary names like "ghc-7.10.3"?

[mgsloan]

In https://github.com/commercialhaskell/stack/issues/2221, we're planning to switch away from using system ghc by default. We could consider trusting that the correct version is installed and skipping the version check.

It turns out that for non system ghcs it is also potentially possible to skip the version check. Someone on IRC mentioned that binaries named things like ghc-7.10.3 exist, and that stack should use these. Skipping the check should save a bit of execution time, potentially something like a tenth of a second.

This would mean not gracefully handling cases like https://github.com/commercialhaskell/stack/issues/2430 . One option would be to perform such sanity checks after a failing build, if the Cabal errors for these cases are insufficient. This way, the user gets feedback quickly. For failing builds, it will wait a bit longer before exiting, to run the sanity checks.

[Blaisorblade]

I think calling ghc-X.X.X, for all GHCs would be huge and would allow having multiple GHCs on the paths; this would make life simpler for all users with trouble with bindists. Right now you can put a GHC path in the stack.yaml, but the result can't be distributed to users typically, since paths are valid locally (except for some scenarios using Docker). Supporting multiple system GHCs, otherwise, would require reading from stack's config a mapping from GHC version to path.

I admit I'm fuzzy on what exactly you have in mind (see below). Interpreting rather liberally, I imagine a few parts to this:

• [ ] call ghc-X.X.X instead of ghc everywhere;
• [ ] same for ghcjs assuming it works (I guess so but I'm not sure);
• [ ] add sanity checks in case of error (maybe just ghc --version, possibly also building and running a minimal program?);
• [ ] also revise sanity checks after GHC setup;
• [ ] remove version check at the start of a build.

On the rest, I'm confused (which is why I didn't answer a week ago). Could you elaborate?

I'm less sure about the changes you propose to the version check, probably because I didn't finish reverse-engineering how it works. In particular, we often don't even call ghc --version but ghc --info. Also, why do we have a version for non system GHCs at all?

In #2221, we're planning to switch away from using system ghc by default. We could consider trusting that the correct version is installed and skipping the version check.

I'm more confused by the link with #2221. Doesn't the version check only make sense for the system GHC? And then _that_ could be dropped by using ghc-X.X.X when looking for a system GHC.

[mgsloan]

same for ghcjs assuming it works (I guess so but I'm not sure);

It works less well for ghcjs right now, because there are a lot of different tarballs all with version 0.2.0. Most tarballs have dates in their name, but many don't have the version number set to something other than 0.2.0.

I'm less sure about the changes you propose to the version check, probably because I didn't finish reverse-engineering how it works. In particular, we often don't even call ghc --version but ghc --info.

Good point, I haven't looked deeply at those details. Could be we can't avoid that. We could cache it?

Also, why do we have a version for non system GHCs at all?

Not sure, I think because it's a mild sanity check, shouldn't be needed though.

I'm more confused by the link with #2221. Doesn't the version check only make sense for the system GHC? And then that could be dropped by using ghc-X.X.X when looking for a system GHC.

If we don't change the default then we'll always have to check the system ghc version.

[Blaisorblade]

If we don't change the default then we'll always have to check the system ghc version.

I think that even if you change the default you'll need some check.
Say I reenable system GHC—and I do that by setting system-ghc: true globally.
That doesn't mean you can use my system GHC for all resolvers, so you still need to learn which is the version of the system GHC.

OTOH, I understand all GHC bindists also provide binaries named ghc-X.X.X, not just the one stack installs. I propose to rely on this _instead_ of the version check, by default. Of course this changes the logic from

• learn GHC's version and match it with what I need
  to
• look for a system GHC of the version I need.
  I assume this is still doable, but might be more invasive.

[mgsloan]

Say I reenable system GHC—and I do that by setting system-ghc: true globally.
That doesn't mean you can use my system GHC for all resolvers, so you still need to learn which is the version of the system GHC.

True!

OTOH, I understand all GHC bindists also provide binaries named ghc-X.X.X, not just the one stack installs. I propose to rely on this instead of the version check, by default.

Yeah, maybe we can just go ahead and always trust it for ghc. In theory the logic change shouldn't affect users

[bitemyapp]

@mgsloan how would this work for the various tools installed with GHC, do they all get suffixed? Some builds are going to fail if not using the associated tool matched to the GHC version.

[Blaisorblade]

Thanks for pointing that out! We should surely use the same version for any invocation of ghc-pkg, runghc and haddock-ghc. Do we need to account for other tools? I'm guessing no for alex and happy (whose versions should be controlled otherwise) but I'm no expert there.

$ ls /Applications/ghc-7.8.4.app/Contents/bin/
cabal  ghc  ghc-7.8.4  ghc-pkg  ghc-pkg-7.8.4  ghci  ghci-7.8.4  haddock  haddock-ghc-7.8.4  hp2ps  hpc  hsc2hs  runghc  runghc-7.8.4  runhaskell
$ ls /usr/local/Cellar/ghc/8.0.1/bin/
ghc  ghc-8.0.1  ghc-pkg  ghc-pkg-8.0.1  ghci  ghci-8.0.1  haddock  haddock-ghc-8.0.1  hp2ps  hpc  hsc2hs  runghc  runghc-8.0.1  runhaskell

[mgsloan]

That is indeed a good point! That makes this a bit trickier, but still a good change I hope.

[bitemyapp]

@mgsloan @Blaisorblade not to derail the intent of the thread, but wouldn't inter-operability with tools expecting ordinary names for these utilities be better served by doing the paths like @hvr's GHC distributions? It puts the version in a subdirectory of a consistent path (/opt/...), you can then use the path to switch between them and regular tooling "just works"

This is roughly what rustup does (path blipping) as well.

[mgsloan]

@bitemyapp Certainly! However, that does not work with standard global installations of GHC. Are they so tied to happy / alex versions?

Stack implements its own approach for switching sub-directories, which is why we haven't had logic like this.

[bitemyapp]

However, that does not work with standard global installations of GHC.

Not certain what you mean. Users of hvr's PPA usually just pick a version and get on with it.

Are they so tied to happy / alex versions?

I've seen incompatible haddock binaries break builds. I am very impatient with unnecessary build breakage.

[mgsloan]

Not certain what you mean. Users of hvr's PPA usually just pick a version and get on with it.

It seems that some users, such as @simonmar do a global install rather than that approach. Their usecase is quite valid, though indeed it may open a can of worms, I'm not certain.

I've seen incompatible haddock binaries break builds. I am very impatient with unnecessary build breakage.

Agreed. You will be entirely safe from this if you have the appropriate GHC installed by stack. I'd recommend not even having a global GHC install.

We are definitely planning to make global GHC a non-default thing in the next release - https://github.com/commercialhaskell/stack/pull/2537

[bitemyapp]

It seems that some users, such as @simonmar do a global install rather than that approach. Their usecase is quite valid, though indeed it may open a can of worms, I'm not certain.

You mean putting it somewhere everything can find it? Yeah, I do that with hvr's thing via symlinks. I used to even have a script I would pass the path of the install to, and then it would symlink into /usr/bin for me.

Agreed. You will be entirely safe from this if you have the appropriate GHC installed by stack. I'd recommend not even having a global GHC install.

Having bumped into --*-profiling, yeah.

We are definitely planning to make global GHC a non-default thing in the next release - #2537

Very much looking forward to it :+1: :)

[mgsloan]

Great!

Note that you can already adopt this default by just putting system-ghc: false in your config.yaml.

[bitemyapp]

Fool me twice... https://github.com/bitemyapp/dotfiles/blob/master/.stack/config.yaml#L5

[mgsloan]

So, @bitemyapp , are you saying that stack exec -- which ghc is reporting a system ghc despite having system-ghc: false? That's very surprising!

[bitemyapp]

No, nothing's wrong, I'm saying I set that configuration quite awhile ago and I think I was one of the earlier people agitating for making system-ghc: false the default (click one of the referenced issues)

just happy it's getting fixed :)

[simonmar]

Can we have an option to tell Stack the path of the GHC binary and have Stack call $GHC --version to find the version? That would basically solve all my problems, and seems easy to implement.

[mgsloan]

Yes, it is easy to implement @simonmar , just an oversight due to our typical usecases using official GHC versions. No promises, but we are planning to get it into the next release, likely a couple weeks from now if not sooner.

[mgsloan]

@simonmar I've looked into implementing an option like that. It'd actually be a bit tricky to support - lets say the user provides --with-ghc /home/mgsloan/oss/haskell/ghc/inplace/bin/ghc-stage2. Should it assume that ghc-pkg is a sibling?

Another approach I've considered is a --with-compiler-bin (to match stack path --compiler-bin), to specify a dir to find exes like ghc / ghc-pkg. This is nearly equivalent to extending the PATH, though. The only advantage to having an explicit option is that we could check that we're actually using the GHC from that path. Is it worth it?

Perhaps in the meantime it's sufficient to just extend the PATH? If any renaming needs to be done (e.g. ghc-stage2), symbolic links could be used.

[hvr]

@mgsloan Note that cabal has been supporting -w SOMEPATH/ghc-stage2 for ages as the primary way to select a GHC instance and uses heuristics to find the matching ghc-pkg so it can't be too hard ;-)

[fommil]

~I maintain all my ghc installs in separate directories and use a shell script to automatically select the correct ghc based on the presence of a .ghc-version file (this is how java and javac is managed in JVM languages, btw).~

~It would be good if I could provide a lookup map for each ghc that stack expects to find, to the directory where I have installed it. I could workaround the ghc-X.Y.Z approach with symlinks, if needed, but I'd rather not.~

I use ghcup now.

[dbaynard]

@mgsloan — is this still relevant? And if so, what would be needed to resolve the issue?

[mgsloan]

Yes, I believe it is still relevant. It's too bad that it didn't get resolved previously. I think there's two things:

1. Allow indicating a path to a GHC - something like cabal's -w option described by @hvr. Would also need to resolve the path to associated tools such as ghc-pkg.

2. Finding binaries called things like ghc-7.10.3. I don't know the details of this, I never use globally installed GHCs so ¯\_(ツ)_/¯

I have been doing a bit of work on GHC lately, so it's on my todo list to make it easier to use stack with a dev version of GHC. Of course, I encourage anyone else to work on that. Not sure when I'd get around to it.

[snoyberg]

I've opened up #4668 which addresses this. I've included an integration test and tested it manually on my machine, but it would be great if someone who has the use case described here could test out that PR.

[ygale]

In my opinion option 1 of @mgsloan is the more generally useful solution. That is #3931. Option 2 might be helpful for some people who don't mind playing havoc with their PATH. #4668 only addresses option 2.

[snoyberg]

I'm honestly not following the use case for option 1. If you want to manually state which GHC to use on each invocation, you can call Stack with PATH=$GHCDIR:$PATH stack .... The feature in Stack that we're talking about is auto-discovery of the correct GHC location.

Maybe I'm missing something fundamental here.

[Blaisorblade]

@snoyberg Well, @ygale objected elsewhere to modifying PATH, but their arguments mystify me.
I imagine maybe beginners would struggle with modifying PATH correctly, but we already have a base setup for beginners, so that doesn't seem to be a problem.

[snoyberg]

Thanks for clarifying @blaisorblade!

[simonmar]

Just to note, I think I also asked for option (1) a while ago to support using stack to test locally-built versions of GHC. As far as I'm aware that's not possible right now.