Haskell-ide-engine: macos release seems broken

[2mol]

Hi folks!

I was trying to troubleshoot HIE for somebody who reported issues getting it to work (for vscode) at https://github.com/alanz/vscode-hie-server/issues/126

Downloading the compiled release (yay for having that option now btw) will not work, seemingly some places have @bubba s path compiled in (/Users/luke/Source/haskell-ide-engine/.stack-work/install/x86_64-osx/...). See attached 4-Haskell HIE (pboy).log

I ran into some other smaller issues as well, I think polishing those up could make the installation experience accessible to a lot more people. I volunteer to work on documentation if we can agree on what to improve.

• it wasn't obvious to me how I can peek into the logs between hie and vscode. What's the recommended way? Found the logs nicely accessible in a dropdown under the output pane of vscode. Maybe we could put this in the Readme under the troubleshooting section at the bottom?
• it seems the hie executable is used instead of hie-wrapper, despite my impression that the latter would be used preferentially. On that note: Why have the hie executable there in the first place?
• I saw a complaint in the logs about hlint.yaml not being found, it's unclear if that was a warning or an error.

[alanz]

cc @bubba

[Rhywun]

I tried to install v0.4.0.1 over v0.2.2.0 using the instructions under "Installation with stack" and something broke on this recommended instruction (after using brew to upgrade stack):

stack install cabal-install

Got "ExitFailure 2" trying to install GHC 8.6.2 but no indication of where to look for a log to explain it, so I gave up. Also, I broke HIE on my machine. I may try to start from scratch but will follow along here to see if anything develops.

Update - I am trying a clean install and made it past installing cabal-install but the same failure occurred after running make build-all in the HIE source directory:

$ make build-all
Downloading nightly-2018-12-01 build plan ...
Downloaded nightly-2018-12-01 build plan.
Preparing to install GHC to an isolated location.
This will not interfere with any system-level installation.
Preparing to download ghc-8.6.2 ...
ghc-8.6.2: download has begun
ghc-8.6.2: 12.97 MiB / 221.33 MiB ( 5.86%) downloaded...
-etc.-
ghc-8.6.2: 221.33 MiB / 221.33 MiB (100.00%) downloaded...
Downloaded ghc-8.6.2.
Unpacking GHC into /Users/me/.stack/programs/x86_64-osx/ghc-8.6.2.temp/ ...
Configuring GHC ...
Installing GHC ...
Received ExitFailure 2 when running
Raw command: /usr/bin/make install
Run from: /Users/me/.stack/programs/x86_64-osx/ghc-8.6.2.temp/ghc-8.6.2/

Preparing to install GHC to an isolated location.
This will not interfere with any system-level installation.
Preparing to download ghc-8.6.2 ...
Already downloaded.
Unpacking GHC into /Users/pat/.stack/programs/x86_64-osx/ghc-8.6.2.temp/ ...
Configuring GHC ...
Installing GHC ...
Received ExitFailure 2 when running
Raw command: /usr/bin/make install
Run from: /Users/me/.stack/programs/x86_64-osx/ghc-8.6.2.temp/ghc-8.6.2/

git submodule sync
Synchronizing submodule url for 'submodules/HaRe'
Synchronizing submodule url for 'submodules/brittany'
Synchronizing submodule url for 'submodules/cabal-helper'
Synchronizing submodule url for 'submodules/ghc-mod'
git submodule update --init
stack install cabal-install
Preparing to install GHC to an isolated location.
This will not interfere with any system-level installation.
Already downloaded.
Installing GHC ...
Received ExitFailure 2 when running
Raw command: /usr/bin/make install
Run from: /Users/me/.stack/programs/x86_64-osx/ghc-8.6.2.temp/ghc-8.6.2/

make: * [cabal] Error 1

Again I don't know where to find a log file that might explain what went wrong.

[2mol]

Hi @Rhywun! Let's split this into a separate issue since the cause will be different from the one affecting the precompiled binary release. I'll then try to redo my installation step by step and see if I can help your case!

[Rhywun]

@2mol

Done, thanks

[bubba]

Update so far: Since the hie binary relies on the libexec and data-dir paths generated by cabal, we'll probably need an install script or something for the binaries to replace these paths. cabal-helper relies on libexec for cabal-helper-wrapper and hlint has some share files, but cabal-helper-wrapper has gone away in recent versions of cabal-helper so we might not need it.

[jasonmcole]

The specific error message I got from make was:

dyld: Library not loaded: /usr/local/opt/gmp/lib/libgmp.10.dylib
  Referenced from: /Users/jasoncole/.stack/programs/x86_64-osx/ghc-8.6.2.temp/ghc-8.6.2/libraries/base/dist-install/build/libHSbase-4.12.0.0-ghc8.6.2.dylib
  Reason: image not found
make[1]: *** [install_packages] Abort trap: 6
make: *** [install] Error 2

Installing gmp with brew install gmp , then re-running stack ./install.hs build-all proceeded without issue.

[Anrock]

@jasonmcole that's known issue with ghc on osx. See README and ghc bug. Affects 8.6.1 and 8.6.2, fixed in 8.6.3 iirc.

[dboardmanrally]

Running the 8.6.4 install script attempts to install ghc 8.6.2 on mac OSX

$ stack ./install.hs hie-8.6.4
stack ./install.hs build-doc-8.6.4

Installing GHC ...
Received ExitFailure 2 when running
Raw command: /usr/bin/make install
Run from: /Users/drew.boardman/.stack/programs/x86_64-osx/ghc-8.6.2.temp/ghc-8.6.2/

Installing GHC ...
Received ExitFailure 2 when running
Raw command: /usr/bin/make install
Run from: /Users/drew.boardman/.stack/programs/x86_64-osx/ghc-8.6.2.temp/ghc-8.6.2/

[fendor]

Yes, the install script uses ghc 8.6.2.
If this causes troubles for you, you can change the resolver in shake.yaml and in install.hs.
That problem, that you have to update two places for this, is currently being worked on and will be resolved soon.

[mouse07410]

@fendor could you please tell where in install.hs is the dependency on ghc-8.6.2? I could not find it.

[mouse07410]

@fendor Here are some logs from failed runs. Help would be greatly appreciated. This is how it fails:

$ stack ./install.hs cabal-hie-8.6.4 -j1 2>&1 | tee cabal-hie-8.6.4b.txt
. . . . .
Installing   ghc-mod-core-5.9.0.0 (lib)
Installing   brittany-0.11.0.0 (lib)
Completed    ghc-mod-core-5.9.0.0 (lib)
Completed    brittany-0.11.0.0 (lib)
Installing   haskell-lsp-types-0.9.0.0 (lib)
Completed    haskell-lsp-types-0.9.0.0 (lib)
Warning: Some package(s) failed to build. Try rerunning with -j1 if you can't
see the error.
Warning: Failed to build documentation for haddock-api-2.22.0 (which is
required by exe:hie-wrapper from haskell-ide-engine-0.8.0.0,
lib:hie-test-utils from haskell-ide-engine-0.8.0.0 and others).
Symlinking 'hie'
Symlinking 'hie-wrapper'
Error when running Shake build system:
  at want, called at src/Development/Shake/Internal/Args.hs:81:69 in shake-0.17.7-25sVUgG7MLNHw9RWEZuoHL:Development.Shake.Internal.Args
* Depends on: cabal-hie-8.6.4
  at need, called at src/Development/Shake/Internal/Derived.hs:92:5 in shake-0.17.7-25sVUgG7MLNHw9RWEZuoHL:Development.Shake.Internal.Derived
* Depends on: /Users/ur20980/.local/bin/hie
  at error, called at src/Development/Shake/Internal/Rules/File.hs:179:58 in shake-0.17.7-25sVUgG7MLNHw9RWEZuoHL:Development.Shake.Internal.Rules.File
* Raised the exception:
Error, file does not exist and no rule available:
  /Users/ur20980/.local/bin/hie

Here are the complete logs:
cabal-build-all.txt
cabal-hie-8.6.4.txt
cabal-hie-8.6.4a.txt
cabal-hie-8.6.4b.txt

$ ghcup debug-info
Script version: 0.0.7

Script variables:
  GHC install location:     /Users/ur20980/.ghcup/ghc
  Binary install location:  /Users/ur20980/.ghcup/bin
  Tarball cache location:   /Users/ur20980/.ghcup/cache
  Downloader:               curl --fail -O <url>
  Script update url:        https://raw.githubusercontent.com/haskell/ghcup/master//ghcup
  GHC download baseurl:     https://downloads.haskell.org/~ghc
  Meta download url         https://raw.githubusercontent.com/haskell/ghcup/master//.download-urls
  Meta download format      1
  Meta version url          https://raw.githubusercontent.com/haskell/ghcup/master//.available-versions
  Meta version format       1

Detected system information:
  Architecture:   x86_64
  Distribution:   Darwin
  Distro alias:   darwin
  Distro version: 18.5.0
$ ghc -V
The Glorious Glasgow Haskell Compilation System, version 8.6.4
$ cabal --version
cabal-install version 2.4.1.0
compiled using version 2.4.1.0 of the Cabal library 
$ stack --version
Version 1.9.3, Git revision 40cf7b37526b86d1676da82167ea8758a854953b (6211 commits) x86_64 hpack-0.31.1
$ haddock --version
Haddock version 2.22.0, (c) Simon Marlow 2006
Ported to use the GHC API by David Waern 2006-2008
$ 

[fendor]

The install.hs depends implicitly on ghc-8.6.2 via its resolver in shake.yaml. You can safely bump it to a newer snapshot.

[mouse07410]

@fendor thanks. Edited shake.yaml, and got the build succeed on one machine (so far).

Could you explain the purpose of stack-8.6.x.yaml files vs. stack.yaml? Are they actually used? If so, how? I mean - do I need to adjust resolver in those that are version-relevant to my installation?

Here's config from the successful build - since two different resolvers seem specified, can you please tell me which one was actually used?

$ head stack-8.6.4.yaml 
resolver: lts-13.15 # GHC 8.6.4
packages:
- .
 . . . . .

$ head stack.yaml
resolver: nightly-2019-04-05 # GHC 8.6.4
packages:
- .
 . . . . .

$ head shake.yaml 
# Used to provide a different environment for the shake build script
#resolver: nightly-2018-12-15 # GHC 8.6.2
resolver: lts-13.15 # GHC 8.6.4
packages:
- .
 . . . . .

$ type hie
hie is hashed (/Users/ur20980/.local/bin/hie)
$ hie --version
Version 0.8.0.0 (2579 commits) x86_64 ghc-8.6.4
$ 

Oh, and I built the master branch via stack ./install.hs hie-8.6.4 (cabal builds still fail).

[samuelpilz]

The different stack.yaml files are used to build hie with different ghc versions. shake.yaml is used to run install.hs in an isolated snapshot.

At what version does the cabal build still fail? You could run the command manually with cabal new-install exe:hie -w <ghc-path> and check if any error is reported. (ghc-path used by install.hs can be found by calling stack install.hs cabal-ghcs)

[mouse07410]

The different stack.yaml files are used to build hie with different ghc versions...

So, if (for example) I use only ghc-8.6.3 or ghc-8.6.4, how should I run the build?

At what version does the cabal build still fail?

Please see the complete logs I posted a couple of messages up - they're from yesterday's failed builds. Cabal build failed with the current (latest stable) Cabal, stack, and GHC-8.6.4 (and I think - 8.6.3 too). I haven't tried running Cabal manually (yet).

$ stack ./install.hs cabal-ghcs

******************************************************************
Found the following GHC paths: 
ghc-8.6.4: /Users/ur20980/.ghcup/bin/ghc-8.6.4
ghc-8.6.3: /Users/ur20980/.ghcup/bin/ghc-8.6.3

******************************************************************

Build completed in 0.00s

P.S. As of now, I have successful builds of the current master on three machines, two with GHC-8.6.4, and one with GHC-8.6.3. All - via stack only (no cabal yet).

[mouse07410]

Also, I want to thank the developers/maintainers of this repo, whose hard work made the build (at least in one form) succeed now, when the previous versions failed.

[samuelpilz]

I have never seen your errors before. Please ensure a fresh repository:

git clean -Xdf && git pull # or do a fresh clone
git submodule sync && git submodule update --init

So, if (for example) I use only ghc-8.6.3 or ghc-8.6.4, how should I run the build?

The officially supported method to install these two versions only is stack install.hs hie-8.6.4 && stack install.hs hie.8.6.3 && stack install.hs build-doc.

if these fail with a similar error message I am interested in the output of stack build --stack-yaml=stack-8.6.4.yaml and cabal new-install --symlink-bindir=$HOME/.local/bin, which are the main commands to build hie, which the install.hs wraps.

[mouse07410]

I have never seen your errors before. Please ensure a fresh repository...

Thanks. Will try later, and report.

The officially supported method to install [for 8.6.4] ... is stack install.hs hie-8.6.4 ...

OK, thanks. Got it. In fact, that's how my installation succeeded.

stack install.hs build-doc

This doesn't work:

$ hie --version
Version 0.7.0.0 (2483 commits) x86_64 ghc-8.6.4
$ stack ./install.hs build-doc
Error when running Shake build system:
  at want, called at src/Development/Shake/Internal/Args.hs:83:69 in shake-0.17.6-CIDPVLgp7pwBLb5kEJCmoZ:Development.Shake.Internal.Args
* Depends on: build-doc
  at error, called at src/Development/Shake/Internal/Rules/File.hs:180:58 in shake-0.17.6-CIDPVLgp7pwBLb5kEJCmoZ:Development.Shake.Internal.Rules.File
* Raised the exception:
Error, file does not exist and no rule available:
  build-doc


$ stack ./install.hs build-docs
# stack (for build-doc-8.2.1)
Downloaded nightly-2017-11-24 build plan.    
Compiler version mismatched, found ghc-8.6.4 (x86_64), but expected minor version match with ghc-8.2.1 (x86_64) (based on resolver setting in /Users/ur20980/src/haskell-ide-engine/stack-8.2.1.yaml).
To install the correct GHC into /Users/ur20980/.stack/programs/x86_64-osx/, try running "stack setup" or use the "--install-ghc" flag.
Error when running Shake build system:
  at want, called at src/Development/Shake/Internal/Args.hs:83:69 in shake-0.17.6-CIDPVLgp7pwBLb5kEJCmoZ:Development.Shake.Internal.Args
* Depends on: build-docs
  at need, called at install.hs:80:25 in main:Main
* Depends on: build-doc-8.2.1
* Raised the exception:
user error (Development.Shake.command_, system command failed
Command: stack --stack-yaml=stack-8.2.1.yaml install hoogle
Exit code: 1
Stderr:
Downloaded nightly-2017-11-24 build plan.    
Compiler version mismatched, found ghc-8.6.4 (x86_64), but expected minor version match with ghc-8.2.1 (x86_64) (based on resolver setting in /Users/ur20980/src/haskell-ide-engine/stack-8.2.1.yaml).
To install the correct GHC into /Users/ur20980/.stack/programs/x86_64-osx/, try running "stack setup" or use the "--install-ghc" flag.
)


$ stack ./install.hs build-doc-8.6.4
# stack (for build-doc-8.6.4)
Copying from /Users/ur20980/src/haskell-ide-engine/.stack-work/install/x86_64-osx/lts-13.15/8.6.4/bin/hoogle to /Users/ur20980/.local/bin/hoogle

Copied executables to /Users/ur20980/.local/bin:
- hoogle
# stack (for build-doc-8.6.4)
Starting generate
Reading Cabal... 7.58s
[156/2343] data-ordlist... 0.01s                    
[284/2343] network-house... 0.05s             
[414/2343] regex-applicative... 0.03s 
. . . . .
[2232/2343] hledger-web... 0.53s        
[2271/2343] hoogle... 0.00s                
Packages missing documentation: abstract-deque-tests al array-memoize base-noprelude bin-package-db bno055-haskell bytestring-builder doctest-discover either-both ghc-heap h2c haskell-gi-overloading hfsevents inliterate interpolator io-manager ipython-kernel lame lens-simple linux-file-extents modular pattern-arrows proto-lens-protobuf-types rocksdb-haskell rocksdb-query rts secp256k1-haskell telegram-bot-simple tensors ucam-webauth ucam-webauth-types Win32-notify zeromq4-patterns
Found 48063 warnings when processing items

Reordering items... 0.28s
Writing tags... 2.29s
Writing names... 1.64s
Writing types... 9.38s
Took 1m45s
Build completed in 1m48s

$

Any idea why the successful doc build above complains about missing documentation?

stack build --stack-yaml=stack-8.6.4.yaml

Also, to clarify the use of stack-8.x.x.yaml: does install.hs actually use them when a specific GHC version is requested? What's the relationship between stack.yaml and stack-8.6.4.yaml?

[samuelpilz]

It seems that your repository could benefit from git pull

[mouse07410]

You may not believe me, but I'm doing "git pull" and "git submodule update --init" every time I start doing anything on this repo. Good hygiene, y'know...

I'd much rather that you explain how stack-8.x.x.yaml relates to stack.yaml, and whether both are used in the stack build.

[fendor]

hie --version hints to an old version of hie, since the current version is 0.8.0.0. Or there is an old hie version on the PATH.
The target build-docs does not exist on HEAD anymore.
Thus, your repository is either not up-to-date or you are not following the master branch.

To the relation, they are not related. stack.yaml is just using the nightly stackage snapshot. It is used for development and CI. The install script does afaik not use it at all.

[mouse07410]

Thank you - I just noticed that on one machine indeed it built (or used) an old version (0.7 instead of 0.8). I'll take care of that.

So, if I'm using "stack install.hs ..." and want to ensure the build uses a specific resolver and GHC-8.6.4, I need to edit shake.yaml and stack-8.6.4.yaml?

[fendor]

Afaik you may to edit the shake.yaml such that install.hs is run with the ghc version you wish. If you use stack install.hs hie-8.6.4 the correct stack*.yaml will be automatically used for the hie executable.

[mouse07410]

Great! But do I need to make sure stack-8.6.4.yaml doesn't pull a different resolver? Or tries to rewrite a different GHC?

[fendor]

No, the only additional resolver is pulled in by shake.yaml. Everything else should work as expected.

[mouse07410]

...or you are not following the master branch...

Indeed. That machine was on a different branch. Thanks for pointing that out. Just fixed.

$ hie --version
Version 0.8.0.0 (2589 commits) x86_64 ghc-8.6.4
$

[jneira]

This seems to be fixed, closing