Haskell-ide-engine: HIE depends on hlint.yaml and cabal-helper-wrapper in its build dir

[lyuinnhe]

Following the installing instructions, the hie ends up using libraries in the project direcotory(i.e. /.stack_work), supposingly they should be installed to the global stack project(i.e %STACK_ROOT%/snapshots)
This causes duplicating libraries in both locations, which is space ineffecient

[nponeccop]

Hie is linked statically. That is, the libraries aren't used at runtime, everything is in .local/bin/hie.

Many libraries are installed at .stack_work, because they don't belong to snapshots. But %STACK_ROOT%/snapshots should still contain some 350+ libs.

For 32-bit 8.0.2 check for example snapshots/i386-linux-nopie/lts-9.14/8.0.2/lib/i386-linux-ghc-8.0.2/ folder. It contains some 300 libs.

[lyuinnhe]

When i renamed the hie project directory, hie ended up not functioning
How is that supposed to be?

[lyuinnhe]

btw are you really sure?
cuz i got only 190 libs in the snapshot

[nponeccop]

I think 190 libs are normal. I may have libs there from other projects as well on my system. For most functions of hie to work you need only hie.exe on your %PATH%.

What exactly stopped working?

[lyuinnhe]

So im using the VS code integration.
when i rename the dir,
Linting stops working, ghc-mod doesnt respond, formatting doesnt work, go to def doesnt work even in the same file
basically the server doesnt respond anything and nothing works
I can check more if you could tell me where the log lies or how to work woth HIE under command line

[lyuinnhe]

Is it normal that with all libs statically linked, hie.exe is still only 119MB large?

[nponeccop]

Yes, the static libs due to various historical and performance reasons use a very inefficient format, so a very little fraction of them is code - the rest are section headers and so on.

119 is normal. 32-bit stack on linux is 44MBs.

You can check dynamic dependencies of a windows exe using this tool: http://dependencywalker.com/.
On linux it's ldd.

[nponeccop]

The log is in %temp%\hie.log on Windows.

To work with command line run hie.exe --lsp and see if it crashes or not. In this configuration it doesn't create any logs but VS code plugin passes extra arguments.

ghc-mod must work with only hie.exe, so let's focus on it.

Delete the log, open vscode and attempt to produce a ghc-mod error, and paste the log here. Also check that you don't have more than one hie.exe and your path doesn't contain subfolders of hie build folder.

[lyuinnhe]

The DepVerifier shows some dlls in c:windows and some not found api-ms-....dll and ext-ms-....dlls

[lyuinnhe]

It doesnt crash, and when run in command line, the output didnt change
The only difference is that it doesnt respond to the addons

[lyuinnhe]

https://gist.github.com/soylens/30096602d210e73793810fab1e117602
the 2 logs and the problematic code

[lyuinnhe]

@nponeccop

[lyuinnhe]

Is there an operation to "merge" the project to the global project?

[nponeccop]

It's 2 am here. Talk to you tomorrow :)

Both of your logs contain bad uris:

2017-12-10 12:56:59.0872056 [ThreadId 3] - ---> {"jsonrpc":"2.0","method":"textDocument/didChange","params":{"textDocument":{"uri":"file:///f%3A/TEMP/DEBUG/tROUBLE-orig.hs","version":3},"contentChanges":[{"range":{"start":{"line":18,"character":12},"end":{"line":18,"character":12}},"rangeLength":0,"text":")"}]}}

There is f%3A instead of correct F:. This is a manifestation of a bug in haskell-lsp. The bug was fixed in https://github.com/alanz/haskell-lsp/pull/63 but it hasn't been merged yet. @alanz?

The issue you experiences is a duplicate of #329. We'll update everything soon.

[lyuinnhe]

Are you sure this is the cause?

[nponeccop]

I'm sure that %3a causes big problems. But I'm not sure that this %3a bug is the only problem you have. Let's fix it and see

[lyuinnhe]

So do I have to do stack clean or whatever before building the later HIE upon update?

[nponeccop]

git pull and stack install is enough

[lyuinnhe]

Thx have a good sleep

[lyuinnhe]

I just noticed that the code range of the diagnostics were L20C0 to L20C100000, this might be the cause for another of my issue

[alanz]

HIE installs a whole lot of stuff into the project 'stack-work directory because it is using git-based dependencies for some things, and these go in the local directory.

Once all the dependencies are in a stackage snapshot, this will cease to be a problem.

[lyuinnhe]

Is it planned to be uploaded to the hackage tree any sooner?

[alanz]

That's why I am currently focusing on cabal-helper. It is blocking a ghc-mod update, which is blocking HaRe and HIE

[lyuinnhe]

So I just checked and all packages pulled by git during stack install are in the hackage tree. So i suppose what you mean is that due to compatability issues, hie uses outdated or dev versions of those packages?

[lyuinnhe]

@alanz BTW is Cabal-helper still alive? It stopped updating into Hackage since Jan 2017 and seems to be failing the builds for quite a time.

[alanz]

yes, see https://gitlab.com/dxld/cabal-helper/pipelines and https://github.com/DanielG/cabal-helper/network

[lyuinnhe]

Thx, I'll keep myself updated to this

[nponeccop]

@soylens Can you reproduce the issue and recollect the logs with updated hie?

Note that it only makes sense to use the 8.2.2 version if your project is 8.2.2 too - ghc versions must match

[lyuinnhe]

wait I saw this as the wrong issue XD

[lyuinnhe]

Nope, it still doesnt work unless the build dir remains in place.
All of brittany, ghc-mod, Hare and hlint still gets installed in .stack-work

[nponeccop]

okay. I ran strace on HIE and found that hie actually uses 2 files from its build directory:

1. cabal-helper-wrapper from $builddir/.stack-work/install/i386-linux-nopie/lts-9.14/8.0.2/libexec/i386-linux-ghc-8.0.2/cabal-helper-0.8.0.0/cabal-helper-wrapper
2. hlint.yaml from $builddir/.stack-work/install/i386-linux-nopie/lts-9.14/8.0.2/share/i386-linux-ghc-8.0.2/hlint-2.0.11/hlint.yaml

Having these ones in the snapshot doesn't help us as snapshots are disposable caches of build products. They take lots of space and are subject to purges. We should take the route of normal per-user installation instead:

• cabal-helper-wrapper and hie go to ~/.local/bin - we only need to change build instructions
• hlint.yaml goes to ~/.local/share/hlint/hlint.yaml - we need to figure out how to configure hlint with --datadir option, as it uses getDataDir call here

@alanz any thoughts?

[lyuinnhe]

Has this problem been resolved? @alanz

[alanz]

We switched to using hlint from hackage, so the hlint.yaml should be the global one.

We put out a new version of cabal-helper (to hackage) but there was a silly bug, now fixed in master.

So until that is published to hackage, it will use the one from the build dir

[marmistrz]

For me under Arch HIE wants:

• hlint.yaml from ~/.stack/snapshots/x86_64-linux-tinfo6-nopie/nightly-2018-02-26/8.2.2/share/x86_64-linux-ghc-8.2.2/hlint-2.1/hlint.yaml
• the ~/.stack/programs directory:

hie: /home/marcin/.stack/programs/x86_64-linux/ghc-tinfo6-nopie-8.2.2/lib/ghc-8.2.2/settings: openFile: does not exist (No such file or directory)

After removing the build directory (~/.cache/pacaur/haskell-ide-engine-git) there are no visible issues. Using the AUR package.

[samuelpilz]

This is still an issue!

Hie complains that it does not find the cabal-helper-wrapper executable it requires, which it expects to be in /home/vagrant/haskell-ide-engine/.stack-work/install/x86_64-linux-tinfo6/nightly-2019-03-07/8.6.4/libexec/x86_64-linux-ghc-8.6.4/cabal-helper-0.9.0.0/cabal-helper-wrapper. I built hie from /home/vagrant/haskell-ide-engine using as much isolation as possible.

@alanz I thought cabal-helper-wrapper has been replaced by something else. Is this not the case?

This currently blocks my efforts to make binary-packages for linux distributions. @mpickering Does your work for the hie-bios solve these issues?

[mpickering]

Yes my branch doesn't use cabal-helper-wrapper so it is probably easier to package.

[samuelpilz]

which one is it? I would like to give it a try. Is it mpickering/haskell-ide-engine:master?

[samuelpilz]

Another issue related to cabal-helper-wrapper: It is possible to set the cabal-helper-libexecdir environment variable and copy the built cabal-helper-wrapper binary into that directory. However, then the settings file of the ghc that compiled the cabal-helper-wrapper needs to be present... sigh.

I really hope that the issues related to cabal-helper-wrapper can be resolved fairly quickly. @mpickering, I appreciate your effort.

[alanz]

Ping @DanielG

[samuelpilz]

It seems that cabal-helper has some nostalgia-issues. It does not work without finding the compiler that was compiling it :wink:

[shinzui]

This is still an issue on macOS. I built the binaries in a tmp dir then deleted the folder which broke hie. I discovered the problem by looking at neovim's language server logs.

Language server logs

2019-04-24 09:09:22.422921 [ThreadId 4] - run entered for hie-wrapper(hie-wrapper) Version 0.8.0.0 (2573 commits) x86_64 ghc-8.6.4
2019-04-24 09:09:22.424822 [ThreadId 4] - Current directory:/Users/shinzui/Gyehoeg/asobiba/haskell/
2019-04-24 09:09:22.703527 [ThreadId 4] - Cradle directory:/Users/shinzui/Gyehoeg/asobiba/haskell
2019-04-24 09:09:22.704841 [ThreadId 4] - Using stack GHC version
2019-04-24 09:09:22.883989 [ThreadId 4] - Project GHC version:8.6.4
2019-04-24 09:09:22.884628 [ThreadId 4] - hie exe candidates :["hie-8.6.4","hie-8.6","hie"]
2019-04-24 09:09:22.885694 [ThreadId 4] - found hie exe at:/Users/shinzui/.local/bin/hie-8.6.4
2019-04-24 09:09:22.886608 [ThreadId 4] - args:[]
2019-04-24 09:09:22.887044 [ThreadId 4] - launching ....



2019-04-24 09:09:22.899466 [ThreadId 4] - Using stack GHC version
2019-04-24 09:09:23.080625 [ThreadId 4] - Run entered for HIE(hie-8.6.4) Version 0.8.0.0 (2573 commits) x86_64 ghc-8.6.4
2019-04-24 09:09:23.081832 [ThreadId 4] - Current directory:/Users/shinzui/Gyehoeg/asobiba/haskell/
2019-04-24 09:09:23.113792 [ThreadId 10] - Using stack GHC version
hie-8.6.4: Could not find $libexecdir/cabal-helper-wrapper

If you are a cabal-helper developer you can set the environment variable
`cabal_helper_libexecdir' to override $libexecdir[1]. The following will
work in the cabal-helper source tree:

    `$ export cabal_helper_libexecdir=$PWD/dist/build/cabal-helper-wrapper`

[1]: /private/tmp/haskell-ide-engine/.stack-work/install/x86_64-osx/lts-13.15/8.6.4/libexec/x86_64-osx-ghc-8.6.4/cabal-helper-0.9.0.0
/private/tmp/haskell-ide-engine/.stack-work/install/x86_64-osx/lts-13.15/8.6.4/libexec/x86_64-osx-ghc-8.6.4/cabal-helper-0.9.0.0/cabal-helper-wrapper
junk/build/cabal-helper-wrapper/cabal-helper-wrapper
/private/tmp/haskell-ide-engine/.stack-work/install/x86_64-osx/lts-13.15/8.6.4/bin/cabal-helper-wrapper

If you don't know what I'm talking about something went wrong with your
installation. Please report this problem here:

    https://github.com/DanielG/cabal-helper/issues
2019-04-24 09:12:29.29119 [ThreadId 4] - run entered for hie-wrapper(hie-wrapper) Version 0.8.0.0 (2573 commits) x86_64 ghc-8.6.4
2019-04-24 09:12:29.293075 [ThreadId 4] - Current directory:/Users/shinzui/Gyehoeg/asobiba/haskell
2019-04-24 09:12:29.48922 [ThreadId 4] - Cradle directory:/Users/shinzui/Gyehoeg/asobiba/haskell
2019-04-24 09:12:29.490459 [ThreadId 4] - Using stack GHC version
2019-04-24 09:12:29.677505 [ThreadId 4] - Project GHC version:8.6.4
2019-04-24 09:12:29.678143 [ThreadId 4] - hie exe candidates :["hie-8.6.4","hie-8.6","hie"]
2019-04-24 09:12:29.679187 [ThreadId 4] - found hie exe at:/Users/shinzui/.local/bin/hie-8.6.4
2019-04-24 09:12:29.680017 [ThreadId 4] - args:[]
2019-04-24 09:12:29.680446 [ThreadId 4] - launching ....

2019-04-24 09:12:29.716384 [ThreadId 4] - Using stack GHC version
2019-04-24 09:12:29.885745 [ThreadId 4] - Run entered for HIE(hie-8.6.4) Version 0.8.0.0 (2573 commits) x86_64 ghc-8.6.4
2019-04-24 09:12:29.887035 [ThreadId 4] - Current directory:/Users/shinzui/Gyehoeg/asobiba/haskell/
2019-04-24 09:12:29.919378 [ThreadId 10] - Using stack GHC version

[samuelpilz]

@shinzui, this is still a known issue and some effort is being made towards fixing this

[jneira]

This is related with those issues:

• #1068 Package hie binaries
• #1053 / #1126 Work to replace cabal-helper with hie-bios
• #1245 Pr to upgrade cabal-helper that removes cabal-helper-wrapper

[fendor]

With the merge of #1126, now only the hlint.yaml files are a problem. These can be fixed quite easily from now on.

[jneira]

And with #1588 hie uses the hlint version that include hlint.yaml within the executable so... closing! 🎉