Ghcide: Support TemplateHaskell

[ndmitchell]

Trying to synthesise one ticket that describes all the TH work. My understanding of the current state is:

• If you have TH or ANN pragmas, and they all refer to code that is imported by packages or in this module, it works.
• If you refer to other modules in this package it breaks, perhaps with #25.

The solution is likely to make sure we have compiled the dependencies to bytecode and send them at the linker before type checking a module that uses TH or ANN.

However, we don't have tests for the current state, and it works by luck rather than deliberately.

[cocreature]

I have also seen some issue around async exceptions and TH which seemed to make GHC very unhappy (I don’t recall the precise error message).

[DavidEichmann]

I've noticed a huge slowdown when TH is enabled. After some searching I've noticed the folks at hie-engine ran into what looks like the same issue (HIE github issue), which was fixed in Infinisil/all-hies
by dynamically linking HIE (https://github.com/Infinisil/all-hies/pull/3). That PR says dynamic linking may not be necessary in for ghc versions with a fix to this GHC issue.

[pwm]

Hi, anyone is in a position to estimate how much work this would entail? Since my team started using ghcide we can’t imagine going back to the stone age ever. You guys did an amazing job! On our end TH support seems to be the last big issue that’s in the way of full adoption.

[ndmitchell]

My guess is you need to:

• Figure out which modules require TH, nice and easy.
• For those that do, first build all the dependencies to object code (moderate amount of work, probably < 100 lines, fits fairly well with the graph approach).
• Then send that object code to the linker (dunno how hard this is, might have problems with state management).

So probably not obviously hard, but I'd almost be surprised if something didn't get in your way.

[domenkozar]

@DavidEichmann in case you were using ghcide-nix, that has been fixed now.

[serras]

I would like to tackle this issue, since out team uses TH in quite some places.
Is there any further information that I can read to work on this?

[ndmitchell]

None that I'm aware of @serras - but best of luck!

[serras]

This seems to be related https://gitlab.haskell.org/ghc/ghc/issues/8025, at least it ends up with the same error as #212

[serras]

I have started working on the issue at https://github.com/serras/ghcide/commit/90c784b6f9503a2686f3e585e678ee65f60e3d9a in a quite crude way, asking to compile all dependencies even if the file does not use Template Haskell.
Now I get errors which look the ones in #212, even though now I am loading some bytecode:

File:     ./src/B.hs
Range:    1:0-100001:0
Source:   typecheck
Severity: DsError
Message:
  Program error:
  ByteCodeLink.lookupCE
  During interactive linking, GHCi couldn't find the following symbol:
  A_a_closure
  This may be due to you not asking GHCi to load extra object files,
  archives or DLLs needed by your current session. Restart GHCi, specifying
  the missing library using the -L/path/to/object/dir and -lmissinglibname
  flags, or simply by naming the relevant files on the GHCi command line.
  Alternatively, this link failure might indicate a bug in GHCi.
  If you suspect the latter, please send a bug report to:
  [email protected]

Is there any way I can check which rules are firing? At least I can be sure that the bytecode for A.hs is generated before continuing my investigation.

[cocreature]

You can use shake profiling to figure out the rules, see https://github.com/digital-asset/ghcide/blob/master/src/Development/IDE/Types/Options.hs#L44 (I think this is not exposed via the CLI so you’ll have to set it in the code somewhere) but that’s more useful for benchmarking than for debugging imho. I would probably just use some printf statements in the rules.

[merijn]

I was testing out this PR and run into similar errors about a missing _closure despite said close being in the relevant library being loaded when I inspect it via nm. Is there a way I can see the ghci configuration ghcide is using to load things?

[domenkozar]

I'm seeing the following using these changes:

https://github.com/hercules-ci/hercules-ci-agent

[merijn]

Yeah, I'm seeing similar errors, and I have some suspicions but I'm not really sure enough how to debug ghcide to investigate further. Essentially, I seem to have both a static library (missing some symbols) and a dynamic library (not missing said symbols) in my dist-newstyle and ghcide seems to try to use the static one, but I'm not really sure why.

[merijn]

I'm trying to trick it into loading the dynamic library to see if that works, but that's not working :

[cocreature]

We don’t do anything explicit atm wtr to linking external libraries in our code. I was hoping that this just magically works™ if we setup the GHC API with the right flags but it looks like we need to figure out the right incantation.

[merijn]

I've been trying to find the right incantation with explicit -l/-L flags in hie.yaml, but so far no luck, but I've also going about it a bit blind since I don't actually know how ghcide sets up it's GHC session

[ndmitchell]

Ghcide uses hie-bios to set up its session, which essentially dumps command line flags into a GHC session. If flags work in ghci, my experience is it usually works in ghcide too.

[merijn]

@ndmitchell I wish :)

ghci seems perfectly happy to work with just .ghc.environment, but I'm seeing all sorts of weird missing symbol errors in ghcide that I can't quite explain. I don't mind digging a bit deeper trying to debug them, but I'm rather unsure/clueless on where to start.

[serras]

@merijn you can have a look at hie-bios documentation and try to turn your current .ghc.environment configuration into a hie.yaml file.

[ndmitchell]

@merijn yeah, .ghc.environment is entirely ignored by everything else, so you'll have to replicate that on the command line. FWIW, I turn off the GHC environment files because they cause lots of confusing behaviour with tools working differently in different places.

[merijn]

I was under the impression that ghcide used them to find dependencies? Because that seems to work fine in most of my simple-ish projects. I do have a hie.yaml to resolve the look source paths, but I'm always a bit mystified what else has to go in there.

[ndmitchell]

@merijn nope, ghcide doesn't look at those files at all. It only uses cabal/stack as configured by hie-bios. In many cases cabal/stack will know the same information as those environment files.

[merijn]

Ah, so maybe that's where things go wrong, because in other projects I haven't had to use a hie.yaml to set things up! I'll start fiddling with hie-bios

[cocreature]

As far as I know, we do pick up .ghc.environment files. The GHC API behaves like GHC itself in that it picks them up implicitly by default.

[ndmitchell]

API's being influenced by random environment files scares me. The more I see about these environment files, the less I like them!