Haskell-language-server: Could not invoke type-checker plugin provided by the same package, even with `-fobject-code`

[konn]

It seems that the current HLS cannot handle type-checker plugins provided by the same package as the invoking module.
It is known that plain GHCi couldn't handle such local type-checker plugins; but in general, specifying -fobject-code GHC option solves this problem.

On the other hand, the current version of haskell-language-server could not handle such local plugins, even with -fobject-code, and results in the following error like the below (with VSCode Haskell Language Server + hls-wrapper):

{
    "resource": "path/to/source",
    "owner": "Haskell HIE",
    "severity": 8,
    "message": "Unexpected usage error\nThe module Data.MySolver did not export the plugin name Data.MySolver.plugin\n",
    "source": "typecheck",
    "startLineNumber": 1,
    "startColumn": 1,
    "endLineNumber": 100001,
    "endColumn": 1
}

Contrary to the message, Data.MySolver package surely exposes the plugin function.

• Data.MySolver module indeed exposes plugin function
• Calling site of Data.MySolver type-checker plugin contains {-# OPTIONS_GHC -fobject-code #-}`.

Since this error occurred in my full-time job project, I haven't done the minimal reproducing example. I will provide minimal examples sooner.

[konn]

OK, I created a small repro here.

Here is the reproduction procedure:

1. Checkout 2a58af80f4859239b47a0c915c4e04030c06b3b6 of haskell-language-server.
2. Build with stack --stack-yaml=stack-8.6.5.yaml build (as cabal-install-hls fails at this time).
3. Copy resulting haskell-language-server binary to ~/.cabal/bin/haskell-language-server-8.6.5,
   or other accessible path.
4. Open this directory with VSCode. Make sure hieExecutablePath in .vscode/setting.json
   is set to the installed location above.
5. run cabal new-build in the shell.
6. Open src/Lib.hs.

7. HLS complains that:

   Program error: attempting to use module ‘fake_uid:Lib.Plugin’ (~/path/to/hls-repro/src/Lib/Plugin.hs) which is not loaded
   

8. On the other hand, cabal new-repl successfully loads all libraries since we specify -fobject-code.

[konn]

I confirmed that this behaviour still exists in 0.1.
I want to contribute if possible. Which part of hls is relevant?

[jneira]

Hi, i just see

It is known that plain GHCi couldn't handle such local type-checker plugins; but in general, specifying -fobject-code GHC option solves this problem.

Could you install the hie-bios executable and run hie-bios debug src\Lib.hs inside the project root? it will output the compiler options used underneath by hls and maybe can help to find the cause of the problem

[konn]

It emits the following (I replaced the real path to hls-repro project with /path/to/hls-repro for brevity and no other edit was taken place)

Root directory:      /path/to/hls-repro
GHC options:         -fbuilding-cabal-package -O0 -outputdir /path/to/hls-repro/dist-newstyle/build/x86_64-osx/ghc-8.6.5/hls-repro-0.1.0.0/build -odir /path/to/hls-repro/dist-newstyle/build/x86_64-osx/ghc-8.6.5/hls-repro-0.1.0.0/build -hidir /path/to/hls-repro/dist-newstyle/build/x86_64-osx/ghc-8.6.5/hls-repro-0.1.0.0/build -stubdir /path/to/hls-repro/dist-newstyle/build/x86_64-osx/ghc-8.6.5/hls-repro-0.1.0.0/build -i -i/path/to/hls-repro/dist-newstyle/build/x86_64-osx/ghc-8.6.5/hls-repro-0.1.0.0/build -i/path/to/hls-repro/src -i/path/to/hls-repro/dist-newstyle/build/x86_64-osx/ghc-8.6.5/hls-repro-0.1.0.0/build/autogen -i/path/to/hls-repro/dist-newstyle/build/x86_64-osx/ghc-8.6.5/hls-repro-0.1.0.0/build/global-autogen -I/path/to/hls-repro/dist-newstyle/build/x86_64-osx/ghc-8.6.5/hls-repro-0.1.0.0/build/autogen -I/path/to/hls-repro/dist-newstyle/build/x86_64-osx/ghc-8.6.5/hls-repro-0.1.0.0/build/global-autogen -I/path/to/hls-repro/dist-newstyle/build/x86_64-osx/ghc-8.6.5/hls-repro-0.1.0.0/build -optP-include -optP/path/to/hls-repro/dist-newstyle/build/x86_64-osx/ghc-8.6.5/hls-repro-0.1.0.0/build/autogen/cabal_macros.h -this-unit-id hls-repro-0.1.0.0-inplace -hide-all-packages -Wmissing-home-modules -no-user-package-db -package-db /Users/hiromi/.cabal/store/ghc-8.6.5/package.db -package-db /path/to/hls-repro/dist-newstyle/packagedb/ghc-8.6.5 -package-db /path/to/hls-repro/dist-newstyle/build/x86_64-osx/ghc-8.6.5/hls-repro-0.1.0.0/package.conf.inplace -package-id base-4.12.0.0 -package-id ghc-8.6.5 -XHaskell2010 Lib Lib.Plugin -hide-all-packages
System libraries:    /Users/hiromi/.ghcup/ghc/8.6.5/lib/ghc-8.6.5
Config Location:     /path/to/hls-repro/hie.yaml
Cradle:              Cradle {cradleRootDir = "/path/to/hls-repro", cradleOptsProg = CradleAction: Cabal}
Dependencies:        hls-repro.cabal cabal.project

[jneira]

@konn so there is no -fobject-code in the GHC options . Those options are precisely the used ones by the hie to start a ghc session, what happens if you call ghc directly with those options?
Maybe setting -fobject-code in the ghc-options stanza in the .cabal file could help?

[konn]

When running the following in the home directory (not in project root), it seems that everything works fine:

$ ghc-8.6.5 -fbuilding-cabal-package -O0 -outputdir /path/to/hls-repro/dist-newstyle/build/x86_64-osx/ghc-8.6.5/hls-repro-0.1.0.0/build -odir /path/to/hls-repro/dist-newstyle/build/x86_64-osx/ghc-8.6.5/hls-repro-0.1.0.0/build -hidir /path/to/hls-repro/dist-newstyle/build/x86_64-osx/ghc-8.6.5/hls-repro-0.1.0.0/build -stubdir /path/to/hls-repro/dist-newstyle/build/x86_64-osx/ghc-8.6.5/hls-repro-0.1.0.0/build -i -i/path/to/hls-repro/dist-newstyle/build/x86_64-osx/ghc-8.6.5/hls-repro-0.1.0.0/build -i/path/to/hls-repro/src -i/path/to/hls-repro/dist-newstyle/build/x86_64-osx/ghc-8.6.5/hls-repro-0.1.0.0/build/autogen -i/path/to/hls-repro/dist-newstyle/build/x86_64-osx/ghc-8.6.5/hls-repro-0.1.0.0/build/global-autogen -I/path/to/hls-repro/dist-newstyle/build/x86_64-osx/ghc-8.6.5/hls-repro-0.1.0.0/build/autogen -I/path/to/hls-repro/dist-newstyle/build/x86_64-osx/ghc-8.6.5/hls-repro-0.1.0.0/build/global-autogen -I/path/to/hls-repro/dist-newstyle/build/x86_64-osx/ghc-8.6.5/hls-repro-0.1.0.0/build -optP-include -optP/path/to/hls-repro/dist-newstyle/build/x86_64-osx/ghc-8.6.5/hls-repro-0.1.0.0/build/autogen/cabal_macros.h -this-unit-id hls-repro-0.1.0.0-inplace -hide-all-packages -Wmissing-home-modules -no-user-package-db -package-db /Users/hiromi/.cabal/store/ghc-8.6.5/package.db -package-db /path/to/hls-repro/dist-newstyle/packagedb/ghc-8.6.5 -package-db /path/to/hls-repro/dist-newstyle/build/x86_64-osx/ghc-8.6.5/hls-repro-0.1.0.0/package.conf.inplace -package-id base-4.12.0.0 -package-id ghc-8.6.5 -XHaskell2010 Lib Lib.Plugin -hide-all-packages

[1 of 2] Compiling Lib.Plugin       ( /path/to/hls-repro/src/Lib/Plugin.hs, /path/to/hls-repro/dist-newstyle/build/x86_64-osx/ghc-8.6.5/hls-repro-0.1.0.0/build/Lib/Plugin.o )
[2 of 2] Compiling Lib              ( /path/to/hls-repro/src/Lib.hs, /path/to/hls-repro/dist-newstyle/build/x86_64-osx/ghc-8.6.5/hls-repro-0.1.0.0/build/Lib.o )

On the other hand, after removing *.{hi,o} files from the directory, ghci panics without -fobject-code in CLI options, even though it is specified in the module header:

$ rm /path/to/hls-repro/**/*.{hi,o}
$ ghci-8.6.5 <flags same as above...>
GHCi, version 8.6.5: http://www.haskell.org/ghc/  :? for help
Some flags have not been recognized: prompt2, ghci| 
Loaded GHCi configuration from /Users/hiromi/.dotfiles/.ghci
[1 of 2] Compiling Lib.Plugin       ( /path/to/hls-repro/src/Lib/Plugin.hs, interpreted )
[2 of 2] Compiling Lib              ( /path/to/hls-repro/src/Lib.hs, interpreted )
ghc: panic! (the 'impossible' happened)
  (GHC version 8.6.5 for x86_64-apple-darwin):
    mkPluginUsage: file not found
  Lib.Plugin /path/to/hls-repro/dist-newstyle/build/x86_64-osx/ghc-8.6.5/hls-repro-0.1.0.0/build/Lib/Plugin.o
  Call stack:
      CallStack (from HasCallStack):
        callStackDoc, called at compiler/utils/Outputable.hs:1160:37 in ghc:Outputable
        pprPanic, called at compiler/deSugar/DsUsage.hs:234:15 in ghc:DsUsage

Please report this as a GHC bug:  http://www.haskell.org/ghc/reportabug

But, once after compile modules with the ghc command, ghci process seems successfully started:

$ ghc-8.6.5 <same as above...>
$ ghci-8.6.5 <same as above...>
[1 of 2] Compiling Lib.Plugin       ( /path/to/hls-repro/src/Lib/Plugin.hs, interpreted )
[2 of 2] Compiling Lib              ( /path/to/hls-repro/src/Lib.hs, interpreted )
Ok, two modules loaded.

[*Lib]
ghci> 

It also succeeds when I remove all the *.{hi,o} but passes -fobject-code as the last CLI Option to ghci:

$ rm /path/to/hls-repro/**/*.{hi,o}
$ ghci-8.6.5 <flags same as above...> -fobject-code
...
Ok, two modules loaded.

[jneira]

aha, i suspect the ghc session opened by hls is similar to ghci, so the project is loaded in the hie in the same state ghci works (with previous compiled files)? or does not in any case?
Not an expert about plugins and the flag -fobject-code, does make sense add it to the .cabal file to check it it works in that way?

[konn]

After adding ghc-options: -fobject-code to the library section of hls-repro.cabal, hie-bios emits the following options:

-fbuilding-cabal-package -O0 -outputdir /path/to/hls-repro/dist-newstyle/build/x86_64-osx/ghc-8.6.5/hls-repro-0.1.0.0/build <.. the same ...> -package-id ghc-8.6.5 -XHaskell2010 Lib Lib.Plugin -fobject-code -hide-all-packages

Now -fobject-code is included in ghc options and, with these options, ghci session starts successfully!
As far as I know, there is no existing Haskell IDE (nor Haddock) which can treat same-package plugins without specifying -fobject-code somewhere in the project (module header, ghc-options, etc.), so this solution seems reasonable to me. Thanks a lot!

[jneira]

great! so i think we can close this one, as the only improvement could be that the interactive session would take in account the flag inlined in the module but this seems to be a fix to make upstream.
Feel free to reopen if it is not the case.

[konn]

Unfortunately, the current implementation needs some more tweaks to get package-local compiler plugins working.
Even if we add -fobject-code to the GHC options in Cabal file and hie-bios properly add it to the flag, HLS then seems incapable of loading the plugin module from the same package!

In particular, with the same hls-repro case, HLS 0.5.0 (as downloaded automatically by vscode-haskell) halts with the following slightly different error:

$ ~/Library/Application\ Support/Code/User/globalStorage/haskell.haskell/haskell-language-server-0.5.0-darwin-8.6.5 src/Lib.hs
haskell-language-server version: 0.5.0.0 (GHC: 8.6.5) (PATH: /Users/hiromi/Library/Application Support/Code/User/globalStorage/haskell.haskell/haskell-language-server-0.5.0-darwin-8.6.5) (GIT hash: 14497f2503a2a0d389fabf3b146d674b9af41a34)
(haskell-language-server)Ghcide setup tester in /Users/hiromi/Documents/Programming/Haskell/git/hls-repro.
Report bugs at https://github.com/haskell/haskell-language-server/issues

Tool versions found on the $PATH
cabal:          3.2.0.0
stack:          2.3.3
ghc:            8.6.5


Step 1/4: Finding files to test in /Users/hiromi/Documents/Programming/Haskell/git/hls-repro
Found 1 files

Step 2/4: Looking for hie.yaml files that control setup
Found 1 cradle

Step 3/4: Initializing the IDE

Step 4/4: Type checking the files
[INFO] Consulting the cradle for "src/Lib.hs"
Output from setting up the cradle Cradle {cradleRootDir = "/Users/hiromi/Documents/Programming/Haskell/git/hls-repro", cradleOptsProg = CradleAction: Cabal}
> Resolving dependencies...
> Build profile: -w ghc-8.6.5 -O1
> In order, the following will be built (use -v for more details):
>  - hls-repro-0.1.0.0 (lib) (configuration changed)
> Configuring library for hls-repro-0.1.0.0..
> Warning: The 'license-file' field refers to the file 'LICENSE' which does not
> exist.
> Preprocessing library for hls-repro-0.1.0.0..
[INFO] Using interface files cache dir: /Users/hiromi/.cache/ghcide/hls-repro-0.1.0.0-inplace-7278d26d0518d440d668e6c3322a81b909d244a6
[INFO] Making new HscEnv[hls-repro-0.1.0.0-inplace]
File:     /Users/hiromi/Documents/Programming/Haskell/git/hls-repro/src/Lib.hs
Hidden:   no
Range:    1:0-2:0
Source:   pragmas
Severity: DsError
Message: 
  Unexpected usage error
  Could not load module ‘Lib.Plugin’
  It is a member of the hidden package ‘hls-repro-0.1.0.0’.
  Perhaps you need to add ‘hls-repro’ to the build-depends in your .cabal file.
Files that failed:
 * /Users/hiromi/Documents/Programming/Haskell/git/hls-repro/src/Lib.hs

Completed (0 files worked, 1 file failed)

Any thoughts?

[konn]

It seems the same error already occurs in ghcide (master branch); so it must be an upstream bug in ghcide, I think.

[konn]

It seems the error comes from preprocessor in Development.IDE.Core.Preprocessor, which is called transitively from getModSummaryRule function.
I first thought that we could use GhcSessionDeps instead of GhcSession in getModSummaryRule, but it resulted in an infinite loop (perhaps due to cyclic dependency of rules?).
I tried to expand the relevant definitions into getModSummaryRule, but just calling useWithStale_ GetDependencies f there also resulted in infinite loop.

[konn]

I filed the upstream issue as haskell/ghcide#868.