Haskell-language-server: SIGSEGV (address boundary error) when encountering Template Haskell splice

[patrickt]

Subject of the issue

I encounter segfaults when attempting to run haskell-language-server-wrapper in the root of my project (or from within my editor). They only go away when I remove this Template Haskell invocation.

I have tried removing the ~/.cache/ghcide folder, ensuring there are no old haskell-language-server binaries on my PATH, and adding a cabal.project to the project. This happens with versions 0.4 and 0.5. I’m using the latest release of macOS.

EDIT: compiling haskell-language-server from source, passing in --enable-executable-dynamic to configure, appears to be a successful workaround.

Your environment

haskell-language-server version: 0.5.0.0 (GHC: 8.10.1) (PATH: /Users/patrickt/.ghcup/bin/haskell-language-server-wrapper-0.5.0) (GIT hash: 14497f2503a2a0d389fabf3b146d674b9af41a34)
Tool versions found on the $PATH
cabal:      3.4.0.0
stack:      2.3.1
ghc:        8.10.2

• Which lsp-client do you use
  Emacs (though this happens at the command line)

• Describe your project (alternative: link to the project)

https://github.com/patrickt/possession

Steps to reproduce

git clone https://github.com/patrickt/possession.git
cd possession
haskell-language-server-wrapper

Expected behaviour

Successful project compilation

Actual behaviour

fish: '/Users/patrickt/.ghcup/bin/hask…' terminated by signal SIGSEGV (Address boundary error)

Include debug information

Debug output:

haskell-language-server version: 0.5.0.0 (GHC: 8.10.2) (PATH: /Users/patrickt/.ghcup/bin/haskell-language-server-8.10.2~0.5.0) (GIT hash: 14497f2503a2a0d389fabf3b146d674b9af41a34)
(haskell-language-server)Ghcide setup tester in /Users/patrickt/src/possession.
Report bugs at https://github.com/haskell/haskell-language-server/issues

Tool versions found on the $PATH
cabal:      3.4.0.0
stack:      2.3.3
ghc:        8.10.2


Step 1/4: Finding files to test in /Users/patrickt/src/possession
Found 16 files

Step 2/4: Looking for hie.yaml files that control setup
Found 1 cradle

Step 3/4: Initializing the IDE

Step 4/4: Type checking the files
[INFO] Consulting the cradle for "src/Game/Ecs.hs"
Output from setting up the cradle Cradle {cradleRootDir = "/Users/patrickt/src/possession", cradleOptsProg = CradleAction: Cabal}
> Resolving dependencies...
> Build profile: -w ghc-8.10.2 -O0
> In order, the following will be built (use -v for more details):
>  - possession-0.1.0.0 (lib) (configuration changed)
> Configuring library for possession-0.1.0.0..
> Preprocessing library for possession-0.1.0.0..
[INFO] Using interface files cache dir: /Users/patrickt/.cache/ghcide/possession-0.1.0.0-inplace-d45d56728bdc453e2db86899d6a423581f0b2ce7
[INFO] Making new HscEnv[possession-0.1.0.0-inplace]
[INFO] Consulting the cradle for "app/Main.hs"
Output from setting up the cradle Cradle {cradleRootDir = "/Users/patrickt/src/possession", cradleOptsProg = CradleAction: Cabal}
fish: '/Users/patrickt/.ghcup/bin/hask…' terminated by signal SIGSEGV (Address boundary error)

[patrickt]

For a commit of the project on which hls executes correctly: 2b14bde26a5f8f0e3f1b31fa6d52410ccab99cf1.
For one that fails: 7ad24b119fcf5655d0692530420d00feb0972125

[patrickt]

If this is relevant, here’s the code generated by the splice:

instance
  ( k_aaaH ~ An_Iso,
    a_aaaI ~ Apecs.Entity,
    b_aaaJ ~ Apecs.Entity
  ) =>
  LabelOptic "player" k_aaaH State State a_aaaI b_aaaJ
  where
  {-# INLINE labelOptic #-}
  labelOptic = (iso (\(State x_aaaK) -> x_aaaK)) State

[patrickt]

Remembered that is probably a duplicate of https://github.com/haskell/ghcide/issues/775

[jneira]

It is almost identical so i would close one of both (the older one?)

[pepeiborra]

Could be related to https://github.com/haskell/ghcide/issues/854

[jneira]

This one may be fixed in master (and the incoming hls 0.5 release) via ghcide-0.5, @patrickt could you check if master fix the issue for you?

[patrickt]

@jneira I’m afraid not; compiling with the latest HEAD gives me

haskell-language-server version: 0.5.1.0 (GHC: 8.10.2) (PATH: /Users/patrickt/.local/bin/haskell-language-server) (GIT hash: 8682517e9ff92caa35e727e28445896f97c61e8d)
(haskell-language-server)Ghcide setup tester in /Users/patrickt/src/possession.
Report bugs at https://github.com/haskell/haskell-language-server/issues

Tool versions found on the $PATH
cabal:      3.4.0.0
stack:      2.5.1
ghc:        8.10.2


Step 1/4: Finding files to test in /Users/patrickt/src/possession
Found 1 files

Step 2/4: Looking for hie.yaml files that control setup
Found 1 cradle

Step 3/4: Initializing the IDE

Step 4/4: Type checking the files
[INFO] Consulting the cradle for "src/UI/State.hs"
Output from setting up the cradle Cradle {cradleRootDir = "/Users/patrickt/src/possession", cradleOptsProg = CradleAction: Cabal}
> Resolving dependencies...
> Build profile: -w ghc-8.10.2 -O0
> In order, the following will be built (use -v for more details):
>  - possession-0.1.0.0 (lib) (configuration changed)
> Configuring library for possession-0.1.0.0..
> Preprocessing library for possession-0.1.0.0..
[INFO] Using interface files cache dir: /Users/patrickt/.cache/ghcide/possession-0.1.0.0-inplace-089966ab63c3b422ee9e2e4bb9ae32c2a6c4a099
[INFO] Making new HscEnv[possession-0.1.0.0-inplace]
File:     /Users/patrickt/src/possession/src/UI/Sidebar.hs
Hidden:   no
Range:    29:1-29:19
Source:   typecheck
Severity: DsWarning
Message: 
  The import of ‘UI.Resource’ is redundant
  except perhaps to import instances from ‘UI.Resource’
  To import instances alone, use: import UI.Resource()
fish: 'haskell-language-server --debug…' terminated by signal SIGSEGV (Address boundary error)

[jneira]

Oh, thanks for trying it anyways, i am afraid that th support needs more work to be reliable enough 😟

[patrickt]

Some more information: here’s the DWARF backtrace (EDIT: the line numbers here don’t seem to be hugely reliable)

* thread #55, stop reason = EXC_BAD_ACCESS (code=1, address=0x20)
  * frame #0: 0x0000000105bcda8b haskell-language-server`ocResolve_MachO [inlined] relocateSection(oc=<unavailable>, curSection=0) at MachO.c:680:40 [opt]
    frame #1: 0x0000000105bcda6d haskell-language-server`ocResolve_MachO(oc=0x000000010be4adb0) at MachO.c:1529 [opt]
    frame #2: 0x0000000105b9d4d4 haskell-language-server`ocTryLoad(oc=0x000000010be4adb0) at Linker.c:1735:9 [opt]
    frame #3: 0x0000000105b9d315 haskell-language-server`lookupSymbol_ at Linker.c:898:17 [opt]
    frame #4: 0x0000000105b9d2f7 haskell-language-server`lookupSymbol_(lbl="_ptcszmcrzm0zi3zi0zi1zm0e46846a_OpticsziInternalziOpticziTypes_zdtcAzuLens_closure") at Linker.c:878 [opt]
    frame #5: 0x0000000105bcddd5 haskell-language-server`ocResolve_MachO at MachO.c:848:24 [opt]
    frame #6: 0x0000000105bcda6d haskell-language-server`ocResolve_MachO(oc=0x000000010be61080) at MachO.c:1529 [opt]
    frame #7: 0x0000000105b9d4d4 haskell-language-server`ocTryLoad(oc=0x000000010be61080) at Linker.c:1735:9 [opt]
    frame #8: 0x0000000105b9d315 haskell-language-server`lookupSymbol_ at Linker.c:898:17 [opt]
    frame #9: 0x0000000105b9d2f7 haskell-language-server`lookupSymbol_(lbl="_ptcszmthzm0zi3zi0zi2zm4f2a2ac9_LanguageziHaskellziTHziOpticsziInternal_zdfSubstTypeTypezuzdcsubstType_closure") at Linker.c:878 [opt]
    frame #10: 0x0000000105bcddd5 haskell-language-server`ocResolve_MachO at MachO.c:848:24 [opt]
    frame #11: 0x0000000105bcda6d haskell-language-server`ocResolve_MachO(oc=0x000000010be5c210) at MachO.c:1529 [opt]
    frame #12: 0x0000000105b9d4d4 haskell-language-server`ocTryLoad(oc=0x000000010be5c210) at Linker.c:1735:9 [opt]
    frame #13: 0x0000000105b9d315 haskell-language-server`lookupSymbol_ at Linker.c:898:17 [opt]
    frame #14: 0x0000000105b9d2f7 haskell-language-server`lookupSymbol_(lbl="_ptcszmthzm0zi3zi0zi2zm4f2a2ac9_OpticsziTHziInternalziUtils_zdwquantifyTypezq_info") at Linker.c:878 [opt]
    frame #15: 0x0000000105bcddd5 haskell-language-server`ocResolve_MachO at MachO.c:848:24 [opt]
    frame #16: 0x0000000105bcda6d haskell-language-server`ocResolve_MachO(oc=0x000000010be5d400) at MachO.c:1529 [opt]
    frame #17: 0x0000000105b9d4d4 haskell-language-server`ocTryLoad(oc=0x000000010be5d400) at Linker.c:1735:9 [opt]
    frame #18: 0x0000000105b9d315 haskell-language-server`lookupSymbol_ at Linker.c:898:17 [opt]
    frame #19: 0x0000000105b9d2f7 haskell-language-server`lookupSymbol_(lbl="_ptcszmthzm0zi3zi0zi2zm4f2a2ac9_OpticsziTHziInternalziProduct_LensRules_con_info") at Linker.c:878 [opt]
    frame #20: 0x0000000105bcddd5 haskell-language-server`ocResolve_MachO at MachO.c:848:24 [opt]
    frame #21: 0x0000000105bcda6d haskell-language-server`ocResolve_MachO(oc=0x000000010be5a230) at MachO.c:1529 [opt]
    frame #22: 0x0000000105b9d4d4 haskell-language-server`ocTryLoad(oc=0x000000010be5a230) at Linker.c:1735:9 [opt]
    frame #23: 0x0000000105b9d315 haskell-language-server`lookupSymbol_ at Linker.c:898:17 [opt]
    frame #24: 0x0000000105b9d2f7 haskell-language-server`lookupSymbol_(lbl="_ptcszmthzm0zi3zi0zi2zm4f2a2ac9_OpticsziTH_makeFieldLabels_closure") at Linker.c:878 [opt]
    frame #25: 0x0000000105b9d560 haskell-language-server`lookupSymbol(lbl="_ptcszmthzm0zi3zi0zi2zm4f2a2ac9_OpticsziTH_makeFieldLabels_closure") at Linker.c:916:21 [opt]
    frame #26: 0x00000001055d6169 haskell-language-server`Lc3pa_info + 49

[wz1000]

@bgamari any idea what is going on here?

[patrickt]

A debug print of the ObjectCode* that causes the crash is here. It looks like the symbol it’s crashing on is __text.

(ObjectCode) $1 = {
  status = OBJECT_NEEDED
  fileName = 0x000000010be4ae80 "/Users/patrickt/.cabal/store/ghc-8.10.2/ptcs-cr-0.3.0.1-0e46846a/lib/libHSptcs-cr-0.3.0.1-0e46846a.a"
  fileSize = 8912
  formatName = 0x0000000105d2db42 "Mach-O"
  archiveMemberName = 0x000000010be4aef0 "/Users/patrickt/.cabal/store/ghc-8.10.2/ptcs-cr-0.3.0.1-0e46846a/lib/libHSptcs-cr-0.3.0.1-0e46846a.a(Types.o)"
  symbols = 0x000000010be4b320
  n_symbols = 41
  image = 0x00000001270cc000 "????"
  info = 0x000000010be4b1a0
  imageMapped = 1
  referenced = 0
  misalignment = 0
  n_sections = 3
  sections = 0x000000010be4b200
  n_segments = 1
  segments = 0x000000010be4ad40
  next = 0x000000010be4a1b0
  proddables = 0x000000010be4b300
  symbol_extras = 0x00000001270ce000
  first_symbol_extra = 0
  n_symbol_extras = 45
  bssBegin = 0x00000001270ce000 ""
  bssEnd = 0x00000001270ce000 ""
  stable_ptrs = 0x0000000000000000
  extraInfos = 0x0000000000000000
  rw_m32 = 0x000000010be4af60
  rx_m32 = 0x000000010be4b080
}
(lldb) p ((ObjectCode*) 0x000000010be4adb0)->info
(ObjectCodeFormatInfo *) $2 = 0x000000010be4b1a0
(lldb) p *(((ObjectCode*) 0x000000010be4adb0)->info)
(ObjectCodeFormatInfo) $3 = {
  header = 0x00000001270cc000
  symCmd = 0x00000001270cc170
  segCmd = 0x00000001270cc020
  dsymCmd = 0x00000001270cc188
  nlist = 0x00000001270cc918
  names = 0x00000001270ccbe8 ""
  macho_sections = 0x00000001270cc068
  macho_symbols = 0x000000010f890200
  n_macho_symbols = 45
  got_start = 0x0000000000000000
  got_size = 0
}

[wz1000]

I think it would be useful if you compiled ghcide with -debug and shared the output of running ghcide with +RTS -Dl

[wz1000]

BTW, the original code snippet you linked to doesn't point to any Template Haskell anymore.

[patrickt]

@wz1000 Fixed the original code snippet. Building with -debug now.

[patrickt]

@wz1000 The result of executing +RTS -Dl and piping stderr to a file was ~600MB, but here’s a gist of the last 500 lines.

[wz1000]

Huh, I can't reproduce this locally on linux with ghcide on GHC 8.10.2.

[ProofOfKeags]

I'd like to advance the theory that this might be restricted to MacOS. My colleague is also experiencing something like this on Mac, but if I try on Linux with her exact same project, I cannot repro.

I'm about to dust off my old Mac to see if there's any credibility to this idea.

[wz1000]

It would be helpful if someone could come up with a test case that fails on the newly added MacOS CI on ghcide.

[ProofOfKeags]

The test case that is failing for my colleague is actually quite easy to produce:

stack new test-yesod yesodweb/postgres <- this with no alterations is enough to produce it. This seems to be the case for her on both 8.8.4/stackage-16.21 and 8.10.2/nightly-2020-11-11.

[pepeiborra]

The backtrace is super useful.

It might be possible to repro without ghcide, if you have a ghc binary built with DYNAMIC_GHC_PROGRAMS=NO. Compiling the project normally might just be enough to reproduce. It would be great if you can try that out and submit a minified GHC bug report.

And if you link ghcide/HLS with -dynamic, it should/will use the system linker instead of the rts linker and avoid this bug.

[patrickt]

@ProofOfKeags @wz1000 Good to hear that this doesn’t repro on Linux: given that this this is dying in the guts of relocating sections in Mach-O executables, we can assume it’s something macOS specific.

[patrickt]

@pepeiborra Wow, compiling with -dynamic fixed things! Thank you!

[ProofOfKeags]

@patrickt @pepeiborra do you just put the -dynamic flag in hls' cabal file? or is this something that goes into the various stack.yaml. Never done dynamic linking before, sorry if this is a dumb question.

[patrickt]

@ProofOfKeags Run cabal configure --enable-optimization --enable-executable-dynamic in the haskell-language-server repo root. Then build, and check to see if the resulting executable handles the file in question correctly.

[ProofOfKeags]

Dynamic linking fixed this for us too. So am I to understand that this issue is in GHC?

[pepeiborra]

Yes, GHC has its own dynamic linker, in the rts, and it gets used in certain cases:

• when GHC the binary is linked statically, by GHCI and TH
• when ghc the library is linked statically, by TH and the ghc interactive api

Implementing a dynamic linker with support for >=3 platforms is a risky and complex exercise. Fortunately there is an easy opt-out (-dynamic) but unfortunately that opt-out makes distribution much harder, and the precompiled HLS binaries are statically linked.

As I mentioned it would be great if you can minify this bug and report it upstream in GHC.

[sordina]

+1 Confirmed the -dynamic option successfully works around the issue for me in haskell-language-server.

[JosephLucas]

+1. It solved an issue with Template Haskell.

HLS was reporting an issue with the linux ld object, that it could not find.
Adding -dynamic seemed to have solved the previous issue, but I got another issue

Cannot load -dynamic objects when GHC is built the normal way
    To fix this, either:
      (1) Use -fexternal-interpreter, or
      (2) Build the program twice: once the normal way, and then
          with -dynamic using -osuf to set a different object file suffix.

Adding -fexternal-interpreter solved this second issue.

Now I still have useless warning waves under quasiquotes of template haskell though.

[pepeiborra]

-fexternal-interpreter is not supported with ghcide/hls, it will crash.

Where did you local GHC installation come from? Is it a custom build with DYNAMIC_GHC_PROGRAMS=NO?

[JosephLucas]

Strange, I think I really use -fexternal-interpreter
-fexternal-interpreter is in package,yaml and PROJECT.cabal.
I confirm that HLS does work. Maybe I don't use the last version of everything...

My ghc comes from /nix/store/lfv2p3l2q8wahr9gx5ks3fdvlvq7rdaf-ghc-8.8.4-with-packages/bin/ghc

[patrickt]

I’ve filed a bug upstream to inform about the dynamic linker issue. Anyone who wanted to help minimize the test case would win my infinite gratitude.