stack 🚀 - GHC panic when building Stack on macOS Sierra

I was able to reproduce both locally and on Homebrew's CI. (We use GHC 8.0.1.)

zmwangx on 13 Sep 2016

Maybe the title could be a little bit more informative. This is only a problem on macOS 10.12 Sierra as far as I can tell.

zmwangx on 13 Sep 2016

Note also
https://ghc.haskell.org/trac/ghc/ticket/12198
https://ghc.haskell.org/trac/ghc/ticket/12479

ilovezfs on 13 Sep 2016

@cartazio any ideas how to fix the GHC problem with Sierra? macOS 10.12 has gone GM and Stack is dead in the water.

ilovezfs on 13 Sep 2016

👍1

Is it stack specific or ghc 8.0 generally? Is there a minimal repro?

cartazio on 13 Sep 2016

Since @bitemyapp's top post reports this for GHC 7.10.3, I suppose it's not specific to GHC 8.0.

zmwangx on 13 Sep 2016

It looks like it's related to split objects or sections?

It would help if there was a simpler application that reported this issue

cartazio on 13 Sep 2016

@cartazio there's doubt that it's about split-sections: https://ghc.haskell.org/trac/ghc/ticket/12479#comment:4

ilovezfs on 13 Sep 2016

@cartazio I think the minimal reproducer is brew instal -s stack at this point, but it looks like it really doesn't get very far

Building stack-1.1.2...
Preprocessing library stack-1.1.2...
[ 1 of 87] Compiling Stack.FileWatch  ( src/Stack/FileWatch.hs, dist/dist-sandbox-68bf8d9c/build/Stack/FileWatch.o )
[ 2 of 87] Compiling System.Process.PagerEditor ( src/System/Process/PagerEditor.hs, dist/dist-sandbox-68bf8d9c/build/System/Process/PagerEditor.o )
[ 3 of 87] Compiling System.Process.Log ( src/System/Process/Log.hs, dist/dist-sandbox-68bf8d9c/build/System/Process/Log.o )
[ 4 of 87] Compiling System.Process.Read ( src/System/Process/Read.hs, dist/dist-sandbox-68bf8d9c/build/System/Process/Read.o )
ghc: panic! (the 'impossible' happened)
  (GHC version 8.0.1 for x86_64-apple-darwin):
    Loading temp shared object failed: dlopen(/var/folders/4d/rttdp_9d2s7f36zplgwnqrgr0000gn/T/ghc94633_0/libghc_44.dylib, 5): no suitable image found.  Did find:
    /var/folders/4d/rttdp_9d2s7f36zplgwnqrgr0000gn/T/ghc94633_0/libghc_44.dylib: malformed mach-o: load commands size (40192) > 32768

ilovezfs on 13 Sep 2016

@borsboom any idea what the problem here is? The Sierra GM has escaped into the wild!

ilovezfs on 15 Sep 2016

_minimal_ _code_ reproducer, stack is too much code

cartazio on 15 Sep 2016

@ilovezfs No idea. It has to be a GHC bug (panics always are) but would be nice to find a workaround. I don't have an Apple developer account, so I don't have any access to Sierra betas. Once it's released I can upgrade my spare mac Mini and try to reproduce.

borsboom on 15 Sep 2016

@borsboom The public beta is already on GM so you don't need to have a developer subscription to have access to pretty much the final version of 10.12.

zmwangx on 15 Sep 2016

If anyone else has some time, I'd probably start trying to make a smaller reproduction by extracting the failing System.Process.Read module to a separate project and minimizing the dependencies. It's an isolated module so that should be pretty easy. If it still fails, just start taking out pieces of code until you find what triggers the panic (if it doesn't fail... well then it's more complicated I guess).

borsboom on 15 Sep 2016

@bitemyapp: where did you get your GHC? Was it a sandboxed one installed by Stack?

borsboom on 15 Sep 2016

Also, from one of the GHC reports, looks like this isn't specific to Stack. yesod-auth also has a similar panic:

   [ 4 of 11] Compiling Yesod.Auth       ( Yesod/Auth.hs, .stack-work/dist/x86_64-osx/Cabal-1.22.4.0/build/Yesod/Auth.o )
    ghc: panic! (the 'impossible' happened)
      (GHC version 7.10.2 for x86_64-apple-darwin):
        Loading temp shared object failed: dlopen(/var/folders/3k/ycnfbqgx33n7qdytkl9ryx7m0000gn/T/ghc64990_0/libghc_21.dylib, 5): no suitable image found.  Did find:
        /var/folders/3k/ycnfbqgx33n7qdytkl9ryx7m0000gn/T/ghc64990_0/libghc_21.dylib: malformed mach-o: load commands size (34176) > 32768

borsboom on 15 Sep 2016

👍7

[ callen@cumae ~/work/codex travisci-build-matrix-stack ✔ ]
$ stack exec -- ghc --version
The Glorious Glasgow Haskell Compilation System, version 7.10.3
[ callen@cumae ~/work/codex travisci-build-matrix-stack ✔ ]
$ stack exec -- which ghc
/Users/callen/.stack/programs/x86_64-osx/ghc-7.10.3/bin/ghc

I did say it was a GHC bug in the title, just trying to keep y'all apprised.

bitemyapp on 15 Sep 2016

If you can do a small single Haskell module repro then I and or other ghc
dev team folk can dig into resolving it. But without a clear minimal repro
that 8.0 can trigger with a small module and clear deps, hard to do.

The moment it's an issue that can be demonstrated In a self contained way,
it'll be a ghc bug that gets full focus by applicable volunteers. Ghc
contributors don't have the bandwidth to boil down a repro from a really
large application

On Thursday, September 15, 2016, Emanuel Borsboom [email protected]
wrote:

If anyone else has some time, I'd probably start trying to make a smaller
reproduction by extracting the failing System.Process.Read module to a
separate project and minimizing the dependencies. It's an isolated module
so that should be pretty easy. If it still fails, just start taking out
pieces of code until you find what triggers the panic (if it doesn't
fail... well then it's more complicated I guess).

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/commercialhaskell/stack/issues/2577#issuecomment-247437655,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAAQwlfNd2gRtHcsG-i2fqj47rYNnKgVks5qqaTKgaJpZM4J546w
.

cartazio on 16 Sep 2016

Also if the bug is in 8.0 and someone can cook up a repro I can see about
sorting out a fix and making sure it's in an 8.0 bug fix. But I can't do
that until there's a small self contained small code base that has the
problem

On Friday, September 16, 2016, Carter Schonwald carter.[email protected]
wrote:

If you can do a small single Haskell module repro then I and or other ghc
dev team folk can dig into resolving it. But without a clear minimal repro
that 8.0 can trigger with a small module and clear deps, hard to do.

The moment it's an issue that can be demonstrated In a self contained way,
it'll be a ghc bug that gets full focus by applicable volunteers. Ghc
contributors don't have the bandwidth to boil down a repro from a really
large application

On Thursday, September 15, 2016, Emanuel Borsboom <
[email protected]

If anyone else has some time, I'd probably start trying to make a smaller
reproduction by extracting the failing System.Process.Read module to a
separate project and minimizing the dependencies. It's an isolated module
so that should be pretty easy. If it still fails, just start taking out
pieces of code until you find what triggers the panic (if it doesn't
fail... well then it's more complicated I guess).

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/commercialhaskell/stack/issues/2577#issuecomment-247437655,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAAQwlfNd2gRtHcsG-i2fqj47rYNnKgVks5qqaTKgaJpZM4J546w
.

cartazio on 16 Sep 2016

Looks like misty has provided some debugging data upstream to ghc. I'll
talk with bengamari. His current theory is that somehow the offending
builds have split sections enabled. And thus are drowning the linker

On Friday, September 16, 2016, Carter Schonwald carter.[email protected]
wrote:

Also if the bug is in 8.0 and someone can cook up a repro I can see about
sorting out a fix and making sure it's in an 8.0 bug fix. But I can't do
that until there's a small self contained small code base that has the
problem

On Friday, September 16, 2016, Carter Schonwald <
carter.[email protected]

If you can do a small single Haskell module repro then I and or
other ghc dev team folk can dig into resolving it. But without a clear
minimal repro that 8.0 can trigger with a small module and clear deps, hard
to do.

The moment it's an issue that can be demonstrated In a self contained
way, it'll be a ghc bug that gets full focus by applicable volunteers. Ghc
contributors don't have the bandwidth to boil down a repro from a really
large application

On Thursday, September 15, 2016, Emanuel Borsboom <
[email protected]> wrote:

If anyone else has some time, I'd probably start trying to make a
smaller reproduction by extracting the failing System.Process.Read module
to a separate project and minimizing the dependencies. It's an isolated
module so that should be pretty easy. If it still fails, just start taking
out pieces of code until you find what triggers the panic (if it doesn't
fail... well then it's more complicated I guess).

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/commercialhaskell/stack/issues/2577#issuecomment-247437655,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAAQwlfNd2gRtHcsG-i2fqj47rYNnKgVks5qqaTKgaJpZM4J546w
.

cartazio on 16 Sep 2016

His current theory is that somehow the offending builds have split sections enabled.

which would be quite weird since as misty also pointed out it seems to be a non-default option that we didn't pass in, but that doesn't mean it's impossible it built that way anyway for some reason ...

ilovezfs on 16 Sep 2016

I'm pretty sure split objects is the default for the libraries. Could you
try disabling that? I could be misremembering. Also I'm totally weak at
linker stuff

On Friday, September 16, 2016, ilovezfs [email protected] wrote:

His current theory is that somehow the offending builds have split
sections enabled.

which would be quite weird since as misty also pointed out it seems to be
a non-default option that we didn't pass in, but that doesn't mean it's
impossible it built that way anyway for some reason ...

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/commercialhaskell/stack/issues/2577#issuecomment-247516337,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAAQwhcph23yUUDuzVrrMcW5uam3W4M0ks5qqhzmgaJpZM4J546w
.

cartazio on 16 Sep 2016

--disable-split-objs and --disable-split-sections?

ilovezfs on 16 Sep 2016

Could be / should be those

On Friday, September 16, 2016, ilovezfs [email protected] wrote:

--disable-split-objs and --disable-split-sections?

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/commercialhaskell/stack/issues/2577#issuecomment-247520287,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAAQwjyBGkFtBJ8zvJI5KsIsa7GtGthcks5qqia7gaJpZM4J546w
.

cartazio on 16 Sep 2016

It's important to note that some builds of ghc base and boot libs may also
have those flags enabled.

In which case this could be a regression in linker on the OS X release. Eg
is it a format limit or a hard coded "safety limit" in the Sierra linker?
Are they still using gnu linker or move to llvm? In latter case is there
some hard coded sanity limits?

Probably easy to test by unpacking the Sierra cli tools and using them on a
previous os release.

On Friday, September 16, 2016, Carter Schonwald carter.[email protected]
wrote:

Could be / should be those

On Friday, September 16, 2016, ilovezfs <[email protected]

--disable-split-objs and --disable-split-sections?

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/commercialhaskell/stack/issues/2577#issuecomment-247520287,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAAQwjyBGkFtBJ8zvJI5KsIsa7GtGthcks5qqia7gaJpZM4J546w
.

cartazio on 16 Sep 2016

Looks like it's because they moved to llvms tool lld, and it hard coded a
limit on number of sections it'll accept. So the near term mitigation
would be to disable split objects on a ghc Mac build , but the Better
medium / long term is
A) we all fire radars on this
B) we make sure that upstream lld is fixed.

Me and other ghc devs are collaborating on IRC right now to confirm that
we've isolated the problem. Affected users could also unpack a pre Sierra
cli tools and put dl in their path as an alternative mitigation (this is me
speculating on an approach I have not tested but would probably work fine )

On Friday, September 16, 2016, Carter Schonwald carter.[email protected]
wrote:

Looks like misty has provided some debugging data upstream to ghc. I'll
talk with bengamari. His current theory is that somehow the offending
builds have split sections enabled. And thus are drowning the linker

On Friday, September 16, 2016, Carter Schonwald <
carter.[email protected]

Also if the bug is in 8.0 and someone can cook up a repro I can see about
sorting out a fix and making sure it's in an 8.0 bug fix. But I can't do
that until there's a small self contained small code base that has the
problem

On Friday, September 16, 2016, Carter Schonwald <
[email protected]> wrote:

If you can do a small single Haskell module repro then I and or
other ghc dev team folk can dig into resolving it. But without a clear
minimal repro that 8.0 can trigger with a small module and clear deps, hard
to do.

The moment it's an issue that can be demonstrated In a self contained
way, it'll be a ghc bug that gets full focus by applicable volunteers. Ghc
contributors don't have the bandwidth to boil down a repro from a really
large application

On Thursday, September 15, 2016, Emanuel Borsboom <
[email protected]> wrote:

If anyone else has some time, I'd probably start trying to make a
smaller reproduction by extracting the failing System.Process.Read module
to a separate project and minimizing the dependencies. It's an isolated
module so that should be pretty easy. If it still fails, just start taking
out pieces of code until you find what triggers the panic (if it doesn't
fail... well then it's more complicated I guess).

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/commercialhaskell/stack/issues/2577#issuecomment-247437655,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAAQwlfNd2gRtHcsG-i2fqj47rYNnKgVks5qqaTKgaJpZM4J546w
.

cartazio on 16 Sep 2016

Ld. not dl

On Friday, September 16, 2016, Carter Schonwald carter.[email protected]
wrote:

Looks like it's because they moved to llvms tool lld, and it hard coded a
limit on number of sections it'll accept. So the near term mitigation
would be to disable split objects on a ghc Mac build , but the Better
medium / long term is
A) we all fire radars on this
B) we make sure that upstream lld is fixed.

Me and other ghc devs are collaborating on IRC right now to confirm that
we've isolated the problem. Affected users could also unpack a pre Sierra
cli tools and put dl in their path as an alternative mitigation (this is me
speculating on an approach I have not tested but would probably work fine
)

On Friday, September 16, 2016, Carter Schonwald <
carter.[email protected]

Looks like misty has provided some debugging data upstream to ghc. I'll
talk with bengamari. His current theory is that somehow the offending
builds have split sections enabled. And thus are drowning the linker

On Friday, September 16, 2016, Carter Schonwald <
[email protected]> wrote:

Also if the bug is in 8.0 and someone can cook up a repro I can see
about sorting out a fix and making sure it's in an 8.0 bug fix. But I can't
do that until there's a small self contained small code base that has the
problem

On Friday, September 16, 2016, Carter Schonwald <
[email protected]> wrote:

If you can do a small single Haskell module repro then I and or
other ghc dev team folk can dig into resolving it. But without a clear
minimal repro that 8.0 can trigger with a small module and clear deps, hard
to do.

The moment it's an issue that can be demonstrated In a self contained
way, it'll be a ghc bug that gets full focus by applicable volunteers. Ghc
contributors don't have the bandwidth to boil down a repro from a really
large application

On Thursday, September 15, 2016, Emanuel Borsboom <
[email protected]> wrote:

If anyone else has some time, I'd probably start trying to make a
smaller reproduction by extracting the failing System.Process.Read module
to a separate project and minimizing the dependencies. It's an isolated
module so that should be pretty easy. If it still fails, just start taking
out pieces of code until you find what triggers the panic (if it doesn't
fail... well then it's more complicated I guess).

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/commercialhaskell/stack/issues/2577#issuecomment-247437655,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAAQwlfNd2gRtHcsG-i2fqj47rYNnKgVks5qqaTKgaJpZM4J546w
.

cartazio on 16 Sep 2016

Is this a setting that is "baked into" the GHC binary when the bindist is built, or can it be controlled by an argument to the bindist's configure when installing the bindist (or, alternatively, by adjusting the lib/ghc-8.0.1/settings or other files after installation)?

borsboom on 16 Sep 2016

The build of GHC and boot lib time thing. Default on most tier want
platforms is to enable split objects for the base and boot libs so that end
executable size is smaller

It's also a flag for userland module builds

On Sep 16, 2016 11:23 AM, "Emanuel Borsboom" [email protected]
wrote:

Is this a setting that is "baked into" the GHC binary when the bindist is
built, or can it be controlled by an argument to the bindist's configure
when installing the bindist (or, alternatively, by adjusting the
lib/ghc-8.0.1/settings or similar file after installation)?

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/commercialhaskell/stack/issues/2577#issuecomment-247629521,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAAQwhj1w5pujW__B5N57yGG7ceUfbHvks5qqrSJgaJpZM4J546w
.

cartazio on 16 Sep 2016

Ah, unfortunate. Still, we can have Stack detect that it's running on macOS >=Sierra and install GHC using alternative bindists.

borsboom on 16 Sep 2016

I'm not sure if that will suffice. We are still investigating

On Friday, September 16, 2016, Emanuel Borsboom [email protected]
wrote:

Ah, unfortunate. Still, we can have Stack detect that it's running on
macOS >=Sierra and install GHC using alternative bindists.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/commercialhaskell/stack/issues/2577#issuecomment-247691097,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAAQwrDpt8h0YR8WHS25Ej-fnCVCvsQdks5qqvGVgaJpZM4J546w
.

cartazio on 17 Sep 2016

It's actually not a GHC bug and doesn't have to do with split objects at all. Sierra has a new limit on the "load commands size" of a shared library (that you can see mentioned in the error message) which includes the paths to its dependencies, and Stack's many dependencies and long directory paths exceed this limit.

There are possible workarounds, but I can't think of anything simple.

rwbarton on 18 Sep 2016

@rwbarton @cartazio From one of our subject matter experts:

I guess this was the hidden message behind this year's WWDC session 406 where they talked about dynamic linker internals and optimizing app startup times.

The apparent new limit of 32K enforced by 10.12 feels very conservative, but then again GHC easily blowing that limit is also a bit scary.

I think their best bet is indeed to implement the idea from the brain-storming and make sure they have only one (or very few) ⁠⁠⁠⁠LC_RPATH⁠⁠⁠⁠ load commands that cover the majority of the path prefix(es) and the remaining ⁠⁠⁠⁠LC_LOAD_DYLIB⁠⁠⁠⁠ commands build on top of that to keep things small. Having a separate ⁠⁠⁠⁠LC_RPATH⁠⁠⁠⁠ command for every single library is a bit insane (but I can totally sympathize with the simplicity of such a solution).

I know basically nothing about GHC, but if the directory layout is somewhat predictable and fixed (aside from a changing prefix), i.e. it's easy to infer where a library should be located relative to the executable/library loading it, then another solution could be to rely on ⁠⁠⁠⁠@loader_path/⁠⁠⁠⁠ and/or ⁠⁠⁠⁠@executable_path/⁠⁠⁠⁠ to avoid the lengthy and repeated prefixes.

I think they will really need to optimize the load commands they are placing in every single binary to stay below the limit. The discussed common prefix idea should be fairly simple to implement. If not directly in the compiler, then at least as a post-processing step for the binaries.

The dyld man page in its entirety, but especially the bottom was highly recommended reading.

For post-processing macho programmatically this has been suggested as a useful resource https://github.com/Homebrew/ruby-macho for various techniques.

ilovezfs on 18 Sep 2016

Apparently I'm a “subject matter expert” and I'm the one being quoted above. I'm am contributor to the above mentioned Ruby library for dissecting and modifying Mach-O files and I'm decently knowledgeable about loaders and linkers in general and on macOS in particular.

I'm also completely clueless about GHC or Haskell in general, but if that's not a problem, I'm happy to help with some of my “wisdom”. (Disclaimer: I've also got a lot of other stuff on my plate at the moment, so I can't really promise super timely responses, but I'll try.)

UniqMartin on 18 Sep 2016

❤14

So it sounds like this is probably something that _should_ be fixed in properly Cabal and GHC, but may be possible to somewhat work around in Stack by shortening the paths it uses? We already do some tricks on Windows like that because of the path length limits there by putting the STACK_ROOT in C:\sr and using a hash instead of some of the longer subdirectory chains.

borsboom on 20 Sep 2016

@borsboom definitely.

ilovezfs on 20 Sep 2016

macOS Sierra has now shipped https://itunes.apple.com/us/app/macos-sierra/id1127487414?mt=12

ilovezfs on 20 Sep 2016

The same issue is being discussed over at the GHC trac.

In the mean time, what do you guys suggest that those of us who need stack on a Mac do? Is there a way to get homebrew to install prebuilt binaries of stack?

davorb on 22 Sep 2016

I'd install El Capitan in a virtual machine until this is resolved.

ilovezfs on 22 Sep 2016

I'd suggest following the manual download instructions for Mac OS X. Those will get you a binary that reportedly runs on Sierra and works for smaller projects. For projects with a large number of dependencies, stick with El Capitan (e.g. in a VM like @ilovezfs suggested).

borsboom on 22 Sep 2016

👍1

Thanks @borsboom. I just updated to macOS Sierra, and wanted to confirm that the manual download binary works for me.

My existing brew install of stack failed to work, which I had expected from reading this issue, and replacing the stack binary with the manual download version got me up and running.

To test, I just built a small lts-7.0 Spock application (103 dependencies), having removed my global .stack & project work directories first, and there are seemingly no problems with my existing brew installed ghc-8.0.1.

simon-nicholls on 23 Sep 2016

@borsboom Manual download binary worked for me as well, thank you for that.

bitemyapp on 23 Sep 2016

I've update get.haskellstack.org and the manual installation instructions to prefer the binary download over Homebrew, and added some Sierra warnings, for now.

borsboom on 23 Sep 2016

👍1

@borsboom tyvm :+1:

bitemyapp on 23 Sep 2016

I think we'll probably just boneyard it for now.

ilovezfs on 23 Sep 2016

But I NEEEED Siri! LOL FML

krambs on 23 Sep 2016

I did some experimentation with shortening the paths like we do on Windows, but no luck: still getting the GHC panic. I can't think of any other workarounds, so I guess we'll just have to wait until it's fixed upstream in GHC/Cabal.

borsboom on 25 Sep 2016

OK, then I shall reveal my workaround :)

We can simply strip the useless rpaths from the El Capitan bottle using install_name_tool and ship a Sierra bottle for this.

ilovezfs on 25 Sep 2016

I'm not using brew-installed stack, but binary manually-installed one, and I'm still having the problem (on a rather big project).

EugeneN on 25 Sep 2016

Correct, a working stack binary will only work on projects that don't themselves hit this same bug that occurs when building stack.

ilovezfs on 25 Sep 2016

@borsboom I've now shipped bottles for Sierra: https://github.com/Homebrew/homebrew-core/commit/c1dce9063bd30c97635f16e54bc8324bbecb2210

ilovezfs on 25 Sep 2016

🎉1

Thanks. The Sierra bottle works in my (study) environment.

simon-nicholls on 25 Sep 2016

I'm not using brew-installed stack, but binary manually-installed one, and I'm still having the problem (on a rather big project).

I was fine compiling a Servant project, but the yesod-simple scaffold still trips this for me.

bitemyapp on 28 Sep 2016

Yes, yesod* is known to be affected. GHC/Cabal needs to fix this. Listing affected projects here won't improve the situation.

ilovezfs on 28 Sep 2016

Sure, I'm reading along with https://ghc.haskell.org/trac/ghc/ticket/12479 to see where they've gotten so far.

bitemyapp on 28 Sep 2016

It looks like a fix is now on the way https://github.com/haskell/cabal/pull/3979.

ilovezfs on 14 Oct 2016

Now that we have haskell/cabal#3982 and haskell/cabal#3983, has there been any thought about "teaching [stack] the new flags?"

shanemikel on 18 Oct 2016

We probably won't be adding new flags to Cabal after all and go with an updated version of https://github.com/haskell/cabal/pull/3979. Instead GHC 8.0.2 will gain some new flags.

23Skidoo on 18 Oct 2016

What's the status on this issue ? If it's resolved, what we should do to fix it on MacOS Sierra ?

nedSaf on 23 Oct 2016

👍13 😕2

It wasn't entirely clear to me (or maybe I just hope I'm wrong) from haskell/cabal#3979, and the other tickets etc, but does that mean that only GHC 8.0.2 and forward will build on macOS Sierra, and older GHC versions will still have the same problem?

Tehnix on 25 Oct 2016

@Tehnix that is usually how it's gone in the past, unfortunately (e.g GHC 7.8.4 is broken on Yosemite and up). These kinds of fixes haven't been back-ported. However, if a volunteer wants to take on backporting, we could have Stack use a backported fixed version when it needs to.

borsboom on 25 Oct 2016

👍1

It's fixed in 8.0.2 and associated cabal

On Sunday, October 23, 2016, Nader Safadi [email protected] wrote:

What's the status on this issue ? If it's resolved, what we should do to
fix it on MacOS Sierra ?

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/commercialhaskell/stack/issues/2577#issuecomment-255594555,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAAQwhd2VLlnY-YbVpUoKrGVPMnOkxJ1ks5q23vDgaJpZM4J546w
.

cartazio on 25 Oct 2016

For maintainers, something to consider Late to the party: On macOS, Homebrew casks are typically used for complex/isolated/versioned (i.e., ChefDK, TeX Live and maybe stack) binary (i.e., .dmg, .pkgs, etc.) distributions. Whereas brew formulas are typically preferred for hackable/component software built from source (although bottles are pre-built binaries of the formula). Some people may want brew formulas, but a complete, vendored-dependencies, binary distribution is usually easier, cheaper and less error-prone to both deploy and support. (Casks are awesome because it's like a centralized CLI Sparkle for full lifecycle mgmt, i.e., update everything: brew cask update.)

steakknife on 25 Oct 2016

👎1

@steakknife are you sure you're in the right thread?

shanemikel on 26 Oct 2016

@shanemikel I encountered this issue some time ago. Most recently addressing commercial haskell people and others whom may need to think about process improvement in release engineering to reduce community/ development support burden. Don't have time to find the appropriate feedback channel. Good luck, hope it works out.

steakknife on 26 Oct 2016

Older ghc will work fine as long as there's no use of th in packages that have as many transitive dynamic library deps as the union of snap and yesod. Also can mitigate by making sure the paths are shorter to the dynamic libs.

Alternatively a build of ghc that uses the ghci static linker for ghci on OS X would also dodge this issue.

cartazio on 26 Oct 2016

@borsboom makes sense if that is what's usually done. I think I'll rather hope for the packages/libraries I use to be updated and perhaps help with that :)

@cartazio Unfortunately my use case is actually specifically for Yesod, so that leaves me on either 8.0.2+ for macOS Sierra, or using docker (which is quite easy with stack luckily, but sometimes hangs on the linking phase).

Tehnix on 26 Oct 2016

The yesod-sqlite template is still busted on Sierra. If GHC < 8.0.2 is busted on OSX Sierra, shouldn't there be a LTS that supports this? LTS Haskell 7.10 is ghc-8.0.1.

chadbrewbaker on 22 Nov 2016

👍2

@chadbrewbaker as of the time of this comment, the progress is at 99%. https://ghc.haskell.org/trac/ghc/roadmap

There's an open issue on this repo for backporting the relevant fixes (which I believe are available now), but... Well you can see for yourself, nobody's done it yet.

shanemikel on 22 Nov 2016

😕1

Sources for Sierra's dyld have now been posted:

https://opensource.apple.com/source/dyld/dyld-421.1/

https://opensource.apple.com/tarballs/dyld/dyld-421.1.tar.gz

https://opensource.apple.com/source/dyld/dyld-421.1/src/ImageLoader.h.auto.html

#define MAX_MACH_O_HEADER_AND_LOAD_COMMANDS_SIZE (32*1024)

https://opensource.apple.com/source/dyld/dyld-421.1/src/ImageLoaderMachO.cpp.auto.html

if ( sizeofcmds > (MAX_MACH_O_HEADER_AND_LOAD_COMMANDS_SIZE-sizeof(macho_header)) )
        dyld::throwf("malformed mach-o: load commands size (%u) > %u", sizeofcmds, MAX_MACH_O_HEADER_AND_LOAD_COMMANDS_SIZE);

Note the changes from El Capitan https://opensource.apple.com/source/dyld/dyld-360.22/src/ImageLoaderMachO.cpp.auto.html

Other Sierra sources: https://opensource.apple.com/release/os-x-1012.html

ilovezfs on 24 Nov 2016

👍4

GHC-8.0.2 RC1 is out http://downloads.haskell.org/~ghc/8.0.2-rc1/

alexanderkjeldaas on 26 Nov 2016

🎉1

@alexanderkjeldaas and that is what the Homebrew Sierra bottle is currently, btw.

ilovezfs on 26 Nov 2016

🎉2

@borsboom given that GHC and Cabal have both been fixed, should the "resolution: upstream issue" tag be removed from this issue, since the only remaining problems are actually internal to stack itself?

ilovezfs on 26 Nov 2016

@ilovezfs are you doing anything special to get brew to download 8.0.2? I'm on macOS Sierra and running brew update && brew install ghc will fetch me 8.0.1_3 even though I can see the formula should have 8.0.2 for macOS Sierra :/ ...

Tehnix on 26 Nov 2016

The name is lying. 8.0.1_3 on Sierra is 8.0.2-rc1. ghc --version will show 8.0.1.20161117

ilovezfs on 26 Nov 2016

❤1

@ilovezfs ah yeah, just figured when I was trying to manually download it. Thanks for the super quick response though! :)

Tehnix on 26 Nov 2016

how does brew solve this issue? does stack use the homebrew distributions on os x?

alexanderkjeldaas on 26 Nov 2016

brew builds the Sierra bottle from the 8.0.2-rc1 source tarball. Before that was available, we used the head of the ghc-8.0 branch. That, combined with the most recent cabal, is sufficient to resolve the problem for GHC + cabal build. The bug is still unfixed for cabal new-build, though.

Note that a more conservative approach would be to backport the fix to 8.0.1:
https://github.com/Homebrew/homebrew-core/pull/6416
(https://gist.githubusercontent.com/ilovezfs/c01cedf6dcf03d35d528e3aab244f42e/raw/1be2b95f935429553a132f0dbb7c2c25aea6c871/gistfile1.txt)

To my knowledge, stack hasn't backported the GHC fixes to their GHC binaries or made the changes to stack itself necessary to take advantage of the GHC and Cabal fixes.

stack will automatically get the GHC side of the equation once there's a stack 8.0.2 but additional changes will be needed to stack itself to actually fix the bug when using stack.

ilovezfs on 26 Nov 2016

@alexanderkjeldaas Note that Stack can use system GHC if that's available on PATH.
See system-ghc option for stack.yaml.

That's what I use to fix my builds on Sierra now.

fizruk on 26 Nov 2016

❤1 👍1

@fizruk all well and good, but stack --system-ghc upgrade --git doesn't seem to work for example. so when stack is fixed, how do we upgrade?

alexanderkjeldaas on 26 Nov 2016

@alexanderkjeldaas what does stack upgrade --git do? If it simply clones stack repo, builds and then installs then I suspect you can just do that manually?

Note that if you don't need to upgrade to the Git version, you can install Stack with brew install haskell-stack. At least until the dust settles with GHC 8.0.2.

fizruk on 26 Nov 2016

By the way, brew can now build stack from source on Sierra, though stack itself cannot build stack yet. If nothing else, it's a neat parlor trick:

brew install --without-bootstrap haskell-stack

ilovezfs on 26 Nov 2016

Another issue: The new homebrew version will not compile the ghcjs releases from http://tolysz.org/ghcjs/untested/ .

Upgrading stack to a ghc version that doesn't build ghcjs is even worse for me than not being able to build large programs with the compiler shipping with stack.

alexanderkjeldaas on 27 Nov 2016

will not compile

meaning?

Are you sure you don't need to bump the ghc version constraint or something?

ilovezfs on 27 Nov 2016

With the released haddock:

    Preprocessing library haddock-api-2.17.3...
    [ 1 of 38] Compiling Paths_haddock_api (
.stack-work/dist/x86_64-osx/Cabal-1.24.1.0/build/autogen/Paths_haddock_api.hs,
.stack-work/dist/x86_64-osx/Cabal-1.24.1.0/build/Paths_haddock_api.o )
    [ 2 of 38] Compiling Haddock.Version  ( src/Haddock/Version.hs,
.stack-work/dist/x86_64-osx/Cabal-1.24.1.0/build/Haddock/Version.o )
    [ 3 of 38] Compiling Haddock.Syb      ( src/Haddock/Syb.hs,
.stack-work/dist/x86_64-osx/Cabal-1.24.1.0/build/Haddock/Syb.o )
    [ 4 of 38] Compiling Haddock.Parser   ( src/Haddock/Parser.hs,
.stack-work/dist/x86_64-osx/Cabal-1.24.1.0/build/Haddock/Parser.o )
    [ 5 of 38] Compiling Haddock.GhcUtils ( src/Haddock/GhcUtils.hs,
.stack-work/dist/x86_64-osx/Cabal-1.24.1.0/build/Haddock/GhcUtils.o )


/private/var/folders/pq/xz5t60ld4v10j_tt7bd25rfhtzqwq8/T/stack51151/haddock-api-2.17.3/src/Haddock/GhcUtils.hs:20:1:
warning: [-Wunused-imports]
        The import of ‘Data.Function’ is redundant
          except perhaps to import instances from ‘Data.Function’
        To import instances alone, use: import Data.Function()


/private/var/folders/pq/xz5t60ld4v10j_tt7bd25rfhtzqwq8/T/stack51151/haddock-api-2.17.3/src/Haddock/GhcUtils.hs:27:1:
warning: [-Wunused-imports]
        The import of ‘RdrName’ is redundant
          except perhaps to import instances from ‘RdrName’
        To import instances alone, use: import RdrName()


/private/var/folders/pq/xz5t60ld4v10j_tt7bd25rfhtzqwq8/T/stack51151/haddock-api-2.17.3/src/Haddock/GhcUtils.hs:28:1:
warning: [-Wunused-imports]
        The import of ‘GhcMonad’ is redundant
          except perhaps to import instances from ‘GhcMonad’
        To import instances alone, use: import GhcMonad()


/private/var/folders/pq/xz5t60ld4v10j_tt7bd25rfhtzqwq8/T/stack51151/haddock-api-2.17.3/src/Haddock/GhcUtils.hs:30:1:
warning: [-Wunused-imports]
        The import of ‘UniqFM’ is redundant
          except perhaps to import instances from ‘UniqFM’
        To import instances alone, use: import UniqFM()
    [ 6 of 38] Compiling Haddock.Backends.Xhtml.Types (
src/Haddock/Backends/Xhtml/Types.hs,
.stack-work/dist/x86_64-osx/Cabal-1.24.1.0/build/Haddock/Backends/Xhtml/Types.o
)
    [ 7 of 38] Compiling Haddock.Backends.Hyperlinker.Types (
src/Haddock/Backends/Hyperlinker/Types.hs,
.stack-work/dist/x86_64-osx/Cabal-1.24.1.0/build/Haddock/Backends/Hyperlinker/Types.o
)
    [ 8 of 38] Compiling Haddock.Types    ( src/Haddock/Types.hs,
.stack-work/dist/x86_64-osx/Cabal-1.24.1.0/build/Haddock/Types.o )


/private/var/folders/pq/xz5t60ld4v10j_tt7bd25rfhtzqwq8/T/stack51151/haddock-api-2.17.3/src/Haddock/Types.hs:451:10:
error:
        Duplicate instance declarations:
          instance NFData Name -- Defined at src/Haddock/Types.hs:451:10
          instance NFData Name -- Defined in ‘Name’


/private/var/folders/pq/xz5t60ld4v10j_tt7bd25rfhtzqwq8/T/stack51151/haddock-api-2.17.3/src/Haddock/Types.hs:452:10:
error:
        Duplicate instance declarations:
          instance NFData OccName -- Defined at src/Haddock/Types.hs:452:10
          instance NFData OccName -- Defined in ‘OccName’


/private/var/folders/pq/xz5t60ld4v10j_tt7bd25rfhtzqwq8/T/stack51151/haddock-api-2.17.3/src/Haddock/Types.hs:453:10:
error:
        Duplicate instance declarations:
          instance NFData ModuleName
            -- Defined at src/Haddock/Types.hs:453:10
          instance NFData ModuleName -- Defined in ‘Module’

With haddock HEAD

    [10 of 38] Compiling Haddock.Interface.LexParseRn (
src/Haddock/Interface/LexParseRn.hs,
.stack-work/dist/x86_64-osx/Cabal-1.24.1.0/build/Haddock/Interface/LexParseRn.o
)


/Users/alekje/.stack/programs/x86_64-osx/x/ghcjs-0.2.1.9007010/.stack-work/downloaded/RdFf5gvtfQ84/haddock-api/src/Haddock/Interface/LexParseRn.hs:70:15:
error:
        • Couldn't match type
‘ghc-boot-th-8.0.1.20161117:GHC.LanguageExtensions.Type.Extension’
                         with ‘LangExt.Extension’
          NB: ‘LangExt.Extension’
                is defined in ‘GHC.LanguageExtensions.Type’
                    in package ‘ghc-boot-th-8.0.1’

‘ghc-boot-th-8.0.1.20161117:GHC.LanguageExtensions.Type.Extension’
                is defined in ‘GHC.LanguageExtensions.Type’
                    in package ‘ghc-boot-th-8.0.1.20161117’
          Expected type: [LangExt.Extension]
            Actual type:
[ghc-boot-th-8.0.1.20161117:GHC.LanguageExtensions.Type.Extension]
        • In the expression:
            map toEnum (toList $ extensionFlags dflags)
            \\ languageExtensions (language dflags)
          In an equation for ‘flags’:
              flags
                = map toEnum (toList $ extensionFlags dflags)
                  \\ languageExtensions (language dflags)
          In the expression:
            do { (hmi, doc) <- case mayStr of {
                                 Nothing -> return failure
                                 Just (L _ (HsDocString fs)) -> do { ... }
};
                 let flags :: [LangExt.Extension]
                     flags
                       = map toEnum (toList $ extensionFlags dflags)
                         \\ languageExtensions (language dflags);
                 return
                   (hmi
                      {hmi_safety = Just $ showPpr dflags safety,
                       hmi_language = language dflags, hmi_extensions =
flags},
                    doc) }

On Sun, Nov 27, 2016 at 9:02 PM, ilovezfs notifications@github.com wrote:

will not compile

meaning?

Are you sure you don't need to bump the ghc version constraint or
something?

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/commercialhaskell/stack/issues/2577#issuecomment-263143765,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAUtqQ85OW3Ea78JF9ah8s6BST73YAroks5rCeHngaJpZM4J546w
.

alexanderkjeldaas on 28 Nov 2016

This is based on
http://tolysz.org/ghcjs/untested/ghc-8.0-2016-11-27-lts-7.10-9007010.tar.gz

The second try with these changes to stack.yaml:

...
packages:
- location: .
- location:
    git:    https://github.com/haskell/haddock.git
    commit: 240bc38b94ed2d0af27333b23392d03eeb615e82
  subdirs:
  - haddock-api

extra-deps:
- haddock-api-2.17.3

On Mon, Nov 28, 2016 at 7:39 AM, Alexander Kjeldaas ak@formalprivacy.com
wrote:

>

With the released haddock:

    Preprocessing library haddock-api-2.17.3...
    [ 1 of 38] Compiling Paths_haddock_api ( .stack-work/dist/x86_64-osx/
Cabal-1.24.1.0/build/autogen/Paths_haddock_api.hs,
.stack-work/dist/x86_64-osx/Cabal-1.24.1.0/build/Paths_haddock_api.o )
    [ 2 of 38] Compiling Haddock.Version  ( src/Haddock/Version.hs,
.stack-work/dist/x86_64-osx/Cabal-1.24.1.0/build/Haddock/Version.o )
    [ 3 of 38] Compiling Haddock.Syb      ( src/Haddock/Syb.hs,
.stack-work/dist/x86_64-osx/Cabal-1.24.1.0/build/Haddock/Syb.o )
    [ 4 of 38] Compiling Haddock.Parser   ( src/Haddock/Parser.hs,
.stack-work/dist/x86_64-osx/Cabal-1.24.1.0/build/Haddock/Parser.o )
    [ 5 of 38] Compiling Haddock.GhcUtils ( src/Haddock/GhcUtils.hs,
.stack-work/dist/x86_64-osx/Cabal-1.24.1.0/build/Haddock/GhcUtils.o )

    /private/var/folders/pq/xz5t60ld4v10j_tt7bd25rfhtzqwq8/T/stack51151/
haddock-api-2.17.3/src/Haddock/GhcUtils.hs:20:1: warning:
[-Wunused-imports]
        The import of ‘Data.Function’ is redundant
          except perhaps to import instances from ‘Data.Function’
        To import instances alone, use: import Data.Function()

    /private/var/folders/pq/xz5t60ld4v10j_tt7bd25rfhtzqwq8/T/stack51151/
haddock-api-2.17.3/src/Haddock/GhcUtils.hs:27:1: warning:
[-Wunused-imports]
        The import of ‘RdrName’ is redundant
          except perhaps to import instances from ‘RdrName’
        To import instances alone, use: import RdrName()

    /private/var/folders/pq/xz5t60ld4v10j_tt7bd25rfhtzqwq8/T/stack51151/
haddock-api-2.17.3/src/Haddock/GhcUtils.hs:28:1: warning:
[-Wunused-imports]
        The import of ‘GhcMonad’ is redundant
          except perhaps to import instances from ‘GhcMonad’
        To import instances alone, use: import GhcMonad()

    /private/var/folders/pq/xz5t60ld4v10j_tt7bd25rfhtzqwq8/T/stack51151/
haddock-api-2.17.3/src/Haddock/GhcUtils.hs:30:1: warning:
[-Wunused-imports]
        The import of ‘UniqFM’ is redundant
          except perhaps to import instances from ‘UniqFM’
        To import instances alone, use: import UniqFM()
    [ 6 of 38] Compiling Haddock.Backends.Xhtml.Types (
src/Haddock/Backends/Xhtml/Types.hs, .stack-work/dist/x86_64-osx/
Cabal-1.24.1.0/build/Haddock/Backends/Xhtml/Types.o )
    [ 7 of 38] Compiling Haddock.Backends.Hyperlinker.Types (
src/Haddock/Backends/Hyperlinker/Types.hs, .stack-work/dist/x86_64-osx/
Cabal-1.24.1.0/build/Haddock/Backends/Hyperlinker/Types.o )
    [ 8 of 38] Compiling Haddock.Types    ( src/Haddock/Types.hs,
.stack-work/dist/x86_64-osx/Cabal-1.24.1.0/build/Haddock/Types.o )

    /private/var/folders/pq/xz5t60ld4v10j_tt7bd25rfhtzqwq8/T/stack51151/
haddock-api-2.17.3/src/Haddock/Types.hs:451:10: error:
        Duplicate instance declarations:
          instance NFData Name -- Defined at src/Haddock/Types.hs:451:10
          instance NFData Name -- Defined in ‘Name’

    /private/var/folders/pq/xz5t60ld4v10j_tt7bd25rfhtzqwq8/T/stack51151/
haddock-api-2.17.3/src/Haddock/Types.hs:452:10: error:
        Duplicate instance declarations:
          instance NFData OccName -- Defined at src/Haddock/Types.hs:452:10
          instance NFData OccName -- Defined in ‘OccName’

    /private/var/folders/pq/xz5t60ld4v10j_tt7bd25rfhtzqwq8/T/stack51151/
haddock-api-2.17.3/src/Haddock/Types.hs:453:10: error:
        Duplicate instance declarations:
          instance NFData ModuleName
            -- Defined at src/Haddock/Types.hs:453:10
          instance NFData ModuleName -- Defined in ‘Module’

With haddock HEAD

    [10 of 38] Compiling Haddock.Interface.LexParseRn (
src/Haddock/Interface/LexParseRn.hs, .stack-work/dist/x86_64-osx/
Cabal-1.24.1.0/build/Haddock/Interface/LexParseRn.o )

    /Users/alekje/.stack/programs/x86_64-osx/x/ghcjs-0.2.1.
9007010/.stack-work/downloaded/RdFf5gvtfQ84/haddock-api/src/Haddock/Interface/LexParseRn.hs:70:15:
error:
        • Couldn't match type ‘ghc-boot-th-8.0.1.20161117:
GHC.LanguageExtensions.Type.Extension’
                         with ‘LangExt.Extension’
          NB: ‘LangExt.Extension’
                is defined in ‘GHC.LanguageExtensions.Type’
                    in package ‘ghc-boot-th-8.0.1’
              ‘ghc-boot-th-8.0.1.20161117:GHC.LanguageExtensions.Type.
Extension’
                is defined in ‘GHC.LanguageExtensions.Type’
                    in package ‘ghc-boot-th-8.0.1.20161117’
          Expected type: [LangExt.Extension]
            Actual type: [ghc-boot-th-8.0.1.20161117:
GHC.LanguageExtensions.Type.Extension]
        • In the expression:
            map toEnum (toList $ extensionFlags dflags)
            \\ languageExtensions (language dflags)
          In an equation for ‘flags’:
              flags
                = map toEnum (toList $ extensionFlags dflags)
                  \\ languageExtensions (language dflags)
          In the expression:
            do { (hmi, doc) <- case mayStr of {
                                 Nothing -> return failure
                                 Just (L _ (HsDocString fs)) -> do { ...
} };
                 let flags :: [LangExt.Extension]
                     flags
                       = map toEnum (toList $ extensionFlags dflags)
                         \\ languageExtensions (language dflags);
                 return
                   (hmi
                      {hmi_safety = Just $ showPpr dflags safety,
                       hmi_language = language dflags, hmi_extensions =
flags},
                    doc) }

On Sun, Nov 27, 2016 at 9:02 PM, ilovezfs notifications@github.com
wrote:

will not compile

meaning?

Are you sure you don't need to bump the ghc version constraint or
something?

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/commercialhaskell/stack/issues/2577#issuecomment-263143765,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAUtqQ85OW3Ea78JF9ah8s6BST73YAroks5rCeHngaJpZM4J546w
.

alexanderkjeldaas on 28 Nov 2016

For someone looking for TL;DR:

This is because macOS Sierra has changed something internal.
This will be fixed in GHC 8.0.2, which is in RC. Which means it will be fixed in Stack when a next resolver using GHC 8.0.2 is released. Current Stack resolvers use GHC 8.0.1.
brew install ghc will install GHC 8.0.2 RC. (it shows 8.0.1 when ghc --version though)
We can force Stack to use the ghc installed by brew with stack --system-ghc.
For example:
$ brew install ghc $ stack --system-ghc build

I have just tried and saw the problem solved. Please correct me if anything is wrong.

hatashiro on 2 Dec 2016

👍20 🎉1

It appears that you can also use GHC 8.0.2-rc1 directly with Stack (no brew needed) with a stack.yaml like this one. Basically you just tell Stack where to get GHC 8.0.2-rc1 for every OS you need. I'll copy the contents here (note that ghc-8.0.1.20161117 is what GHC 8.0.2-rc1 is called):

compiler-check: match-exact
resolver: lts-7.10
compiler: ghc-8.0.1.20161117
setup-info:
  ghc:
    linux64:
      8.0.1.20161117:
        url: http://downloads.haskell.org/~ghc/8.0.2-rc1/ghc-8.0.1.20161117-x86_64-deb8-linux.tar.xz
        content-length: 112047972
        sha1: 6a6e4c9c53c71cc84b6966a9f61948542fd2f15a
    macosx:
      8.0.1.20161117:
        url: https://downloads.haskell.org/~ghc/8.0.2-rc1/ghc-8.0.1.20161117-x86_64-apple-darwin.tar.xz
        content-length: 113379688
        sha1: 53ed03d986a49ea680c291540ce44ce469514d7c
    windows64:
      8.0.1.20161117:
        url: https://downloads.haskell.org/~ghc/8.0.2-rc1/ghc-8.0.1.20161117-x86_64-unknown-mingw32.tar.xz
        content-length: 155652048
        sha1: 74118dd8fd8b5e4c69b25df1644273fbe13177c7

fizruk on 2 Dec 2016

❤5 👍2

@alexanderkjeldaas
It is an interesting find. But once the lts moves to ghc 8.0.2 all packages will be compiling together.
Thus this issue will go away. For now, you would need to find versions which work and you could create a new ghcjs bundle.

tolysz on 2 Dec 2016

Now that GHC 8.0.2 is out, can anyone confirm that this fixes the problem? I'm still on old OS X, but I have clients who are on Sierra.

chrisdone on 17 Feb 2017

I can confirm this fixes the problem

lorenzo on 17 Feb 2017

🎉5

Fixed problem for me as well.

fizruk on 17 Feb 2017

Seems alright here, modulo abandoning 7.8 and 7.10 on my Mac.

bitemyapp on 17 Feb 2017

chrisdone on 18 Feb 2017

Can something be done to speed up compatibility with latest OS releases? Six months after official release (not counting OS public beta) feels on the long side.

ddacunha on 18 Feb 2017

@ddacunha take it to the GHC team

bitemyapp on 18 Feb 2017

👍2

Should the comment on the install doc be updated? https://docs.haskellstack.org/en/stable/install_and_upgrade/#mac-os-x says "macOS Sierra warning: There are new limitations in the dynamic linker that are causing problems for GHC when building projects with many dependencies. See #2577 for more information."

vertexcite on 20 Feb 2017

@vertexcite It could be updated to mention that this is fixed in GHC 8.0.2/LTS 8. A PR would be welcome!

borsboom on 20 Feb 2017

@borsboom PR #3016

vertexcite on 21 Feb 2017

I'm still seeing this issue on macOS Sierra 10.12.3.
In my .stack/global-project/stack.yaml, I set the resolver to lts-8.5.
Then I ran stack upgrade in my home directory and I got:

[ 11 of 124] Compiling System.Process.Read ( src/System/Process/Read.hs, .stack-work/dist/x86_64-osx/Cabal-1.22.5.0/build/System/Process/Read.o )
ghc: panic! (the 'impossible' happened)
  (GHC version 7.10.3 for x86_64-apple-darwin):
        Loading temp shared object failed: dlopen(/var/folders/cg/wclcdwbs4cd9qppgccbc5rwh0000gn/T/ghc94925_0/libghc_68.dylib, 5): no suitable image found.  Did find:
        /var/folders/cg/wclcdwbs4cd9qppgccbc5rwh0000gn/T/ghc94925_0/libghc_68.dylib: malformed mach-o: load commands size (50712) > 32768

In the error message I see "GHC version 7.10.3", which is really odd. If I run stack path | grep ghc I only see references to 8.0.2, and nothing about 7.10.3.

Anyone know what the problem might be?

jasonstolaruk on 19 Mar 2017

https://github.com/Homebrew/homebrew-core/blob/master/Formula/haskell-stack.rb#L45

You need to pass --stack-yaml=stack-8.0.yaml

ilovezfs on 19 Mar 2017

Thank you for the quick reply! Unfortunately that didn't work and I'm still seeing the same error message...

jasonstolaruk on 19 Mar 2017

@jasonstolaruk stack upgrade in older versions of Stack are going to try to use the preferred LTS version to build stack-1.4.0, which is lts-6, regardless of your global project's default resolver. Newer versions of stack upgrade will just download a binary instead. Your best bet is to do a manual download from https://github.com/commercialhaskell/stack/releases. Future upgrades should be easier.

borsboom on 19 Mar 2017

@borsboom Ah, that would explain it! Yeah, I think my current version of stack is fairly old. I'll try that, thanks!

jasonstolaruk on 19 Mar 2017

@borsboom @jasonstolaruk I just got this with Stack 1.4.1, LTS 8.4, and Sierra. Not sure where to report, so chiming in here again.

    [ 2 of 16] Compiling Settings.StaticFiles ( src/Settings/StaticFiles.hs, .stack-work/dist/x86_64-osx/Cabal-1.24.2.0/build/Settings/StaticFiles.o )
    ghc: panic! (the 'impossible' happened)
      (GHC version 8.0.2 for x86_64-apple-darwin):
        Loading temp shared object failed: dlopen(/var/folders/gq/rh0h8yns2p1bb5d31mdx69b40000gn/T/ghc37966_0/libghc_33.dylib, 5): no suitable image found.  Did find:
        /var/folders/gq/rh0h8yns2p1bb5d31mdx69b40000gn/T/ghc37966_0/libghc_33.dylib: malformed mach-o: load commands size (33560) > 32768

Jason, are you sorted now?

It's one project of several at work that is tripping over this. Everything else builds.

Should I file a new ticket? Should it be with Stack?

bitemyapp on 3 Apr 2017

Just to confirm, still getting this with a Stack binary provisioned by the website via curl or manual download.

$ stack --version
Version 1.4.0, Git revision e714f1dd3fade19496d91bd6a017e435a96a6bcd (4640 commits) x86_64 hpack-0.17.0

bitemyapp on 3 Apr 2017

@bitemyapp Not sure where that should be filed. Feel free to open a ticket, but it'll take a minimal repro for us to investigate it. Seems more likely to be a ghc issue.

mgsloan on 3 Apr 2017

@bitemyapp Yep, I was able to build just now. Looks like I'm good.

macOS Sierra 10.12.3
Stack Version 1.4.0, Git revision e714f1dd3fade19496d91bd6a017e435a96a6bcd (4640 commits) x86_64 hpack-0.17.0
resolver: nightly-2017-04-02

jasonstolaruk on 3 Apr 2017

This isn't and never was a GHC issue, it's a new limit added by Apple, for which a workaround was added to Cabal, which required a new feature in GHC.

If your project has enough dependencies it's possible to hit the limit again even with the workaround in place. The output of otool -l on the temporary shared object (invoke GHC with -keep-tmp-files) will clarify what is happening.

rwbarton on 3 Apr 2017

I don't think it's possible for Stack to do anything about this. Even if it is possible to add a workaround to Stack, it's properly fixed in Cabal and/or GHC. It's annoying when Apple does this sort of thing, but unfortunately we're stuck with it (it's not a macOS bug; they intentionally added that limit, and we will just have to adapt).

borsboom on 4 Apr 2017

@chrisdone I can confirm that this still comes up for me on ghc-8.0.2 and lts-8.5 while building a yesod-based application:

    ghc: panic! (the 'impossible' happened)
      (GHC version 8.0.2 for x86_64-apple-darwin):
        Loading temp shared object failed: dlopen(/var/folders/l2/zpkx9mks1n12dbj0cy16nfn40000gn/T/ghc34456_0/libghc_10.dylib, 5): no suitable image found.  Did find:
        /var/folders/l2/zpkx9mks1n12dbj0cy16nfn40000gn/T/ghc34456_0/libghc_10.dylib: malformed mach-o: load commands size (33280) > 32768

    Please report this as a GHC bug:  http://www.haskell.org/ghc/reportabug

As @rwbarton points out, you can still hit the limit even with the workaround added to Cabal.

In my case, I may be able to shorted the length of some paths because I currently soft link ~/.stack to elsewhere (in my dotfiles repo) so that it picks up my .stack/config.yaml.

I wonder if there might be any benefit in soft linking ~/.stack to say /s to shorten paths further?

steshaw on 4 Apr 2017

A quick follow-up. For this app, I wasn't able to get it to build when I moved .stack to the usual location (ie. /Users/steshaw/.stack). However, when I soft linked /s to ~/.stack, I managed to get a successful build.

The tail end of the output from otool -l shows that the difference is made when addressing packages that come with ghc. Also, there is only a couple of hundred bytes to spare for sizeofcmds.

$ otool -l <file>.dylib | grep /s/programs
         path /s/programs/x86_64-osx/ghc-8.0.2/lib/ghc-8.0.2/process-1.4.3.0 (offset 12)
         path /s/programs/x86_64-osx/ghc-8.0.2/lib/ghc-8.0.2/directory-1.3.0.0 (offset 12)
         path /s/programs/x86_64-osx/ghc-8.0.2/lib/ghc-8.0.2/filepath-1.4.1.1 (offset 12)
         path /s/programs/x86_64-osx/ghc-8.0.2/lib/ghc-8.0.2/template-haskell-2.11.1.0 (offset 12)
         path /s/programs/x86_64-osx/ghc-8.0.2/lib/ghc-8.0.2/pretty-1.1.3.3 (offset 12)
         path /s/programs/x86_64-osx/ghc-8.0.2/lib/ghc-8.0.2/ghc-boot-th-8.0.2 (offset 12)
         path /s/programs/x86_64-osx/ghc-8.0.2/lib/ghc-8.0.2/unix-2.7.2.1 (offset 12)
         path /s/programs/x86_64-osx/ghc-8.0.2/lib/ghc-8.0.2/time-1.6.0.1 (offset 12)
         path /s/programs/x86_64-osx/ghc-8.0.2/lib/ghc-8.0.2/transformers-0.5.2.0 (offset 12)
         path /s/programs/x86_64-osx/ghc-8.0.2/lib/ghc-8.0.2/binary-0.8.3.0 (offset 12)
         path /s/programs/x86_64-osx/ghc-8.0.2/lib/ghc-8.0.2/containers-0.5.7.1 (offset 12)
         path /s/programs/x86_64-osx/ghc-8.0.2/lib/ghc-8.0.2/bytestring-0.10.8.1 (offset 12)
         path /s/programs/x86_64-osx/ghc-8.0.2/lib/ghc-8.0.2/deepseq-1.4.2.0 (offset 12)
         path /s/programs/x86_64-osx/ghc-8.0.2/lib/ghc-8.0.2/array-0.5.1.1 (offset 12)
         path /s/programs/x86_64-osx/ghc-8.0.2/lib/ghc-8.0.2/base-4.9.1.0 (offset 12)
         path /s/programs/x86_64-osx/ghc-8.0.2/lib/ghc-8.0.2/integer-gmp-1.0.0.1 (offset 12)
         path /s/programs/x86_64-osx/ghc-8.0.2/lib/ghc-8.0.2/ghc-prim-0.5.0.0 (offset 12)
         path /s/programs/x86_64-osx/ghc-8.0.2/lib/ghc-8.0.2/rts (offset 12)

steshaw on 4 Apr 2017

      cmdsize 80
         path /s/programs/x86_64-osx/ghc-8.0.2/lib/ghc-8.0.2/directory-1.3.0.0 (offset 12)
Load command 125
          cmd LC_RPATH
      cmdsize 80
         path /s/programs/x86_64-osx/ghc-8.0.2/lib/ghc-8.0.2/filepath-1.4.1.1 (offset 12)
Load command 126
          cmd LC_RPATH
      cmdsize 80
         path /s/programs/x86_64-osx/ghc-8.0.2/lib/ghc-8.0.2/ghc-boot-th-8.0.2 (offset 12)
Load command 127
          cmd LC_RPATH
      cmdsize 80
         path /s/programs/x86_64-osx/ghc-8.0.2/lib/ghc-8.0.2/ghc-prim-0.5.0.0 (offset 12)
Load command 128
          cmd LC_RPATH
      cmdsize 80
         path /s/programs/x86_64-osx/ghc-8.0.2/lib/ghc-8.0.2/integer-gmp-1.0.0.1 (offset 12)
Load command 129
          cmd LC_RPATH
      cmdsize 80
         path /s/programs/x86_64-osx/ghc-8.0.2/lib/ghc-8.0.2/pretty-1.1.3.3 (offset 12)
Load command 130
          cmd LC_RPATH
      cmdsize 80
         path /s/programs/x86_64-osx/ghc-8.0.2/lib/ghc-8.0.2/process-1.4.3.0 (offset 12)

...

$ otool -l .stack-work/install/x86_64-osx/lts-8.4/8.0.2/lib/x86_64-osx-ghc-8.0.2/libHSyesod-core-1.4.32-G0tj9L3iIRq7jY7gWqui14-ghc8.0.2.dylib | grep /s/programs
         path /s/programs/x86_64-osx/ghc-8.0.2/lib/ghc-8.0.2/array-0.5.1.1 (offset 12)
         path /s/programs/x86_64-osx/ghc-8.0.2/lib/ghc-8.0.2/base-4.9.1.0 (offset 12)
         path /s/programs/x86_64-osx/ghc-8.0.2/lib/ghc-8.0.2/binary-0.8.3.0 (offset 12)
         path /s/programs/x86_64-osx/ghc-8.0.2/lib/ghc-8.0.2/bytestring-0.10.8.1 (offset 12)
         path /s/programs/x86_64-osx/ghc-8.0.2/lib/ghc-8.0.2/containers-0.5.7.1 (offset 12)
         path /s/programs/x86_64-osx/ghc-8.0.2/lib/ghc-8.0.2/deepseq-1.4.2.0 (offset 12)
         path /s/programs/x86_64-osx/ghc-8.0.2/lib/ghc-8.0.2/directory-1.3.0.0 (offset 12)
         path /s/programs/x86_64-osx/ghc-8.0.2/lib/ghc-8.0.2/filepath-1.4.1.1 (offset 12)
         path /s/programs/x86_64-osx/ghc-8.0.2/lib/ghc-8.0.2/ghc-boot-th-8.0.2 (offset 12)
         path /s/programs/x86_64-osx/ghc-8.0.2/lib/ghc-8.0.2/ghc-prim-0.5.0.0 (offset 12)
         path /s/programs/x86_64-osx/ghc-8.0.2/lib/ghc-8.0.2/integer-gmp-1.0.0.1 (offset 12)
         path /s/programs/x86_64-osx/ghc-8.0.2/lib/ghc-8.0.2/pretty-1.1.3.3 (offset 12)
         path /s/programs/x86_64-osx/ghc-8.0.2/lib/ghc-8.0.2/process-1.4.3.0 (offset 12)
         path /s/programs/x86_64-osx/ghc-8.0.2/lib/ghc-8.0.2/rts (offset 12)
         path /s/programs/x86_64-osx/ghc-8.0.2/lib/ghc-8.0.2/template-haskell-2.11.1.0 (offset 12)
         path /s/programs/x86_64-osx/ghc-8.0.2/lib/ghc-8.0.2/time-1.6.0.1 (offset 12)
         path /s/programs/x86_64-osx/ghc-8.0.2/lib/ghc-8.0.2/transformers-0.5.2.0 (offset 12)
         path /s/programs/x86_64-osx/ghc-8.0.2/lib/ghc-8.0.2/unix-2.7.2.1 (offset 12)

Thank you for sharing this tactic @steshaw, still not quite enough in my case:

(GHC version 8.0.2 for x86_64-apple-darwin):
        Loading temp shared object failed: dlopen(/var/folders/gq/rh0h8yns2p1bb5d31mdx69b40000gn/T/ghc76548_0/libghc_33.dylib, 5): no suitable image found.  Did find:
        /var/folders/gq/rh0h8yns2p1bb5d31mdx69b40000gn/T/ghc76548_0/libghc_33.dylib: malformed mach-o: load commands size (33288) > 32768

Other than the irritation of getting dyld to compile (you have to start vendoring the darwin userland into a build env), is there a reason people haven't made a fork that disables the byte limit? /cc @borsboom

bitemyapp on 4 Apr 2017

It's the loader itself that has the limit: https://github.com/commercialhaskell/stack/issues/2577#issuecomment-262725233

ilovezfs on 4 Apr 2017

@ilovezfs any idea as to why Apple did this?

bitemyapp on 4 Apr 2017

@bitemyapp I believe it's purely an optimization measure to improve load times, but there may also be undisclosed security issues with the prior versions for all we know. (Also, no one should run into the limit in the first place when libraries are arranged as intended.)

ilovezfs on 4 Apr 2017

The LC_LOAD_DYLIB load command for one library is typically 72 bytes long (depending on the library's file name length), so even with no other load commands at all (in particular, not counting any LC_RPATH entries) that puts a limit of around 500 on the number of dependencies. My understanding is that many Yesod-based projects already comfortably exceed half this number so it's a limit that does not seem very far off.

rwbarton on 4 Apr 2017

@rwbarton: would it help to shorten the library paths within ~/.stack? On Windows replace some of the deeper paths with a short hash to work around path length limits, and we could do something like that on macOS too (although I'd much rather not, since it makes it harder for a human to navigate ~/.stack).

borsboom on 5 Apr 2017

@borsboom My load commands typically look something like this:

          cmd LC_LOAD_DYLIB
      cmdsize 72
         name @rpath/libHSghc-boot-th-8.0.2-ghc8.0.2.dylib (offset 24)
   time stamp 2 Wed Dec 31 19:00:02 1969

It's using @rpath to handle the path prefix. I don't know how the bytes are calculated by the runtime loader, but I'm not sure what more you can do.

This is a touch frustrating as the project that won't build is a pretty young Yesod project with not many extra dependencies beyond Yesod + lens.

bitemyapp on 5 Apr 2017

👍1

Stack could create a library in intermediate steps, while the tree of dynamic libraries would be bigger, the limit could be mitigated.

I'm looking into possible solutions, for Nix one would be just to patch Apple source.

domenkozar on 20 Jul 2017

Can anyone confirm that this is fixed in _macOS High Sierra_? I was previously on _Sierra_ but had to wipe my machine and downgrade to _El Capitan_ to get work done. I'm wondering if it's safe to upgrade to _High Sierra_ for a yesod project with lots of dependencies (342 ?!?).

As far as I can tell, the GHC bug is fixed ... but it does look like someone had a similar problem on _Sierra_ with it 7 weeks ago. Probably not fixed?

steshaw on 10 Dec 2017

@steshaw it's not fixed, as the GHC fix only makes it harder to hit the limit in linker.

In Nix, we have a hack to reexport symbols to never reach the limit in one executable, so that should work with Stack.

I suggest you try out your project on Sierra (High Sierra doesn't change anything) and try to use Nix to provide the linker hack.

Long term solution is for haskell libraries not to use such long paths, now that stack/nix use separate folders and naming doesn't have to account for namespacing.

domenkozar on 10 Dec 2017

It’s still broken for me on High Sierra. I use docker container with ubuntu
and some rsync-ing to get the job done.

On Sun, Dec 10, 2017 at 06:58 Domen Kožar notifications@github.com wrote:

@steshaw https://github.com/steshaw it's not fixed, as the GHC fix only
makes it harder to hit the limit in linker.

In Nix, we have a hack to reexport symbols to never reach the limit in one
executable, so that should work with Stack.

I suggest you try out your project on Sierra (High Sierra doesn't change
anything) and try to use Nix to provide the linker hack.

Long term solution is for haskell libraries not to use such long paths,
now that stack/nix use separate folders and naming doesn't have to account
for namespacing.

—
You are receiving this because you are subscribed to this thread.
Reply to this email directly, view it on GitHub
https://github.com/commercialhaskell/stack/issues/2577#issuecomment-350543254,
or mute the thread
https://github.com/notifications/unsubscribe-auth/ABNscrZrcjU1uAvSLzIUBgwHMorxpRPFks5s-8dhgaJpZM4J546w
.

EugeneN on 10 Dec 2017

I am running into this as well. Would it be reasonable to use Nix's workaround inside Stack? I am not familiar with Stack's internals, but if anyone thinks that would be a good approach I can investigate it.

Also, perhaps this issue should be reopened?

asivitz on 27 Feb 2018

@asivitz Stack had already done the rpath shortening trick, I think this is what remains as a more permanent fix.

https://github.com/NixOS/nixpkgs/pull/27536

bitemyapp on 27 Feb 2018

Yikes, I think that's a little more involved than I hoped.

Btw, I tried to make a controlled reproduction case by generating a giant stack project with a bunch of sub-packages, but I couldn't cause the panic. I wonder what I'm missing? https://gist.github.com/asivitz/f4b983b2374a6155ac4faaf9b61aca59

asivitz on 27 Feb 2018

@asivitz I think they need to be dependencies rather than modules? Make a Stack project that depends on acme-everything and see if that trips it.

bitemyapp on 27 Feb 2018

They are dependencies actually. It's a top-level library that depends on 400 sub-packages. It's pretty weird because when I looked at the output for my failure it was around 300 deps, and this one has 400 (and longer filenames also) and it doesn't trip it.

asivitz on 27 Feb 2018

@asivitz Oh my bad, I only glanced at the script. I've found reproducing the failure with Stack to be inconsistent in this way too, FWIW.

Did you try executing it or using some Template Haskell in it? I think you can force a linker failure during compile-time if you use TH.

bitemyapp on 27 Feb 2018

@bitemyapp Nope! Didn't try that. I'll give it a shot, thanks!

asivitz on 28 Feb 2018

@asivitz AIUI this is the runtime linker in the Darwin kernel that is imposing the kilobyte limit on the paths. macOS doesn't actually permit static linking at all. Given that, the only way to trip it that I'm aware of is for TH to force runtime linking at compile time or to run the program.

bitemyapp on 28 Feb 2018

Ah, ok that's super helpful, thanks. I added some TH and am now able to reproduce it reliably. I updated the gist.

asivitz on 28 Feb 2018

👍1

We have worked around this issue by following in the Nix folks' footsteps and using a wrapper script around ld to recursively subdivide the dependencies into a tree of re-exporting delegate libraries.

The script, an example using @asivitz's repro, and more info about this issue is available here:
https://github.com/Simspace/ld-wrapper-macos

Hope this helps!

jship on 19 Apr 2018

❤8 👍7

I wonder: would it be possible to modify the GHC settings file to use this wrapper for all builds (rather than having to add the flag to every project)? If so, it would be possible to have Stack use patched GHC bindists on macOS that includes this fix.

borsboom on 19 Apr 2018

Sure, but you have to add -fuse-ld=ld-wrapper-macos.sh to the C compiler flags section of the settings file, you can't simply set the ld command option.

edit: for stack, the path for the settings file looks like ~/.stack/programs/x86_64-osx/ghc-8.2.2/lib/ghc-8.2.2/settings

gelisam on 19 Apr 2018

So we could, in theory, fix this for macOS users by providing patched bindists that include the wrapper script and a modified configure to update settings to use it on macOS >= 10.12. I'm always hesitant about this sort of thing, because it's one more thing we have to support going forward and would be best fixed by the GHC team themselves, but in this case there are enough macOS users and this causes enough pain that it might be worth doing.

borsboom on 19 Apr 2018

👍1

I've changed my settings to have the following flags and it built my project on High Sierra. Thank you!

("C compiler link flags", " -m64 -fuse-ld=ld-wrapper-macos.sh")

steshaw on 20 Apr 2018

🎉7

@steshaw I'm getting

clang: error: argument unused during compilation: '-fuse-ld=ld-wrapper-macos.sh' [-Werror,-Wunused-command-line-argument]

I'm on stack 1.6.5. Have you seen this problem?

brandon-leapyear on 10 May 2018

Just ignore that warning instead of making it fail with -Werror; the warning comes from clang, who noticed that you gave it linking flags in a clang invocation which doesn't perform any linking. This in turn occurs because adding -fuse-ld=ld-wrapper-macos.sh to C compiler link flags causes ghc to add this flag to all of its invocations of clang, both those which perform linking and those which don't.

gelisam on 10 May 2018

👍1

@brandon-leapyear @gelisam I don't see that error. Perhaps it's because I updated C compiler link flags rather than C compiler flags in ~/.stack/programs/x86_64-osx/ghc-8.0.2/lib/ghc-8.0.2/settings.

I do get some warnings during the linking stage like this:

/Users/steshaw/.stack/programs/x86_64-osx/ghc-8.0.2/lib/ghc-8.0.2/include/rts/storage/ClosureMacros.h:552:32: error:
     warning: macro expansion producing 'defined' has undefined behavior [-Wexpansion-to-defined]
#if ZERO_SLOP_FOR_LDV_PROF && !ZERO_SLOP_FOR_SANITY_CHECK
                               ^
...
9 warnings generated.

Seems those can be safely ignored.

steshaw on 11 May 2018

👍2

Stack: GHC panic when building Stack on macOS Sierra

Most helpful comment

All 140 comments

Related issues