Stack: --profile and -dynamic builds fail

[blitzcode]

I frequently use -dynamic in my GHC options to reduce link times during development. Profile builds seem to first rebuild their dependencies properly, but then fail to find them during compilation of the actual project. It seems like only static versions of the profile libraries are build.

Here's a project of mine with quite a few dependencies and -dynamic set in the Cabal file:

[~] stack --version
Version 1.0.0 x86_64

[~] git clone [email protected]:blitzcode/ray-marching-distance-fields.git
...
[~] cd ray-marching-distance-fields/
[~/ray-marching-distance-fields] stack build --profile
ansi-terminal-0.6.2.3: copying precompiled package
base-orphans-0.4.5: configure
base-orphans-0.4.5: build
ObjectName-1.1.0.1: configure
ObjectName-1.1.0.1: build
...
rmdf-0.1.0.0: configure
Configuring rmdf-0.1.0.0...
rmdf-0.1.0.0: build
Preprocessing executable 'rmdf' for rmdf-0.1.0.0...

/Users/Tim/ray-marching-distance-fields/App.hs:16:8:
    Could not find module ‘Control.Monad.Reader’
    Perhaps you haven't installed the "p_dyn" libraries for package                ‘mtl-2.2.1@mtl_Aue4leSeVkpKLsfHIV51E8’?
    Use -v to see a list of the files searched for.

/Users/Tim/ray-marching-distance-fields/App.hs:17:8:
    Could not find module ‘Control.Monad.State’
    Perhaps you haven't installed the "p_dyn" libraries for package ‘mtl-2.2.1@mtl_Aue4leSeVkpKLsfHIV51E8’?
    Use -v to see a list of the files searched for.

/Users/Tim/ray-marching-distance-fields/App.hs:18:8:
    Could not find module ‘Control.Monad.STM’
    Perhaps you haven't installed the "p_dyn" libraries for package ‘stm-2.4.4.1@stm_EyxFXFfnfY05IrjGaqCxeD’?
    Use -v to see a list of the files searched for.

...

[sjakobi]

I'm seeing a similar error when building the stack project with --profile:

~/s/stack (master) $ stack clean
~/s/stack (master) $ stack build --profile
stack-1.0.5: unregistering (local file changes: src/Control/Concurrent/Execute.hs src/Data/Aeson/Extended.hs src/Data/Attoparsec/Args.hs src/Data...)
stack-1.0.5: configure
Configuring stack-1.0.5...
stack-1.0.5: build
Preprocessing library stack-1.0.5...
[ 1 of 86] Compiling Stack.FileWatch  ( src/Stack/FileWatch.hs, .stack-work/dist/x86_64-linux/Cabal-1.22.7.0/build/Stack/FileWatch.o )
...
[86 of 86] Compiling Stack.SDist      ( src/Stack/SDist.hs, .stack-work/dist/x86_64-linux/Cabal-1.22.7.0/build/Stack/SDist.p_o )
In-place registering stack-1.0.5...
Preprocessing executable 'stack' for stack-1.0.5...
[1 of 2] Compiling Paths_stack      ( .stack-work/dist/x86_64-linux/Cabal-1.22.7.0/build/autogen/Paths_stack.hs, .stack-work/dist/x86_64-linux/Cabal-1.22.7.0/build/stack/stack-tmp/Paths_stack.p_o )
[2 of 2] Compiling Main             ( src/main/Main.hs, .stack-work/dist/x86_64-linux/Cabal-1.22.7.0/build/stack/stack-tmp/Main.p_o )
/home/simon/src/stack/src/main/Main.hs:113:10:
    cannot find normal object file ‘.stack-work/dist/x86_64-linux/Cabal-1.22.7.0/build/stack/stack-tmp/Paths_stack.dyn_o’
    while linking an interpreted expression

--  While building package stack-1.0.5 using:
      /home/simon/.stack/setup-exe-cache/x86_64-linux/setup-Simple-Cabal-1.22.7.0-ghc-7.10.3 --builddir=.stack-work/dist/x86_64-linux/Cabal-1.22.7.0 build lib:stack exe:stack --ghc-options " -ddump-hi -ddump-to-file"
    Process exited with code: ExitFailure 1

There's no dynamic flag involved here AFAIK.

~/s/stack (master) $ stack --version
Version 1.0.5, Git revision 57eb614df50826f38b923e1892cda904288a8ec9 (dirty) (3489 commits) x86_64 hpack-0.12.0

Is there any reason to believe that this is a GHC or Cabal issue or is this likely a bug in stack itself?

[mgsloan]

Hmm, that's very strange! I have a similar stack version, same ghc, same Cabal, but it doesn't repro. It's a long shot, but is it solved by the most recent HEAD?

It'd be helpful to have a dockerfile repo + ideally an uploaded container

[sjakobi]

Hmm, that's very strange! I have a similar stack version, same ghc, same Cabal, but it doesn't repro. It's a long shot, but is it solved by the most recent HEAD?

57eb614 is the latest HEAD.

It'd be helpful to have a dockerfile repo + ideally an uploaded container

I'll give this a try in the next days.

[sjakobi]

I've also reproduced this locally with stack-1.0.4.3 and stack-1.0.2.

Using stack-HEAD and stack-7.8.yaml, i.e. GHC-7.8.4 and Cabal-1.18.1.5, leads to the same error.

[sjakobi]

I've run into another instance of this bug, this time closer to what @blitzcode experienced:

• CLI output here
• Builds logs just in case here

In this case the missing profiling libraries are supposed to come from a _procompiled package_! I think this might be a hint at the root of the problem.

BTW, during this run I had set apply-ghc-options to locals in the stack.yaml. Because of this I was a bit surprised that the snapshot libraries had to be recompiled at all. Is this a misunderstanding on the intended behaviour of --profile on my part?

$ stack --version
Version 1.0.4.3, Git revision db1afb275ebd64016e8ef0db122f0f03da9e5906 (3242 commits) x86_64

[mgsloan]

IIRC, --profile builds still require compilation of all dependencies with profiling, regardless of the apply-ghc-options setting

[schell]

It seems I'm also running into this.

stack --version
Version 1.1.2 x86_64 hpack-0.14.0

My build was failing like so

    /Users/schell/Code/shapes/shapes-math/src/Shapes/Linear/Double.hs:1:1:
        cannot find normal object file ‘.stack-work/dist/x86_64-osx/Cabal-1.22.5.0/build/math-bench/math-bench-tmp/Shapes/Linear/Template.dyn_o’
        while linking an interpreted expression

The executable that was failing in this case is math-bench. The file seems to have some template haskell in it - I'm not sure if that is the issue, but math-bench also has some ghc-options in its cabal file that may be (part of) the culprit

  ghc-options:         -O2
                       -threaded
                       -fexcess-precision
                       -optc-O3
                       -optc-ffast-math
                       -funfolding-use-threshold=16

Since I have access to the cabal file I just set buildable: False and stack seems to be able to build now.

[jacereda]

I've hit this problem trying to build HaRe with --profile on both Mac and Windows.

[jchia]

Could this be related? https://github.com/haskell/cabal/issues/2827

[chreekat]

I also just hit this while building stack HEAD (078cfad) with --executable-profiling. Kind of annoying, since I wasn't getting it just earlier today. :)

I don't have useful report info yet, I'm just pinging the thread. I'm going to blow away .stack-work and see what changes.

[chreekat]

My repro steps were:

1. Install from get.haskellstack.org.
   $ stack --version Version 1.3.3, Git revision f5180d48084091cb975da2bfa70f4aaa51d26abb (dirty) (4595 commits) x86_64 hpack-0.17.0

2. Go into stack repo and build with profiling.

   $ rm -fr .stack-work
   $ stack build --executable-profiling
   <snip>
       Preprocessing executable 'stack' for stack-1.3.3...
       [1 of 2] Compiling Paths_stack      ( .stack-work/dist/x86_64-linux/Cabal-1.22.7.0/build/autogen/Paths_stack.hs, .stack-work/dist/x86_64-linux/Cabal-1.22.7.0/build/stack/stack-tmp/Paths_stack.p_o )
       [2 of 2] Compiling Main             ( src/main/Main.hs, .stack-work/dist/x86_64-linux/Cabal-1.22.7.0/build/stack/stack-tmp/Main.p_o )
       /home/b/src/Haskell/stack/stack/src/main/Main.hs:127:10:
           cannot find normal object file ‘.stack-work/dist/x86_64-linux/Cabal-1.22.7.0/build/stack/stack-tmp/Paths_stack.dyn_o’
           while linking an interpreted expression
       Completed 26 action(s).
   
       --  While building package stack-1.3.3 using:
             /home/b/src/Haskell/stack/stack/.stack-work/dist/x86_64-linux/Cabal-1.22.7.0/setup/setup --builddir=.stack-work/dist/x86_64-linux/Cabal-1.22.7.0 build lib:stack exe:stack --ghc-options " -ddump-hi -ddump-to-file"
           Process exited with code: ExitFailure 1
   

3. Look around for Paths_stack.

   $ find .stack-work -name '*Paths_stack*'
   .stack-work/dist/x86_64-linux/Cabal-1.22.7.0/build/stack/stack-tmp/Paths_stack.p_o
   .stack-work/dist/x86_64-linux/Cabal-1.22.7.0/build/stack/stack-tmp/Paths_stack.p_hi
   .stack-work/dist/x86_64-linux/Cabal-1.22.7.0/build/stack/stack-tmp/.stack-work/dist/x86_64-linux/Cabal-1.22.7.0/build/autogen/Paths_stack.dump-hi
   .stack-work/dist/x86_64-linux/Cabal-1.22.7.0/build/Paths_stack.p_o
   .stack-work/dist/x86_64-linux/Cabal-1.22.7.0/build/Paths_stack.p_hi
   .stack-work/dist/x86_64-linux/Cabal-1.22.7.0/build/autogen/Paths_stack.hs
   .stack-work/dist/x86_64-linux/Cabal-1.22.7.0/build/.stack-work/dist/x86_64-linux/Cabal-1.22.7.0/build/autogen/Paths_stack.dump-hi
   .stack-work/dist/x86_64-linux/Cabal-1.22.7.0/build/Paths_stack.o
   .stack-work/dist/x86_64-linux/Cabal-1.22.7.0/build/Paths_stack.hi
   .stack-work/dist/x86_64-linux/Cabal-1.22.7.0/build/Paths_stack.dyn_hi
   .stack-work/dist/x86_64-linux/Cabal-1.22.7.0/build/Paths_stack.dyn_o
   

   So there's one in there, just not where stack wants to find it.

   Next is the fun part... overcoming the issue.

4. Rebuild/install without profiling. No cleaning or anything.
   $ stack install ... $ stack --version Version 1.3.3, Git revision 078cfadeb37a39501eae24732e5c757cc8aca31b (4608 commits) x86_64 hpack-0.17.0

5. Build with profiling, again.

   stack-1.3.3: unregistering
   stack-1.3.3: configure (lib + exe)
   Configuring stack-1.3.3...
   stack-1.3.3: build (lib + exe)
   Preprocessing library stack-1.3.3...
   Preprocessing executable 'stack' for stack-1.3.3...
   [2 of 2] Compiling Main             ( src/main/Main.hs, .stack-work/dist/x86_64-linux/Cabal-1.22.7.0/build/stack/stack-tmp/Main.p_o )
   Linking .stack-work/dist/x86_64-linux/Cabal-1.22.7.0/build/stack/stack ...
   stack-1.3.3: copy/register
   Installing library in
   /home/b/src/Haskell/stack/stack/.stack-work/install/x86_64-linux/lts-6.25/7.10.3/lib/x86_64-linux-ghc-7.10.3/stack-1.3.3-Il62v6TwN0R40oGLVzWaiu
   Installing executable(s) in
   /home/b/src/Haskell/stack/stack/.stack-work/install/x86_64-linux/lts-6.25/7.10.3/bin
   Registering stack-1.3.3...
   

   Success!

[mgsloan]

@chreekat Could this be the same issue described in https://github.com/commercialhaskell/stack/issues/2903 ? Namely, there's appears to be poor interaction between --profile and TH

[chreekat]

@mgsloan I would say most likely. Interesting in this case that it's the Paths_\.hs module that causes the trouble, which is autogenerated (but I forget who does the autogeneration).

[mgsloan]

Interesting in this case that it's the Paths_.hs module that causes the trouble, which is autogenerated (but I forget who does the autogeneration).

The autogeneration of Paths_ is part of Cabal. No TH involved. It is rather curious that it's failing due to that.

[chrissound]

build:
  library-profiling: true
  executable-profiling: true

I was getting similar error messages with the above in my stack.yaml - had to set the values to false to successfully build. Not too sure how relevant this is..

[snoyberg]

This appears to be an upstream issue, either in Cabal or GHC. Closing in favor of haskell/cabal#5961

[snoyberg]

And I remembered the resolution for this problem: can you try adding other-extensions: TemplateHaskell to the appropriate stanza and see if that fixes things?