Stack: Internal libraries break `stack haddock`

Created on 27 Apr 2018  路  45Comments  路  Source: commercialhaskell/stack

The issue reported in https://github.com/haskell/cabal/issues/4969 appears not to be fixed in stack: building haddock documentation for a package with internal libraries causes an error while resolving dependencies of the main library. The resolution proposed there, to use cabal new-haddock, has no obvious stack equivalent.

haddock further investigation required help wanted upstream issue bug
Source
picture ryanreich
👍6

Most helpful comment

That change will be released with Stack 2. You can see the blog post for more details, but the earliest release date is May 29. In the meanwhile, you can try using the prerelease version. You can try stack upgrade --git, or watch for an announcement in the next few days about a release candidate.

picture snoyberg on 14 May 2019
👍2

All 45 comments

I've been following links to issues between projects and the long chain eventually led me to https://github.com/commercialhaskell/stack/pull/3955 which is merged. But this issue is still occurring (as of lts-11.7 and stack-1.7.1) so I'm at a bit of a loss. 

    Preprocessing library for haddock-library-1.4.5..
    Running Haddock on library for haddock-library-1.4.5..
    Cabal-simple_mPHDZzAJ_2.0.1.0_ghc-8.2.2: internal error when calculating
    transitive package dependencies.
    Debug info: []
istathar picture istathar on 2 May 2018

Thanks for the report, I was able to reproduce and I'll try to solve it (I think it's close to a solution for #3996)

mihaimaruseac picture mihaimaruseac on 2 May 2018

@mihaimaruseac Trying to build pandoc, for what it's worth. I'm sure there's a simpler minimal reproducible case.

istathar picture istathar on 3 May 2018
👍1

There is, I reproduced it from the example in #3996 (although there is no documentation there to be generated)

mihaimaruseac picture mihaimaruseac on 3 May 2018

Does anyone know of a workaround? Should revert to stack-1.6? I'm quite unable to build my projects right now because of this.

istathar picture istathar on 10 May 2018

Sorry, I stopped working on it to get HCAR out. I'll restart working on this after Tuesday/Wednesday when i hope to get HCAR public.

mihaimaruseac picture mihaimaruseac on 12 May 2018

@mihaimaruseac certainly. Most appreciated. I'm just a bit surprised that I couldn't find an obvious way to get around this. Usually in stack if something goes wrong it's worst-case nuke the _.stack-work_ directory. :). Look forward to trying any code you want me to test if/when the time comes.

istathar picture istathar on 13 May 2018

I just bumped into this problem as well. I haven't found a workaround.

NorfairKing picture NorfairKing on 13 May 2018

@mihaimaruseac, any luck?

NorfairKing picture NorfairKing on 18 May 2018

Not yet, still working on it.

mihaimaruseac picture mihaimaruseac on 18 May 2018

I made #4033 for #3966, next step is this. With the repo to reproduce #3966 I have a successful stack haddock if the internal library is a dependency of the main library but not if not.

mihaimaruseac picture mihaimaruseac on 21 May 2018

@afcowie, do you recall which version of Pandoc exercised this bug? I tried it now with the latest but it no longer has internal libraries so it works for me.

mihaimaruseac picture mihaimaruseac on 25 May 2018

I found a workaround: with the fix in #4033 and if the internal library is not a dependency of the main library, stack haddock finishes successfully.

I'll keep looking into this, but so far it seems that both in the case when the sublib is a dependency of the main library and not, Stack calls cabal haddock with the exact same arguments, after properly building the code. Surely I'm missing something.

mihaimaruseac picture mihaimaruseac on 25 May 2018

do you recall which version of Pandoc exercised this bug?

The one in resolver lts-12.20. I just tried still happening six months later

$ stack haddock pandoc

with lts-12.20 and no change. Which resolver are you using, @mihaimaruseac?

istathar picture istathar on 25 May 2018

Oh, I stack unpack'ed the latest version and tried building it directly.

mihaimaruseac picture mihaimaruseac on 26 May 2018

Related issue pointing to why the Stackage curator doesn't run haddock on haddock-library. https://github.com/commercialhaskell/stackage/issues/3236

Seems to be a bug in Cabal, not in Stack. I'll be trying the rest of today to get a workaround, but I don't know how many chances of success I have.

mihaimaruseac picture mihaimaruseac on 26 May 2018

Tried making Stack run the steps suggested by https://github.com/haskell/cabal/issues/4969#issuecomment-373467229 but it fails due to GHC_PACKAGE_PATH.

I'm going to put this on the back of my mind and work on something else for a while and maybe I'll get more ideas in meanwhile. Or maybe cabal gets updated to make the new-build commands the default and then Stack can pick it up and adapt.

Sorry for not being able to solve this :(

mihaimaruseac picture mihaimaruseac on 26 May 2018

I think this could be fixed by linking the Setup program with the recent Cabal library (2.3.0.0+), but can't find any way to do it with Stack.

ip1981 picture ip1981 on 20 Jun 2018

Or by disabling haddock for _some_ packages.

ip1981 picture ip1981 on 20 Jun 2018

@mihaimaruseac should #4111 fix this?

dbaynard picture dbaynard on 27 Jun 2018

I don't think so. I'm going to test it later today or tomorrow, but I'm almost confident it doesn't as this is blocked on calling Cabal in a totally different way than Stack does.

mihaimaruseac picture mihaimaruseac on 27 Jun 2018

Checked now, this is still an issue.

mihaimaruseac picture mihaimaruseac on 27 Jun 2018

Is it possible to selectively disable the generation of haddocks on failing librairies to circumvent this issue ?

nrolland picture nrolland on 30 Nov 2018

Is it possible to selectively disable the generation of haddocks on failing librairies to circumvent this issue ?

More specifically, what file do I need to touch (ie create zero size) so that _stack_ doesn't think that it needs to build haddock content for haddock-library. I'd be happy to use that as a workaround for now.

AfC

istathar picture istathar on 10 Dec 2018

I don't think it's Stack. Stack just calls Setup.hs to build the haddocks, so probably it's a question for Cabal/GHC

mihaimaruseac picture mihaimaruseac on 11 Dec 2018

It looks like this Cabal problem was fixed in version 2.4.0.1, at least I got stack haddock succeeding with test/integration/tests/3899-dont-rebuild-sublibraries/files by switching from 8.2.2 to 8.6.3, I plan to try haddock-library a bit later

qrilka picture qrilka on 27 Dec 2018
👍2

For what it's worth, pandoc 2.5 depends on haddock-library 1.7.0 which has dropped the troublesome internal (vendored?) dependency on attoparsec. So the problem is going to go away for people like me when that version hits Stackage. Which is great for me, but soon I won't have a clear idea if the underlying problem is fixed. I hope @qrilka's report is correct.

istathar picture istathar on 27 Dec 2018
👍2

stack haddock works fine for haddock-library with LTS-13.0

qrilka picture qrilka on 27 Dec 2018

The underlying problem most likely isn't fixed, it's just that haddock-libary devs were convinced to not use internal deps.

Didn't get a chance to investigate the underlying problem yet, I'm too far back on open source work :(

mihaimaruseac picture mihaimaruseac on 27 Dec 2018

@mihaimaruseac why do you think it's not fixed?
Here is a log of successful build:

qrilka@qdesktop ~/ws/h/haddock $ git checkout ghc-8.4
袙械褌泻邪 芦ghc-8.4禄 芯褌褋谢械卸懈胁邪械褌 胁薪械褕薪褞褞 胁械褌泻褍 芦ghc-8.4禄 懈蟹 芦origin禄.
袩械褉械泻谢褞褔械薪芯 薪邪 薪芯胁褍褞 胁械褌泻褍 芦ghc-8.4禄
qrilka@qdesktop ~/ws/h/haddock $ PAGER= git diff haddock-library
diff --git a/haddock-library/haddock-library.cabal b/haddock-library/haddock-library.cabal
index eb961041..e6819f79 100644
--- a/haddock-library/haddock-library.cabal
+++ b/haddock-library/haddock-library.cabal
@@ -22,9 +22,9 @@ library
   default-language:     Haskell2010

   build-depends:
-      base         >= 4.5     && < 4.12
+      base         >= 4.5     && < 4.13
     , bytestring   >= 0.9.2.1 && < 0.11
-    , containers   >= 0.4.2.1 && < 0.6
+    , containers   >= 0.4.2.1 && < 0.7
     , transformers >= 0.3.0   && < 0.6
     , text         >= 1.2.3.0  && < 1.3
     , parsec       >= 3.1.13.0 && < 3.2
qrilka@qdesktop ~/ws/h/haddock $ grep ^version haddock-library/haddock-library.cabal 
version:              1.6.0
qrilka@qdesktop ~/ws/h/haddock $ stack haddock
haddock-library-1.6.0: unregistering (rebuilding with haddocks)
haddock-library-1.6.0: configure (lib)
Configuring haddock-library-1.6.0...
haddock-library-1.6.0: build (lib)
Preprocessing library for haddock-library-1.6.0..
Building library for haddock-library-1.6.0..
haddock-library-1.6.0: haddock
Preprocessing library for haddock-library-1.6.0..
Running Haddock on library for haddock-library-1.6.0..
Haddock coverage:
Warning: 'Doc' is out of scope.
    If you qualify the identifier, haddock can try to link it
    it anyway.
  19% (  3 / 16) in 'Documentation.Haddock.Types'
  Missing documentation for:
    MetaDoc (src/Documentation/Haddock/Types.hs:41)
    overDoc (src/Documentation/Haddock/Types.hs:59)
    overDocF (src/Documentation/Haddock/Types.hs:62)
    Version (src/Documentation/Haddock/Types.hs:65)
    Package (src/Documentation/Haddock/Types.hs:66)
    Hyperlink (src/Documentation/Haddock/Types.hs:68)
    Picture (src/Documentation/Haddock/Types.hs:73)
    Header (src/Documentation/Haddock/Types.hs:78)
    Example (src/Documentation/Haddock/Types.hs:83)
    TableCell (src/Documentation/Haddock/Types.hs:88)
    TableRow (src/Documentation/Haddock/Types.hs:94)
    Table (src/Documentation/Haddock/Types.hs:98)
    DocH (src/Documentation/Haddock/Types.hs:103)
  57% (  8 / 14) in 'Documentation.Haddock.Parser.Monad'
  Missing documentation for:
    Module header
    ParserState (src/Documentation/Haddock/Parser/Monad.hs:23)
    initialParserState (src/Documentation/Haddock/Parser/Monad.hs:27)
    setSince (src/Documentation/Haddock/Parser/Monad.hs:30)
    Parser (src/Documentation/Haddock/Parser/Monad.hs:33)
    parseOnly (src/Documentation/Haddock/Parser/Monad.hs:38)
  67% (  2 /  3) in 'Documentation.Haddock.Markup'
  Missing documentation for:
    markup (src/Documentation/Haddock/Markup.hs:9)
  44% (  4 /  9) in 'Documentation.Haddock.Doc'
  Missing documentation for:
    Module header
    docParagraph (src/Documentation/Haddock/Doc.hs:57)
    docAppend (src/Documentation/Haddock/Doc.hs:41)
    docConcat (src/Documentation/Haddock/Doc.hs:11)
    emptyMetaDoc (src/Documentation/Haddock/Doc.hs:35)
Warning: '#' is out of scope.
    If you qualify the identifier, haddock can try to link it
    it anyway.
 100% (  6 /  6) in 'Documentation.Haddock.Parser'
  67% (  2 /  3) in 'Documentation.Haddock.Utf8'
  Missing documentation for:
    Module header
Warning: Documentation.Haddock.Parser: could not find link destinations for:
    parseText parseParagraph innerList definitionList more parseValid
Warning: Documentation.Haddock.Utf8: could not find link destinations for:
    encode decode
Documentation created:
.stack-work/dist/x86_64-linux-tinfo6/Cabal-2.4.0.1/doc/html/haddock-library/index.html,
.stack-work/dist/x86_64-linux-tinfo6/Cabal-2.4.0.1/doc/html/haddock-library/haddock-library.txt
haddock-library-1.6.0: copy/register
Installing library in /home/qrilka/ws/h/haddock/.stack-work/install/x86_64-linux-tinfo6/lts-13.0/8.6.3/lib/x86_64-linux-ghc-8.6.3/haddock-library-1.6.0-IoJUVQoM6qZAhuN6fYIkNo
Registering library for haddock-library-1.6.0..
Updating Haddock index for local packages in
/home/qrilka/ws/h/haddock/.stack-work/install/x86_64-linux-tinfo6/lts-13.0/8.6.3/doc/index.html
Updating Haddock index for local packages and dependencies in
/home/qrilka/ws/h/haddock/.stack-work/install/x86_64-linux-tinfo6/lts-13.0/8.6.3/doc/all/index.html
Haddock index for snapshot packages already up to date at:
/home/qrilka/.stack/snapshots/x86_64-linux-tinfo6/lts-13.0/8.6.3/doc/index.html
qrilka@qdesktop ~/ws/h/haddock $
qrilka picture qrilka on 28 Dec 2018

Try haddock-library 1.4.5

mihaimaruseac picture mihaimaruseac on 28 Dec 2018

works fine as well - https://gist.github.com/qrilka/88f785ad331f577043c355fddd8690fa

qrilka picture qrilka on 28 Dec 2018

No, it doesn't: https://gist.github.com/mihaimaruseac/8eb45ef4ba223c6c801c8cd6d99000e8

You might have some env or binary in path which made it work, but it doesn't on a fresh install of HEAD of Stack.

mihaimaruseac picture mihaimaruseac on 29 Dec 2018

I see "ghc-8.2.2" in you gist @mihaimaruseac - how do you get it with LTS-13.0? I guess that's why you have that problem. BTW I still have old stack-1.7.1 on by $PATH by default, the problem isn't about Stack, it's about the change in how Cabal works.

qrilka picture qrilka on 29 Dec 2018

Selected resolver: lts-11.22

I was under the impression that this is a Cabal internals issue that Stack needs a work around for.

mihaimaruseac picture mihaimaruseac on 29 Dec 2018

Oh, missed that line, sorry, from a couple of Cabal issues I've read it looks like Stack could have some workaround even with some previous Cabal versions (though I'm not sure how) but it appears that now we don't need to :)

qrilka picture qrilka on 29 Dec 2018

Even manually switching to the LTR-13.0 resolver doesn't work

[haddock-library-1.4.5] 位 cat stack.yaml 
resolver: lts-13.0
packages:
- .
allow-newer: true
[haddock-library-1.4.5] 位 stack haddock
WARNING: Ignoring out of range dependency (allow-newer enabled): base-4.12.0.0. haddock-library requires: >=4.5 && <4.11
haddock-library-1.4.5: configure (lib + internal-lib)
Configuring haddock-library-1.4.5...
haddock-library-1.4.5: build (lib + internal-lib)
Preprocessing library 'attoparsec' for haddock-library-1.4.5..
Building library 'attoparsec' for haddock-library-1.4.5..
[ 1 of 11] Compiling Data.Attoparsec.ByteString.FastSet ( vendor/attoparsec-0.13.1.0/Data/Attoparsec/ByteString/FastSet.hs, .stack-work/dist/x86_64-linux-tinfo6/Cabal-2.4.0.1/build/attoparsec/Data/Attoparsec/ByteString/FastSet.o )
...
[11 of 11] Compiling Data.Attoparsec.Number ( vendor/attoparsec-0.13.1.0/Data/Attoparsec/Number.hs, .stack-work/dist/x86_64-linux-tinfo6/Cabal-2.4.0.1/build/attoparsec/Data/Attoparsec/Number.o )
Preprocessing library for haddock-library-1.4.5..
Building library for haddock-library-1.4.5..
[1 of 7] Compiling Documentation.Haddock.Types ( src/Documentation/Haddock/Types.hs, .stack-work/dist/x86_64-linux-tinfo6/Cabal-2.4.0.1/build/Documentation/Haddock/Types.o )
...
[7 of 7] Compiling Documentation.Haddock.Parser ( src/Documentation/Haddock/Parser.hs, .stack-work/dist/x86_64-linux-tinfo6/Cabal-2.4.0.1/build/Documentation/Haddock/Parser.o )
haddock-library-1.4.5: haddock
Preprocessing library 'attoparsec' for haddock-library-1.4.5..
Running Haddock on library 'attoparsec' for haddock-library-1.4.5..
Haddock coverage:
...
Documentation created:
.stack-work/dist/x86_64-linux-tinfo6/Cabal-2.4.0.1/doc/html/haddock-library/index.html,
.stack-work/dist/x86_64-linux-tinfo6/Cabal-2.4.0.1/doc/html/haddock-library/haddock-library.txt
Preprocessing library for haddock-library-1.4.5..
Running Haddock on library for haddock-library-1.4.5..
Haddock coverage:
...
Documentation created:
.stack-work/dist/x86_64-linux-tinfo6/Cabal-2.4.0.1/doc/html/haddock-library/index.html,
.stack-work/dist/x86_64-linux-tinfo6/Cabal-2.4.0.1/doc/html/haddock-library/haddock-library.txt
haddock-library-1.4.5: copy/register
Installing internal library attoparsec in /tmp/mm/haddock-library-1.4.5/.stack-work/install/x86_64-linux-tinfo6/c25e7537cc9b99b49d8aa57ac6709440591f58c477a96ef94f6114e6a668f7fd/8.6.3/lib/x86_64-linux-ghc-8.6.3/haddock-library-1.4.5-4aIdHtThdzDHvICUM0c4eh-attoparsec
Installing library in /tmp/mm/haddock-library-1.4.5/.stack-work/install/x86_64-linux-tinfo6/c25e7537cc9b99b49d8aa57ac6709440591f58c477a96ef94f6114e6a668f7fd/8.6.3/lib/x86_64-linux-ghc-8.6.3/haddock-library-1.4.5-62JBiw4RYQM30IALTFPWTm
Registering library 'attoparsec' for haddock-library-1.4.5..
Registering library for haddock-library-1.4.5..
Updating Haddock index for local packages in
/tmp/mm/haddock-library-1.4.5/.stack-work/install/x86_64-linux-tinfo6/c25e7537cc9b99b49d8aa57ac6709440591f58c477a96ef94f6114e6a668f7fd/8.6.3/doc/index.html
Updating Haddock index for local packages and dependencies in
/tmp/mm/haddock-library-1.4.5/.stack-work/install/x86_64-linux-tinfo6/c25e7537cc9b99b49d8aa57ac6709440591f58c477a96ef94f6114e6a668f7fd/8.6.3/doc/all/index.html
Received ExitFailure 1 when running
Raw command: /home/mihai/.stack/programs/x86_64-linux/ghc-tinfo6-8.6.3/bin/haddock --optghc=-package-db=/home/mihai/.stack/snapshots/x86_64-linux-tinfo6/c25e7537cc9b99b49d8aa57ac6709440591f58c477a96ef94f6114e6a668f7fd/8.6.3/pkgdb --optghc=-package-db=/tmp/mm/haddock-library-1.4.5/.stack-work/install/x86_64-linux-tinfo6/c25e7537cc9b99b49d8aa57ac6709440591f58c477a96ef94f6114e6a668f7fd/8.6.3/pkgdb --gen-contents --gen-index -i ../array-0.5.3.0,../array-0.5.3.0/array.haddock -i ../ghc-prim-0.5.3,../ghc-prim-0.5.3/ghc-prim.haddock -i ../bytestring-0.10.8.2,../bytestring-0.10.8.2/bytestring.haddock -i ../z-haddock-library-z-attoparsec-1.4.5,../z-haddock-library-z-attoparsec-1.4.5/z-haddock-library-z-attoparsec.haddock -i ../transformers-0.5.5.0,../transformers-0.5.5.0/transformers.haddock -i ../haddock-library-1.4.5,../haddock-library-1.4.5/haddock-library.haddock -i ../base-4.12.0.0,../base-4.12.0.0/base.haddock -i ../deepseq-1.4.4.0,../deepseq-1.4.4.0/deepseq.haddock -i ../integer-gmp-1.0.2.0,../integer-gmp-1.0.2.0/integer-gmp.haddock
Run from: /tmp/mm/haddock-library-1.4.5/.stack-work/install/x86_64-linux-tinfo6/c25e7537cc9b99b49d8aa57ac6709440591f58c477a96ef94f6114e6a668f7fd/8.6.3/doc/all/
Standard output:

haddock: internal error: ../z-haddock-library-z-attoparsec-1.4.5/z-haddock-library-z-attoparsec.haddock: openBinaryFile: does not exist (No such file or directory)
mihaimaruseac picture mihaimaruseac on 29 Dec 2018

@mihaimaruseac I think I have found was causes this for you: see special hack which was added exactly for this library - https://github.com/commercialhaskell/stack/blob/v1.9.0.1/src/Stack/Build/Execute.hs#L1328
I have stack-1.7.1 locally so I don't have that enabled and so my Stack works OK, I guess that thing needs get reviewed.

qrilka picture qrilka on 29 Dec 2018

I added that, in #4111 as it was solving other issues with internal libraries.

mihaimaruseac picture mihaimaruseac on 30 Dec 2018

Oh, but it looks like the error changed from the beginning of the issue, so maybe there's some chance of fixing this. If I get some time I'll investigate.

mihaimaruseac picture mihaimaruseac on 30 Dec 2018

One final point as to why this is not solved yet: I got a container with stahck-1.7.1 and, while stack haddock succeeds, no documentation for the internal library is generated (it is ignored completely).

mihaimaruseac picture mihaimaruseac on 30 Dec 2018

I completely forgot about the docs themselves. I see docs for the internal library getting generated:

qrilka@qdesktop ~/ws/h/haddock-library-1.4.5 $ ls `stack path --local-doc-root`/haddock-library-1.4.5
Data-Attoparsec-ByteString-Char8.html    Documentation-Haddock-Parser.html  haddock-bundle.min.js    meta.json       src
Data-Attoparsec-ByteString.html          Documentation-Haddock-Types.html   haddock-library.haddock  minus.gif       synopsis.png
Documentation-Haddock-Doc.html           Documentation-Haddock-Utf8.html    haddock-library.txt      ocean.css
Documentation-Haddock-Markup.html        LICENSE                            hslogo-16.png            plus.gif
Documentation-Haddock-Parser-Monad.html  doc-index.html                     index.html               quick-jump.css

But it looks like main library index doesn't include them (it overwrites the one from the internal library).
I tried cabal new-haddock (but with Cabal 2.2) - it generates 2 haddock directories and the one about internal attoparsec talks about "haddock-library-1.4.5: Library exposing some functionality of Haddock." which is at least very confusing...

qrilka picture qrilka on 31 Dec 2018

I am still experiencing this issue. We have a large monorepo that still fails stack build --haddock on 1.9.3.1 because of an internal library in one of the submodules.

I get: haddock: internal error: ../z-avatar-app-z-avatar-internal-0/z-avatar-app-z-avatar-internal.haddock: openBinaryFile: does not exist (No such file or directory) when trying to do the haddock build in root.

You can find the offending .cabal file here, if that's helpful: https://github.com/futurice/haskell-mega-repo/blob/master/avatar-app/avatar-app.cabal
and stack.yaml: https://github.com/futurice/haskell-mega-repo/blob/master/stack.yaml

Any idea when the fix in #4596 will be released?

jarcane picture jarcane on 14 May 2019

That change will be released with Stack 2. You can see the blog post for more details, but the earliest release date is May 29. In the meanwhile, you can try using the prerelease version. You can try stack upgrade --git, or watch for an announcement in the next few days about a release candidate.

snoyberg picture snoyberg on 14 May 2019
👍2

Thank you. I was able to build correctly on the Git build of Stack, so looks like we'll just have to wait until the update.

jarcane picture jarcane on 15 May 2019
Was this page helpful?
0 / 5 - 0 ratings

Related issues

domenkozar picture domenkozar  路  3Comments
mgsloan picture mgsloan  路  3Comments
tinkyholloway picture tinkyholloway  路  4Comments
igrep picture igrep  路  3Comments
Cosmius picture Cosmius  路  3Comments