Stack: Stack cannot parse cabal-install output (again)

[drvink]

General summary/comments

The output of cabal install --dry-run can break across multiple lines, but Stack's parser expects one line per package.

See also #2084.

Steps to reproduce

$ stack init --resolver ghc-8.2.1

This .cabal should reproduce it:

name:                junk
version:             0.1.0.0
license:             BSD3
license-file:        LICENSE
author:              urmom
maintainer:          [email protected]
build-type:          Simple
extra-source-files:  ChangeLog.md
cabal-version:       >=2.0

library
  build-depends:       base >=4.9 && <5
                     , async
                     , attoparsec
                     , hashtables
                     , haskell-src-exts
                     , mono-traversable
                     , network
                     , optparse-applicative
                     , prelude-extras
                     , regex-base
                     , regex-posix
                     , safe
                     , shake
                     , split
                     , text
                     , vector
                     , bytestring
                     , process
                     , unix
                     , parsec
                     , regex-compat-tdfa
                     , regex-tdfa
                     , QuickCheck
                     , ListLike
                     , megaparsec
  default-language:    Haskell2010

Expected

It should work even in the face of this brain-dead wrapped output. Note that it works fine when there aren't any lines long enough that cabal decides to wrap them.

The beginning of the relevant output from cabal-install looks like the following (notice the correlation between the lines Stack reports as not having been able to parse and the fact that what is "logically" one line is hard-wrapped onto multiple lines):

Resolving dependencies...
In order, the following would be installed:
clock-0.7.2 (via: extra-1.6) (new package)
colour-2.3.3 (via: ansi-terminal-0.7.1.1) (new package)
dlist-0.8.0.3 (via: ListLike-4.5.1) (new package)
fmlist-0.9 (via: ListLike-4.5.1) (new package)
integer-logarithms-1.0.2 (via: scientific-0.3.5.2) (new package)
js-flot-0.8.3 (via: shake-0.16 shake-0.16) (new package)
js-jquery-3.2.1 (via: shake-0.16 shake-0.16) (new package)
mtl-2.2.1 (via: megaparsec-6.2.0 regex-tdfa-1.2.2 parsec-3.1.11
regex-base-0.93.2) (new package)
network-2.6.3.2 (via: junk-0.1.0.0) (new package)
old-locale-1.0.0.7 (via: cpphs-1.20.8 cpphs-1.20.8 old-time-1.1.0.3) (new
package)
parser-combinators-0.2.0 (via: megaparsec-6.2.0) (new package)

Actual

Asking cabal to calculate a build plan...
The following lines from cabal-install output could not be parsed:
mtl-2.2.1 (via: megaparsec-6.1.1 regex-tdfa-1.2.2 parsec-3.1.11
regex-base-0.93.2) (new package)
old-locale-1.0.0.7 (via: cpphs-1.20.8 cpphs-1.20.8 old-time-1.1.0.3) (new
package)
primitive-0.6.2.0 (via: shake-0.16 shake-0.16 hashtables-1.2.2.0
scientific-0.3.5.2 vector-algorithms-0.7.0.1 tf-random-0.5 vector-0.12.0.1)
(new package)
random-1.1 (via: shake-0.16 shake-0.16 QuickCheck-2.10.0.1 tf-random-0.5) (new
package)
text-1.2.2.2 (via: junk-0.1.0.0 megaparsec-6.1.1 attoparsec-0.13.2.0
mono-traversable-1.0.2.1 case-insensitive-1.2.0.10 scientific-0.3.5.2
ListLike-4.5.1 hashable-1.2.6.1 parsec-3.1.11 polyparse-1.12) (new package)
regex-base-0.93.2 (via: junk-0.1.0.0 regex-compat-tdfa-0.95.1.4
regex-tdfa-1.2.2 regex-posix-0.95.2) (new package)
vector-0.12.0.1 (via: junk-0.1.0.0 mono-traversable-1.0.2.1 hashtables-1.2.2.0
ListLike-4.5.1 vector-algorithms-0.7.0.1) (new package)
hashable-1.2.6.1 (via: mono-traversable-1.0.2.1 shake-0.16 shake-0.16
case-insensitive-1.2.0.10 hashtables-1.2.2.0 scientific-0.3.5.2
unordered-containers-0.2.8.0) (new package)
unordered-containers-0.2.8.0 (via: mono-traversable-1.0.2.1 shake-0.16
shake-0.16) (new package)
CallStack (from HasCallStack):
  error, called at src/Stack/Solver.hs:187:16 in stack-1.5.1-8L0HpUwgvNa7pLuInyRw8N:Stack.Solver

Stack version

Version 1.5.1, Git revision 600c1f01435a10d127938709556c1682ecfd694e (4861 commits) x86_64 hpack-0.17.1

Method of installation

• Official binary, downloaded from stackage.org or fpcomplete's package repository

[ikciwor]

I have same problem. I can't actually work now because of this :/

[mgsloan]

@drvink Hmm, are you sure that you're using version 1.5.1? In particular, I wouldn't expect stack init to use the solver at all in recent versions. I've tried on master, which I don't think has many changes to new / init, and got:

mgsloan@computer:~/fpco/test-stack/issue3453$ stack init --resolver ghc-8.2.1
Looking for .cabal or package.yaml files to use to init the project.
Using cabal packages:
- junk.cabal

Selected resolver: ghc-8.2.1
Resolver 'ghc-8.2.1' does not have a matching compiler to build some or all of your package(s).
ghc-8.2.1 cannot be used for these packages:
    - junk
base not found
    - junk requires >=4.9 && <5

This may be resolved by:
    - Using '--omit-packages to exclude mismatching package(s).
    - Using '--resolver' to specify a matching snapshot/resolver

@ikciwor Sorry that you feel that you can't work due to this. What version of stack are you using? The solver is not a crucial part of stack's functioning. You should be able to get along fine by writing your own stack.yaml... Typically this doesn't require much effort if you use a stackage snapshot. Alternatively, you can install the previous versions of cabal-install. Basically, there are many alternatives you can take, no reason to cease your work...

[drvink]

@mgsloan Yup:

$ sha256sum stack-1.5.1-linux-x86_64-static.tar.gz
6a250cbab60870f6b933c780c96c5cafb02d066a6f2d7182bb00ec005262842b  stack-1.5.1-linux-x86_64-static.tar.gz
$ stack --version
Version 1.5.1, Git revision 600c1f01435a10d127938709556c1682ecfd694e (4861 commits) x86_64 hpack-0.17.1

[drvink]

@mgsloan Tried again just now with an empty ~/.stack and was able to repro with the .cabal I gave in the initial post. Try with stack init --install-ghc --resolver ghc-8.2.1, then run stack update, then stack init --resolver ghc-8.2.1.

[mgsloan]

Ah, indeed, it looks like there is a difference here between master and stack-1.5.1. The difference is caused by https://github.com/commercialhaskell/stack/issues/3463

Unfortunately, I don't have time at the moment to dig into solving this. I currently just have cabal 1.24 on my path. Hopefully a straightforward change to get things to work with cabal-2.0? If it's non trivial, perhaps it makes sense to consider whether we can use some other way of scraping cabal solver results, perhaps via cabal freeze files, which should be a more stable format for the info.

[DanBurton]

Bump. I'm seeing this issue with stack-1.6.1 and cabal-install-2.0.0.1. Repro, assuming these versions of said programs are on the PATH:

$ stack unpack riak-1.1.2.2 && cd riak-1.1.2.2
$ stack init --resolver nightly-2017-12-07 --solver

Expected: the command completes successfully.
Actual: the command fails with error message "The following lines from cabal-install output could not be parsed:"

When cabal-install-1.24.0.2 is used instead, the command works as expected.

[mgsloan]

I have investigated this a bit, and I'm not sure how using cabal-install-1.24 is even working because it seems like the cabal.config format has been changed to reject constraint: specifications. Indicative of how little we need the solver, that things have been broken for a while. Maybe cabal-install-1.24 was using the pkg db and this tended to correspond to project config? Hmm hmm, shouldn't be since it only provides in the global DB

I was hoping to fix this, but throwing up my hands for now after I discovered that constraint and similar specs in cabal config files no longer work. (they work in cabal 1.18, haven't tested cabal 1.22)

I think it should be possible to use freeze instead of install, and parse the freeze file instead of verbose output. I think this is the best way forward, since the freeze file is intended for machine use. Just noticed @lwm self-assigned this - great! I think the next steps are:

• [ ] Fixing the generation of cabal.config to work with cabal-install >= 1.24 , checking the behavior of 1.22 . Check cabal-install version to determine what input to give it

• [ ] Switch over to cabal freeze

• [ ] Ideally, have integration tests that check this stuff works on various versions of cabal-install

[decentral1se]

Ah! I think I won't get time - I removed myself :/ Thanks for explanation though 👍

[steshaw]

This was giving me hell when trying to stack init --solver a new project. I found this issue. Seems you can install cabal-install 1.24.0.2 using stack --resolver lts-9 install cabal-install. This fixed the problem for me. HTH.

[qrilka]

This looks to be a duplicate of #4319 (or the other way around :) )

[snoyberg]

We've removed the solver support in #4670