Cardano-node: [BUG] - cabal install fails for 1.19.0

[mark-stopka]

Exernal

Summary
```¨
cabal install cardano-cli \
--installdir=/usr/local/bin \
--install-method=copy \
-f -systemd \
-f -external-libsodium-vrf

and

cabal install cardano-node \
--install-method=copy \
--installdir=/usr/local/bin \
-f -systemd \
-f -external-libsodium-vrf

fail in 1.18.1 release with

cabal: Error: Could not find module: Shelley.Spec.Ledger.Bench.Gen with any
suffix: ["gc","chs","hsc","x","y","ly","cpphs","hs","lhs","hsig","lhsig"]. If
the module is autogenerated it should be added to 'autogen-modules'.
```

Likely caused by https://github.com/input-output-hk/cardano-ledger-specs/pull/1749.

Steps to reproduce
Steps to reproduce the behavior:

N/A, obvious from summary

Expected behavior
cabal install will build and install binaries to specified path as with previous versions of cardano-node.

System info (please complete the following information):

• OS: CentOS
• Version: 8
• Node version: 1.18.1

Workaround
Since few reached out to me, there is an easy workaround if you really need this particular release for your pool, you can use cabal build all and copy the binaries manually.

[mark-stopka]

@disassembler, could we add cabal install into pre-release build test suite?

[tiredpixel]

Seconded; Cardano Node 1.18.1 fails for me on my customised Docker images that I'm using for my highly-available stake pool [CRAB]. This is coming from my CI, but I've also verified it with a local build manually within an environment I use for Haskell programming. The build environment is Concourse CI, so basically Garden using runc, within virtualised non-Docker containers, on CentOS 8 (64-bit).

#15 [10/10] RUN cabal v2-update &&     cabal v2-install -O2 -j         carda...
#15 1.893 Config file path source is default config file.
#15 1.893 Config file /home/x/.cabal/config not found.
#15 1.894 Writing default configuration to /home/x/.cabal/config
#15 1.920 Cloning into '/home/x/repo/lib/cardano-node/dist-newstyle/src/cardano-b_-c97d51a3264adedf'...
#15 3.462 HEAD is now at 7d795c3... Merge pull request #144 from input-output-hk/dcoutts/hashing-api
#15 3.468 Cloning into '/home/x/repo/lib/cardano-node/dist-newstyle/src/cardano-c_-613c120a593cd275'...
#15 4.636 HEAD is now at 2547ad1... Merge pull request #65 from dcoutts/dcoutts/drop-vrf-p256-openssl
#15 4.639 Cloning into '/home/x/repo/lib/cardano-node/dist-newstyle/src/cardano-l_-afc0f56298b7d554'...
#15 7.828 HEAD is now at ea88d98a... Tries to fix a bottle neck at epoch boundaries.
#15 7.852 Cloning into '/home/x/repo/lib/cardano-node/dist-newstyle/src/cardano-p_-6616435bbe3d1976'...
#15 10.09 HEAD is now at 316c854... Merge #113
#15 10.09 Cloning into '/home/x/repo/lib/cardano-node/dist-newstyle/src/cardano-s_-c79e68d577f5acc3'...
#15 11.10 HEAD is now at 43a036c... Replace Universum with Cardano.Prelude
#15 11.10 Cloning into '/home/x/repo/lib/cardano-node/dist-newstyle/src/goblins-f4f7645a78b07334'...
#15 12.10 HEAD is now at 26d35ad... Add SeedGoblin instance for `Maybe a`
#15 12.11 Cloning into '/home/x/repo/lib/cardano-node/dist-newstyle/src/iohk-moni_-c4029568d4836e27'...
#15 28.55 HEAD is now at efa4b5e... Merge #568
#15 28.55 Cloning into '/home/x/repo/lib/cardano-node/dist-newstyle/src/ouroboros_-86f4da3e33dbe3ab'...
#15 37.46 HEAD is now at ec74d1c46... Merge #2466
#15 37.50 Cloning into '/home/x/repo/lib/cardano-node/dist-newstyle/src/cborg-fa2f3608b845333'...
#15 40.66 HEAD is now at 42a8319... Merge pull request #223 from Jimbo4350/jordan/update-pprint
#15 40.90 Downloading the latest package list from hackage.haskell.org
#15 66.28 Wrote tarball sdist to
#15 66.28 /home/x/repo/lib/cardano-node/dist-newstyle/sdist/cardano-api-1.18.1.tar.gz
#15 66.31 Wrote tarball sdist to
#15 66.31 /home/x/repo/lib/cardano-node/dist-newstyle/sdist/cardano-cli-1.18.1.tar.gz
#15 66.32 Wrote tarball sdist to
#15 66.32 /home/x/repo/lib/cardano-node/dist-newstyle/sdist/cardano-config-0.1.0.0.tar.gz
#15 66.34 Wrote tarball sdist to
#15 66.34 /home/x/repo/lib/cardano-node/dist-newstyle/sdist/cardano-node-1.18.1.tar.gz
#15 66.34 Wrote tarball sdist to
#15 66.34 /home/x/repo/lib/cardano-node/dist-newstyle/sdist/cardano-binary-1.5.0.tar.gz
#15 66.35 Wrote tarball sdist to
#15 66.35 /home/x/repo/lib/cardano-node/dist-newstyle/sdist/cardano-binary-test-1.3.0.tar.gz
#15 66.35 Wrote tarball sdist to
#15 66.35 /home/x/repo/lib/cardano-node/dist-newstyle/sdist/cardano-crypto-class-2.0.0.tar.gz
#15 66.35 Wrote tarball sdist to
#15 66.35 /home/x/repo/lib/cardano-node/dist-newstyle/sdist/cardano-slotting-0.1.0.0.tar.gz
#15 66.37 Wrote tarball sdist to
#15 66.37 /home/x/repo/lib/cardano-node/dist-newstyle/sdist/cardano-crypto-praos-2.0.0.tar.gz
#15 66.39 Wrote tarball sdist to
#15 66.39 /home/x/repo/lib/cardano-node/dist-newstyle/sdist/cardano-crypto-1.1.0.tar.gz
#15 66.42 Wrote tarball sdist to
#15 66.42 /home/x/repo/lib/cardano-node/dist-newstyle/sdist/cardano-ledger-0.1.0.0.tar.gz
#15 66.43 Wrote tarball sdist to
#15 66.43 /home/x/repo/lib/cardano-node/dist-newstyle/sdist/cardano-crypto-wrapper-1.3.0.tar.gz
#15 66.44 Wrote tarball sdist to
#15 66.44 /home/x/repo/lib/cardano-node/dist-newstyle/sdist/cardano-ledger-test-1.3.0.tar.gz
#15 66.44 Wrote tarball sdist to
#15 66.44 /home/x/repo/lib/cardano-node/dist-newstyle/sdist/cardano-crypto-test-1.3.0.tar.gz
#15 66.44 Wrote tarball sdist to
#15 66.44 /home/x/repo/lib/cardano-node/dist-newstyle/sdist/byron-spec-chain-0.1.0.0.tar.gz
#15 66.46 Wrote tarball sdist to
#15 66.46 /home/x/repo/lib/cardano-node/dist-newstyle/sdist/byron-spec-ledger-0.1.0.0.tar.gz
#15 66.46 Wrote tarball sdist to
#15 66.46 /home/x/repo/lib/cardano-node/dist-newstyle/sdist/small-steps-0.1.0.0.tar.gz
#15 66.47 Wrote tarball sdist to
#15 66.47 /home/x/repo/lib/cardano-node/dist-newstyle/sdist/small-steps-test-0.1.0.0.tar.gz
#15 66.47 Wrote tarball sdist to
#15 66.47 /home/x/repo/lib/cardano-node/dist-newstyle/sdist/shelley-spec-non-integral-0.1.0.0.tar.gz
#15 66.47 cabal: Error: Could not find module: Shelley.Spec.Ledger.Bench.Gen with any
#15 66.47 suffix: ["gc","chs","hsc","x","y","ly","cpphs","hs","lhs","hsig","lhsig"]. If
#15 66.47 the module is autogenerated it should be added to 'autogen-modules'.
#15 66.47 
#15 ERROR: executor failed running [/bin/sh -c cabal v2-update &&     cabal v2-install -O2 -j         cardano-cli         cardano-node]: buildkit-runc did not terminate successfully
------
 > [10/10] RUN cabal v2-update &&     cabal v2-install -O2 -j         cardano-cli         cardano-node:
------
error: failed to solve: rpc error: code = Unknown desc = executor failed running [/bin/sh -c cabal v2-update &&     cabal v2-install -O2 -j         cardano-cli         cardano-node]: buildkit-runc did not terminate successfully   

[tiredpixel]

A bisect shows the failure is at 391dd255, which indeed points to input-output-hk/cardano-ledger-specs as @mark-stopka suggested, although I haven't been able to verify the point of breakage there, yet. Specifically, changes like this:

diff --git a/cabal.project b/cabal.project
index 4d65ceae..0d716d27 100644
--- a/cabal.project
+++ b/cabal.project
@@ -161,78 +161,78 @@ source-repository-package
 source-repository-package
   type: git
   location: https://github.com/input-output-hk/cardano-ledger-specs
-  tag: 5011965ff8b44e3f7954e8e664fd73c020d13aeb
-  --sha256: 0h8c9dqmhknyj1brz67i3vp2zkjm01pqgr46b3nl3frnh9vxrnws
+  tag: ea88d98a28e4587f16d900a33b0d9c49f26f61fe
+  --sha256: 1fpzpx73m3729m9ahf50fd1zgkgfm8qasbj8vnqlab0cqqnz5lm1

Looks like it's the Czech team tonight. ;)

[tiredpixel]

A bisect within cardano-ledger-specs cross-building with cardano-node 1.18.1 (modified) shows the failure is at input-output-hk/cardano-ledger-specs@ea88d98a28e4587f16d900a33b0d9c49f26f61fe , specifically these parts:

https://github.com/input-output-hk/cardano-ledger-specs/blame/release/1.18.x/shelley/chain-and-ledger/executable-spec/bench/Main.hs#L15

diff --git a/shelley/chain-and-ledger/executable-spec/bench/Main.hs b/shelley/chain-and-ledger/executable-spec/bench/Main.hs
index 5ee8ad9f..4b94c884 100644
--- a/shelley/chain-and-ledger/executable-spec/bench/Main.hs
+++ b/shelley/chain-and-ledger/executable-spec/bench/Main.hs
@@ -1,11 +1,33 @@
+{-# LANGUAGE DataKinds #-}
+{-# OPTIONS_GHC -Wno-unused-binds #-}
+{-# OPTIONS_GHC -Wno-unused-imports #-}
+
 module Main where

+import BenchUTxOAggregate (expr, genTestCase)
 import Control.DeepSeq (NFData)
-import Control.Iterate.SetAlgebra (keysEqual)
+import Control.Iterate.SetAlgebra (dom, forwards, keysEqual, (▷), (◁))
+import Control.Iterate.SetAlgebraInternal (compile, compute, run)
 import Criterion.Main (Benchmark, bench, bgroup, defaultMain, env, whnf)
+import Data.Map (Map)
 import qualified Data.Map.Strict as Map
 import Data.Word (Word64)
+import Shelley.Spec.Ledger.Bench.Gen (genTx)
+import Shelley.Spec.Ledger.Coin (Coin (..))

https://github.com/input-output-hk/cardano-ledger-specs/blob/release/1.18.x/shelley/chain-and-ledger/executable-spec/shelley-spec-ledger.cabal#L252

diff --git a/shelley/chain-and-ledger/executable-spec/shelley-spec-ledger.cabal b/shelley/chain-and-ledger/executable-spec/shelley-spec-ledger.cabal
index 3259df37..32b4567b 100644
--- a/shelley/chain-and-ledger/executable-spec/shelley-spec-ledger.cabal
+++ b/shelley/chain-and-ledger/executable-spec/shelley-spec-ledger.cabal
@@ -30,6 +30,7 @@ library
                      Shelley.Spec.Ledger.Credential
                      Shelley.Spec.Ledger.Delegation.Certificates
                      Shelley.Spec.Ledger.Delegation.PoolParams
+                     Shelley.Spec.Ledger.DeserializeShort
                      Shelley.Spec.Ledger.EpochBoundary
                      Shelley.Spec.Ledger.Genesis
                      Shelley.Spec.Ledger.Hashing
@@ -152,6 +153,7 @@ test-suite shelley-spec-ledger-test
       Test.Shelley.Spec.Ledger.NonTraceProperties.Generator
       Test.Shelley.Spec.Ledger.NonTraceProperties.Mutator
       Test.Shelley.Spec.Ledger.NonTraceProperties.PropertyTests
+      Test.Shelley.Spec.Ledger.NonTraceProperties.Serialization
       Test.Shelley.Spec.Ledger.NonTraceProperties.Validity
       Test.Shelley.Spec.Ledger.Orphans
       Test.Shelley.Spec.Ledger.PropertyTests
@@ -246,42 +248,68 @@ benchmark mainbench
   hs-source-dirs:   bench,
                     test
   main-is:          Main.hs
-  other-modules:    Test.Shelley.Spec.Ledger.BenchmarkFunctions,
+  other-modules:    BenchUTxOAggregate,
+                    Shelley.Spec.Ledger.Bench.Gen
+                    Test.Cardano.Crypto.VRF.Fake,

[tiredpixel]

Ah. There's a typo. :) I've succeeded in building past that point—although I haven't run it to the end, including any tests, as I don't have the Cardano Node project setup for development. Nevertheless, patch incoming. 🦀

[tiredpixel]

Okay, I spoke too soon—ignore that; it's not the typo I thought it was. input-output-hk/cardano-ledger-specs@ea88d98a mentions Shelley.Spec.Ledger.Bench.Gen, but it's not present in hs-source-dirs bench/ or test/ (unlike BenchUTxOAggregate, which was added at the same time). Searching for the file yields https://github.com/input-output-hk/cardano-ledger-specs/blob/b24402e4bec675d93c028b3a31ad92d0619934ec/shelley/chain-and-ledger/shelley-spec-ledger-test/bench/Shelley/Spec/Ledger/Bench/Gen.hs , but input-output-hk/cardano-ledger-specs@b24402e4 is not present in release/1.18.x or producer-release/1.18.x.

[mark-stopka]

Issue present in proposed 1.19.0 (https://github.com/input-output-hk/cardano-node/pull/1721) also...

Wrote tarball sdist to
/usr/src/cardano-node/dist-newstyle/sdist/cardano-api-1.19.0.tar.gz
Wrote tarball sdist to
/usr/src/cardano-node/dist-newstyle/sdist/cardano-cli-1.19.0.tar.gz
Wrote tarball sdist to
/usr/src/cardano-node/dist-newstyle/sdist/cardano-config-0.1.0.0.tar.gz
Wrote tarball sdist to
/usr/src/cardano-node/dist-newstyle/sdist/cardano-node-1.19.0.tar.gz
cabal: filepath wildcard 'ChangeLog.md' does not match any files.

[michaeljfazio]

Bug still present in 1.19.0 :(

[tiredpixel]

Seconded. Different error, but again install failure:

#15 55.85 Wrote tarball sdist to
#15 55.85 /home/x/repo/lib/cardano-node/dist-newstyle/sdist/cardano-api-1.19.0.tar.gz
#15 55.89 Wrote tarball sdist to
#15 55.89 /home/x/repo/lib/cardano-node/dist-newstyle/sdist/cardano-cli-1.19.0.tar.gz
#15 55.90 Wrote tarball sdist to
#15 55.90 /home/x/repo/lib/cardano-node/dist-newstyle/sdist/cardano-config-0.1.0.0.tar.gz
#15 55.92 Wrote tarball sdist to
#15 55.92 /home/x/repo/lib/cardano-node/dist-newstyle/sdist/cardano-node-1.19.0.tar.gz
#15 55.92 cabal: filepath wildcard 'ChangeLog.md' does not match any files.

Any chance of getting a fixed version pushed? Confirmation from a core dev that they're experiencing the same would also be appreciated—although I'm pretty certain it isn't a local problem, because I'm building in a clean environment using a similar method to how I usually compile my own Haskell code.

[boaz-yehezkel]

Any update on this issue? We have been seeing this on fresh docker builds following a very similar process as the coin cashew setup. I will have a play and see if there is a short term workaround in the mean time.

[mark-stopka]

Thx @rhyslbw, I can confirm that the linked issue works when cherry-picked onto 1.19.0!

[tiredpixel]

I second that 1.19.0 with cherry-picked 4e71321f5fa0c43eddfa26b968b5e050c8b955b1 from https://github.com/input-output-hk/cardano-node/pull/1755 fixes the compile failures. I'll upgrade and roll out the 1.19.0-patch to CRAB pool shortly, once I've finished QA. In case anyone else wants to point to a branch with the patch, you can use https://github.com/tiredpixel/cardano-node/tree/1.19.0-patch .