Cabal: `cxx-options` and `cxx-sources` build info fields not fully implemented

[recursion-ninja]

Usage of cxx-options & cxx-sources sometimes fail

Now that stack-1.7.1 depends on Cabal-2.2 I tried some more complicated use cases of the cxx-options and cxx-sources build info fields in one of my projects. Unfortunately it looks like the feature (#3700, #4810) of separate compilation of C and C++ sources with separate compilation flags isn't completely implemented.

What does work

• Building a library that uses the cxx-options and cxx-sources fields
• Building an executable, benchmark, or test-suite that does not use the cxx-options and cxx-sources fields
• Building an executable, benchmark, or test-suite that does not use the cxx-options and cxx-sources fields but depends on a library that uses the cxx-options and cxx-sources fields

What not does work

• Building an executable, benchmark, or test-suite that uses the cxx-options and cxx-sources fields

Reproducability

See this tag of my example repository for a minimal working example of the defect.

There are 7 build targets included in the example repository tag:

| Type | Name | Status |
| :--------- | :------------- | :--------- |
| library | tcm-memo | PASS |
| executable | exe-no-lib | FAIL |
| executable | exe-with-lib | PASS |
| test-suite | test-no-lib | FAIL |
| test-suite | test-with-lib | PASS |
| benchmark | bench-no-lib | FAIL |
| benchmark | bench-with-lib | PASS |

Expected successes:

cabal new-build exe-with-lib && cabal new-test test-with-lib && cabal new-bench bench-with-lib

Expected failures:

cabal new-build exe-no-lib || cabal new-test test-no-lib || cabal new-bench bench-no-lib

The library builds successfully.

The executable, test-suite,and benchmark that depend on the library build successfully.

The executable, test-suite, and benchmark that _do not_ depend on the library and instead directly build the C & C++ sources, each fail with the same undefined references during linking phase. The functions for which undefined references were found are the functions exported from the C++ file via the C ABI. This is because the C++ sources from the cxx-sources are not added to the linking command in the non-library build targets. This can be deduced by "diffing" the invoked linking command from building the library (tcm-memo) and a build target which uses the cxx-sources field (exe-no-lib).

For example:

cabal new-build tcm-memo -v && cabal new-build exe-no-lib -v

The above two cabal invocations produce the following two linking commands (spaces replaced with newlines for clarity).

Library linking:

/usr/local/bin/ghc
-shared
-dynamic
'-lstdc++'
'-dynload
deploy'
-optl-Wl,-rpath,/home/user/.cabal/store/ghc-8.2.2/adjunctions-4.4-b55b9d4a557890c96e2fbffe0c42bd4a22efc0c79efa53d400e1f27c12d4436d/lib
-optl-Wl,-rpath,/usr/local/haskell/ghc-8.2.2-x86_64/lib/ghc-8.2.2/array-0.5.2.0
-optl-Wl,-rpath,/usr/local/haskell/ghc-8.2.2-x86_64/lib/ghc-8.2.2/base-4.10.1.0
-optl-Wl,-rpath,/home/user/.cabal/store/ghc-8.2.2/base-orphans-0.7-5abe8b92ad61c32bfecb410f722b960b842ff2c5e4fe512713270722e3fb70d3/lib
-optl-Wl,-rpath,/home/user/.cabal/store/ghc-8.2.2/bifunctors-5.5.2-a8014cbadff882dde44a6428ea8f5fe7147ff8b65e8cf1372f9742ef8c0593b2/lib
-optl-Wl,-rpath,/usr/local/haskell/ghc-8.2.2-x86_64/lib/ghc-8.2.2/binary-0.8.5.1
-optl-Wl,-rpath,/usr/local/haskell/ghc-8.2.2-x86_64/lib/ghc-8.2.2/bytestring-0.10.8.2
-optl-Wl,-rpath,/home/user/.cabal/store/ghc-8.2.2/comonad-5.0.3-5899c2bd073eb27135d2ed87cbfeb27b71429926ad4a193349571bc27fd854c4/lib
-optl-Wl,-rpath,/usr/local/haskell/ghc-8.2.2-x86_64/lib/ghc-8.2.2/containers-0.5.10.2
-optl-Wl,-rpath,/home/user/.cabal/store/ghc-8.2.2/contravariant-1.4.1-2d6ae61ba4ef9efb0c9f27b7fb76d8a26d3b4ced644bc5e5702b360ae2605471/lib
-optl-Wl,-rpath,/usr/local/haskell/ghc-8.2.2-x86_64/lib/ghc-8.2.2/deepseq-1.4.3.0
-optl-Wl,-rpath,/home/user/.cabal/store/ghc-8.2.2/distributive-0.5.3-8bf54ef91955ab01582f07f9d9037a18db0e04bbb4998466278cc8220990fd50/lib
-optl-Wl,-rpath,/home/user/.cabal/store/ghc-8.2.2/exceptions-0.10.0-8fb69e89d379a2aa497316ca8e63bfa0c4c11e9e5f066975c8e5abd8a6910f7c/lib
-optl-Wl,-rpath,/usr/local/haskell/ghc-8.2.2-x86_64/lib/ghc-8.2.2/filepath-1.4.1.2
-optl-Wl,-rpath,/home/user/.cabal/store/ghc-8.2.2/free-5.0.2-8a8aa3c58095154af833309defdffadd1ba2aeaf1d037ffa9a98eb9a98521833/lib
-optl-Wl,-rpath,/usr/local/haskell/ghc-8.2.2-x86_64/lib/ghc-8.2.2/ghc-boot-th-8.2.2
-optl-Wl,-rpath,/usr/local/haskell/ghc-8.2.2-x86_64/lib/ghc-8.2.2/ghc-prim-0.5.1.1
-optl-Wl,-rpath,/usr/local/haskell/ghc-8.2.2-x86_64/lib/ghc-8.2.2/integer-gmp-1.0.1.0
-optl-Wl,-rpath,/home/user/.cabal/store/ghc-8.2.2/kan-extensions-5.1-294a184500320c5424d37dd8e30aa8e83df4d82816fbadc6956727fc5b051628/lib
-optl-Wl,-rpath,/home/user/.cabal/store/ghc-8.2.2/lens-4.16.1-e4201867f2a1df5ce3f212bb5a348e9b1d7923b2e8038bc8f078b48e8bb8dc5c/lib
-optl-Wl,-rpath,/usr/local/haskell/ghc-8.2.2-x86_64/lib/ghc-8.2.2/pretty-1.1.3.3
-optl-Wl,-rpath,/home/user/.cabal/store/ghc-8.2.2/profunctors-5.2.2-acffd904fcce222bf341f7750c916b923bb356ea901d56320f5115b091fe36fb/lib
-optl-Wl,-rpath,/home/user/.cabal/store/ghc-8.2.2/reflection-2.1.3-30c0765ffa631b6f647feb50514789726ffc2c8156acadc7e9881790512ddf5a/lib
-optl-Wl,-rpath,/usr/local/haskell/ghc-8.2.2-x86_64/lib/ghc-8.2.2/rts
-optl-Wl,-rpath,/home/user/.cabal/store/ghc-8.2.2/semigroupoids-5.2.2-1f8823939fa7839942413bb02bdf7a5b9e045603512fd89f4aafc7577cecd82a/lib
-optl-Wl,-rpath,/home/user/.cabal/store/ghc-8.2.2/semigroups-0.18.4-6d0e4010b7e2b0f82be335bd6f004e822f6126b715a265852fc83dddcd1701b8/lib
-optl-Wl,-rpath,/home/user/.cabal/store/ghc-8.2.2/tagged-0.8.5-02260dbd6f77a91418dd0c72f2f1045416f0d374469782a354be855e98b932d9/lib
-optl-Wl,-rpath,/usr/local/haskell/ghc-8.2.2-x86_64/lib/ghc-8.2.2/template-haskell-2.12.0.0
-optl-Wl,-rpath,/home/user/.cabal/store/ghc-8.2.2/th-abstraction-0.2.6.0-a7da03ec29b43bc7228fb15650820ed9902bcbb309b1fdee01ac69f395749357/lib
-optl-Wl,-rpath,/usr/local/haskell/ghc-8.2.2-x86_64/lib/ghc-8.2.2/time-1.8.0.2
-optl-Wl,-rpath,/usr/local/haskell/ghc-8.2.2-x86_64/lib/ghc-8.2.2/transformers-0.5.2.0
-optl-Wl,-rpath,/home/user/.cabal/store/ghc-8.2.2/transformers-base-0.4.5.2-f1a489b2ed59749f6b1850ca2b31100e9af6d4b28b5988f1652ea8a7140cb60f/lib
-optl-Wl,-rpath,/home/user/.cabal/store/ghc-8.2.2/transformers-compat-0.6.1.5-1468053241ed3f68c187da5eecaec0da23cc7edb7e48d98faa2337cff8cdf2d2/lib
-optl-Wl,-rpath,/usr/local/haskell/ghc-8.2.2-x86_64/lib/x86_64-linux-ghc-8.2.2
-optl-Wl,-rpath,/home/user/.cabal/store/ghc-8.2.2/void-0.7.2-4bea1ef24a4d9a74394b0f938c44386a1924dea12f53d1e7246ac8641989e875/lib
-this-unit-id
tcm-memo-1.0.0.0-inplace
-hide-all-packages
-no-auto-link-packages
-no-user-package-db
-package-db
/home/user/.cabal/store/ghc-8.2.2/package.db
-package-db
/home/user/tcm-memo/dist-newstyle/packagedb/ghc-8.2.2
-package-db
/home/user/tcm-memo/dist-newstyle/build/x86_64-linux/ghc-8.2.2/tcm-memo-1.0.0.0/package.conf.inplace
-package-id
base-4.10.1.0
-package-id
lens-4.16.1-e4201867f2a1df5ce3f212bb5a348e9b1d7923b2e8038bc8f078b48e8bb8dc5c
-package-id
deepseq-1.4.3.0
-package-id
hashable-1.2.6.1-JDYnvpSucMf1h1i2CUUtVb
-package-id
QuickCheck-2.10.1-BUFYQzp5Pjm7JbQeTzW89l
/home/user/tcm-memo/dist-newstyle/build/x86_64-linux/ghc-8.2.2/tcm-memo-1.0.0.0/build/Bio/Character/Exportable/Class.dyn_o
/home/user/tcm-memo/dist-newstyle/build/x86_64-linux/ghc-8.2.2/tcm-memo-1.0.0.0/build/Data/TCM/Memoized.dyn_o
/home/user/tcm-memo/dist-newstyle/build/x86_64-linux/ghc-8.2.2/tcm-memo-1.0.0.0/build/Data/TCM/Memoized/FFI.dyn_o
/home/user/tcm-memo/dist-newstyle/build/x86_64-linux/ghc-8.2.2/tcm-memo-1.0.0.0/build/memoized-tcm/costMatrixWrapper.dyn_o
/home/user/tcm-memo/dist-newstyle/build/x86_64-linux/ghc-8.2.2/tcm-memo-1.0.0.0/build/memoized-tcm/dynamicCharacterOperations.dyn_o
/home/user/tcm-memo/dist-newstyle/build/x86_64-linux/ghc-8.2.2/tcm-memo-1.0.0.0/build/memoized-tcm/costMatrix.dyn_o
-o
/home/user/tcm-memo/dist-newstyle/build/x86_64-linux/ghc-8.2.2/tcm-memo-1.0.0.0/build/libHStcm-memo-1.0.0.0-inplace-ghc8.2.2.so
-hide-all-packages

Other target linking:

/usr/local/bin/ghc
--make
-fbuilding-cabal-package
-O
-static
-outputdir
/home/user/tcm-memo/dist-newstyle/build/x86_64-linux/ghc-8.2.2/tcm-memo-1.0.0.0/x/exe-no-lib/build/exe-no-lib/exe-no-lib-tmp
-odir
/home/user/tcm-memo/dist-newstyle/build/x86_64-linux/ghc-8.2.2/tcm-memo-1.0.0.0/x/exe-no-lib/build/exe-no-lib/exe-no-lib-tmp
-hidir
/home/user/tcm-memo/dist-newstyle/build/x86_64-linux/ghc-8.2.2/tcm-memo-1.0.0.0/x/exe-no-lib/build/exe-no-lib/exe-no-lib-tmp
-stubdir
/home/user/tcm-memo/dist-newstyle/build/x86_64-linux/ghc-8.2.2/tcm-memo-1.0.0.0/x/exe-no-lib/build/exe-no-lib/exe-no-lib-tmp
-i
-i/home/user/tcm-memo/dist-newstyle/build/x86_64-linux/ghc-8.2.2/tcm-memo-1.0.0.0/x/exe-no-lib/build/exe-no-lib/exe-no-lib-tmp
-ilib
-iapp
-i/home/user/tcm-memo/dist-newstyle/build/x86_64-linux/ghc-8.2.2/tcm-memo-1.0.0.0/x/exe-no-lib/build/exe-no-lib/autogen
-i/home/user/tcm-memo/dist-newstyle/build/x86_64-linux/ghc-8.2.2/tcm-memo-1.0.0.0/x/exe-no-lib/build/global-autogen
-I/home/user/tcm-memo/dist-newstyle/build/x86_64-linux/ghc-8.2.2/tcm-memo-1.0.0.0/x/exe-no-lib/build/exe-no-lib/autogen
-I/home/user/tcm-memo/dist-newstyle/build/x86_64-linux/ghc-8.2.2/tcm-memo-1.0.0.0/x/exe-no-lib/build/global-autogen
-I/home/user/tcm-memo/dist-newstyle/build/x86_64-linux/ghc-8.2.2/tcm-memo-1.0.0.0/x/exe-no-lib/build/exe-no-lib/exe-no-lib-tmp
-Imemoized-tcm
-I/home/user/tcm-memo/dist-newstyle/build/x86_64-linux/ghc-8.2.2/tcm-memo-1.0.0.0/x/exe-no-lib/build/memoized-tcm
-optP-include
-optP/home/user/tcm-memo/dist-newstyle/build/x86_64-linux/ghc-8.2.2/tcm-memo-1.0.0.0/x/exe-no-lib/build/exe-no-lib/autogen/cabal_macros.h
'-lstdc++'
-hide-all-packages
-Wmissing-home-modules
-no-user-package-db
-package-db
/home/user/.cabal/store/ghc-8.2.2/package.db
-package-db
/home/user/tcm-memo/dist-newstyle/packagedb/ghc-8.2.2
-package-db
/home/user/tcm-memo/dist-newstyle/build/x86_64-linux/ghc-8.2.2/tcm-memo-1.0.0.0/x/exe-no-lib/package.conf.inplace
-package-id
base-4.10.1.0
-package-id
lens-4.16.1-e4201867f2a1df5ce3f212bb5a348e9b1d7923b2e8038bc8f078b48e8bb8dc5c
-package-id
deepseq-1.4.3.0
-package-id
hashable-1.2.6.1-JDYnvpSucMf1h1i2CUUtVb
-package-id
QuickCheck-2.10.1-BUFYQzp5Pjm7JbQeTzW89l
-package-id
safe-0.3.17-1aa8741433d7acf0c307abe3f2f18a9b367a760b47a4a2ae03fbd585994d1bf3
-XHaskell2010
Bio.Character.Exportable.Class
Data.TCM.Memoized
Data.TCM.Memoized.FFI
app/Main.hs
/home/user/tcm-memo/dist-newstyle/build/x86_64-linux/ghc-8.2.2/tcm-memo-1.0.0.0/x/exe-no-lib/build/exe-no-lib/exe-no-lib-tmp/memoized-tcm/costMatrixWrapper.o
/home/user/tcm-memo/dist-newstyle/build/x86_64-linux/ghc-8.2.2/tcm-memo-1.0.0.0/x/exe-no-lib/build/exe-no-lib/exe-no-lib-tmp/memoized-tcm/dynamicCharacterOperations.o
-o
/home/user/tcm-memo/dist-newstyle/build/x86_64-linux/ghc-8.2.2/tcm-memo-1.0.0.0/x/exe-no-lib/build/exe-no-lib/exe-no-lib
-O2
-Wall
-hide-all-packages

While there is a lot going on in these linking commands, the important part is the presence of this line in the library linking command and it's absence in the non-library build target's linking command:

/home/user/tcm-memo/dist-newstyle/build/x86_64-linux/ghc-8.2.2/tcm-memo-1.0.0.0/build/memoized-tcm/costMatrix.dyn_o

This is the linking instruction for the _only_ file listed in the cxx-sources field. Including this linking instruction to the invoked linking command for non-library targets which use the cxx-sources field _should_ resolve this linking issue. I haven't yet investigated if the C++ sources are being successfully built with the cxx-options field in the non-library build targets.

Next steps

I would greatly appreciate pointers as to where in the Cabal library's codebase I need to shim in linking references to the C++ sources. @23Skidoo, @amigalemming, @ezyang; any of you have an ideas?

[recursion-ninja]

I really would like to get this defect resolved before before the Cabal-2.4 release. What is the time table on that?

[quasicomputational]

AIUI, it'll go with GHC 8.6 which is itself targetting August.

[23Skidoo]

What @quasicomputational said.

[23Skidoo]

@recursion-ninja Thanks for the very detailed and well-written bug report. First step to fixing this would be to add the test cases from your repro repo to the test suite, marking the ones that are currently failing as known-fail. The relevant code is in D.S.GHC, I suggest that you look at the buildOrReplLib and gbuild functions.

[recursion-ninja]

@23Skidoo I'll work on adding these build successes and failures properly to Cabal's test suite this week.