Libelektra: haskell: cmake issues

[ingwinlu]

Steps to Reproduce the Problem

Build without internet access or right now (since hackage is down) + haskell binding

Expected Result

cmake should fail if required packages are not available

Actual Result

cmake passes and make tries to call cabal during build

System Information

• Elektra Version: master

Further Log Files and Output

[ 92%] Linking C executable ../../bin/testmod_augeas
Scanning dependencies of target testmod_hosts
[ 92%] Building C object src/plugins_tests/CMakeFiles/testmod_hosts.dir/hosts/testmod_hosts.c.o
[ 92%] Built target testmod_dbusrecv
cabal: Failed to download http://hackage.haskell.org/00-index.tar.gz : HTTP
code 503
src/bindings/haskell/CMakeFiles/c2hs_haskell.dir/build.make:71: recipe for target 'src/bindings/haskell/libHSlibelektra-haskell.so' failed
make[2]: *** [src/bindings/haskell/libHSlibelektra-haskell.so] Error 1
CMakeFiles/Makefile2:17443: recipe for target 'src/bindings/haskell/CMakeFiles/c2hs_haskell.dir/all' failed
make[1]: *** [src/bindings/haskell/CMakeFiles/c2hs_haskell.dir/all] Error 2
make[1]: *** Waiting for unfinished jobs....
Scanning dependencies of target testmod_ini
[ 92%] Building C object src/plugins_tests/CMakeFiles/testmod_ini.dir/ini/testmod_ini.c.o

Full Log

[markus2330]

Thank you for reporting!

[ingwinlu]

@e1528532 any idea why we call cabal update here?

Any chance to remove that from the build process and instead do it earlier?

[ingwinlu]

[100%] Building CXX object src/bindings/intercept/env/tests/CMakeFiles/test_context.dir/test_context.cpp.o
[100%] Linking CXX executable ../../../../../bin/test_fork
[100%] Linking CXX executable ../../../../../bin/test_context
[100%] Built target test_fork
[100%] Built target test_context
[100%] Linking CXX executable ../../../../../bin/test_getenv
[100%] Built target test_getenv
[100%] Linking CXX executable ../../../../bin/testtool_samemountpoint
[100%] Built target testtool_samemountpoint
cabal: Error: some packages failed to install:
HUnit-1.6.0.0 depends on call-stack-0.1.0 which failed to install.
call-stack-0.1.0 failed while downloading the package. The exception was:
user error (Failed to download
http://hackage.haskell.org/package/call-stack-0.1.0.tar.gz : HTTP code 504)
hspec-2.5.0 depends on call-stack-0.1.0 which failed to install.
hspec-core-2.5.0 depends on call-stack-0.1.0 which failed to install.
hspec-discover-2.5.0 failed while downloading the package. The exception was:
user error (Failed to download
http://hackage.haskell.org/package/hspec-discover-2.5.0.tar.gz : HTTP code
504)
hspec-expectations-0.8.2 depends on call-stack-0.1.0 which failed to install.
quickcheck-io-0.2.0 depends on call-stack-0.1.0 which failed to install.
unrecognized 'configure' option `--dynlibdir=$libdir/$abi'
src/bindings/haskell/CMakeFiles/c2hs_haskell_tests.dir/build.make:74: recipe for target 'src/bindings/haskell/dist/build/testhaskell_basic/testhaskell_basic' failed
make[2]: *** [src/bindings/haskell/dist/build/testhaskell_basic/testhaskell_basic] Error 1
CMakeFiles/Makefile2:17404: recipe for target 'src/bindings/haskell/CMakeFiles/c2hs_haskell_tests.dir/all' failed
make[1]: *** [src/bindings/haskell/CMakeFiles/c2hs_haskell_tests.dir/all] Error 2
Makefile:160: recipe for target 'all' failed
make: *** [all] Error 2
script returned exit code 2

[ingwinlu]

FindHaskell.cmake also has some problems when cabal has never been run on the system before:
GHC_DYNAMIC_LIBRARY_DIR can not be set because ls -F "${CABAL_LOCATION}/lib" fails as it is not existing.

[ingwinlu]

Failed again on me today with the following output:

cabal: Error: some packages failed to install:
QuickCheck-2.11.3-19mdm5lhmxMFf4AANg83KE depends on QuickCheck-2.11.3 which
failed to install.
hspec-2.5.0-FQn7auLuKhR3ku9rnURxaQ depends on hspec-2.5.0 which failed to
install.
hspec-core-2.5.0-9KjLOgn2eZXAM9cROrdMVV depends on hspec-core-2.5.0 which
failed to install.
primitive-0.6.3.0-CXy1O9sQYlI58rn9KQkFyo failed while downloading the package.
The exception was:
user error (hTryLock: lock already exists:
/home/jenkins/.cabal/packages/hackage.haskell.org/hackage-security-lock)
quickcheck-io-0.2.0-4xxaMfY1GFe36gllJF7Ubr depends on quickcheck-io-0.2.0
which failed to install.
random-1.1-LLUGZ7T9DqQ5vN0Jbcd0We failed while downloading the package. The
exception was:
user error (hTryLock: lock already exists:
/home/jenkins/.cabal/packages/hackage.haskell.org/hackage-security-lock)
tf-random-0.5-H0EZh6X3duX25gCCL9n0I3 depends on tf-random-0.5 which failed to
install.

traditionally cmake would check for the existence of those deps and than not need to touch them anymore. What is the reason we try to install during build?

[markus2330]

@e1528532 ping

[ingwinlu]

https://github.com/ElektraInitiative/libelektra/blob/master/src/bindings/haskell/CMakeLists.txt#L107 should probably use our test macros

[e1528532]

The thing is that all the haskell things are built via cabal using sandboxes to avoid having a users global cabal packages leak into the build process.
Cmake doesn't really know about all these things. It would require a cmake module that takes care about the whole haskell compilation process manually and bypass cabal but as there isn't really such a module that offers all required functionality i simply call cabal which then downloads the dependencies into a sandbox and builds the whole thing. the sandbox is reused between all the different plugins as they often use the same libraries to keep the impact low.
I agree though that there still seems to be some kind of race condition / concurrency issue though i already tried to serialize the cabal dependency installation procedure. I'll investigate that when i have time but right now it works sufficiently enough for me. There are some changes in the typechecker-regex-prototype branch which are related to that they might help make it more stable when its merged.

[markus2330]

I think what @ingwinlu wanted to say is that you simply should call the exact same commands during the cmake stage instead of during the build process.

So you simply should use execute_process instead of add_custom_target

I agree though that there still seems to be some kind of race condition / concurrency issue though i already tried to serialize the cabal dependency installation procedure

That would be solved because the cmake and the build process are completely separated.

If hackage is down, however, the job would still fail, though.

now it works sufficiently enough for me

If it does not work reliable and sometimes makes build jobs fail, it is a problem. I do not know if this is the case. @ingwinlu can you comment on the frequency of this problem?

[ingwinlu]

I am aware of sandboxes, let me go into what I don't get into more detail:

cmake checks if all dependencies can be satisfied and prepare everything for a build. For C that would mean check if a libraries exist, for java it would mean that the jdk can be found etc.

Why do we care about where it is installed in the haskell plugin when we don't do that anywhere else?
Imaging cmake trying to download gcc if it isn't found or running apt-get update.

It might be useful for developers to specify such a sandbox or a global/userwide installation if required, but per default cmake should only check if the packages can be found and fail if it does not and NOT try to install them at all.

So in our case I only expect to have cabal list --installed --simple-output be grabbed for the installed package + version.
You can then let users specify the environment(global + user + dedicated sandbox) via a cmake variable or rely on the users cabal settings to pinpoint to the sandbox.

[e1528532]

So in our case I only expect to have cabal list --installed --simple-output be grabbed for the installed package + version.
You can then let users specify the environment(global + user + dedicated sandbox) via a cmake variable or rely on the users cabal settings to pinpoint to the sandbox.

Fair point i have to say. I don't remember the exact reasons anymore why it ended up like that but the whole thing was quite complicated and really took a lot of time to get it working (apparently, partially working ;) ).
But i just tried, you can really just create a sandbox anywhere and install the dependencies into them. I think i've never tried that for some reason as i wanted the build process to be as automated as possible. So maybe it is indeed a good solution to require the user doing that manually like its done for the other deps. He can then simply go to some folder (maybe we assume it to be in the build directory by default) and then install the deps there, and then use this sandbox for the further build process.

I think we should try this approach out but currently i'm quite busy so i'm not sure when i find time for that.

[ingwinlu]

I think we need to prioritize because we build haskell much more frequent now. We might up getting our build servers banned from hackage.

[markus2330]

I am not sure if this is related, but now we regularly get over 3.5Gb/day. Not really a big issue but if it keeps that way I should again change the threshold so that I will see actual problems.

[ingwinlu]

should not be related as all haskell builds are not done on the community
server.

however the new multibranch build was running quite some time without usage
of the mirror repository which might have caused high io. also it is
currently catching up with all open pr's which might be a lot of io as well.

On 14 May 2018 at 09:10, markus2330 [e-mail] wrote:

I am not sure if this is related, but now we regularly get over 3.5Gb/day.
Not really a big issue but if it keeps that way I should again change the
threshold so that I will see actual problems.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/ElektraInitiative/libelektra/issues/1902#issuecomment-388718109,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AEOv-pDfLnaPNUGBPGy_eqt6ZPSA7fX4ks5tyS3vgaJpZM4TRl-a
.

[e1528532]

the sandbox gets cached both on travis and the haskell build job, so i am not sure either if this is related, as it should get built only once. you could still drop haskell from the jenkins build all please command if you are worried, i can still build it manually for the relevant PRs/branches.

[markus2330]

No, I am not worried but we should also not waste bandwidth or power.

i can still build it manually for the relevant PRs/branches.

We exactly want to avoid such a situation by the build server :smile:

[ingwinlu]

\ https://build.libelektra.org/jenkins/blue/organizations/jenkins/libelektra/detail/PR-2000/1/pipeline

[ingwinlu]

cabal: The timestamps file is corrupted. Please delete and recreate the
sandbox.
src/bindings/haskell/CMakeFiles/c2hs_haskell_tests.dir/build.make:74: recipe for target 'src/bindings/haskell/dist/build/testhaskell_basic/testhaskell_basic' failed
make[2]: *** [src/bindings/haskell/dist/build/testhaskell_basic/testhaskell_basic] Error 1
CMakeFiles/Makefile2:14647: recipe for target 'src/bindings/haskell/CMakeFiles/c2hs_haskell_tests.dir/all' failed

[e1528532]

the whole build procedure got improved alot and the dependency fetching issue is also resolved, closing.