Libelektra: Running tests

[alexsaber]

Question

How to execute all the tests? I want to run exactly the same tests when the CI build server runs.
Can you provide a step-by-step guide for someone without experience in C/C++ development?

What I tried

Like it says in TESTING.MD and COMPILE.MD I tried to execute "make run_all" and "make run_nokdbtests"

The problem

The tests are executed, but a lot more of them fail compared to the output of the build server.

The output

I built the project like this
mkdir build
cd build
cmake -DBUILD_SHARED=OFF -DBUILD_FULL=ON .. --> otherwise I got errors
make -j 5

And I ran tests like this
make run_all

-----a lot of lines omitted here------

-----below is the last seen output-------
The command kdb-full export failed while accessing the key database with the info:
Sorry, the error (#49) occurred ;(
Description: unknown command detected in dumpfile
Reason: key
Ingroup: plugin
Module: dump
At: /home/osboxes/TU/libelektra/src/plugins/dump/dump.cpp:191
Mountpoint: user
Configfile: /home/osboxes/.config/default.ecf

fatal: Could not export user config

64% tests passed, 63 tests failed out of 177

Label Time Summary:
kdbtests = 37.02 secproc (78 tests)
memleak = 36.32 secproc (70 tests)

Total Test time (real) = 39.77 sec

The following tests FAILED:
9 - testtool_mergingkdb (Failed)
26 - testcpp_kdb (Failed)
33 - testshell_markdown_base64 (Failed)
35 - testshell_markdown_boolean (Failed)
37 - testshell_markdown_camel (Failed)
40 - testshell_markdown_conditionals (Failed)
41 - testshell_markdown_constants (Failed)
43 - testshell_markdown_csvstorage (Failed)
45 - testshell_markdown_dini (Failed)
47 - testshell_markdown_directoryvalue (Failed)
50 - testshell_markdown_enum (Failed)
57 - testshell_markdown_hexnumber (Failed)
59 - testshell_markdown_hosts (Failed)
61 - testshell_markdown_iconv (Failed)
62 - testmod_ini (Failed)
63 - testshell_markdown_ini (Failed)
65 - testshell_markdown_ipaddr (Failed)
67 - testshell_markdown_line (Failed)
71 - testshell_markdown_mathcheck (Failed)
74 - testshell_markdown_mini (Failed)
76 - testshell_markdown_mmapstorage (Failed)
77 - testshell_markdown_multifile (Failed)
78 - testshell_markdown_network (Failed)
81 - testshell_markdown_ni (Failed)
82 - testshell_markdown_path (Failed)
83 - testshell_markdown_quickdump (Failed)
86 - testshell_markdown_range (Failed)
88 - testshell_markdown_reference (Failed)
90 - testshell_markdown_rename (Failed)
92 - testshell_markdown_shell (Failed)
95 - testshell_markdown_type (Failed)
97 - testshell_markdown_uname (Failed)
101 - testscr_check_distribution (Failed)
103 - testscr_check_error (Failed)
108 - testscr_check_kdb_internal_check (Failed)
110 - testscr_check_merge (Failed)
112 - testscr_check_mount (Failed)
117 - testscr_check_real_world (Failed)
118 - testscr_check_resolver (Failed)
119 - testscr_check_spec (Failed)
122 - testshell_db_changes (Failed)
123 - testshell_host (Failed)
124 - testshell_listtest (Failed)
125 - testshell_mathcheck (Failed)
126 - testshell_profiletest (Failed)
127 - testshell_script (Failed)
128 - testshell_selftest (Failed)
129 - testshell_markdown_kdb_complete (Failed)
130 - testshell_markdown_kdb_global_umount (Failed)
131 - testshell_markdown_kdb_ls (Failed)
132 - testshell_markdown_kdb_find (Failed)
133 - testshell_markdown_tutorial_arrays (Failed)
134 - testshell_markdown_tutorial_cascading (Failed)
135 - testshell_markdown_tutorial_validation (Failed)
136 - testshell_markdown_msr_syntax (Failed)
137 - testshell_markdown_readme_msr (Failed)
138 - testshell_markdown_issue_template (Failed)
139 - testshell_markdown_ini_crash_test (Failed)
173 - testkdb_conflict (Failed)
174 - testkdb_error (Failed)
175 - testkdb_nested (Failed)
176 - testkdb_simple (Failed)
177 - testkdb_highlevel (Failed)
Errors while running CTest
tests/CMakeFiles/run_all.dir/build.make:57: recipe for target 'tests/CMakeFiles/run_all' failed
make[3]: * [tests/CMakeFiles/run_all] Error 8
CMakeFiles/Makefile2:12208: recipe for target 'tests/CMakeFiles/run_all.dir/all' failed
make[2]: [tests/CMakeFiles/run_all.dir/all] Error 2
CMakeFiles/Makefile2:12215: recipe for target 'tests/CMakeFiles/run_all.dir/rule' failed
make[1]: [tests/CMakeFiles/run_all.dir/rule] Error 2
Makefile:3304: recipe for target 'run_all' failed
make: * [run_all] Error 2

[kodebach]

The command kdb-full export failed while accessing the key database with the info:
Sorry, the error (#49) occurred ;(
Description: unknown command detected in dumpfile
Reason: key
Ingroup: plugin
Module: dump
At: /home/osboxes/TU/libelektra/src/plugins/dump/dump.cpp:191
Mountpoint: user
Configfile: /home/osboxes/.config/default.ecf

This error message looks like the file /home/osboxes/.config/default.ecf is broken. If you don't have anything important in your KDB, just delete the file, otherwise please post the file so I can take a closer look. (Note: the file is binary, so opening it in a text editor and copying the contents might not work)

It might also be a permissions problem. For development purposes, I would recommend using the cmake options: -DKDB_DB_SYTEM="~/.config/kdb/system" and -DKDB_DB_SPEC="~/.config/kdb/spec".

---

The easiest way to execute all tests similar to the way that the Jenkins Server does is by following these instructions to build/download and run the debian-stretch docker container (the docker run command has to be executed in the root directory of the git repository). Then inside the container you just run:

mkdir build-docker && cd build-docker

# compile
cmake .. -DBINDINGS="ALL;-DEPRECATED" -DPLUGINS="ALL;-DEPRECATED" -DTOOLS="ALL" -DENABLE_DEBUG=ON -DKDB_DB_HOME="$PWD" -DKDB_DB_SYSTEM="$PWD/.config/kdb/system" -DKDB_DB_SPEC="$PWD/.config/kdb/system"
make -j 10

# run all tests
make run_all

# run tests with valgrind
make run_memcheck

The build server also runs the tests for the installed version of kdb and some other things that are more complicated to replicate. If you really want to run everything (almost never necessary) take a look at the Jenkinsfile.

[alexsaber]

@kodebach
Thank you very much for such an extended answer!
I followed your advice and went for testing using Docker. I could successfully build my own image by execution following
docker build -t buildelektra-stretch-full \ --build-arg JENKINS_USERID=id -u\ --build-arg JENKINS_GROUPID=id -g\ -f scripts/docker/debian/stretch/Dockerfile \ scripts/docker/debian/stretch/

Then I started a container like this
docker run -it --rm \ -v "$PWD:/home/jenkins/workspace" \ -w /home/jenkins/workspace \ buildelektra-stretch-full

Inside the container, I built the project as you described it, but I get the following error when it is almost finished:

[ 97%] Generating libHShaskell-ghc8.0.1.so
cabal: Error: some packages failed to install:
libelektra-haskell-0.8.26 failed during the final install step. The exception
was:
ExitFailure 1
unrecognized 'configure' option `--dynlibdir=$libdir/$abi'
src/plugins/haskell/CMakeFiles/haskell.dir/build.make:64: recipe for target 'src/plugins/haskell/libHShaskell-ghc8.0.1.so' failed
make[2]: * [src/plugins/haskell/libHShaskell-ghc8.0.1.so] Error 1
CMakeFiles/Makefile2:5313: recipe for target 'src/plugins/haskell/CMakeFiles/haskell.dir/all' failed
make[1]: [src/plugins/haskell/CMakeFiles/haskell.dir/all] Error 2
Makefile:160: recipe for target 'all' failed
make: ** [all] Error 2

Do you have an idea what might cause the error?

[sanssecours]

Inside the container, I built the project as you described it, but I get the following error when it is almost finished…

I also used the steps described above and the build system reports the same error.

Do you have an idea what might cause the error?

The problem seems to be the Haskell binding. I would recommend to not include this binding:

cmake ..                                        \
      -DBINDINGS="ALL;-DEPRECATED;-haskell"     \
      -DPLUGINS="ALL;-DEPRECATED"               \
      -DTOOLS="ALL"                             \
      -DENABLE_DEBUG=ON                         \
      -DKDB_DB_HOME="$PWD"                      \
      -DKDB_DB_SYSTEM="$PWD/.config/kdb/system" \
      -DKDB_DB_SPEC="$PWD/.config/kdb/system"

. If I use the commands above, the build step (make -j 10) finishes successfully.

[markus2330]

@sanssecours thank you for the info. Can you please fix the Dockerfiles in our repo? (exclude haskell)

[sanssecours]

Can you please fix the Dockerfiles in our repo? (exclude haskell)

Since the Haskell build still works on the Jenkins build server, I do not think that is a good idea.

[markus2330]

I think it is more important that building Elektra works for new developer than continuously testing something which is not maintained anyway (which is unfortunately the case for Haskell).

If it is again maintained, we readd it to the tests of course. Then such problems can also be fixed by the Haskell maintainer.

[sanssecours]

I think it is more important that building Elektra works for new developer than continuously testing something which is not maintained anyway (which is unfortunately the case for Haskell).

While I agree, I would not recommend the full Dockerfile for Debian for new developers anyway. In my opinion it makes much more sense to use the Alpine Docker image, which is

• much smaller,
• takes a lot less time to build,
• uses less outdated packages

.

[kodebach]

I would not recommend the full Dockerfile for Debian for new developers anyway

Me neither, but the question was how to execute all tests. AFAIK that is only possible with the Debian image.

[alexsaber]

Thank you very much! I can now execute all tests.

[markus2330]

@sanssecours you can also add that the official repo has Elektra packages as advantage. Seems like the image is also used already, see #2593

[alexsaber]

Thank you all for your support! I created a tutorial which explains how to run the tests here --> https://github.com/ElektraInitiative/libelektra/pull/2768

Thus this issue/question is resolved.