Libelektra: testshell_markdown_*: multiple tests working with the same file lead to errors

[mpranj]

Steps to Reproduce the Problem

cmake -GNinja -DBINDINGS="swig_python;cpp;swig_ruby" -DBUILD_FULL=ON -DENABLE_COVERAGE=OFF -DENABLE_OPTIMIZATIONS=ON -DENABLE_DEBUG=ON -DENABLE_LOGGER=ON -DBUILD_STATIC=ON -DCMAKE_INSTALL_PREFIX="$INSTALL_DIR" -DKDB_DB_SYSTEM="$SYSTEM_DIR" -DINSTALL_SYSTEM_FILES=OFF -DPLUGINS="ALL" -DTOOLS="ALL" -DKDB_DEFAULT_STORAGE="dump" -DKDB_DB_FILE="default.ecf" -DKDB_DB_INIT="elektra.ecf" ../workspace/libelektra/

ninja

ctest -j 18 --force-new-ctest-process --output-on-failure --no-compress-output -R "^testshell_*"
ctest -j 18 --force-new-ctest-process --output-on-failure --no-compress-output -R "^testshell_*"

Running the test multiple times is not key, but it increases the chances.

It seems that the parallelism of our tests introduces some problems. Multiple tests work on the same exact file and in this case testshell_markdown_tutorial_storage_plugins has not yet cleaned up config.yaml while testshell_markdown_camel is already working on it.

In some tests there are things like echo "bla:" >> config.yaml, which will obviously lead to problems too.

Expected Result

All tests should run successfully.

Actual Result

44/67 Test  #55: testshell_markdown_camel ......................***Failed    0.46 sec
Input: /home/jenkins/workspace/libelektra/src/plugins/camel/README.md
kdb mount config.yaml /tests/camel camel
kdb set /tests/camel/key value

ERROR - RET:
Return value “5” does not match “0”

kdb get /tests/camel/key

ERROR - RET:
Return value “5” does not match “0”


ERROR - STDOUT:
“”
does not match
“value”

kdb set /tests/camel/kittens "warm & fuzzy"

ERROR - RET:
Return value “5” does not match “0”

kdb get /tests/camel/kittens

ERROR - RET:
Return value “5” does not match “0”


ERROR - STDOUT:
“”
does not match
“warm & fuzzy”

kdb set /tests/camel/empty ""

ERROR - RET:
Return value “5” does not match “0”

kdb export /tests/camel camel

ERROR - RET:
Return value “5” does not match “0”


ERROR - STDOUT:
“”
does not match
“{⏎  "empty" : ""⏎, "key" : "value"⏎, "kittens" : "warm & fuzzy"⏎}”

kdb rm -r /tests/camel

ERROR - RET:
Return value “5” does not match “0”

kdb umount /tests/camel
shell_recorder /tmp/tmp.vvkypOpMbu RESULTS: 12 test(s) done 10 error(s).

—— Protocol ————————————————————————————————————————————————————
CMD: kdb mount config.yaml /tests/camel camel
RET: 0

CMD: kdb set /tests/camel/key value
RET: 5
=== FAILED return value does not match expected pattern 0
STDERR: The command kdb set failed while accessing the key database with the info:
Sorry, the error (#10) occurred ;(
Description: general parse error
Reason: /home/jenkins/.config/config.yaml:1:1: Expected “{” but found “r”
Ingroup: kdb
Module: 
At: /home/jenkins/workspace/libelektra/src/plugins/camel/camel.c:317
Mountpoint: user/tests/camel
Configfile: /home/jenkins/.config/config.yaml
ERROR: 10

CMD: kdb get /tests/camel/key
RET: 5
=== FAILED return value does not match expected pattern 0
STDERR: The command kdb get failed while accessing the key database with the info:
Sorry, the error (#10) occurred ;(
Description: general parse error
Reason: /home/jenkins/.config/config.yaml:1:1: Expected “{” but found “r”
Ingroup: kdb
Module: 
At: /home/jenkins/workspace/libelektra/src/plugins/camel/camel.c:317
Mountpoint: user/tests/camel
Configfile: /home/jenkins/.config/config.yaml
=== FAILED stdout does not match expected pattern value
ERROR: 10

CMD: kdb set /tests/camel/kittens "warm & fuzzy"
RET: 5
=== FAILED return value does not match expected pattern 0
STDERR: The command kdb set failed while accessing the key database with the info:
Sorry, the error (#10) occurred ;(
Description: general parse error
Reason: /home/jenkins/.config/config.yaml:1:1: Expected “{” but found “r”
Ingroup: kdb
Module: 
At: /home/jenkins/workspace/libelektra/src/plugins/camel/camel.c:317
Mountpoint: user/tests/camel
Configfile: /home/jenkins/.config/config.yaml
ERROR: 10

CMD: kdb get /tests/camel/kittens
RET: 5
=== FAILED return value does not match expected pattern 0
STDERR: The command kdb get failed while accessing the key database with the info:
Sorry, the error (#10) occurred ;(
Description: general parse error
Reason: /home/jenkins/.config/config.yaml:1:1: Expected “{” but found “r”
Ingroup: kdb
Module: 
At: /home/jenkins/workspace/libelektra/src/plugins/camel/camel.c:317
Mountpoint: user/tests/camel
Configfile: /home/jenkins/.config/config.yaml
=== FAILED stdout does not match expected pattern warm & fuzzy
ERROR: 10

CMD: kdb set /tests/camel/empty ""
RET: 5
=== FAILED return value does not match expected pattern 0
STDERR: The command kdb set failed while accessing the key database with the info:
Sorry, the error (#10) occurred ;(
Description: general parse error
Reason: /home/jenkins/.config/config.yaml:1:1: Expected “{” but found “r”
Ingroup: kdb
Module: 
At: /home/jenkins/workspace/libelektra/src/plugins/camel/camel.c:317
Mountpoint: user/tests/camel
Configfile: /home/jenkins/.config/config.yaml
ERROR: 10

CMD: kdb export /tests/camel camel
RET: 5
=== FAILED return value does not match expected pattern 0
STDERR: The command kdb export failed while accessing the key database with the info:
Sorry, the error (#10) occurred ;(
Description: general parse error
Reason: /home/jenkins/.config/config.yaml:1:1: Expected “{” but found “r”
Ingroup: kdb
Module: 
At: /home/jenkins/workspace/libelektra/src/plugins/camel/camel.c:317
Mountpoint: user/tests/camel
Configfile: /home/jenkins/.config/config.yaml
=== FAILED stdout does not match expected pattern {⏎  "empty" : ""⏎, "key" : "value"⏎, "kittens" : "warm & fuzzy"⏎}
ERROR: 10

CMD: kdb rm -r /tests/camel
RET: 5
=== FAILED return value does not match expected pattern 0
STDERR: The command kdb rm failed while accessing the key database with the info:
Sorry, the error (#10) occurred ;(
Description: general parse error
Reason: /home/jenkins/.config/config.yaml:1:1: Expected “{” but found “r”
Ingroup: kdb
Module: 
At: /home/jenkins/workspace/libelektra/src/plugins/camel/camel.c:317
Mountpoint: user/tests/camel
Configfile: /home/jenkins/.config/config.yaml
ERROR: 10

CMD: kdb umount /tests/camel
RET: 0
————————————————————————————————————————————————————————————————

System Information

• Elektra Version: master
• Docker Container: debian-stretch

Further Log Files and Output

cat .config/config.yaml

root:
  ___dirdata: 🐓
  level1:
    level2:
      level3: 🐣

[markus2330]

Thank you for reporting this problem!

@sanssecours Do you think we can solve this problem in general within the shellrecorder? (E.g. by putting all relative files into some temporary folder?)

[sanssecours]

Do you think we can solve this problem in general within the shellrecorder? (E.g. by putting all relative files into some temporary folder?)

That might be possible. However, if we do that correctly, we have to parse the kdb commands we use in Markdown Shell Recorder tests.

[mpranj]

The original reported problem is not solved by 9764b1aaf3dc1bd470b93293b3cd13967503b610 imo. We can still keep it closed if everyone is satisfied with the current state.

[markus2330]

Thank you for reporting this!

[sanssecours]

I ran the following script on my computer using a similar configuration as given above:

for repetitions in $(seq 50); do
    printf 'Repetition: %s\n' "$repetitions"
    ctest -j 18 --force-new-ctest-process --output-on-failure \
        --no-compress-output -R "^testshell_*" || exit 1
done

and none of the tests failed. It might make sense to update the ”steps to reproduce” in the issue description, to give a better hint on how to reproduce the problematic situation. I also do not think that parallelism is the problem, since all of the Markdown Shell Recorder tests run sequentially, as far as I know.

[mpranj]

You're right. I tested it again and 9764b1aaf3dc1bd470b93293b3cd13967503b610 seems to fix the problem indeed. My assumption that it has something to do with parallelism was most likely wrong.

Thank you for fixing it!