Virtual-environments: OpenMP threads count >0 tests stop working on Ubuntu 18.04 & 20.04

[myselfhimself]

Description
Since December 2019 pytest test cases in the gmic-py Python c++ binding project would succeed on Ubuntu 18.04.
Since the beginning of july, those tests fail.
Running the nektos/act tool with an ubuntu 18.04 Docker image mimicking your virtual-environments config show 100% passing OpenMP tests with an identical workflow file and show that your online cloud Github Actions environment has issues possibly linked to its Ubuntu 18.04 VM configuration or its kernel (since Ubuntu 20.04 shows the same errors).
Have you changed something on your hosts' kernel or ubuntu images possibly preventing OpenMP from spawning threads?

Area for Triage:
Area: Containers
Area: C/C++
Area: Deployment/Release

Question, Bug, or Feature?:
Bug

Virtual environments affected

• [ ] macOS 10.15
• [ ] Ubuntu 16.04 LTS
• [x] Ubuntu 18.04 LTS
• [x] Ubuntu 20.04 LTS
• [ ] Windows Server 2016 R2
• [ ] Windows Server 2019

Expected behavior
When C/C++ OpenMP test for number of threads being run, the number of those should be >0
To help you understand the layers on top: gmic-py under test using the pytest testing framework, bind the libgmic c++ library which embeds openmp pragmas being called through a meta-language for image processing named G'MIC.

Actual behavior
Flakiness of testing with 5% chance or less of having those number of threads >0, when it should be 100% (and is so locally with nektos/act or without any docker image and handmade testing on my bare-bones Ubuntu 18.04).

Repro steps

1. Run the cpython debug workflow file
2. Look at the logs for the 10-times repeated test_run_gmic_ensure_openmp_linked_and_working tests, notice they are 99-100% failing.
3. On a Linux OS: git clone the gmic-py project and switch to the numpy_io branch or this very recent same-branch commit. Install https://github.com/nektos/act for Linux, add -P ubuntu-latest=nektos/act-environments-ubuntu:18.04 into a current directory .actrc file, possibly tune your Docker configuration if your /var/lib/docker drive cannot tolerate 6-18GB images (eg. put a user home-based folder instead), run act in order to mimick the Github Actions job. Observe 100% passing of the test_run_gmic_ensure_openmp_linked_and_working tests.

• The related issue on my side is here: https://github.com/myselfhimself/gmic-py/issues/47
  with related flakiness comment here: https://github.com/myselfhimself/gmic-py/issues/47#issuecomment-678232479
• Full example logs here: (look for: test_run_gmic_ensure_openmp_linked_and_working ; the test cases are run 10 times to help showcase flakiness)
  The related pytest test case ensuring that the number of openmp threads open is >0 is here: https://github.com/myselfhimself/gmic-py/blob/numpy_io/tests/test_gmic_py.py#L52
• A C ifdef check ensures that openmp's compilation flags is enabled. This check succeeds 100% as needed.
• An handmade ldd of the compiled binding shared library ensures that openmp's .so file is properly linked. Succeeds 100% as needed. Excerpt here:
  libgomp.so.1 => /usr/lib/x86_64-linux-gnu/libgomp.so.1 (0x00007ff2a807c000)

[myselfhimself]

Hello,

I have simplified the environment and reproduced the issue in a separate tiny repository, with a simple example OpenMP C file, which gets gcc-compiled, ldd-checked and run.

Please head on to that other repository https://github.com/myselfhimself/github-actions-openmp-troubleshoot and look at the latest actions run.

When running the troubleshoot.yml workflow file locally (eg. with nektos/act using a Ubuntu 18.04 image), the log tail is:

|   linux-vdso.so.1 (0x00007ffc06e67000)
|   libgomp.so.1 => /usr/lib/x86_64-linux-gnu/libgomp.so.1 (0x00007f6810779000)
|   libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f6810388000)
|   libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007f6810184000)
|   libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f680ff65000)
|   /lib64/ld-linux-x86-64.so.2 (0x00007f6810bb2000)
| Hello World from thread = 0
| Number of threads = 12
| Hello World from thread = 10
| Hello World from thread = 7
| Hello World from thread = 11
| Hello World from thread = 2
| Hello World from thread = 8
| Hello World from thread = 6
| Hello World from thread = 4
| Hello World from thread = 3
| Hello World from thread = 9
| Hello World from thread = 5
| Hello World from thread = 1

while on the online Github Actions log I get:

    linux-vdso.so.1 (0x00007ffe71921000)
    libgomp.so.1 => /usr/lib/x86_64-linux-gnu/libgomp.so.1 (0x00007fd653bb5000)
    libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007fd6537c4000)
    libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007fd6535c0000)
    libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007fd6533a1000)
    /lib64/ld-linux-x86-64.so.2 (0x00007fd653ff6000)
Hello World from thread = 0
Number of threads = 2
Hello World from thread = 1

Why is the allowed number of threads so low on Github Actions?
Now looking into making the allowed number of threads higher by system configuration...

[myselfhimself]

Just added an environment variable for setting OpenMP's number of threads before running the compiled code: https://github.com/myselfhimself/github-actions-openmp-troubleshoot/runs/1012676930?check_suite_focus=true

OMP_NUM_THREADS:16
OMP_SCHEDULE:
OMP_DYNAMIC:
OMP_NESTED:
    linux-vdso.so.1 (0x00007ffc2898f000)
    libgomp.so.1 => /usr/lib/x86_64-linux-gnu/libgomp.so.1 (0x00007f61f28de000)
    libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f61f24ed000)
    libdl.so.2 => /lib/x86_64-linux-gnu/libdl.so.2 (0x00007f61f22e9000)
    libpthread.so.0 => /lib/x86_64-linux-gnu/libpthread.so.0 (0x00007f61f20ca000)
    /lib64/ld-linux-x86-64.so.2 (0x00007f61f2d1f000)
Hello World from thread = 1
Hello World from thread = 3
Hello World from thread = 0
Number of threads = 16
Hello World from thread = 2
Hello World from thread = 8
Hello World from thread = 9
Hello World from thread = 10
Hello World from thread = 11
Hello World from thread = 12
Hello World from thread = 13
Hello World from thread = 14
Hello World from thread = 15
Hello World from thread = 7
Hello World from thread = 6
Hello World from thread = 5
Hello World from thread = 4

That has fixed the issue in the troubleshooting-oriented repository. Now testing it for the gmic-py project.

[myselfhimself]

Increasing the OMP_NUM_THREADS environment variable from undefined to 16 (I should test lower numbers maybe) in gmic-py did solve my issue.
If this seems critical on your side, you may want to tune lower level configuration. I will not bother more here.
Thanks for the awesome project and tools!

[maxim-lobanov]

Thank you for providing solution for your issue, it could be helpful for people who face with the same.
Sorry that we didn't have a chance to take a look earlier. Usually, such issues take more time to take a look because they require specific skillset.

Btw, usually, image changes can be tracked by history of readme file: https://github.com/actions/virtual-environments/commits/main/images/linux/Ubuntu1804-README.md

I am closing this issue but please let us know if you still need any assistance

[myselfhimself]

Thank you for taking the time to answer and pointing me to the Linux images
changelog. I am wondering if the amount of CPUs or Kernel has changed over
time and thus the OpenMP auto-setup of threads number would have been
impacted. So maybe the images themselves are not so related.
This issue's legacy will definitely help people with similar problems as
mine.
Best wishes :)

po 24. 8. 2020 v 8:59 odesílatel Maxim Lobanov [e-mail]
napsal:

Thank you for providing solution for your issue, it could be helpful for
people who face with the same.
Sorry that we didn't have a chance to take a look earlier. Usually, such
issues take more time to take a look because they require specific skillset.

Btw, usually, image changes can be tracked by history of readme file:
https://github.com/actions/virtual-environments/commits/main/images/linux/Ubuntu1804-README.md

I am closing this issue but please let us know if you still need any
assistance

—
You are receiving this because you authored the thread.
Reply to this email directly, view it on GitHub
https://github.com/actions/virtual-environments/issues/1465#issuecomment-678944320,
or unsubscribe
https://github.com/notifications/unsubscribe-auth/AAJU5QTGVQP2BE3VT3NNI23SCIFULANCNFSM4QHGMNWQ
.