Openenclave: pthread and oethread tests timing out

[radhikaj]

The recent additions of TestErrnoMultiThreadsSameenclave and TestErrnoMultiThreadsDifferentenclave are causing pthread and oethread tests to timeout from time to time causing discrepancies in CI. We are investigating this. In the mean time, we have turned off these tests. See PR #2919.

[jhand2]

(gdb) backtrace
#0  0x00007fdd63f22d2d in __pthread_timedjoin_ex ()
   from /lib/x86_64-linux-gnu/libpthread.so.0
#1  0x00007fdd63c4e933 in std::thread::join() ()
   from /usr/lib/x86_64-linux-gnu/libstdc++.so.6
#2  0x000000000040f478 in test_errno_multi_threads_diffenclave(_oe_enclave*, _oe_enclave*) ()
#3  0x000000000040d41d in main ()

Backtrace from my local repro of this issue.

[yanxue-intel]

Is the TestErrnoMultiThreadsDifferentenclave triggers the OE thread deadlock?
if yes, I will develop more cases about multi-enclaves with multi-threads for OE.

[jhand2]

@yanxue-intel yes it appears so. Before we make more test cases, I would be interested to know why the test causes deadlock. Would you have time to look into this?

I was able to repro by running the test many times in a row and then attaching a debugger when it started hanging.

[nonpolarity]

Is the TestErrnoMultiThreadsDifferentenclave triggers the OE thread deadlock?
if yes, I will develop more cases about multi-enclaves with multi-threads for OE.

@yanxue-intel Besides the deadlock problem, I think you are right we need more multi-threads test. While I commented out the wait function, the test can still pass. It seems no effective test for wait/wake.

[jhand2]

Repro steps:

• Build SDK with -DCMAKE_BUILD_TYPE=Release
• Build SDK with -DLVI_MITIGATION=ControlFlow`
• Run this test until it hangs with the ctest option --repeat-until-fail <n_times>
• Attach a debugger and get a backtrace to see where hang it happening

Note that it may be possible to get a repro without the first two steps, but I haven't had a chance to try it out yet. It would likely be easier to debug if the SDK isn't compiled with those options.

[yanxue-intel]

Is the TestErrnoMultiThreadsDifferentenclave triggers the OE thread deadlock?
if yes, I will develop more cases about multi-enclaves with multi-threads for OE.

@yanxue-intel Besides the deadlock problem, I think you are right we need more multi-threads test. While I commented out the wait function, the test can still pass. It seems no effective test for wait/wake.

Is the untrust part has mutex? maybe pthread_join hangs in untrust part.

[nonpolarity]

@yanxue-intel Please check my pr #2947. Although the multithreads test use mutex to avoid race condition https://github.com/openenclave/openenclave/blob/3f6a1ac6fe1fe3984aad2854b54d315b1efae9d2/tests/thread/enc/cond_tests.cpp#L15, the test can not guarantee the sequence of thread wait/wake that wake executes after wait.