Cuml: [BUG] cuml logger wrapper doesn't seem to work well in multi-threaded environments

[teju85]

Describe the bug
If any of the logger methods are called from inside a multi-threaded C++ code block (eg: from inside an 'omp pragma') and further when this code is called from python, it is resulting in a hang.

It still needs to be established whether this hang manifests only when called from python or even from C++.

Steps/Code to reproduce bug

1. pick any omp parallel code block
2. insert any logger call, anywhere inside this block. eg: CUML_LOG_WARN("This is a warning message from thread=%d", omp_get_thread_num());
3. call this resulting code block from python

Expected behavior
Logging methods should work seamlessly even when called from multi-threaded regions.

Kudos to @vinaydes for finding this root-cause.

[vinaydes]

Thanks, @teju85 for creating the issue.

For testing if it hangs for C++ code, I added following to C++ logger test

  #pragma omp parallel for num_threads(2)
  for(int i = 0; i < 2; i++)  {
    CUML_LOG_WARN("Test %d", omp_get_thread_num());
  }
  CUML_LOG_WARN("Out of the loop\n");

C++ test did not hang. So for now, it looks like it hangs only for Python.
Also printing omp_get_thread_num() is completely optional. It is there to see which thread is stuck.

[wphicks]

I think I can claim responsibility for this one. If we disable the callback to use Python's print for logging, the hang goes away. I'll look into it and see if I can figure out what's going on.

[wphicks]

Fix is available here: #3037. Reviews would be very welcome; note the comment on that PR about the difficulty with creating a clean unit test for this issue.