Onednn: Relationships between thread number settings of OMP, MKL, Tensorflow session

[t13m]

Hi, dear developers,
I am using tensorflow with mkl-dnn integrated in my production environment for inference, where the cpu resource need to be controlled. There are several concepts of parallelism in this system make me confused. Can anyone explain it or list some useful references would be very helpful!

The concepts or questions are:

• when running my program, the OMP_THREAD_NUM environment variable
• when creating a tensorflow session, the inter_op_parallelism and intra_op_parallelism and the thread pool settings, how would these three settings interact with OMP_THREAD_NUM. My guess is no matter what these settings are, eventually there will be OMP_THREAD_NUM threads, is it right?
• when creating several tensorflow session, are they limited to one shared OMP_THREAD_NUM? or limited to their own OMP_THREAD_NUM?

My goal is like this, several tensorflow sessions run simultaneously, every session has seperated limited resource. Using tensorflow without mkldnn integrated, this goal can be achieved by inter_op_parallelism and intra_op_parallelism and session thread pool settings. Now the mkldnn imports the setting of OMP_THREAD_NUM, what should I do with it?

Thank you very much

[t13m]

I notice that there are both threaded and single-thread version of MKL library, the mkl-dnn integration for tensorflow seems bundled with threaded version (mklml). Is it possible to use a single threaded version? Should it be done to avoid oversubscription in multithread programs?

[rsdubtso]

I think there is a detailed writeup in the TensorFlow guide regarding interaction betwee OpenMP and Eigen threads. In short, it may happen that OpenMP regions will be nested inside Eigen threads which will result in oversubscription. If your model does not run well with inter_op_parallelism == intra_op_parallelism == 1 and default number of OpenMP threads (which is equal to the number of logical CPUs in the current process's CPU mask), then it may make sense to try setting OMP_NUM_THREADS to 1. The value of the environment variable is shared by all threads in the process, so each session would get the same value.

[t13m]

Thank you rsdubtso.
So the scenario can be like follows as I understand. Is it right?

OMP_NUM_THREADS=2
Session(inter_op_parallelism=3, intra_op_parallelism=4)

Session ---  inter_op_thread_pool(3)
           |          |--- inter op thread # 1 (mkl_matmul for example, also controled by OMP_NUM_THREADS)
           |          |--- inter op thread # 2
           |          |--- inter op thread # 3
           |
              --  intra_op_thread_pool(4)
                          |--- eigen thread # 1
                          |            |--- omp thread # 1
                          |            |--- omp thread # 2
                          |--- eigen thread # 2
                          |--- eigen thread # 3
                          |--- eigen thread # 4

[rsdubtso]

Right. Moreover, the topmost inter_op_thread_pool, thread # 1 will have its own OpenMP team, and thread # 1 in the second thread pool will have its own. Therefore, in the worst case you would end up with an OpenMP team per eigen thread, which is rarely what you want.

[t13m]

Thank you so much! It really helps a lot. And sorry for it being probably off-topic (question of tensorflow in mkl-dnn's repository :P), I don't think it is possible for me to figure all these out without digging into lots of code.

[rsdubtso]

No problem, glad to help! Closing as resolved.

[cfRod]

I think there is a detailed writeup in the TensorFlow guide regarding interaction betwee OpenMP and Eigen threads. In short, it may happen that OpenMP regions will be nested inside Eigen threads which will result in oversubscription. If your model does not run well with inter_op_parallelism == intra_op_parallelism == 1 and default number of OpenMP threads (which is equal to the number of logical CPUs in the current process's CPU mask), then it may make sense to try setting OMP_NUM_THREADS to 1. The value of the environment variable is shared by all threads in the process, so each session would get the same value.

Is the link still available to the TensorFlow guide? I got a 404 page.