Models: limiting number of CPUs with intra_op_parallelism_threads

[xinyuwl]

I'm trying to run models/official/mnist/mnist.py on CPU.
I tried to limit the number of CPU threads used by Tensorflow to 1 by creating config like:

session_config = tf.ConfigProto(
inter_op_parallelism_threads=1,#flags_obj.inter_op_parallelism_threads,
intra_op_parallelism_threads,1,#flags_obj.intra_op_parallelism_threads,
allow_soft_placement=True)

But unfortunately, that didn't help. I've tried various variations on this, However, none of it seems to work: all 48 logical cores on the cluster are used. In my opinion, the main problem we are having here is the fact that we are not able to control the number of threads here. Although we set it to 1 with various TF options you can actually see that this job is creating many more threads on the node.

P.S. I'm using Python 2.7 and Tensorflow 1.11.0.

[tensorflowbutler]

Thank you for your post. We noticed you have not filled out the following field in the issue template. Could you update them if they are relevant in your case, or leave them as N/A? Thanks.
What is the top-level directory of the model you are using
Have I written custom code
OS Platform and Distribution
TensorFlow installed from
TensorFlow version
Bazel version
CUDA/cuDNN version
GPU model and memory
Exact command to reproduce

[azraelkuan]

+1 same problem using tf1.12 built from source and python36

[mayou36]

Same for me, binaries 1.12 and python36

@xinyuwl may you fill out the specs tensorflowbutler asked you for? So it's easier to debug

[xxxzhi]

+1 same problem using tf1.14.0 and python2.7.16.
Have I written custom code: No
OS Platform and Distribution: Ubuntu18.04
TensorFlow installed from: pip install tensorflow-gpu==1.14.0
TensorFlow version: 1.14.0
Bazel version
CUDA/cuDNN version: 10.0/7.4.2
GPU model and memory: GTX 2080 Ti /12G
Exact command to reproduce: https://github.com/vt-vl-lab/iCAN tools/Train_ResNet_HICO.py

However, When I switch to python3.7, I found the cpu usage keep below 100% no matter how I change intra_op_paralleism_threads

[xxxzhi]

by the way, I find there are many processes when I run the code in python2.
image

When I use python3.7, the code run extremely slower than python2

[tensorflowbutler]

Hi There,
We are checking to see if you still need help on this, as this seems to be considerably old issue. Please update this issue with the latest information, code snippet to reproduce your issue and error you are seeing.
If we don't hear from you in the next 7 days, this issue will be closed automatically. If you don't need help on this issue any more, please consider closing this.