Onednn: Could not initialize a memory descriptor in mkl_concat_op.cc:813

[tonyreina]

Environment

Intel MKL-DNN includes hardware-specific optimizations and may behave
differently on depending on the compiler and build environment. Include
the following information to help reproduce the issue:

• CPU make and model (try lscpu; if your lscpu does not list CPU flags,
  try running cat /proc/cpuinfo | grep flags | sort -u)
  Architecture: x86_64
  CPU op-mode(s): 32-bit, 64-bit
  Byte Order: Little Endian
  CPU(s): 112
  On-line CPU(s) list: 0-111
  Thread(s) per core: 2
  Core(s) per socket: 28
  Socket(s): 2
  NUMA node(s): 2
  Vendor ID: GenuineIntel
  CPU family: 6
  Model: 85
  Model name: Intel(R) Xeon(R) Platinum 8180 CPU @ 2.50GHz
  Stepping: 4
  CPU MHz: 999.908
  BogoMIPS: 5005.94
  Virtualization: VT-x
  L1d cache: 32K
  L1i cache: 32K
  L2 cache: 1024K
  L3 cache: 39424K
  NUMA node0 CPU(s): 0-27,56-83
  NUMA node1 CPU(s): 28-55,84-111
• OS version (uname -a)
  Linux merlin-param01.merlin.cluster.bds 3.10.0-862.2.3.el7.x86_64 #1 SMP Wed May 9 18:05:47 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux
• Compiler version (gcc --version)
  gcc (GCC) 7.2.0
• MKLROOT value (echo MKLROOT=$MKLROOT)
  MKLROOT=
• CMake version (cmake --version)
  cmake3 version 3.11.2
• CMake output log
• git hash (git log -1 --format=%H)

Steps to reproduce

1. Clone the tf_cnn_benchmarks (git clone https://github.com/tensorflow/benchmarks.git)
2. cd benchmarks/scripts/tf_cnn_benchmarks
3. python tf_cnn_benchmarks.py --forward_only --device cpu --model googlenet --data_format=NHWC

Actual behavior

Describe the behavior you see.

AbortedError (see above for traceback): Operation received an exception:Status: 5, message: could not initialize a memory descriptor, in file tensorflow/core/kernels/mkl_concat_op.cc:813
[[Node: tower_0/v/cg/incept_v10_3/concat = _MklConcatV2[N=4, T=DT_FLOAT, Tidx=DT_INT32, _kernel="MklOp", _device="/job:localhost/replica:0/task:0/device:CPU:0"](tower_0/v/cg/incept_v10_3/conv21/Relu, tower_0/v/cg/incept_v10_3/conv23/Relu, tower_0/v/cg/incept_v10_3/conv25/Relu, tower_0/v/cg/incept_v10_3/conv26/Relu, tower_0/v/cg/incept_v10/concat/axis, tower_0/v/cg/incept_v10_3/conv21/Relu:1, tower_0/v/cg/incept_v10_3/conv23/Relu:1, tower_0/v/cg/incept_v10_3/conv25/Relu:1, tower_0/v/cg/incept_v10_3/conv26/Relu:1, DMT/_58)]]

Expected behavior

Describe the behavior you expect.

Looks like a bug in mkl concat op.

[tonyreina]

This is with conda install -c anaconda tensorflow which gives latest TF 1.9 with MKL-DNN

[vpirogov]

Thank you for the report, @tonyreina. The defect is likely related to the integration between Tensorflow and Intel MKL-DNN. In general, the defects related to applications that use MKL-DNN should be reported to corresponding application maintainers first. Summoning @agramesh1.

[vpirogov]

Let's try @claynerobison as well.

[tonyreina]

Thanks Vadim. As an update. The bug seems to be happening on GoogleNet and Inception topologies. U-Net works fine with the concat op. It also appears that on AVX-2 that the GoogleNet and Inception topologies work, but when I try the wheel against AVX-512 machine I get the described error message about the concat op failing.

[agramesh1]

@tonyreina sending you slack message to follow up on this issue.

[vpirogov]

The issue was submitted against TF, closing this one

[appyfizzA]

@agramesh1 @tonyreina
how did you solve this?