Bazel: Include path errors despite build recipe having the location and isystem being set

[s-sajid-ali]

Description of the problem / feature request:

Further context at : https://github.com/tensorflow/tensorflow/issues/35014.

While building TF from source, bazel generates local configs for dependencies that are already installed. When it attempts to build the nasm library from source, it fails because in attempting to compile nasm/output/outelf.c (which has a #include eval.h), bazel tries to link it to the wrong header. The nasm build recipe at tensorflow/third_party/nasm/BUILD.bazel shows that the includes for nasm are :

    includes = [
        "asm",
        "include",
        "output",
        "x86",
    ],

and the folder asm does indeed have the relevant header eval.h.

The exact command being executed by the auto-generated crosstool compiler toolchain is :

ERROR: /tmp/spack/tf/3d4affddb55a47900a919f49b073413c/external/nasm/BUILD.bazel:8:1: C++ compilation of rule '@nasm//:nasm' failed (Exit 1): crosstool_wrapper_driver_is_not_gcc failed: error executing command 
  (cd /tmp/spack/tf/3d4affddb55a47900a919f49b073413c/execroot/org_tensorflow && \
  exec env - \
    PATH=/home/sajid/packages/spack/opt/spack/linux-centos8-broadwell/gcc-7.3.0/python-3.7.4-ihubkpw2r66wyuqpyelixazj4v76jd7i/bin:/home/sajid/packages/spack/lib/spack/env/gcc:/home/sajid/packages/spack/lib/spack/env/case-insensitive:/home/sajid/packages/spack/lib/spack/env:/home/sajid/packages/spack/opt/spack/linux-centos8-broadwell/gcc-7.3.0/python-3.7.4-ihubkpw2r66wyuqpyelixazj4v76jd7i/bin:/home/sajid/packages/spack/opt/spack/linux-centos8-broadwell/gcc-7.3.0/python-3.7.4-ihubkpw2r66wyuqpyelixazj4v76jd7i/bin:/home/sajid/packages/spack/opt/spack/linux-centos8-broadwell/gcc-7.3.0/python-3.7.4-ihubkpw2r66wyuqpyelixazj4v76jd7i/bin:/home/sajid/packages/spack/opt/spack/linux-centos8-broadwell/gcc-7.3.0/cuda-10.1.243-wciaq2rrrkc4w6vcraocheah55wrekha/bin:/home/sajid/packages/spack/opt/spack/linux-centos8-broadwell/gcc-7.3.0/python-3.7.4-ihubkpw2r66wyuqpyelixazj4v76jd7i/bin:/home/sajid/packages/spack/opt/spack/linux-centos8-broadwell/gcc-7.3.0/python-3.7.4-ihubkpw2r66wyuqpyelixazj4v76jd7i/bin:/home/sajid/packages/spack/opt/spack/linux-centos8-broadwell/gcc-7.3.0/python-3.7.4-ihubkpw2r66wyuqpyelixazj4v76jd7i/bin:/home/sajid/packages/spack/opt/spack/linux-centos8-broadwell/gcc-7.3.0/python-3.7.4-ihubkpw2r66wyuqpyelixazj4v76jd7i/bin:/home/sajid/packages/spack/opt/spack/linux-centos8-broadwell/gcc-7.3.0/py-wheel-0.33.1-op2hwc4hqsgmc4qhevj3tq3tvw7civj2/bin:/home/sajid/packages/spack/opt/spack/linux-centos8-broadwell/gcc-7.3.0/python-3.7.4-ihubkpw2r66wyuqpyelixazj4v76jd7i/bin:/home/sajid/packages/spack/opt/spack/linux-centos8-broadwell/gcc-7.3.0/py-numpy-1.17.4-uxmerwuy33filyrx4sw5rkfawty2vlup/bin:/home/sajid/packages/spack/opt/spack/linux-centos8-broadwell/gcc-7.3.0/curl-7.63.0-jrvggtmsfzn64chuvcenb7a4g7lznfvj/bin:/home/sajid/packages/spack/opt/spack/linux-centos8-broadwell/gcc-7.3.0/python-3.7.4-ihubkpw2r66wyuqpyelixazj4v76jd7i/bin:/home/sajid/packages/spack/opt/spack/linux-centos8-broadwell/gcc-7.3.0/swig-4.0.0-mvastxbkyr3gnidrgierbrulpj6eys5u/bin:/home/sajid/packages/spack/opt/spack/linux-centos8-broadwell/gcc-7.3.0/python-3.7.4-ihubkpw2r66wyuqpyelixazj4v76jd7i/bin:/home/sajid/packages/spack/opt/spack/linux-centos8-broadwell/gcc-7.3.0/python-3.7.4-ihubkpw2r66wyuqpyelixazj4v76jd7i/bin:/home/sajid/packages/spack/opt/spack/linux-centos8-broadwell/gcc-7.3.0/py-numpy-1.17.4-uxmerwuy33filyrx4sw5rkfawty2vlup/bin:/home/sajid/packages/spack/opt/spack/linux-centos8-broadwell/gcc-7.3.0/python-3.7.4-ihubkpw2r66wyuqpyelixazj4v76jd7i/bin:/home/sajid/packages/spack/opt/spack/linux-centos8-broadwell/gcc-7.3.0/python-3.7.4-ihubkpw2r66wyuqpyelixazj4v76jd7i/bin:/home/sajid/packages/spack/opt/spack/linux-centos8-broadwell/gcc-7.3.0/jdk-1.8.0_231-b11-ynnz6lmhhw5mngtkw74hwcnn5ungicak/bin:/home/sajid/packages/spack/opt/spack/linux-centos8-broadwell/gcc-7.3.0/bazel-0.26.1-fgoxd6mcj3iyn7z43j7fq7dyjb5okpm7/bin:/home/sajid/packages/spack/opt/spack/linux-centos8-broadwell/gcc-7.3.0/python-3.7.4-ihubkpw2r66wyuqpyelixazj4v76jd7i/bin:/home/sajid/packages/spack/opt/spack/linux-centos8-broadwell/gcc-7.3.0/python-3.7.4-ihubkpw2r66wyuqpyelixazj4v76jd7i/bin:/home/sajid/packages/spack/opt/spack/linux-centos8-broadwell/gcc-7.3.0/py-setuptools-41.4.0-ctklysjoeronag3sfh44ptqfrcs6cuov/bin:/home/sajid/packages/spack/opt/spack/linux-centos8-broadwell/gcc-7.3.0/python-3.7.4-ihubkpw2r66wyuqpyelixazj4v76jd7i/bin:/home/sajid/packages/spack/opt/spack/linux-centos8-broadwell/gcc-7.3.0/py-setuptools-41.4.0-ctklysjoeronag3sfh44ptqfrcs6cuov/bin:/home/sajid/packages/spack/opt/spack/linux-centos8-broadwell/gcc-7.3.0/python-3.7.4-ihubkpw2r66wyuqpyelixazj4v76jd7i/bin:/home/sajid/packages/spack/opt/spack/linux-centos8-broadwell/gcc-7.3.0/cuda-10.1.243-wciaq2rrrkc4w6vcraocheah55wrekha/bin:/home/sajid/packages/spack/opt/spack/linux-centos8-broadwell/gcc-7.3.0/py-wheel-0.33.1-op2hwc4hqsgmc4qhevj3tq3tvw7civj2/bin:/home/sajid/packages/spack/opt/spack/linux-centos8-broadwell/gcc-7.3.0/py-numpy-1.17.4-uxmerwuy33filyrx4sw5rkfawty2vlup/bin:/home/sajid/packages/spack/opt/spack/linux-centos8-broadwell/gcc-7.3.0/curl-7.63.0-jrvggtmsfzn64chuvcenb7a4g7lznfvj/bin:/home/sajid/packages/spack/opt/spack/linux-centos8-broadwell/gcc-7.3.0/swig-4.0.0-mvastxbkyr3gnidrgierbrulpj6eys5u/bin:/home/sajid/packages/spack/opt/spack/linux-centos8-broadwell/gcc-7.3.0/bazel-0.26.1-fgoxd6mcj3iyn7z43j7fq7dyjb5okpm7/bin:/home/sajid/packages/spack/opt/spack/linux-centos8-broadwell/gcc-7.3.0/py-setuptools-41.4.0-ctklysjoeronag3sfh44ptqfrcs6cuov/bin:/home/sajid/packages/spack/opt/spack/linux-centos8-broadwell/gcc-7.3.0/python-3.7.4-ihubkpw2r66wyuqpyelixazj4v76jd7i/bin:/home/sajid/packages/spack/opt/spack/linux-centos8-broadwell/gcc-8.2.1/gcc-9.2.0-yaowtafvg5f7v3f2en47gpta5b6ucvlq/bin:/home/sajid/packages/spack/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin \
    PWD=/proc/self/cwd \
    SPACK_CC=/home/sajid/packages/spack/opt/spack/linux-centos8-broadwell/gcc-8.2.1/gcc-7.3.0-uhzsamzf47moqf6gu2exxqsyg5gw2pbx/bin/gcc \
    SPACK_CC_RPATH_ARG=-Wl,-rpath, \
    SPACK_COMPILER_IMPLICIT_RPATHS=/home/sajid/packages/spack/opt/spack/linux-centos8-broadwell/gcc-8.2.1/gcc-7.3.0-uhzsamzf47moqf6gu2exxqsyg5gw2pbx/lib/gcc/x86_64-pc-linux-gnu/7.3.0:/home/sajid/packages/spack/opt/spack/linux-centos8-broadwell/gcc-8.2.1/gcc-7.3.0-uhzsamzf47moqf6gu2exxqsyg5gw2pbx/lib64 \
    [email protected] \
    SPACK_CXX=/home/sajid/packages/spack/opt/spack/linux-centos8-broadwell/gcc-8.2.1/gcc-7.3.0-uhzsamzf47moqf6gu2exxqsyg5gw2pbx/bin/g++ \
    SPACK_CXX_RPATH_ARG=-Wl,-rpath, \
    SPACK_DEBUG_LOG_DIR=/home/sajid \
    SPACK_DEBUG_LOG_ID=py-tensorflow-a73jdej \
    SPACK_DTAGS_TO_ADD=--disable-new-dtags \
    SPACK_DTAGS_TO_STRIP=--enable-new-dtags \
    SPACK_ENV_PATH=/home/sajid/packages/spack/lib/spack/env:/home/sajid/packages/spack/lib/spack/env/case-insensitive:/home/sajid/packages/spack/lib/spack/env/gcc \
    SPACK_F77=/home/sajid/packages/spack/opt/spack/linux-centos8-broadwell/gcc-8.2.1/gcc-7.3.0-uhzsamzf47moqf6gu2exxqsyg5gw2pbx/bin/gfortran \
    SPACK_F77_RPATH_ARG=-Wl,-rpath, \
    SPACK_FC=/home/sajid/packages/spack/opt/spack/linux-centos8-broadwell/gcc-8.2.1/gcc-7.3.0-uhzsamzf47moqf6gu2exxqsyg5gw2pbx/bin/gfortran \
    SPACK_FC_RPATH_ARG=-Wl,-rpath, \
    SPACK_INCLUDE_DIRS=/home/sajid/packages/spack/opt/spack/linux-centos8-broadwell/gcc-7.3.0/openssl-1.1.1d-4bo5gg2eookuk2ubczsa4ciaz5c4mbwz/include:/home/sajid/packages/spack/opt/spack/linux-centos8-broadwell/gcc-7.3.0/bzip2-1.0.8-5zisvxvepuimx27rqxi53srnkqj6jqlk/include:/home/sajid/packages/spack/opt/spack/linux-centos8-broadwell/gcc-7.3.0/cudnn-7.6.5.32-10.1-linux-x64-sabespboxtmnfvkohzmmackhv7tjvbqb/include:/home/sajid/packages/spack/opt/spack/linux-centos8-broadwell/gcc-7.3.0/cuda-10.1.243-wciaq2rrrkc4w6vcraocheah55wrekha/include:/home/sajid/packages/spack/opt/spack/linux-centos8-broadwell/gcc-7.3.0/gdbm-1.18.1-ek2j6ymdsfdhwtcsgm5typyeqr2d427s/include:/home/sajid/packages/spack/opt/spack/linux-centos8-broadwell/gcc-7.3.0/readline-8.0-sdz3prqngfpcyiooqriil7iyy3mp77nr/include:/home/sajid/packages/spack/opt/spack/linux-centos8-broadwell/gcc-7.3.0/libxml2-2.9.9-s5wjt2encrutai46rptb2qdbuqm74rcj/include/libxml2:/home/sajid/packages/spack/opt/spack/linux-centos8-broadwell/gcc-7.3.0/libffi-3.2.1-mhdz37flhbznp5bla7dxtvtuwqxlzdpr/lib/libffi-3.2.1/include:/home/sajid/packages/spack/opt/spack/linux-centos8-broadwell/gcc-7.3.0/gettext-0.20.1-3x3th3xdaosnjd7ttly2mym5hukdqln5/include:/home/sajid/packages/spack/opt/spack/linux-centos8-broadwell/gcc-7.3.0/python-3.7.4-ihubkpw2r66wyuqpyelixazj4v76jd7i/include/python3.7m:/home/sajid/packages/spack/opt/spack/linux-centos8-broadwell/gcc-7.3.0/ncurses-6.1-bnt5ehudftgrg5smastj4ejhjvyxfjmj/include:/home/sajid/packages/spack/opt/spack/linux-centos8-broadwell/gcc-7.3.0/xz-5.2.4-oc7pfgyjo477pshkef56rkv4ovz7ykei/include:/home/sajid/packages/spack/opt/spack/linux-centos8-broadwell/gcc-7.3.0/nccl-2.4.8-1-phuasbwfolmvjlo5mmizcuz4vqeuayhk/include:/home/sajid/packages/spack/opt/spack/linux-centos8-broadwell/gcc-7.3.0/expat-2.2.9-v4k636wrhszysgefq2sijry72mj5l2au/include:/home/sajid/packages/spack/opt/spack/linux-centos8-broadwell/gcc-7.3.0/libbsd-0.10.0-jhowpdxutwgnbbpqv5dl3ifkfmrmuck5/include:/home/sajid/packages/spack/opt/spack/linux-centos8-broadwell/gcc-7.3.0/curl-7.63.0-jrvggtmsfzn64chuvcenb7a4g7lznfvj/include:/home/sajid/packages/spack/opt/spack/linux-centos8-broadwell/gcc-7.3.0/sqlite-3.30.1-kifllx2du5fe2yqreol2ldfk7zzqrhm2/include:/home/sajid/packages/spack/opt/spack/linux-centos8-broadwell/gcc-7.3.0/libiconv-1.16-qeeepkzh7y5dnj75fidj2abx23x6nwbl/include:/home/sajid/packages/spack/opt/spack/linux-centos8-broadwell/gcc-7.3.0/zlib-1.2.11-ah6qvpjr2sam2rpn527ivwmz4vyfcqbf/include \
    SPACK_LINKER_ARG=-Wl, \
    SPACK_LINK_DIRS=/home/sajid/packages/spack/opt/spack/linux-centos8-broadwell/gcc-7.3.0/openssl-1.1.1d-4bo5gg2eookuk2ubczsa4ciaz5c4mbwz/lib:/home/sajid/packages/spack/opt/spack/linux-centos8-broadwell/gcc-7.3.0/bzip2-1.0.8-5zisvxvepuimx27rqxi53srnkqj6jqlk/lib:/home/sajid/packages/spack/opt/spack/linux-centos8-broadwell/gcc-7.3.0/cudnn-7.6.5.32-10.1-linux-x64-sabespboxtmnfvkohzmmackhv7tjvbqb/lib64:/home/sajid/packages/spack/opt/spack/linux-centos8-broadwell/gcc-7.3.0/cuda-10.1.243-wciaq2rrrkc4w6vcraocheah55wrekha/targets/x86_64-linux/lib/stubs:/home/sajid/packages/spack/opt/spack/linux-centos8-broadwell/gcc-7.3.0/cuda-10.1.243-wciaq2rrrkc4w6vcraocheah55wrekha/lib64:/home/sajid/packages/spack/opt/spack/linux-centos8-broadwell/gcc-7.3.0/gdbm-1.18.1-ek2j6ymdsfdhwtcsgm5typyeqr2d427s/lib:/home/sajid/packages/spack/opt/spack/linux-centos8-broadwell/gcc-7.3.0/readline-8.0-sdz3prqngfpcyiooqriil7iyy3mp77nr/lib:/home/sajid/packages/spack/opt/spack/linux-centos8-broadwell/gcc-7.3.0/libxml2-2.9.9-s5wjt2encrutai46rptb2qdbuqm74rcj/lib:/home/sajid/packages/spack/opt/spack/linux-centos8-broadwell/gcc-7.3.0/libffi-3.2.1-mhdz37flhbznp5bla7dxtvtuwqxlzdpr/lib64:/home/sajid/packages/spack/opt/spack/linux-centos8-broadwell/gcc-7.3.0/libffi-3.2.1-mhdz37flhbznp5bla7dxtvtuwqxlzdpr/lib:/home/sajid/packages/spack/opt/spack/linux-centos8-broadwell/gcc-7.3.0/gettext-0.20.1-3x3th3xdaosnjd7ttly2mym5hukdqln5/lib:/home/sajid/packages/spack/opt/spack/linux-centos8-broadwell/gcc-7.3.0/python-3.7.4-ihubkpw2r66wyuqpyelixazj4v76jd7i/lib:/home/sajid/packages/spack/opt/spack/linux-centos8-broadwell/gcc-7.3.0/ncurses-6.1-bnt5ehudftgrg5smastj4ejhjvyxfjmj/lib:/home/sajid/packages/spack/opt/spack/linux-centos8-broadwell/gcc-7.3.0/xz-5.2.4-oc7pfgyjo477pshkef56rkv4ovz7ykei/lib:/home/sajid/packages/spack/opt/spack/linux-centos8-broadwell/gcc-7.3.0/nccl-2.4.8-1-phuasbwfolmvjlo5mmizcuz4vqeuayhk/lib:/home/sajid/packages/spack/opt/spack/linux-centos8-broadwell/gcc-7.3.0/expat-2.2.9-v4k636wrhszysgefq2sijry72mj5l2au/lib:/home/sajid/packages/spack/opt/spack/linux-centos8-broadwell/gcc-7.3.0/libbsd-0.10.0-jhowpdxutwgnbbpqv5dl3ifkfmrmuck5/lib:/home/sajid/packages/spack/opt/spack/linux-centos8-broadwell/gcc-7.3.0/curl-7.63.0-jrvggtmsfzn64chuvcenb7a4g7lznfvj/lib:/home/sajid/packages/spack/opt/spack/linux-centos8-broadwell/gcc-7.3.0/sqlite-3.30.1-kifllx2du5fe2yqreol2ldfk7zzqrhm2/lib:/home/sajid/packages/spack/opt/spack/linux-centos8-broadwell/gcc-7.3.0/libiconv-1.16-qeeepkzh7y5dnj75fidj2abx23x6nwbl/lib:/home/sajid/packages/spack/opt/spack/linux-centos8-broadwell/gcc-7.3.0/zlib-1.2.11-ah6qvpjr2sam2rpn527ivwmz4vyfcqbf/lib \
    SPACK_ROOT=/home/sajid/packages/spack \
    SPACK_RPATH_DIRS=/home/sajid/packages/spack/opt/spack/linux-centos8-broadwell/gcc-7.3.0/py-tensorflow-2.0.0-a73jdejmdsiybrq3ocyrzxhq4zbv2cjz/lib:/home/sajid/packages/spack/opt/spack/linux-centos8-broadwell/gcc-7.3.0/py-tensorflow-2.0.0-a73jdejmdsiybrq3ocyrzxhq4zbv2cjz/lib64:/home/sajid/packages/spack/opt/spack/linux-centos8-broadwell/gcc-7.3.0/openssl-1.1.1d-4bo5gg2eookuk2ubczsa4ciaz5c4mbwz/lib:/home/sajid/packages/spack/opt/spack/linux-centos8-broadwell/gcc-7.3.0/bzip2-1.0.8-5zisvxvepuimx27rqxi53srnkqj6jqlk/lib:/home/sajid/packages/spack/opt/spack/linux-centos8-broadwell/gcc-7.3.0/cudnn-7.6.5.32-10.1-linux-x64-sabespboxtmnfvkohzmmackhv7tjvbqb/lib64:/home/sajid/packages/spack/opt/spack/linux-centos8-broadwell/gcc-7.3.0/cuda-10.1.243-wciaq2rrrkc4w6vcraocheah55wrekha/targets/x86_64-linux/lib/stubs:/home/sajid/packages/spack/opt/spack/linux-centos8-broadwell/gcc-7.3.0/cuda-10.1.243-wciaq2rrrkc4w6vcraocheah55wrekha/lib64:/home/sajid/packages/spack/opt/spack/linux-centos8-broadwell/gcc-7.3.0/gdbm-1.18.1-ek2j6ymdsfdhwtcsgm5typyeqr2d427s/lib:/home/sajid/packages/spack/opt/spack/linux-centos8-broadwell/gcc-7.3.0/readline-8.0-sdz3prqngfpcyiooqriil7iyy3mp77nr/lib:/home/sajid/packages/spack/opt/spack/linux-centos8-broadwell/gcc-7.3.0/libxml2-2.9.9-s5wjt2encrutai46rptb2qdbuqm74rcj/lib:/home/sajid/packages/spack/opt/spack/linux-centos8-broadwell/gcc-7.3.0/libffi-3.2.1-mhdz37flhbznp5bla7dxtvtuwqxlzdpr/lib64:/home/sajid/packages/spack/opt/spack/linux-centos8-broadwell/gcc-7.3.0/libffi-3.2.1-mhdz37flhbznp5bla7dxtvtuwqxlzdpr/lib:/home/sajid/packages/spack/opt/spack/linux-centos8-broadwell/gcc-7.3.0/gettext-0.20.1-3x3th3xdaosnjd7ttly2mym5hukdqln5/lib:/home/sajid/packages/spack/opt/spack/linux-centos8-broadwell/gcc-7.3.0/python-3.7.4-ihubkpw2r66wyuqpyelixazj4v76jd7i/lib:/home/sajid/packages/spack/opt/spack/linux-centos8-broadwell/gcc-7.3.0/ncurses-6.1-bnt5ehudftgrg5smastj4ejhjvyxfjmj/lib:/home/sajid/packages/spack/opt/spack/linux-centos8-broadwell/gcc-7.3.0/xz-5.2.4-oc7pfgyjo477pshkef56rkv4ovz7ykei/lib:/home/sajid/packages/spack/opt/spack/linux-centos8-broadwell/gcc-7.3.0/nccl-2.4.8-1-phuasbwfolmvjlo5mmizcuz4vqeuayhk/lib:/home/sajid/packages/spack/opt/spack/linux-centos8-broadwell/gcc-7.3.0/expat-2.2.9-v4k636wrhszysgefq2sijry72mj5l2au/lib:/home/sajid/packages/spack/opt/spack/linux-centos8-broadwell/gcc-7.3.0/libbsd-0.10.0-jhowpdxutwgnbbpqv5dl3ifkfmrmuck5/lib:/home/sajid/packages/spack/opt/spack/linux-centos8-broadwell/gcc-7.3.0/curl-7.63.0-jrvggtmsfzn64chuvcenb7a4g7lznfvj/lib:/home/sajid/packages/spack/opt/spack/linux-centos8-broadwell/gcc-7.3.0/sqlite-3.30.1-kifllx2du5fe2yqreol2ldfk7zzqrhm2/lib:/home/sajid/packages/spack/opt/spack/linux-centos8-broadwell/gcc-7.3.0/libiconv-1.16-qeeepkzh7y5dnj75fidj2abx23x6nwbl/lib:/home/sajid/packages/spack/opt/spack/linux-centos8-broadwell/gcc-7.3.0/zlib-1.2.11-ah6qvpjr2sam2rpn527ivwmz4vyfcqbf/lib \
    SPACK_SHORT_SPEC='[email protected]%[email protected]~android~aws~computecpp+cuda cuda_arch=61 ~dynamic_kernels+gcp~gdr~hdfs~ignite~ios~jemalloc~kafka~mkl~monolithic~mpi+nccl~ngraph~numa~opencl patches=c49766a976e5c25c3827036828df0a2630e511bd783ac9cdcc6fc13068b22fac ~rocm~tensorrt~verbs~xla arch=linux-centos8-broadwell/a73jdej' \
    SPACK_SYSTEM_DIRS=/bin:/usr/bin:/usr/local/bin:/bin64:/usr/bin64:/usr/local/bin64:/include:/usr/include:/usr/local/include:/lib:/usr/lib:/usr/local/lib:/lib64:/usr/lib64:/usr/local/lib64:/:/usr:/usr/local \
    SPACK_TARGET_ARGS='-march=broadwell -mtune=broadwell' \
  external/local_config_cuda/crosstool/clang/bin/crosstool_wrapper_driver_is_not_gcc -MD -MF bazel-out/host/bin/external/nasm/_objs/nasm/outelf.d '-frandom-seed=bazel-out/host/bin/external/nasm/_objs/nasm/outelf.o' -DHAVE_SNPRINTF -DHAVE_SYS_TYPES_H -iquote external/nasm -iquote bazel-out/host/bin/external/nasm -iquote external/bazel_tools -iquote bazel-out/host/bin/external/bazel_tools -isystem external/nasm/asm -isystem bazel-out/host/bin/external/nasm/asm -isystem external/nasm/include -isystem bazel-out/host/bin/external/nasm/include -isystem external/nasm/output -isystem bazel-out/host/bin/external/nasm/output -isystem external/nasm/x86 -isystem bazel-out/host/bin/external/nasm/x86 -Wno-builtin-macro-redefined '-D__DATE__="redacted"' '-D__TIMESTAMP__="redacted"' '-D__TIME__="redacted"' -fPIE -U_FORTIFY_SOURCE '-D_FORTIFY_SOURCE=1' -fstack-protector -Wall -fno-omit-frame-pointer -no-canonical-prefixes -fno-canonical-system-headers -DNDEBUG -g0 -O2 -ffunction-sections -fdata-sections -g0 '-march=native' -w '-std=c99' -c external/nasm/output/outelf.c -o bazel-out/host/bin/external/nasm/_objs/nasm/outelf.o)
Execution platform: @bazel_tools//platforms:host_platform
In file included from external/nasm/output/outelf.c:49:0:
/home/sajid/packages/spack/opt/spack/linux-centos8-broadwell/gcc-7.3.0/python-3.7.4-ihubkpw2r66wyuqpyelixazj4v76jd7i/include/python3.7m/eval.h:10:12: error: unknown type name 'PyObject'
 PyAPI_FUNC(PyObject *) PyEval_EvalCode(PyObject *, PyObject *, PyObject *);
            ^~~~~~~~
/home/sajid/packages/spack/opt/spack/linux-centos8-broadwell/gcc-7.3.0/python-3.7.4-ihubkpw2r66wyuqpyelixazj4v76jd7i/include/python3.7m/eval.h:12:12: error: unknown type name 'PyObject'
 PyAPI_FUNC(PyObject *) PyEval_EvalCodeEx(PyObject *co,
            ^~~~~~~~
/home/sajid/packages/spack/opt/spack/linux-centos8-broadwell/gcc-7.3.0/python-3.7.4-ihubkpw2r66wyuqpyelixazj4v76jd7i/include/python3.7m/eval.h:21:12: error: unknown type name 'PyObject'
 PyAPI_FUNC(PyObject *) _PyEval_EvalCodeWithName(
            ^~~~~~~~
/home/sajid/packages/spack/opt/spack/linux-centos8-broadwell/gcc-7.3.0/python-3.7.4-ihubkpw2r66wyuqpyelixazj4v76jd7i/include/python3.7m/eval.h:31:12: error: unknown type name 'PyObject'
 PyAPI_FUNC(PyObject *) _PyEval_CallTracing(PyObject *func, PyObject *args);
            ^~~~~~~~
SUBCOMMAND: # @nasm//:nasm [action 'Compiling external/nasm/output/outdbg.c [for host]']

Where it's clear that despite being given the relevant include locations in the BUILD.bazel recipe and the exact command executed having the correct include location (as indicated by the presence of -isystem external/nasm/asm). bazel tries to include a completely irrelevant header (that of eval.h from python standard library instead).

What operating system are you running Bazel on?

centos-8.0, but I'm able to reproduce the error on rhel-7.4

What's the output of bazel info release?

Unable to run info outside WORKSPACE but it's compiled from source and has the following dependency tree :

[sajid@xrmlite ~]$ spack find -ldv bazel
==> 1 installed package
-- linux-centos8-broadwell / [email protected] --------------------------
fgoxd6m [email protected] patches=3e6448a0dde42bbd72568d29c5646d370dd62ca300cdd10a630908c086844167,75fad08d2a118372ed86696832942c7903bb716af28e5af969d8e20857655cf4,aa926467d3fc2bcd338ccc6355bc9f56adfd18dd3b4e1813a4ce8daee9e34600
ynnz6lm     [email protected]_231-b11
ihubkpw     [email protected]+bz2+ctypes+dbm+lzma~nis~optimizations+pic+pyexpat+pythoncmd+readline+shared+sqlite3+ssl~tix~tkinter~ucs4~uuid+zlib
5zisvxv         [email protected]+shared
v4k636w         [email protected]+libbsd
jhowpdx             [email protected]
ek2j6ym         [email protected]
sdz3prq             [email protected]
bnt5ehu                 [email protected]~symlinks~termlib
3x3th3x         [email protected]+bzip2+curses+git~libunistring+libxml2+tar+xz
s5wjt2e             [email protected]~python
qeeepkz                 [email protected]
oc7pfgy                 [email protected]
ah6qvpj                 [email protected]+optimize+pic+shared
ijowy5t             [email protected]
mhdz37f         [email protected]
4bo5gg2         [email protected]+systemcerts
kifllx2         [email protected]~column_metadata+fts~functions~rtree

with some patches as described in the spack recipe to inject relevant compiler paths and libraries.

Have you found anything relevant by searching the web?

Possibly related : https://github.com/bazelbuild/bazel/issues/2143, https://github.com/bazelbuild/bazel/issues/9254

That flag_groups are expanded top to bottom and left to right which means that the order should be quote_include_paths, include_paths, system_include_paths, the usual default as per both the main compiler config and the auto-generated crosstool config and isystem is indeed passing the relevant include flag in this case. I also see that bazel has a search order for libraries to link to at https://github.com/bazelbuild/rules_cc/blob/0e66ef31d611b9fe5b762bc7b363e15134c53c85/cc/private/toolchain/unix_cc_toolchain_config.bzl#L1122 but since nasm has no dependency, this should have no relevance in this case.

Any other information, logs, or outputs that you want to share?

I'm sharing the following files :

• command.log
• java.log
• spack-build.out
• BUILD from the following locations :
• local_config_python
• local_config_cc
• local_config_cuda/cuda
• local_config_cuda/crosstool
• cc_toolchain_config.bzl and cc_wrapper.sh from local_config_cc
• cc_toolchain_config.bzl from local_config_cuda/crosstool

available at this link : https://northwestern.box.com/s/yxhh3uzj72nj3nqn81xcvkozwzqslrhs

[coreyjadams]

I am also seeing this problem when installing tensorflow with spack. I'm trying to install it on super computers at the Argonne Leadership Computing Facility, in preparation for our next system Aurora.

It will be very important to successfully install tensorflow with spack on our HPC systems - can you help us resolve this issue?

[pramodk]

Same issue here. Any suggestions for bazel developers please?

[tetromino]

As far as I can tell, you are using a cc wrapper script ("crosstool_wrapper_driver_is_not_gcc") which comes with spack's modified variant of tensorflow.

Bazel seems to be invoking this wrapper script with the correct options.

The C++ compiler invoked by the wrapper is doing something very strange (reading eval.h from /home/sajid/packages/spack/opt/spack/linux-centos8-broadwell/gcc-7.3.0/python-3.7.4-ihubkpw2r66wyuqpyelixazj4v76jd7i/include/python3.7m - a path which was not in any of the command-line arguments passed to the wrapper).

This suggests that the /home/sajid/...include/python3.7m path is being passed to the wrapper out-of-band (through env variables, global config files, or was hard-coded in the wrapper script itself). My first suspicion is that one of the SPACK_* env variables in the invocation is causing the wrapper script to add the wrong include paths to the underlying compiler command line; or maybe the wrapper script changes its current working directory to Spack's python, invokes the compiler from there, and adds "." to the list of include paths.

You may want to start by figuring out what crosstool_wrapper_driver_is_not_gcc is doing - for example, by using strace -f or by adding a bunch of print() statements in the wrapper (which is probably written in Python) to see exactly what compiler command line it's invoking and why. You would probably want to do this debugging by invoking crosstool_wrapper_driver_is_not_gcc (and checking the effect of the SPACK_* env variables etc.) from the shell, not from Bazel; and by compiling a simpler example - something like

#include <stdio.h>
#include "eval.h"
int main(int argc, char **argv) {
  printf("Hello, world!\n");
  return 0;
}

At this point, it doesn't seem like Bazel itself is doing anything wrong (but rules-cpp team members would know better). The one actionable item for the Bazel team might be to improve documentation for how to write a custom crosstool.

[tetromino]

The /home/sajid/...include/python3.7m is in your SPACK_INCLUDE_DIRS env variable.

You need to figure out what the effect of SPACK_INCLUDE_DIRS is on the underlying gcc command line invoked by spack's gcc wrapper (which in turn is invoked by tensorflow's crosstool_wrapper_driver_is_not_gcc).

[pramodk]

@adamjstewart : FYI, see above. As you know Spack internals better, considering above comment, if you have any suggestions for for me / @s-sajid-ali to try, let us know.

[adamjstewart]

@tetromino Yes, Spack is doing a few things differently. As you've surmised, Spack uses compiler wrappers (actually written in Bash) so that when you run gcc or whatever compiler you happen to use, it calls a wrapper script. This wrapper script automatically injects -I, -L, and -Wl,-rpath flags for dependencies. This compiler wrapper requires certain environment variables to be set, which requires additional patches to bazel. First, we use the following patch to pass these environment variables (because Bazel cleans the environment):

--- a/src/main/java/com/google/devtools/build/lib/bazel/rules/BazelRuleClassProvider.java
+++ b/src/main/java/com/google/devtools/build/lib/bazel/rules/BazelRuleClassProvider.java
@@ -181,6 +181,13 @@ public class BazelRuleClassProvider {
           env.put("PATH", null);
         }

+        Map<String, String> spackEnv = System.getenv();
+        for (String envName : spackEnv.keySet()) {
+          if (envName.startsWith("SPACK_")) {
+            env.put(envName, spackEnv.get(envName));
+          }
+        }
+
         // Shell environment variables specified via options take precedence over the
         // ones inherited from the fragments. In the long run, these fragments will
         // be replaced by appropriate default rc files anyway.

Then, we use the following patch to inject include directories based on the dependencies in the DAG:

--- a/tools/cpp/unix_cc_configure.bzl
+++ b/tools/cpp/unix_cc_configure.bzl
@@ -145,11 +145,18 @@ def get_escaped_cxx_inc_directories(repository_ctx, cc, lang_flag, additional_fl
     else:
         inc_dirs = result.stderr[index1 + 1:index2].strip()

-    return [
+    default_inc_directories = [
         _prepare_include_path(repository_ctx, _cxx_inc_convert(p))
         for p in inc_dirs.split("\n")
     ]

+    env = repository_ctx.os.environ
+    if "SPACK_INCLUDE_DIRS" in env:
+        for path in env["SPACK_INCLUDE_DIRS"].split(":"):
+            default_inc_directories.append(path)
+
+    return default_inc_directories
+
 def _is_compiler_option_supported(repository_ctx, cc, option):
     """Checks that `option` is supported by the C compiler. Doesn't %-escape the option."""
     result = repository_ctx.execute([

Finally, we force the build to run with the requested compilers:

--- a/compile.sh
+++ b/compile.sh
@@ -63,7 +63,7 @@
 log "Building output/bazel"
 # We set host and target platform directly because we are building for the local
 # host.
-bazel_build "src:bazel_nojdk${EXE_EXT}" \
+CC=$SPACK_CC CXX=$SPACK_CXX bazel_build "src:bazel_nojdk${EXE_EXT}" \
   --action_env=PATH \
   --host_platform=@local_config_platform//:host \
   --platforms=@local_config_platform//:host \

These patches were more or less developed ad hoc, so there may be other locations that require patches that we haven't found yet, or better places to do the same thing. It would be great if Bazel itself allowed certain environment variables to be passed to the build environment (for compiler wrappers for example) so that we don't have to patch it ourselves (the patches keep changing with version).

@pramodk the first idea I have is to try building things without Spack's compiler wrappers. This can be done by setting CC and friends to self.compiler.cc instead of spack_cc. This will likely break a lot of things because aside from CUDA/NCCL I don't see a way to specify the installation prefix for most dependencies. You could then try setting CPATH and other env vars that Bazel does respect to the full list of dependencies.

Another way of moving this issue forward would be to try to repeat the exact same build outside of Spack. If you can get it to break, then we know it's not a Spack/compiler wrapper issue. If you can't, then you know it's our fault. As for what the problem is, you can try to manually tweak env vars like SPACK_INCLUDE_DIRS to not include problematic directories.

Another idea that may help: currently, Spack recursively adds all dependencies in the DAG to the -I, -L, and -Wl,-rpath flags. For some packages, like dealii, this caused an issue where the RPATH on macOS was longer than the system could handle. To solve this, we added the transitive_rpaths = False functionality to that package. You could try setting this in py-tensorflow (not sure if it only influences RPATHs or also handles includes/links). At one point, we introduced a PR that actually did this by default for all packages, but it ended up breaking a lot of packages that didn't explicitly declare their dependencies. Now that we have better testing, we could think about re-introducing that PR if it helps solve the TF issue. I believe @scheibelp or @becker33 could point you to the specific PR or how to reproduce it these days just to test the TF build.

[tetromino]

@adamjstewart - I appreciate your effort with packaging Bazel for Spack, but I see a number of problems with what Spack is doing.

1. [Removing point about invalidating caches here - I must apologize, I believe I misunderstood the patch above.] Instead of fighting against Bazel, work with it by generating toolchains appropriate for the Spack environment.
2. Spack's injected include paths are too rich. There is no need for _every_ g++ invocation to scan through python headers - it should be limited to the g++ invocations building a source file which includes a python header. Otherwise, you are guaranteed to see header name collisions (like the one here).
3. Spack is making the name collision even more likely by injecting those include paths using -I instead of -isystem.

I would suggest a redesign of how you package Bazel.

[adamjstewart]

Maybe @tgamblin can comment on the above.

Instead of fighting against Bazel, work with it by generating toolchains appropriate for the Spack environment.

I would love to do this, however, I barely know how Bazel works and this seems like a lot of work compared to what we already do (and what kind of works). Also, TF is currently the only package in Spack that uses Bazel. If it were a more popular build system, I wouldn't mind investing more time into better Spack support, but right now we only care about getting a working TF installation. With that said, if anyone wants to take a stab at this, I'm more than happy to provide support and advice on how the Spack internals work.

[tetromino]

I think the quickest small fix is injecting include paths using -isystem instead of -I when Spack's g++ wrapper is run from Bazel.

[tgamblin]

@tetromino: thanks for taking such an in-depth look at this. I haven't had a chance to dive into this yet (thanks goes to @adamjstewart for getting the TF package working in the first place), but I really do want to get TF working in Spack.

RE: @adamjstewart:

I barely know how Bazel works and this seems like a lot of work compared to what we already do (and what kind of works)

I do think we should take the time to look into the toolchains -- especially if that's the direction the Bazel developers want to support. If you're busy I can try to do that in the next few weeks per @tetromino's suggestion. I think I won't be able to say much more until I read up on generating toolchains.

RE: @tetromino:

Spack's injected include paths are too rich. There is no need for every g++ invocation to scan through python headers - it should be limited to the g++ invocations building a source file which includes a python header. Otherwise, you are guaranteed to see header name collisions (like the one here).

I think "guaranteed" is a bit strong here, as this approach has worked quite well for keeping Spack builds for other build systems hermetic in some very diverse HPC environments. Consider that in Spack, we operate at package granularity -- we're a package manager, not a build system. We don't own most of the projects we build. We can't patch every CMake/Bazel/Makefile/whatever in each project, and no maintainer has time to analyze the dependencies at the source file level.

In normal Spack builds, the includes are ordered by precedence (pre-order) in the DAG, and any -I's passed on the command line in the root project take precedence over injected ones. This does a pretty good job of avoiding conflicts -- if there are similarly named headers, the one the project needs is almost always found first.

Anyway all that is kind of beside the point -- I think the immediate issue here is that we're injecting the headers the wrong way in Bazel. @tetromino's suggestion to try -isystem is a really good one and might solve the issue. Let's try that.

[tetromino]

In normal Spack builds, the includes are ordered by according to precedence (pre-order) in the DAG, and any -I's passed on the command line in the root project take precedence over injected ones.

Bazel tries to be fancy and uses -iquote and -isystem for various paths depending on circumstances; I assume that this doesn't play well with injected -I :)