Models: Executing genrule //syntaxnet:parser_ops_pygenrule failed (Exit 255): bash failed: error executing command

[SaintNazaire]

OS Platform and Distribution

• Windows 10 professional
• Docker image based on Ubuntu:16.04.4 LTS (xenial)
• Docker 18.03.0-ce-win59
• Python versions 2.7.12 & 3.5.2
• CUDA/cuDNN version not relevant (CPU install)
• GPU model and memory not relevant (CPU install)

TensorFlow installed from

• Github /tensorflow/tensorflow branch:master

TensorFlow version

Tensorflow 1.8

Bazel version

Bazel 0.12.0

Exact command to reproduce

cd /models/research/syntaxnet && bazel build ... --subcommands --explain=/tmp/bazel_build_log.txt --verbose_explanations=True --verbose_failures

What is the top-level directory of the model you are using

/models/research/syntaxnet

Have I written custom code (as opposed to using a stock example script provided in TensorFlow)

Spent past three months migrating step by step Syntaxnet towards Bazel and Tensorflow current releases instead of Bazel 0.5.4 and Tensorflow 1.3.

• Bazel 0.12.0 (initially 0.9.0)
• Tensorflow r1.8 (intially r1.6)

Describe the problem

After clearing a lot of obstacles I reach the far end of the build (>7000) process with much warning but no error up to executing genrules related to "tf_gen_op_wrapper_py". The example of log I have provided below relates to //syntaxnet:parser_ops_pygenrule but I get the same error with //dragnn/core:dragnn_ops_pygenrule or //dragnn/core:dragnn_bulk_ops_pygenrule which also throw 255 exception errors.

In BUILD this "tf_gen_op_wrapper_py" looks like for example:

tf_gen_op_wrapper_py(
    name = "dragnn_bulk_ops",
    deps = [":dragnn_bulk_ops_op_lib"],
)

For //syntaxnet: it is located in /models/research/syntaxnet/syntaxnet/syntaxnet.bzl and for //dragnn/core: it is located in /models/research/syntaxnet/dragnn/tensorflow_ops.bzl

Code for tf_gen_op_wrapper_py as follows:

def tf_gen_op_wrapper_py(name, out=None, hidden=None, visibility=None, deps=[],
                         require_shape_functions=False, hidden_file=None,
                         generated_target_name=None):
  # Construct a cc_binary containing the specified ops.
  tool_name = "gen_" + name + "_py_wrappers_cc"
  if not deps:
    deps = ["@org_tensorflow//tensorflow/core:" + name + "_op_lib"]
  native.cc_binary(
      name = tool_name,
      linkopts = ["-lm"],
      copts = tf_copts(),
      linkstatic = 1,   # Faster to link this one-time-use binary dynamically
      deps = (["@org_tensorflow//tensorflow/core:framework",
               "@org_tensorflow//tensorflow/python:python_op_gen_main"] + deps),
      visibility = ["@org_tensorflow//tensorflow:internal"],
  )

  # Invoke the previous cc_binary to generate a python file.
  if not out:
    out = "ops/gen_" + name + ".py"

  if hidden:
    # `hidden` is a list of op names to be hidden in the generated module.
    native.genrule(
        name=name + "_pygenrule",
        outs=[out],
        tools=[tool_name],
        cmd=("$(location " + tool_name + ") " + ",".join(hidden)
             + " " + ("1" if require_shape_functions else "0") + " > $@"))
  elif hidden_file:
    # `hidden_file` is file containing a list of op names to be hidden in the
    # generated module.
    native.genrule(
        name=name + "_pygenrule",
        outs=[out],
        srcs=[hidden_file],
        tools=[tool_name],
        cmd=("$(location " + tool_name + ") @$(location "
             + hidden_file + ") " + ("1" if require_shape_functions else "0")
             + " > $@"))
  else:
    # No ops should be hidden in the generated module.
    native.genrule(
        name=name + "_pygenrule",
        outs=[out],
        tools=[tool_name],
        cmd=("$(location " + tool_name + ") "
             + ("1" if require_shape_functions else "0") + " > $@"))

  # Make a py_library out of the generated python file.
  if not generated_target_name:
    generated_target_name = name
  native.py_library(name=generated_target_name,
                    srcs=[out],
                    srcs_version="PY2AND3",
                    visibility=visibility,
                    deps=[
                        "@org_tensorflow//tensorflow/python:framework_for_generated_wrappers",
                    ],)

I tried to find out other issues opened mentioning tf_gen_op_wrapper_py and I found this reference to mostly static link issues that were raised due to CI failures in early January.

bazelbuild/bazel#4474

@calberti @andorardo @bogatyy @markomernick is that Syntaxnet territory? Have you ever encountered the problem? Do you have any info as to what triggers the issue?

@dslomov is tha related to the mostly static link issue in Bazel?

Error logs

SUBCOMMAND: # //syntaxnet:parser_ops_pygenrule [action 'Executing genrule //syntaxnet:parser_ops_pygenrule']
(cd /root/.cache/bazel/_bazel_root/3b4c7ccb85580bc382ce4a52e9580003/execroot/__main__ && \
exec env - \
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin \
PYTHON_BIN_PATH=/usr/bin/python \
PYTHON_LIB_PATH=/usr/lib/python2.7 \
TF_DOWNLOAD_CLANG=0 \
TF_NEED_CUDA=0 \
TF_NEED_OPENCL_SYCL=0 \
/bin/bash -c 'source external/bazel_tools/tools/genrule/genrule-setup.sh; bazel-out/host/bin/syntaxnet/gen_parser_ops_py_wrappers_cc 0 > bazel-out/k8-opt/genfiles/syntaxnet/ops/gen_parser_ops.py')
ERROR: /models/research/syntaxnet/syntaxnet/BUILD:777:1: Executing genrule //syntaxnet:parser_ops_pygenrule failed (Exit 255): bash failed: error executing command

[tensorflowbutler]

Thank you for your post. We noticed you have not filled out the following field in the issue template. Could you update them if they are relevant in your case, or leave them as N/A? Thanks.
What is the top-level directory of the model you are using
OS Platform and Distribution
TensorFlow installed from
TensorFlow version
Bazel version
Exact command to reproduce

[SaintNazaire]

Those fields were filled though not in standard issue so the robotic butler didn't get it

[markomernick]

We have an upcoming release that will update the code to be compatible with
the new 1.8 release of Tensorflow. Stay tuned!

On Thu, Apr 26, 2018 at 3:13 AM SaintNazaire [e-mail]
wrote:

Those fields were filled though not in standard issue so the robotic
buttler didn't get it

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/tensorflow/models/issues/4085#issuecomment-384537093,
or mute the thread
https://github.com/notifications/unsubscribe-auth/AZQ42H_yW5DS-iKV9En5S74iCNTDYJRLks5tsXORgaJpZM4TjqZR
.

--

Mark Omernick | Research & Machine Intelligence | [email protected] |
XXX-XXX-XXXX

[SaintNazaire]

Thank you @markomernick.

Grand, just a quick couple questions.

1. Could you explain what was wrong?
2. We've heard of an update already back in november last year, is the update you mention days or weeks ahead?

It would also be very nice if you could share a little bit on the status of the project, including refactoring and issues, that would help to learn and eventually contribute.

If you don't have time no worries you can dump me stuff, I'll try to organize it for you.

[SaintNazaire]

Ok I think I got it.

There are three tf_gen_op_wrapper_py definitions:

• DRAGNN's in tensorflow_ops.bzl which is located in /models/research/syntaxnet/dragnn/
• SYNTAXNETs' in syntaxnet.bzl which is located in /models/research/syntaxnet/syntaxnet/
• Tensorflow is tensorflow.bzl which is located in /models/research/tensorflow/tensorflow/

I then compared the various versions and it is very different, just to mention one change from framework_for_generated_wrappers to framework_for_generated_wrappers_v2.

SYNTAXNET's native.py_library call:

native.py_library(name=name,
                    srcs=[out],
                    srcs_version="PY2AND3",
                    visibility=visibility,
                    deps=[
                        "@org_tensorflow//tensorflow/python:framework_for_generated_wrappers",
                    ],)

DRAGNN's native.py_library call:

native.cc_binary(
      name = tool_name,
      linkopts = ["-lm"],
      copts = tf_copts(),
      linkstatic = 1,   # Faster to link this one-time-use binary dynamically
      deps = (["@org_tensorflow//tensorflow/core:framework",
               "@org_tensorflow//tensorflow/python:python_op_gen_main"] + deps),
      visibility = ["@org_tensorflow//tensorflow:internal"],
  )

Tensorflow's native.py_library call:

native.py_library(
      name=generated_target_name,
      srcs=[out],
      srcs_version="PY2AND3",
      visibility=visibility,
      deps=[
          clean_dep("//tensorflow/python:framework_for_generated_wrappers_v2"),
      ],)

Questions to the Syntaxnet people @calberti or @andorardo:

1. why did you declare different tf_gen_op_wrapper_py functions? Is it not possible to merge all three (eventually use the most complete and recent version, i.e. Tensorflow) hence reducing the cost of upkeep and reducing the risk of incompatible versions ?

2. Should it be better to try tf.load_op_library() [dynamic linking] instead of tf_op_gen_wrapper_py rule [static linking] ?

Many thanks!

[terryykoo]

Hi, sorry about the confusing situation!

In answer to Q1, my understanding is that in the distant past, TF's version of tf_gen_op_wrapper_py() and other build rules did not use the clean_dep() function. So TF's native.py_library() call looked something like this:

native.py_library(
....
deps = [
...
"//tensorflow/python:framework_for_generated_wrappers_v2",
]
)

The "//tensorflow/..." build target turned into a dangling reference when importing TF as a submodule, as SyntaxNet does. So we had to clone those build rules and s=//tensorflow=@org_tensorflow//tensorflow=.

I've been independently working on upgrading SyntaxNet to use r1.8 and have that working internally. I'll try to get it pushed out "soon". Among other things, I ended up deleting most of our internally-cloned build rules.

[SaintNazaire]

Ok @terryykoo it worked fine.

Solution was to simply get rid of all the duplicate functions in syntaxnet.bzl and tensorflow_ops.bzl and replace with loading from tensorflow 1.8

load(
    "@org_tensorflow//tensorflow:tensorflow.bzl",
    "tf_binary_additional_srcs",
    "clean_dep",
    "if_not_windows",
    "if_windows"
)