Models: [SyntaxNet] How to install Tensorflow correctly for SyntaxNet

[LangLEvoI]

System information

• What is the top-level directory of the model you are using: models/research/syntaxnet
• Have I written custom code (as opposed to using a stock example script provided in TensorFlow): models/research/syntaxnet/examples/dragnn/tutorial_1.py
• OS Platform and Distribution (e.g., Linux Ubuntu 16.04): Mint 18.3
• TensorFlow installed from (source or binary): Installing by conda install tensorflow
• TensorFlow version (use command below): ('v1.4.0-rc1-11-g130a514', '1.4.0')
• Bazel version (if compiling from source): 0.54, from source
• CUDA/cuDNN version: N/a
• GPU model and memory: N/a
• Exact command to reproduce: python models/research/syntaxnet/examples/dragnn/tutorial_1.py

Describe the problem

Failing to run example script which is tutorial_1.py, because there is not tensor in eager:

(anaconda2-5.0.1) admin@admin-virtual-machine ~/models/research/syntaxnet/examples/dragnn $ python tutorial_1.py
Traceback (most recent call last):
  File "tutorial_1.py", line 11, in <module>
    from dragnn.python import graph_builder
  File "/home/admin/.pyenv/versions/anaconda2-5.0.1/lib/python2.7/site-packages/dragnn/python/graph_builder.py", line 24, in <module>
    from dragnn.python import component
  File "/home/admin/.pyenv/versions/anaconda2-5.0.1/lib/python2.7/site-packages/dragnn/python/component.py", line 24, in <module>
    from dragnn.python import dragnn_ops
  File "/home/admin/.pyenv/versions/anaconda2-5.0.1/lib/python2.7/site-packages/dragnn/python/dragnn_ops.py", line 19, in <module>
    from dragnn.core.ops.gen_dragnn_bulk_ops import *
  File "/home/admin/.pyenv/versions/anaconda2-5.0.1/lib/python2.7/site-packages/dragnn/core/ops/gen_dragnn_bulk_ops.py", line 11, in <module>
    from tensorflow.python.eager import tensor as _tensor
ImportError: cannot import name tensor

[asimshankar]

This seems like a version mismatch between the dragnn module and the TensorFlow package.

Did you follow the process in https://github.com/tensorflow/models/tree/master/research/syntaxnet#installation ? In particular, note that the syntaxnet repository is pinned to a specific commit of the TensorFlow repository and the installation process builds TensorFlow from source at that commit. So it may not be compatible with a release version of TensorFlow installed via conda install tensorflow.

Hope that helps, or do let us know if you did indeed follow the installation process outlined above. If so, perhaps list out the exact sequence of steps. Thanks.

[robinqhuang]

@asimshankar , I built from source by following "Manual installation" section. I had the same issue.

[LangLEvoI]

@asimshankar Thanks for your reply, it's actually a version conflict. So I removed all relative libraries and reinstalled them according to Manual installation. But there is still a new error in tutorial_1.py.

Traceback (most recent call last):
  File "/home/admin/models/research/syntaxnet/examples/dragnn/tutorial_1.py", line 11, in <module>
    from dragnn.python import graph_builder
  File "/home/admin/.pyenv/versions/anaconda2-5.0.1/lib/python2.7/site-packages/dragnn/python/graph_builder.py", line 24, in <module>
    from dragnn.python import component
  File "/home/admin/.pyenv/versions/anaconda2-5.0.1/lib/python2.7/site-packages/dragnn/python/component.py", line 24, in <module>
    from dragnn.python import dragnn_ops
  File "/home/admin/.pyenv/versions/anaconda2-5.0.1/lib/python2.7/site-packages/dragnn/python/dragnn_ops.py", line 19, in <module>
    from dragnn.core.ops.gen_dragnn_bulk_ops import *
  File "/home/admin/.pyenv/versions/anaconda2-5.0.1/lib/python2.7/site-packages/dragnn/core/ops/gen_dragnn_bulk_ops.py", line 11, in <module>
    from tensorflow.python.eager import tensor as _tensor
  File "/home/admin/.pyenv/versions/anaconda2-5.0.1/lib/python2.7/site-packages/tensorflow/python/eager/tensor.py", line 26, in <module>
    from tensorflow.python.framework.ops import _tensor_from_handle
ImportError: cannot import name _tensor_from_handle

Though it lacks the method _tensor_from_handle in installed Tensorflow lib, I can find it in the specific commit repo (c52cdc0) which I have cloned. It is def _tensor_from_handle(handle): in ln 902 of models/research/syntaxnet/tensorflow/tensorflow/python/framework/ops.py

So maybe I need to install TF correctly. Some main installations are following,

Before building and testing Syntaxnet, I have installed some libs by homepage. Then I configured tensorflow,

(anaconda2-5.0.1) admin@admin-virtual-machine ~/models/research/syntaxnet/tensorflow $ ./configure
You have bazel 0.5.4 installed.
Please specify the location of python. [Default is /home/admin/.pyenv/versions/anaconda2-5.0.1/bin/python]: 
Found possible Python library paths:
/home/admin/.pyenv/versions/anaconda2-5.0.1/lib/python2.7/site-packages
Please input the desired Python library path to use.  Default is /home/admin/.pyenv/versions/anaconda2-5.0.1/lib/python2.7/site-packages
Do you wish to build TensorFlow with jemalloc as malloc support? [Y/n]: 
jemalloc as malloc support will be enabled for TensorFlow.

Do you wish to build TensorFlow with Google Cloud Platform support? [y/N]: 
No Google Cloud Platform support will be enabled for TensorFlow.

Do you wish to build TensorFlow with Hadoop File System support? [y/N]: 
No Hadoop File System support will be enabled for TensorFlow.

Do you wish to build TensorFlow with XLA JIT support? [y/N]: 
No XLA JIT support will be enabled for TensorFlow.

Do you wish to build TensorFlow with GDR support? [y/N]: 
No GDR support will be enabled for TensorFlow.

Do you wish to build TensorFlow with VERBS support? [y/N]: 
No VERBS support will be enabled for TensorFlow.

Do you wish to build TensorFlow with OpenCL support? [y/N]: 
No OpenCL support will be enabled for TensorFlow.

Do you wish to build TensorFlow with CUDA support? [y/N]: 
No CUDA support will be enabled for TensorFlow.

Do you wish to build TensorFlow with MPI support? [y/N]: 
No MPI support will be enabled for TensorFlow.

Please specify optimization flags to use during compilation when bazel option "--config=opt" is specified [Default is -march=native]: 
Add "--config=mkl" to your bazel command to build with MKL support.
Please note that MKL on MacOS or windows is still not supported.
If you would like to use a local MKL instead of downloading, please set the environment variable "TF_MKL_ROOT" every time before build.
Configuration finished

And then, I tested them by Bazel, all passed.

(anaconda2-5.0.1) amdin@amdin-virtual-machine ~/synet/models/research/syntaxnet $ bazel test ...
.
WARNING: /home/admin/.cache/bazel/_bazel_admin/c94f55ca99e288eef8cf2dbdb1d0caae/external/org_tensorflow/tensorflow/contrib/learn/BUILD:15:1: in py_library rule @org_tensorflow//tensorflow/contrib/learn:learn: target '@org_tensorflow//tensorflow/contrib/learn:learn' depends on deprecated target '@org_tensorflow//tensorflow/contrib/session_bundle:exporter': No longer supported. Switch to SavedModel immediately.
WARNING: /home/admin/.cache/bazel/_bazel_admin/c94f55ca99e288eef8cf2dbdb1d0caae/external/org_tensorflow/tensorflow/contrib/learn/BUILD:15:1: in py_library rule @org_tensorflow//tensorflow/contrib/learn:learn: target '@org_tensorflow//tensorflow/contrib/learn:learn' depends on deprecated target '@org_tensorflow//tensorflow/contrib/session_bundle:gc': No longer supported. Switch to SavedModel immediately.
INFO: Analysed 235 targets (193 packages loaded).
INFO: Found 177 targets and 58 test targets...
INFO: Elapsed time: 219.790s, Critical Path: 5.42s
INFO: Build completed successfully, 1 total action
//dragnn/components/stateless:stateless_component_test          (cached) PASSED in 2.8s
//dragnn/components/syntaxnet:syntaxnet_component_test          (cached) PASSED in 0.7s
//dragnn/components/syntaxnet:syntaxnet_link_feature_extractor_test (cached) PASSED in 0.1s
//dragnn/components/syntaxnet:syntaxnet_transition_state_test   (cached) PASSED in 0.6s
//dragnn/core:beam_test                                         (cached) PASSED in 2.8s
//dragnn/core:compute_session_impl_test                         (cached) PASSED in 4.0s
//dragnn/core:compute_session_pool_test                         (cached) PASSED in 0.2s
//dragnn/core:dragnn_bulk_op_kernels_test                       (cached) PASSED in 0.2s
//dragnn/core:dragnn_op_kernels_test                            (cached) PASSED in 0.1s
//dragnn/core:index_translator_test                             (cached) PASSED in 0.1s
//dragnn/core:input_batch_cache_test                            (cached) PASSED in 1.4s
//dragnn/core:resource_container_test                           (cached) PASSED in 0.1s
//dragnn/io:sentence_input_batch_test                           (cached) PASSED in 0.6s
//dragnn/python:bulk_component_test                             (cached) PASSED in 6.9s
//dragnn/python:composite_optimizer_test                        (cached) PASSED in 9.2s
//dragnn/python:digraph_ops_test                                (cached) PASSED in 15.5s
//dragnn/python:dragnn_model_saver_lib_test                     (cached) PASSED in 27.4s
//dragnn/python:evaluation_test                                 (cached) PASSED in 7.6s
//dragnn/python:graph_builder_test                              (cached) PASSED in 59.7s
  Stats over 5 runs: max = 59.7s, min = 21.8s, avg = 31.4s, dev = 14.3s
//dragnn/python:lexicon_test                                    (cached) PASSED in 5.6s
//dragnn/python:network_units_test                              (cached) PASSED in 5.6s
//dragnn/python:render_parse_tree_graphviz_test                 (cached) PASSED in 7.2s
//dragnn/python:render_spec_with_graphviz_test                  (cached) PASSED in 16.4s
//dragnn/python:sentence_io_test                                (cached) PASSED in 5.2s
//dragnn/python:spec_builder_test                               (cached) PASSED in 5.4s
//dragnn/python:transformer_units_test                          (cached) PASSED in 5.5s
//dragnn/python:visualization_test                              (cached) PASSED in 15.5s
//dragnn/tools/benchmarks:beam_benchmark                        (cached) PASSED in 0.3s
//dragnn/tools:model_trainer_test                               (cached) PASSED in 19.8s
//examples/dragnn:test_run_all_tutorials                        (cached) PASSED in 12.1s
//syntaxnet:arc_standard_transitions_test                       (cached) PASSED in 0.4s
//syntaxnet:beam_reader_ops_test                                (cached) PASSED in 37.7s
//syntaxnet:binary_segment_state_test                           (cached) PASSED in 0.2s
//syntaxnet:binary_segment_transitions_test                     (cached) PASSED in 0.2s
//syntaxnet:char_ngram_string_extractor_test                    (cached) PASSED in 0.5s
//syntaxnet:char_properties_test                                (cached) PASSED in 0.1s
//syntaxnet:char_shift_transitions_test                         (cached) PASSED in 0.2s
//syntaxnet:generic_features_test                               (cached) PASSED in 0.3s
//syntaxnet:graph_builder_test                                  (cached) PASSED in 21.2s
//syntaxnet:head_label_transitions_test                         (cached) PASSED in 0.2s
//syntaxnet:head_transitions_test                               (cached) PASSED in 0.2s
//syntaxnet:label_transitions_test                              (cached) PASSED in 0.1s
//syntaxnet:lexicon_builder_test                                (cached) PASSED in 6.4s
//syntaxnet:morphology_label_set_test                           (cached) PASSED in 0.2s
//syntaxnet:once_transitions_test                               (cached) PASSED in 0.1s
//syntaxnet:parser_features_test                                (cached) PASSED in 0.1s
//syntaxnet:parser_trainer_test                                 (cached) PASSED in 84.1s
//syntaxnet:reader_ops_test                                     (cached) PASSED in 13.4s
//syntaxnet:segmenter_utils_test                                (cached) PASSED in 0.2s
//syntaxnet:sentence_features_test                              (cached) PASSED in 0.5s
//syntaxnet:shared_store_test                                   (cached) PASSED in 0.5s
//syntaxnet:tagger_transitions_test                             (cached) PASSED in 0.1s
//syntaxnet:text_formats_test                                   (cached) PASSED in 5.8s
//syntaxnet/util:check_test                                     (cached) PASSED in 4.4s
//syntaxnet/util:registry_test                                  (cached) PASSED in 4.2s
//syntaxnet/util:resources_test                                 (cached) PASSED in 3.1s
//syntaxnet:whole_sentence_features_test                        (cached) PASSED in 0.1s
//util/utf8:unicodetext_unittest                                (cached) PASSED in 0.3s

Lastly, I installed the whl which was generated by command.

(anaconda2-5.0.1) amdin@amdin-virtual-machine ~/tmp $ pip install syntaxnet-0.2-cp27-cp27mu-linux_x86_64.whl
Processing ./syntaxnet-0.2-cp27-cp27mu-linux_x86_64.whl
Collecting tensorflow (from syntaxnet==0.2)
  Downloading tensorflow-1.4.1-cp27-cp27mu-manylinux1_x86_64.whl (40.7MB)
    100% |████████████████████████████████| 40.8MB 21kB/s 
Requirement already satisfied: pygraphviz in /home/amdin/.pyenv/versions/anaconda2-5.0.1/lib/python2.7/site-packages (from syntaxnet==0.2)
Requirement already satisfied: enum34>=1.1.6 in /home/amdin/.pyenv/versions/anaconda2-5.0.1/lib/python2.7/site-packages (from tensorflow->syntaxnet==0.2)
Requirement already satisfied: backports.weakref>=1.0rc1 in /home/amdin/.pyenv/versions/anaconda2-5.0.1/lib/python2.7/site-packages (from tensorflow->syntaxnet==0.2)
Requirement already satisfied: wheel in /home/amdin/.pyenv/versions/anaconda2-5.0.1/lib/python2.7/site-packages (from tensorflow->syntaxnet==0.2)
Requirement already satisfied: mock>=2.0.0 in /home/amdin/.pyenv/versions/anaconda2-5.0.1/lib/python2.7/site-packages (from tensorflow->syntaxnet==0.2)
Requirement already satisfied: tensorflow-tensorboard<0.5.0,>=0.4.0rc1 in /home/amdin/.pyenv/versions/anaconda2-5.0.1/lib/python2.7/site-packages (from tensorflow->syntaxnet==0.2)
Requirement already satisfied: numpy>=1.12.1 in /home/amdin/.pyenv/versions/anaconda2-5.0.1/lib/python2.7/site-packages (from tensorflow->syntaxnet==0.2)
Requirement already satisfied: protobuf>=3.3.0 in /home/amdin/.pyenv/versions/anaconda2-5.0.1/lib/python2.7/site-packages (from tensorflow->syntaxnet==0.2)
Requirement already satisfied: six>=1.10.0 in /home/amdin/.pyenv/versions/anaconda2-5.0.1/lib/python2.7/site-packages (from tensorflow->syntaxnet==0.2)
Requirement already satisfied: pbr>=0.11 in /home/amdin/.pyenv/versions/anaconda2-5.0.1/lib/python2.7/site-packages (from mock>=2.0.0->tensorflow->syntaxnet==0.2)
Requirement already satisfied: funcsigs>=1 in /home/amdin/.pyenv/versions/anaconda2-5.0.1/lib/python2.7/site-packages (from mock>=2.0.0->tensorflow->syntaxnet==0.2)
Requirement already satisfied: bleach==1.5.0 in /home/amdin/.pyenv/versions/anaconda2-5.0.1/lib/python2.7/site-packages (from tensorflow-tensorboard<0.5.0,>=0.4.0rc1->tensorflow->syntaxnet==0.2)
Requirement already satisfied: futures>=3.1.1; python_version < "3.2" in /home/amdin/.pyenv/versions/anaconda2-5.0.1/lib/python2.7/site-packages (from tensorflow-tensorboard<0.5.0,>=0.4.0rc1->tensorflow->syntaxnet==0.2)
Requirement already satisfied: markdown>=2.6.8 in /home/amdin/.pyenv/versions/anaconda2-5.0.1/lib/python2.7/site-packages (from tensorflow-tensorboard<0.5.0,>=0.4.0rc1->tensorflow->syntaxnet==0.2)
Requirement already satisfied: werkzeug>=0.11.10 in /home/amdin/.pyenv/versions/anaconda2-5.0.1/lib/python2.7/site-packages (from tensorflow-tensorboard<0.5.0,>=0.4.0rc1->tensorflow->syntaxnet==0.2)
Requirement already satisfied: html5lib==0.9999999 in /home/amdin/.pyenv/versions/anaconda2-5.0.1/lib/python2.7/site-packages (from tensorflow-tensorboard<0.5.0,>=0.4.0rc1->tensorflow->syntaxnet==0.2)
Requirement already satisfied: setuptools in /home/amdin/.pyenv/versions/anaconda2-5.0.1/lib/python2.7/site-packages (from protobuf>=3.3.0->tensorflow->syntaxnet==0.2)
Installing collected packages: tensorflow, syntaxnet
Successfully installed syntaxnet-0.2 tensorflow-1.4.1

Thanks!

[rogerpasky]

After a few days of reverse-engineering, I'm afraid I must respectfully disagree with @ReeRay because his solution may work with tutorial_1.py but it's unlikely to make Syntaxnet work properly elsewhere. This is what I found:

• As @asimshankar said, in order to build and install Syntaxnet you must stick to the very Tensorflow version it comes with (1.3). It does not work properly even if you force a pip install tensorflow==1.3.
• Following the README.mdinstructions, doing /tmp/syntaxnet_pkg/syntaxnet-0.2-whatever-your-configuration-is.whl leads you to a pip Tensorflow dependency which will install the latest available version (1.5 in my case, 1.4.1 in the previous replay). It won't work.
• However, I found doing a previous bazel-bin/dragnn/tools/build_pip_package --include-tensorflow --output-dir=/tmp/syntaxnet_pkg package creation (notice the --include-tensorflow extra option) you'll generate a different package/tmp/syntaxnet_pkg/syntaxnet_with_tensorflow-0.2-whatever-your-configuration-is.whl, which is the one the container distribution uses to include the very Tensorflow version it comes along with Syntaxnet source code.
• Follow the next pip package deployment with the new name (sudo -H pip --no-cache-dir install /tmp/syntaxnet_pkg/syntaxnet_with_tensorflow-0.2-whatever-your-configuration-is.whl, having in mind I added -H option to sudo and --no-cache-dir to pip just to avoid annoyances, and they can be avoided). The other package is useless, please do install just this one.
• Installation is done. You are almost there. If you are willing to use the English and Spanish languages that are distributed in syntaxnet/examples/dragnn/data (for instance with the nearby *.ipynbnotebooks) you MUST convert the checkpointfiles to the appropriate protocol buffer version using models/research/syntaxnet/tensorflow/tensorflow/contrib/rnn/python/tools/checkpoint_convert.py. This particular undocumented need took me four days of source code digging. I hope you will enjoy those four extra days of life you have just gain.

[veranimus]

@rogerpasky Wow, thanks. This saved me a lot of time.