Addons: Entity could not be transformed

[kyleabeauchamp]

System information

• Have I written custom code (as opposed to using a stock example script provided in TensorFlow):
• OS Platform and Distribution (e.g., Linux Ubuntu 16.04): Linux Ubuntu 18.04
• TensorFlow installed from (source or binary): binary
• TensorFlow version (use command below): 2.0.0-dev20190317 (gpu nightly)
• TensorFlow Addons installed from (source, PyPi): source
• TensorFlow Addons version: master
• Python version and type (eg. Anaconda Python, Stock Python as in Mac, or homebrew installed Python etc): anaconda (conda-forge) python 3.7.1
• Bazel version (if compiling from source): 0.20.0
• GCC/Compiler version (if compiling from source): 7.3.0
• Is GPU used? (yes/no): yes
• GPU model (if used): RTX 2070

I'm seeing the following warning appear when trying to use the transform op. The simplest way to reproduce is to simply run the tests via python:

python ~/src/addons/tensorflow_addons/image/transform_test.py
[...]
W0317 18:03:00.753756 139906711668544 tf_logging.py:161] Entity <function image_projective_transform_v2 at 0x7f3e73a05a60> could not be transformed and will be staged without change. Error details can be found in the logs when running with the env variable AUTOGRAPH_VERBOSITY >= 1. Please report this to the AutoGraph team. Cause: Unexpected error transforming <function image_projective_transform_v2 at 0x7f3e73a05a60>. If you believe this is due to a bug, please set the verbosity to 10 (on Linux, `export AUTOGRAPH_VERBOSITY=10`) and attach the full output when filing the bug report. Caused by: Unable to locate the source code of <function image_projective_transform_v2 at 0x7f3e73a05a60>. Note that functions defined in certain environments, like the interactive Python shell do not expose their source code. If that is the case, you should to define them in a .py source file. If you are certain the code is graph-compatible, wrap the call using @tf.autograph.do_not_convert. Original error: could not get source code

export AUTOGRAPH_VERBOSITY=10
python ~/src/addons/tensorflow_addons/image/transform_test.py
[...]
I0317 18:11:38.742995 140338201401152 ag_logging.py:132] Error transforming <function image_projective_transform_v2 at 0x7fa2ea6b2a60>
Traceback (most recent call last):
  File "~/miniconda3/envs/tf2/lib/python3.7/site-packages/tensorflow/python/autograph/pyct/parser.py", line 51, in parse_entity
    source = tf_inspect.getsource_no_unwrap(entity)
  File "~/miniconda3/envs/tf2/lib/python3.7/site-packages/tensorflow/python/util/tf_inspect.py", line 408, in getsource_no_unwrap
    lines, lnum = _inspect.findsource(obj)
  File "~/miniconda3/envs/tf2/lib/python3.7/inspect.py", line 786, in findsource
    raise OSError('could not get source code')
OSError: could not get source code

Is this expected? I expect this is likely an artifact of a broken source build on my end, but I imagine others may experience similar issues until the CI/CD system is building nightly versions for all platforms and python versions.

[seanpmorgan]

Hi Kyle, without looking too closely I see you're running py37 and said you installed from binary. As seen in #75 we don't have a working py37 build yet.

Are you sure you're in the right environment?

[seanpmorgan]

Oh, it seems you're running the test from the source code. The binaries found in tensorflow_addons/custom_ops are not built until you compile. For that reason it's probably recommended you use a pip install (or conda in the future). Alternatively you could build and test using bazel

[kyleabeauchamp]

I pip-installed TF from the wheel and compiled tf-addons from source using the following:

conda activate tf2

python --version
Python 3.7.1

./configure.sh
bazel build build_pip_pkg
bazel-bin/build_pip_pkg artifacts
pip install artifacts/tensorflow_addons-*.whl

[kyleabeauchamp]

I should also note that I had to make some bazel config changes to get the build working on my system (namely, setting D_GLIBCXX_USE_CXX11_ABI=1 throughout).

[kyleabeauchamp]

Last note: I've seen the same warning in very basic non-test calculations, but I figured executing the test script manually is a pretty good way to reproduce the issue initially.

edit
Here's a minimal example that triggers the same warning:

import numpy as np
import tensorflow as tf
import tensorflow_addons as tfa
from tensorflow_addons.image.transform_ops import angles_to_projective_transforms

minval = 10 * np.pi / 180.
maxval = -1.0 * minval

n_images = 10
x = tf.eye(28, 28, [n_images])
random_angles = tf.random.uniform(shape=[1], minval=minval, maxval=maxval)
transforms = angles_to_projective_transforms(random_angles, 28, 28)
transforms = transforms[0:1]
rotated_images = tfa.image.transform(x, transforms)

[facaiy]

@kyleabeauchamp Thanks for reporting this problem, Kyle! Would you like to fix it? I'm kind of busy this week, so I might have time to look into the issue next week.

[kyleabeauchamp]

I'll certainly open a PR if I end up figuring out the solution, but it's not obvious that I'll figure it out before you, as I only recently started ramping up on 2.0.

[facaiy]

@alextp Hi, Alexandre, do you know what the warning means? thanks :-)

[facaiy]

@seanpmorgan @jharmsen Hi, is it related to #90?

[jharmsen]

I tried a git revert f36d04f5f65573f1c88c5b7110cf939b1f654e99 and the issue was still present.
I can poke around a little more.

[seanpmorgan]

A lot of the test coverage we moved over from contrib is incomplete and I think this bug is an example of something that should be caught (It's on the agenda to review the tests of the ported code). If I were to guess it seems like it's related to decorating the function with autograph, but I haven't dug in past that stack trace. Also a little busy, but I can take a look in a day or two if the error isn't immediately obvious to @alextp

[alextp]

@mdanatg I think the warnings from autograph need to be a little clearer. Can you comment here?

[mdanatg]

Yes, we're reviewing the warning messages, and planning to adjust this one as well. But I'm encouraged that the instructions in the warning allowed you to re-run the code with the diagnostic flag - that helps a lot.

To be clear, this is just a warning and probably doesn't affect the functionality of your code. So we probably want to include a flag that suppresses it for now.

Digging deeper for the image_projective_transform_v2 function that causes the error, it looks like a custom op - these won't have any Python source code attached, so autograph should not attempt to convert them in the first place. We'll look into that.

[seanpmorgan]

So just a recap:

• Regarding what I said about test coverage catching this... probably unlikely because its difficult to assert no warnings when addons depends on TF which depends on many other packages. I apologize for reading through the issue too quickly and not seeing the error came after the AUTOGRAPH env variable. That said we still need to re-evaluate the coverage for tf.contrib ports.
• @mdanatg Can you update this issue when we can expect custom ops to be ignored by autograph?
• @kyleabeauchamp Thanks again for reporting this. We'll leave it open until a fix because the unable to locate source code warning does sound quite damning

[mdanatg]

Will do! Feel free to add me to the Assignees list but we linked an internal bug to the issue.

[seanpmorgan]

@mdanatg Just wondering if you could give an update on this issue? I checked today's nightly and looks like the issue is still there. With TF2 release candidate somewhere on the horizon I thought it's worth revisiting.

[mdanatg]

It seems that the issue did not update automatically, probably because this is a separate repository. The issue should be resolved at head, but if it still appears then we need to look a bit closely.

Unfortunately I wasn't able to build addons from source. This is the error I'm getting when running python tensorflow_addons/image/transform_ops_test.py:

...
File "...tensorflow/python/framework/load_library.py", line 61, in load_op_library
    lib_handle = py_tf.TF_LoadLibrary(library_filename)
tensorflow.python.framework.errors_impl.NotFoundError:
.../tensorflow_addons/custom_ops/image/_image_ops.so: undefined symbol: _ZN10tensorflow8internal21CheckOpMessageBuilder9NewStringEv

Alternately, if you run the script instead, with AUTOGRAPH_VERBOSITY=10 and link the complete log output, that might be enough for me to figure out what's wrong.

[seanpmorgan]

Ah, sorry thats an ugly error that comes from compiling with gcc >=5. We're waiting on a new build toolchain that will fix this see #119 .

A temporary fix should be to build in this docker container which will make it compatible with the tensorflow pip package:
https://github.com/tensorflow/addons/blob/master/CONTRIBUTING.md#development-environment

[mdanatg]

Thanks, the docker container worked, and I found the culprit as well. It seems that our own op loader generates some dynamic code using exec, which discards source code information. I'll have a fix shortly to handle those properly.

The warning is otherwise absolutely benign, but sorry for the annoyance it created!

[seanpmorgan]

Thanks for looking into this! No real annoyance... most of our users at the moment understand we're still working out some stuff. After TF2 release I'd imagine this be more problematic.

[mdanatg]

I just checked the test with the nightly and the warnings should go away now - can you confirm?

[seanpmorgan]

Confirmed. Thank you for your help!

[georgesterpu]

Hello
I am trying to get started with tensorflow-addons, and currently receiving the following error message:

 from tensorflow_addons import seq2seq
  File "/home/john_tukey/miniconda3/envs/tf20py36/lib/python3.6/site-packages/tensorflow_addons/__init__.py", line 75, in <module>
    from tensorflow_addons import image
  File "/home/john_tukey/miniconda3/envs/tf20py36/lib/python3.6/site-packages/tensorflow_addons/image/__init__.py", line 22, in <module>
    from tensorflow_addons.image.distance_transform import euclidean_dist_transform
  File "/home/john_tukey/miniconda3/envs/tf20py36/lib/python3.6/site-packages/tensorflow_addons/image/distance_transform.py", line 26, in <module>
    get_path_to_datafile("custom_ops/image/_image_ops.so"))
  File "/home/john_tukey/miniconda3/envs/tf20py36/lib/python3.6/site-packages/tensorflow/python/framework/load_library.py", line 61, in load_op_library
    lib_handle = py_tf.TF_LoadLibrary(library_filename)
tensorflow.python.framework.errors_impl.NotFoundError: /home/john_tukey/miniconda3/envs/tf20py36/lib/python3.6/site-packages/tensorflow_addons/custom_ops/image/_image_ops.so: undefined symbol: _ZTVN10tensorflow14kernel_factory17OpKernelRegistrar18PtrOpKernelFactoryE
libgcc_s.so.1 must be installed for pthread_cancel to work

Packages installed from pip

$ conda list
# packages in environment at /home/john_tukey/miniconda3/envs/tf20py36:
#
# Name                    Version                   Build  Channel
absl-py                   0.7.1                    pypi_0    pypi
artifacts                 20190320                 pypi_0    pypi
astor                     0.7.1                    pypi_0    pypi
ca-certificates           2019.1.23                     0  
certifi                   2019.3.9                 py36_0  
gast                      0.2.2                    pypi_0    pypi
google-pasta              0.1.6                    pypi_0    pypi
grpcio                    1.20.1                   pypi_0    pypi
h5py                      2.9.0                    pypi_0    pypi
keras-applications        1.0.7                    pypi_0    pypi
keras-preprocessing       1.0.9                    pypi_0    pypi
libgcc-ng                 8.2.0                hdf63c60_1  
markdown                  3.1                      pypi_0    pypi
numpy                     1.16.3                   pypi_0    pypi
openssl                   1.0.2r               h7b6447c_0  
pip                       19.1.1                   py36_0  
protobuf                  3.7.1                    pypi_0    pypi
python                    3.6.0                         0  
readline                  6.2                           2  
setuptools                41.0.1                   py36_0  
six                       1.12.0                   pypi_0    pypi
sqlite                    3.13.0                        0  
tb-nightly                1.14.0a20190511          pypi_0    pypi
tensorflow-addons         0.3.1                    pypi_0    pypi
tensorflow-estimator-2-0-preview 1.14.0.dev2019051100          pypi_0    pypi
termcolor                 1.1.0                    pypi_0    pypi
tf-nightly-2-0-preview    2.0.0.dev20190511          pypi_0    pypi
tk                        8.5.18                        0  
werkzeug                  0.15.2                   pypi_0    pypi
wheel                     0.33.2                   py36_0  
wrapt                     1.11.1                   pypi_0    pypi
xz                        5.2.4                h14c3975_4  
zlib                      1.2.11               h7b6447c_3

Same story with other calls to
tf.load_op_library( get_path_to_datafile("custom_ops/ [...] ))

Any advice @seanpmorgan @mdanatg ?
Thank you

[seanpmorgan]

Hi! Could you create a separate issue for this? I believe it is related to an ABI version mistmatch but we'll need some more information. Such as was addons and tensorflow both installed from pip? It'll be easier to troubleshoot in a separate thread because this isn't related.

[georgesterpu]

Sure, creating a separate issue right now.

[ruifengma]

feature_extractor_url ="https://tfhub.dev/google/tf2-preview/mobilenet_v2/classification/2" #@param {type:"string"}

feature_extrator_layer = hub.KerasLayer(feature_extractor_url, input_shape=IMAGE_SHAPE+(3,))

model = tf.keras.Sequential([
  feature_extractor_layer,
  layers.Dense(image_data_train.num_classes, activation='softmax')
])

model.summary()

WARNING:tensorflow:Entity > could not be transformed and will be executed as-is. Please report this to the AutoGraph team. When filing the bug, set the verbosity to 10 (on Linux, export AUTOGRAPH_VERBOSITY=10) and attach the full output. Cause: module 'gast' has no attribute 'Num'
WARNING:tensorflow:Entity > could not be transformed and will be executed as-is. Please report this to the AutoGraph team. When filing the bug, set the verbosity to 10 (on Linux, export AUTOGRAPH_VERBOSITY=10) and attach the full output. Cause: module 'gast' has no attribute 'Num'
WARNING: Entity > could not be transformed and will be executed as-is. Please report this to the AutoGraph team. When filing the bug, set the verbosity to 10 (on Linux, export AUTOGRAPH_VERBOSITY=10) and attach the full output. Cause: module 'gast' has no attribute 'Num'

[mdanatg]

@ruifengma could you open a separate bug for these warnings? They're due to a different cause, happy to elaborate on a separate thread.