Hub: TypeError: Cannot mix str and non-str arguments

[andrewm4894]

Hi,

I'm trying to run a job to load a module and do an embedding from within an airflow task in google cloud composer.

It all works when i run the script in python itself but when i try have it run in airflow i get the below error on the line

embed = hub.Module("https://tfhub.dev/google/nnlm-en-dim50/1")

Error i get in the airflow log:

[2018-06-12 20:47:19,845] {base_task_runner.py:98} INFO - Subtask: [2018-06-12 20:47:19,841] {tf_hub_encode_posts.py:118} INFO - ... save posts into list ...
[2018-06-12 20:47:19,845] {base_task_runner.py:98} INFO - Subtask: [2018-06-12 20:47:19,842] {tf_hub_encode_posts.py:121} INFO - ... look at posts ...
[2018-06-12 20:47:19,847] {base_task_runner.py:98} INFO - Subtask: [2018-06-12 20:47:19,842] {tf_hub_encode_posts.py:122} INFO - [u'11 Stars Who Looked Sexy Playing Home-Wreckers On-Screen: Taylor Swift & More', u"Shania Twain Performs 'Life's About To Get Good' In Sexy Cheetah Print Suit On James Corden", u'Selena Gomez & Justin Bieber: It\u2019s \u2018Painful\u2019 For Her To See Him With Hailey Baldwin', u"Meghan Markle Is Adjusting To Royal Life 'Faster Than Expected' -- Which Royal Is Her New Bestie?", u'Dennis Rodman Mocked After He Breaks Down On CNN After Trump Meets With Kim Jong Un', u'Kate Upton Shows Off Major Sideboob In Sexy Nude Video To Thank Fans For Their Birthday Wishes', u'Hailey Baldwin Playfully Dries Justin Bieber Off With A Towel After Taking A Dip In The Pool', u"'Criminal Minds': Kirsten Vangsness Reveals What\u2019s Really On Her Computer Screen On The Show", u"Mac Miller 'Devastated' Over Ariana's Engagement To Pete Davidson: 'It's A Punch To The Gut'", u'Bode Miller: 5 Things To Know About Olympian Whose 19-Month-Old Daughter Tragically Drowned']
[2018-06-12 20:47:19,847] {base_task_runner.py:98} INFO - Subtask: [2018-06-12 20:47:19,842] {tf_hub_encode_posts.py:125} INFO - ... begin - get module from tf-hub ...
[2018-06-12 20:47:19,858] {base_task_runner.py:98} INFO - Subtask: [2018-06-12 20:47:19,857] {tf_logging.py:160} INFO - Using /tmp/tfhub_modules to cache modules.
[2018-06-12 20:47:19,858] {base_task_runner.py:98} INFO - Subtask: [2018-06-12 20:47:19,858] {tf_logging.py:116} INFO - Downloading TF-Hub Module 'https://tfhub.dev/google/nnlm-en-dim50/1'.
[2018-06-12 20:47:19,935] {base_task_runner.py:98} INFO - Subtask: [2018-06-12 20:47:19,935] {sendgrid.py:84} INFO - Email with subject Airflow alert: <TaskInstance: tf_hub_encode_posts.hollywoodlife_encode_posts_nnlm_en_dim50 2018-06-12 20:30:00 [up_for_retry]> is successfully sent to recipients: [{'to': [{'email': '[email protected]'}]}]
[2018-06-12 20:47:19,974] {base_task_runner.py:98} INFO - Subtask: Traceback (most recent call last):
[2018-06-12 20:47:19,975] {base_task_runner.py:98} INFO - Subtask:   File "/usr/local/bin/airflow", line 27, in <module>
[2018-06-12 20:47:19,975] {base_task_runner.py:98} INFO - Subtask:     args.func(args)
[2018-06-12 20:47:19,975] {base_task_runner.py:98} INFO - Subtask:   File "/usr/local/lib/python2.7/site-packages/airflow/bin/cli.py", line 392, in run
[2018-06-12 20:47:19,976] {base_task_runner.py:98} INFO - Subtask:     pool=args.pool,
[2018-06-12 20:47:19,976] {base_task_runner.py:98} INFO - Subtask:   File "/usr/local/lib/python2.7/site-packages/airflow/utils/db.py", line 50, in wrapper
[2018-06-12 20:47:19,976] {base_task_runner.py:98} INFO - Subtask:     result = func(*args, **kwargs)
[2018-06-12 20:47:19,976] {base_task_runner.py:98} INFO - Subtask:   File "/usr/local/lib/python2.7/site-packages/airflow/models.py", line 1492, in _run_raw_task
[2018-06-12 20:47:19,976] {base_task_runner.py:98} INFO - Subtask:     result = task_copy.execute(context=context)
[2018-06-12 20:47:19,977] {base_task_runner.py:98} INFO - Subtask:   File "/usr/local/lib/python2.7/site-packages/airflow/operators/python_operator.py", line 89, in execute
[2018-06-12 20:47:19,977] {base_task_runner.py:98} INFO - Subtask:     return_value = self.execute_callable()
[2018-06-12 20:47:19,977] {base_task_runner.py:98} INFO - Subtask:   File "/usr/local/lib/python2.7/site-packages/airflow/operators/python_operator.py", line 94, in execute_callable
[2018-06-12 20:47:19,977] {base_task_runner.py:98} INFO - Subtask:     return self.python_callable(*self.op_args, **self.op_kwargs)
[2018-06-12 20:47:19,977] {base_task_runner.py:98} INFO - Subtask:   File "/home/airflow/gcs/dags/tf_hub_encode_posts.py", line 127, in fn_encode_posts
[2018-06-12 20:47:19,978] {base_task_runner.py:98} INFO - Subtask:     embed = hub.Module("https://tfhub.dev/google/nnlm-en-dim50/1")
[2018-06-12 20:47:19,978] {base_task_runner.py:98} INFO - Subtask:   File "/usr/local/lib/python2.7/site-packages/tensorflow_hub/module.py", line 105, in __init__
[2018-06-12 20:47:19,979] {base_task_runner.py:98} INFO - Subtask:     self._spec = as_module_spec(spec)
[2018-06-12 20:47:19,979] {base_task_runner.py:98} INFO - Subtask:   File "/usr/local/lib/python2.7/site-packages/tensorflow_hub/module.py", line 31, in as_module_spec
[2018-06-12 20:47:19,980] {base_task_runner.py:98} INFO - Subtask:     return native_module.load_module_spec(spec)
[2018-06-12 20:47:19,980] {base_task_runner.py:98} INFO - Subtask:   File "/usr/local/lib/python2.7/site-packages/tensorflow_hub/native_module.py", line 99, in load_module_spec
[2018-06-12 20:47:19,981] {base_task_runner.py:98} INFO - Subtask:     path = compressed_module_resolver.get_default().get_module_path(path)
[2018-06-12 20:47:19,982] {base_task_runner.py:98} INFO - Subtask:   File "/usr/local/lib/python2.7/site-packages/tensorflow_hub/resolver.py", line 385, in get_module_path
[2018-06-12 20:47:19,982] {base_task_runner.py:98} INFO - Subtask:     return self._get_module_path(handle)
[2018-06-12 20:47:19,983] {base_task_runner.py:98} INFO - Subtask:   File "/usr/local/lib/python2.7/site-packages/tensorflow_hub/resolver.py", line 467, in _get_module_path
[2018-06-12 20:47:19,983] {base_task_runner.py:98} INFO - Subtask:     return resolver.get_module_path(handle)
[2018-06-12 20:47:19,983] {base_task_runner.py:98} INFO - Subtask:   File "/usr/local/lib/python2.7/site-packages/tensorflow_hub/resolver.py", line 385, in get_module_path
[2018-06-12 20:47:19,984] {base_task_runner.py:98} INFO - Subtask:     return self._get_module_path(handle)
[2018-06-12 20:47:19,984] {base_task_runner.py:98} INFO - Subtask:   File "/usr/local/lib/python2.7/site-packages/tensorflow_hub/compressed_module_resolver.py", line 105, in _get_module_path
[2018-06-12 20:47:19,985] {base_task_runner.py:98} INFO - Subtask:     self._lock_file_timeout_sec())
[2018-06-12 20:47:19,985] {base_task_runner.py:98} INFO - Subtask:   File "/usr/local/lib/python2.7/site-packages/tensorflow_hub/resolver.py", line 313, in atomic_download
[2018-06-12 20:47:19,985] {base_task_runner.py:98} INFO - Subtask:     download_fn(handle, tmp_dir)
[2018-06-12 20:47:19,986] {base_task_runner.py:98} INFO - Subtask:   File "/usr/local/lib/python2.7/site-packages/tensorflow_hub/compressed_module_resolver.py", line 86, in download
[2018-06-12 20:47:19,986] {base_task_runner.py:98} INFO - Subtask:     request = url.Request(_append_compressed_format_query(handle))
[2018-06-12 20:47:19,986] {base_task_runner.py:98} INFO - Subtask:   File "/usr/local/lib/python2.7/site-packages/tensorflow_hub/compressed_module_resolver.py", line 62, in _append_compressed_format_query
[2018-06-12 20:47:19,987] {base_task_runner.py:98} INFO - Subtask:     return urlparse.urlunparse(parsed)
[2018-06-12 20:47:19,987] {base_task_runner.py:98} INFO - Subtask:   File "/usr/local/lib/python2.7/site-packages/future/backports/urllib/parse.py", line 387, in urlunparse
[2018-06-12 20:47:19,987] {base_task_runner.py:98} INFO - Subtask:     _coerce_args(*components))
[2018-06-12 20:47:19,988] {base_task_runner.py:98} INFO - Subtask:   File "/usr/local/lib/python2.7/site-packages/future/backports/urllib/parse.py", line 115, in _coerce_args
[2018-06-12 20:47:19,988] {base_task_runner.py:98} INFO - Subtask:     raise TypeError("Cannot mix str and non-str arguments")
[2018-06-12 20:47:19,988] {base_task_runner.py:98} INFO - Subtask: TypeError: Cannot mix str and non-str arguments

From looking around i think it's related to python2 and using future. It seems like maybe also to do with how the call is being made to urllib by tensorflow hub.

Here are related issue that explain it a bit more.
https://github.com/jfinkels/flask-restless/issues/594
https://github.com/rst2pdf/rst2pdf/issues/596

Apologies i could not come up with a easily reproducible error as its running in airflow (composer) and a few libs deep from what i can see.

[andresusanopinto]

This is related with something passing a "bytes" parameter as the path of the module.
I am not familiar with airflow, but why would it be replacing a str with a bytes?

[andrewm4894]

hmm - i'm not sure maybe for python 2 vs 3 compatibility or something
perhaps?

I'll share this issue with the airflow dev list to see what they think.

On Tue, Jun 19, 2018 at 9:42 AM André Susano Pinto [e-mail]
wrote:

This is related with something passing a "bytes" parameter as the path of
the module.
I am not familiar with airflow, but why would it be replacing a str with a
bytes?

—
You are receiving this because you authored the thread.

Reply to this email directly, view it on GitHub
https://github.com/tensorflow/hub/issues/76#issuecomment-398322318, or mute
the thread
https://github.com/notifications/unsubscribe-auth/ACE89Dpsd8zjVbPPd4EENzKGlHAn3yVIks5t-LmGgaJpZM4UlJBl
.

[andrewm4894]

started a thread here to see if airflow devs might have an idea.

https://lists.apache.org/thread.html/aa21a6a91ead325f11862a47766811a4bc121222b7b5abc698eec96d@%3Cdev.airflow.apache.org%3E

I'll update here if i hear anything.

[andresusanopinto]

Sorry, i missed this was in python2.

TLDR:
Do you happen to use from __future__ import unicode_literals in your file?
You can try to use: "hub.Module(str(...))".

Long:
I looked a bit more and this seems due to https://github.com/PythonCharmers/python-future/issues/273
However I don't understand how our code is stumbling on it since we do not ues python future when backporting urllib. Anyhow using the colab.research.google.com environment I tried all combinations of imports and I could not reproduce this. In particular knowing the versions of urlparse and future where this happen could help debugging.

[andrewm4894]

I tried wrapping it in str() but got the same error.

It was only when i tried to port my script to airflow that i ran into this too.

The code is being run by Apache Airflow in a PythonOperator.

I guess maybe it's more of an Airflow thing then a tf hub thing perhaps. (Partly i raised it here as i'm sure others might end up using Airflow to schedule jobs that import from tf-hub so in case was to be something other may end up coming across).

I'll try follow up with the Airflow people maybe.

Feel free to close and thanks for taking a look.

[andrewm4894]

p.s. looks like probably yes from __future__ import unicode_literals is being used

https://github.com/apache/incubator-airflow/search?q=from+__future__+import+unicode_literals&unscoped_q=from+__future__+import+unicode_literals

[andresusanopinto]

I found that the same error happens in cloud datalab.

There the issue is that the python2 instalation contains a backport of "from urllib.parse import urlencode" which outputs a "" instead of "str". This breaks the later call into "urlparse.urlunparse(parsed)".

[rmothukuru]

@andrewm4894 ,
Can you please confirm if this issue has been resolved with the code that was committed by @arnoegw

[rmothukuru]

Automatically closing due to lack of recent activity. Please update the issue when new information becomes available, and we will reopen the issue. Thanks!