Smart_open: Getting error when trying to write .txt file from local fs to webhdfs.

[andrewrobertmadsen]

Getting the following error when trying to write a text file from local to webhdfs:
TypeError: can only concatenate str (not "bytes") to str

Hoping that I am just doing something wrong. My code is as follows:

from smart_open import open

def smart_copy(source_file, sync_file):
    with open(source_file, 'rb') as source:
        with open(sync_file, 'wb') as sync:
            for line in source:
                sync.write(line)

smart_copy('./test_file.txt', 'webhdfs://{username}@{host}:{port}/user/XXXX/smart_copy/test_file.txt')

The stack trace for the error is:

---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-40-b63bd711d13c> in <module>
----> 1 smart_copy('./test_file.txt', 'webhdfs://[email protected]:XXXXX/user/XXXX/smart_copy/test_file.txt')

<ipython-input-38-694f70cf0776> in smart_copy(source_file, sync_file)
      3     '''
      4     with open(source_file, 'rb') as source:
----> 5         with open(sync_file, 'wb') as sync:
      6             for line in source:
      7                 sync.write(line)

~/Github/tools/.venv/lib/python3.7/site-packages/smart_open/smart_open_lib.py in open(uri, mode, buffering, encoding, errors, newline, closefd, opener, ignore_ext, transport_params)
    346     except KeyError:
    347         binary_mode = mode
--> 348     binary, filename = _open_binary_stream(uri, binary_mode, transport_params)
    349     if ignore_ext:
    350         decompressed = binary

~/Github/tools/.venv/lib/python3.7/site-packages/smart_open/smart_open_lib.py in _open_binary_stream(uri, mode, transport_params)
    560         elif parsed_uri.scheme == "webhdfs":
    561             kw = _check_kwargs(smart_open_webhdfs.open, transport_params)
--> 562             return smart_open_webhdfs.open(parsed_uri.uri_path, mode, **kw), filename
    563         elif parsed_uri.scheme.startswith('http'):
    564             #

~/Github/tools/.venv/lib/python3.7/site-packages/smart_open/webhdfs.py in open(uri, mode, min_part_size)
     40         return BufferedInputBase(uri)
     41     elif mode == 'wb':
---> 42         return BufferedOutputBase(uri, min_part_size=min_part_size)
     43     else:
     44         raise NotImplementedError('webhdfs support for mode %r not implemented' % mode)

~/Github/tools/.venv/lib/python3.7/site-packages/smart_open/webhdfs.py in __init__(self, uri_path, min_part_size)
    129                                      params=payload, allow_redirects=False)
    130         if not init_response.status_code == httplib.TEMPORARY_REDIRECT:
--> 131             raise WebHdfsException(str(init_response.status_code) + "\n" + init_response.content)
    132         uri = init_response.headers['location']
    133         response = requests.put(uri, data="", headers={'content-type': 'application/octet-stream'})

TypeError: can only concatenate str (not "bytes") to str

test_file.txt is just an ascii text file. Using python 3.7.

Any guidance that you could provide would be awesome. End use case is copying files from s3 to webHDFS and back again.

Thanks!!!

[andrewrobertmadsen]

Looking at this further, it looks like init_response.content on line 131 of webhdfs.py is returning a byte string instead of a string which is causing the error. I changed this to 'init_response.text' and now I am seeing the intended error message. Happy to open a PR to address this issue. Looks like it may occur again on 135 of the same module.

[piskvorky]

Hmm, how come the unit tests pass?

@andrewrobertmadsen thanks for reporting, a PR will be welcome of course! But let's include unit test(s) too, to avoid such regressions in the future: something that fails now, and passes after your PR.

[andrewrobertmadsen]

That’s a good question. I’m guessing the tests pass because it still raises an exception, just not the informative one that was intended. I will have to look first to verify and then I’ll submit a PR. Thanks for looking at this!

[mpenkov]

@andrewrobertmadsen Ping. Are you able to work on this? Hacktoberfest started today ;)

[sachinmukherjee]

Can I work on this?

[mpenkov]

@sachinmukherjee Sure, go ahead

[EinKaiser]

@mpenkov Hi, can I work on this parallely along with @sachinmukherjee too?

[mpenkov]

@EinKaiser I'd prefer to avoid doubling up on a small ticket like this, so I encourage you to look through the issue list and pick another ticket. There are plenty to go around. Ping me if you can't find anything suitable.

[sachinmukherjee]

@mpenkov How to reproduce this error?

[wadleo]

@sachinmukherjee are you still working on this? If not, I can pick it up.

[sachinmukherjee]

@wadleo I got struck in reproducing the error. You can take this up.

[canada11]

I'm having a similar issue which I think is related.

smart_open 1.9.0
python 3.6

Code:

destination = f"webhdfs://<host>:<port>/etl/test.csv"
with open(destination, 'wb') as landing_file:
    pass

Stacktrace:

  File "example.py", line 32
    with open(destination, 'wb') as landing_file:
  File "/usr/local/lib/python3.6/dist-packages/smart_open/smart_open_lib.py", line 352, in open
    binary, filename = _open_binary_stream(uri, binary_mode, transport_params)
  File "/usr/local/lib/python3.6/dist-packages/smart_open/smart_open_lib.py", line 566, in _open_binary_stream
    return smart_open_webhdfs.open(parsed_uri.uri_path, mode, **kw), filename
  File "/usr/local/lib/python3.6/dist-packages/smart_open/webhdfs.py", line 42, in open
    return BufferedOutputBase(uri, min_part_size=min_part_size)
  File "/usr/local/lib/python3.6/dist-packages/smart_open/webhdfs.py", line 131, in __init__
    raise WebHdfsException(str(init_response.status_code) + "\n" + init_response.content)
TypeError: must be str, not bytes

Just putting this here as an FYI, since the bug is still open and I think they might be related

[canada11]

Yeah, same issue, I dug into it and made the changes that Andrew made and they helped. I made a pull request. #382

[mrk-its]

@canada11 ups, we duplicated work a bit :) While testing I've found & fixed few other issues, take a look on description of my PR.

[canada11]

Awesome, yours looks better. I'll close mine so that just your pull gets merged.