smart_open() printing lots of output during transfer when not desired

[canada11]

When I use smart_open() (connecting to s3) it prints a ton of stuff like this:

...
133
142
170
154
144
137
147
...

Can I turn that off? What is it? There's a lot of it, I don't want it to print this.

Also some docs around the project would be greatly appreciated.
For example another question (because there's no docs):

How do I know if the s3 connection is using multipart upload or uploading serially? Maybe you guys always have it do multipart? I wouldn't know though, all I see is it seems like you have to get some parameters correct for it to use multipart upload

[piskvorky]

smart_open never prints anything. If you include a minimal reproducible example, we'll have look.

[canada11]

(python 3.6.7, boto3 1.9.103, botocore 1.12.103)

from smart_open import smart_open
with smart_open('s3://<bucket>/test.csv', 'wb', profile_name='<your-profile>') as s3_landing_file:
    for line in smart_open('https://raw.githubusercontent.com/kavanagh/zipcodes/master/public/data/zcta_county_rel_10.csv'):
        s3_landing_file.write(line)

[canada11]

While hitting ctrl+c while it's outputting to the screen, I had this output:

  File "<stdin>", line 2, in <module>
  File "/<local_path>/lib/python3.6/site-packages/smart_open/http.py", line 82, in read
    logger.debug('read: %r', locals())
  File "/usr/lib/python3.6/logging/__init__.py", line 1286, in debug
    def debug(self, msg, *args, **kwargs):

Maybe debug was turned on somewhere in smart_open?

[mpenkov]

The output you are seeing is a stack trace. This always gets printed to standard error when you interrupt your Python program by pressing ctrl+c.

What do you see if you don't hit ctrl+c? Please post the full output.

[canada11]

I tried to get the output in a file by running a python script and I noticed that there was no output when run as a script. But the output happens when I run the code by hand in a python shell interpreter. I copied the output from the interpreter and it is 44413 lines that all are this:

>>>s3_landing_file.write(line)
186
159
144
150
150
...
147
148
142
154
>>>

I gave the first and last 5 lines of the output and everything in between for 44413 lines all look like that.

So yeah, new update: it seems to be in the python interpreter only?

[mpenkov]

That’s really strange.

Let’s try to narrow the problem down a bit. I see you’re using smart_open twice in your example. Once to read, once to write. Those two things are independent, so you try to reproduce the problem using only one of them.

Also, what is special about the number 44413 (+5) in your data? It looks like something inside a loop is doing the printing. It’d help knowing what it is and where to look.

[piskvorky]

@canada11 how did you install smart_open? What version? Did you make any edits to its source code?

[canada11]

I have smart-open 1.8.0. I installed it with pip3 install smart-open. I didn't make any edits to the source code.

I was thinking it would be nice to use smart-open to stream from a url into s3 w/o writing the data to a local area first. That's why I used it to open two sources and read from one and write the reads into the other. It's quite handy if it can work.

I couldn't get it to print in a similar way when I did them separately.
And there's nothing special about 44413, that's just how many lines of ouput I got when doing the example code I provided above. I gave the count instead of pasting all of the lines, especially since they're all the same. i.e. they are all just a number around 150.

[mpenkov]

Turns out this is expected behavior. The Python interpreter outputs the result of the last command, if it is not None. For example:

>>> 1 + 2
3

The .write method returns the number of bytes written. So

>>> fout.write(b'hello world')
9

You can suppress that by assigning to a throwaway variable:

>>> _ = fout.write(b'hello world')