Google-cloud-python: Offer progress indication

[kofalt]

I might just be thick, but when searching through the web and documentation, I could not find a way to get gcloud to report upload / transfer progress to the callee:

# Sit blocked & mute for N minutes
bucket.blob(path).upload_from_filename(local_path)

One pattern I saw recently from docker-py allows you to set an optional stream parameter, in which case the function returns an iterator with json strings:

progress = docker_client.pull(repository=x, tag=y, stream=True)

for line in progress:
    update = json.loads(line)
    # do something with line - fields include current bytes, total bytes

A quick search indicates this could be related to #1077.

[dhermes]

Certainly related to #1077, good find. You want progress for display purposes or for other reasons?

[kofalt]

I'd like to report back to the user or tool consuming my script, yeah.
Updates could list bytes transferred, % complete (when known), and ideally a bandwidth rate.

As an example: I haven't filed a bug yet, as I haven't reproduced, but I noticed one run (using upload_from_filename) was hanging for hours - I checked slurm and there seemed to be no outbound traffic. It would be nice to see that when running interactively, or have some other logic trigger.

And, if you're waiting on long transfers, it's nice to know how long you have to get coffee :smile:

[dhermes]

Hanging for hours is scary, but I've seen httplib2 hang for 10+ minutes when the connection gets b0rked in a multi-thread application (#1214).

[kofalt]

Hmm, noted. I'll definitely check that out. I'm not sure sure if that's it because I don't believe we're doing any concurrency, but if I see it again & get a handle on it I'll report back or file a ticket.

[dhermes]

Thanks! I think the Upload object has most of the info in it so covering #1077 probably covers this.

[lukesneeringer]

Hello,
One of the challenges of maintaining a large open source project is that sometimes, you can bite off more than you can chew. As the lead maintainer of google-cloud-python, I can definitely say that I have let the issues here pile up.

As part of trying to get things under control (as well as to empower us to provide better customer service in the future), I am declaring a "bankruptcy" of sorts on many of the old issues, especially those likely to have been addressed or made obsolete by more recent updates.

My goal is to close stale issues whose relevance or solution is no longer immediately evident, and which appear to be of lower importance. I believe in good faith that this is one of those issues, but I am scanning quickly and may occasionally be wrong. If this is an issue of high importance, please comment here and we will reconsider. If this is an issue whose solution is trivial, please consider providing a pull request.

Thank you!

[bachirelkhoury]

Has there been any development on this, or perhaps you can offer an alternative?
we have a very large upload (Gb) and would need to monitor and track the progress of the upload.
Is there anyway to do that?

[PetrochukM]

Same!

[tseaver]

Please open a new issue linking this one, and describing the change you'd like against the current software. Following up on an issue closed for almost two years is not going to get appropriate attention.

[kofalt]

I no longer use this tool in this way, but my colleagues would still benefit from this feature. If someone wants to do as @tseaver suggests I would appreciate it :)