Google-cloud-python: GAX support: need new releases of 'google-gax' and 'gax-google-pubsub-v1'

[tseaver]

I'm blocked on finishing / pushing my GAX pubsub changes on new releases. See:

• https://pypi.python.org/pypi/google-gax
• https://pypi.python.org/pypi/gax-google-pubsub-v1

CC @tbetbetbe

[tbetbetbe]

Sorry - these were due last week - I could not get to them before I moved.
The new ETA is this time tomorrow.

On 24 May 2016 at 00:50, Tres Seaver [email protected] wrote:

I'm blocked on finishing / pushing my GAX pubsub changes on new releases.
See:

• https://pypi.python.org/pypi/google-gax
• https://pypi.python.org/pypi/gax-google-pubsub-v1

CC @tbetbetbe https://github.com/tbetbetbe

—
You are receiving this because you were mentioned.
Reply to this email directly or view it on GitHub
https://github.com/GoogleCloudPlatform/gcloud-python/issues/1814

This email may be confidential or privileged. If you received this
communication by mistake, please don't forward it to anyone else (it may
contain confidential or privileged information), please erase all copies of
it, including all attachments, and please let the sender know it went to
the wrong person. Thanks.

[tseaver]

@tbetbetbe can you update the ETA?

[tbetbetbe]

Sorry about the delay on this. As of now, this is my highest priority
task, I'll update this thread within the next day to report completion.

On 25 May 2016 at 13:18, Tres Seaver [email protected] wrote:

@tbetbetbe https://github.com/tbetbetbe can you update the ETA?

—
You are receiving this because you were mentioned.
Reply to this email directly or view it on GitHub
https://github.com/GoogleCloudPlatform/gcloud-python/issues/1814#issuecomment-221469247

This email may be confidential or privileged. If you received this
communication by mistake, please don't forward it to anyone else (it may
contain confidential or privileged information), please erase all copies of
it, including all attachments, and please let the sender know it went to
the wrong person. Thanks.

[tseaver]

@tbetbetbe Any update?

[tseaver]

Looks like we have a new release of google-gax. We still need another release of gax-google-pubsub-v1, as well as an initial release of gax-google-logging-v2.

[tseaver]

Doesn't look as though the tag for gax-python 0.11.0 has been pushed to Github yet.

[tseaver]

Doesn't look as though the tag for gax-python 0.11.0 has been pushed to Github yet.

I see the v0.11.0 tag now.

@tbetbetbe are releases for the (generated, I presume) gax-google-pubsub-v1 and gax-google-logging-v2 libraries pending something in the upstream? The protos haven't changed recently AFAIK. The generation is done via https://github.com/googleapis/toolkit, right?

[tbetbetbe]

That's correct.

What I'm trying to do is make sure that their
dependencies, grpc-google-pubsub-v1 and grpc-google-logging-v2 are updated
first.
These are not generated by toolkit, rather they are generated by protoc +
the grpc python plugin. That's been updated, and I want to make sure that
we use the updated version.

On 6 June 2016 at 21:22, Tres Seaver [email protected] wrote:

Doesn't look as though the tag for gax-python 0.11.0 has been pushed to
Github yet.

I see the v0.11.0 tag
https://github.com/googleapis/gax-python/releases/tag/v0.11.0 now.

@tbetbetbe https://github.com/tbetbetbe are releases for the
(generated, I presume) gax-google-pubsub-v1 and gax-google-logging-v2
libraries pending something in the upstream? The protos haven't changed
recently AFAIK. The generation is done via https://github.com/googleapis/
toolkit, right?

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
https://github.com/GoogleCloudPlatform/gcloud-python/issues/1814#issuecomment-224060707,
or mute the thread
https://github.com/notifications/unsubscribe/AI18Fhpm_YonsgU4K5PjTbK_LUqqTBHKks5qJHOPgaJpZM4IkoFY
.

This email may be confidential or privileged. If you received this
communication by mistake, please don't forward it to anyone else (it may
contain confidential or privileged information), please erase all copies of
it, including all attachments, and please let the sender know it went to
the wrong person. Thanks.

[tbetbetbe]

@jmuk, can you finish this off please, after submitting https://github.com/googleapis/packman/pull/40 ?

I've successfully updated/published the following

• grpc-google-pubsub-v1
• grpc-google-logging-v2
• grpc-google-devtools-cloudtrace-v1

Now, they all depend on grpcio-v0.14.0 and are generated with the latest grpc python plugin, which fixes issues some python package users are seeing where there are missing imports. (FYI: @jgeiger)

I think the best route would be to submit https://github.com/googleapis/artman/pull/23, and use the changes it contains to use artman to create the updated grpc-gax packages.

The created packages can be submitted using the googleapis-package user name.

[jmuk]

Well, it turns out that googleapis/artman#23 is not useful by itself for now. Without https://github.com/googleapis/toolkit/issues/126, the automatically generated whl file will have the config .json files at a wrong place.

[jmuk]

now googleapis/artman#23 is merged, FYI

[tseaver]

Yay! It looks as though https://github.com/googleapis/toolkit/pull/192 has landed as well (resolving https://github.com/googleapis/toolkit/issues/126, AFAICT).

[bjwatson]

@jgeewax @tseaver I've pushed new releases of https://pypi.python.org/pypi/gax-google-pubsub-v1 and https://pypi.python.org/pypi/grpc-google-pubsub-v1 (and sent you an email describing the same). I back-rev'd them to 0.7.6, because this is still beta code.

I missed https://github.com/googleapis/toolkit/pull/192 in the 0.7.6 release, but I have a bash script I've been using that works around it. Now that I know the recipe, I can easily cut 0.7.7 with that PR, so please let me know if you're missing anything.

Please close this issue if it's been resolved to your satisfaction. Thanks!

[tseaver]

@bjwatson Hmm, that release has a lower version number than the 1.0.16 version, which is still installable from PyPI (even though invisible in the web UI). pip finds the older-but-higher-numbered release:

$ /opt/Python-2.7.11/bin/virtualenv /tmp/gax
New python executable in /tmp/gax/bin/python
Installing setuptools, pip, wheel...done.
 $ /tmp/gax/bin/pip install --no-cache-dir gax-google-pubsub-v1
Collecting gax-google-pubsub-v1
  Downloading gax-google-pubsub-v1-1.0.16.tar.gz
Collecting googleapis-common-protos>=1.1.0 (from gax-google-pubsub-v1)
  Downloading googleapis-common-protos-1.1.0.tar.gz
Collecting google-gax==0.9.4 (from gax-google-pubsub-v1)
  Downloading google-gax-0.9.4.tar.gz
Collecting grpc-google-pubsub-v1>=1.0.15 (from gax-google-pubsub-v1)
  Downloading grpc-google-pubsub-v1-1.0.19.tar.gz
Collecting oauth2client>=1.4.11 (from gax-google-pubsub-v1)
  Downloading oauth2client-2.1.0.tar.gz (69kB)
    100% |████████████████████████████████| 71kB 1.2MB/s 
Collecting protobuf>=3.0.0b2 (from googleapis-common-protos>=1.1.0->gax-google-pubsub-v1)
  Downloading protobuf-3.0.0b3-py2-none-any.whl (334kB)
    100% |████████████████████████████████| 337kB 2.3MB/s 
Collecting grpcio==0.13.1 (from google-gax==0.9.4->gax-google-pubsub-v1)
  Downloading grpcio-0.13.1.tar.gz (5.2MB)
    100% |████████████████████████████████| 5.2MB 2.9MB/s 
Collecting ply==3.8 (from google-gax==0.9.4->gax-google-pubsub-v1)
  Downloading ply-3.8.tar.gz (157kB)
    100% |████████████████████████████████| 163kB 3.1MB/s 
Collecting httplib2>=0.9.1 (from oauth2client>=1.4.11->gax-google-pubsub-v1)
  Downloading httplib2-0.9.2.zip (210kB)
    100% |████████████████████████████████| 215kB 3.2MB/s 
Collecting pyasn1>=0.1.7 (from oauth2client>=1.4.11->gax-google-pubsub-v1)
  Downloading pyasn1-0.1.9-py2.py3-none-any.whl
Collecting pyasn1-modules>=0.0.5 (from oauth2client>=1.4.11->gax-google-pubsub-v1)
  Downloading pyasn1_modules-0.0.8-py2.py3-none-any.whl
Collecting rsa>=3.1.4 (from oauth2client>=1.4.11->gax-google-pubsub-v1)
  Downloading rsa-3.4.2-py2.py3-none-any.whl (46kB)
    100% |████████████████████████████████| 51kB 7.4MB/s 
Collecting six>=1.6.1 (from oauth2client>=1.4.11->gax-google-pubsub-v1)
  Downloading six-1.10.0-py2.py3-none-any.whl
Requirement already satisfied (use --upgrade to upgrade): setuptools in /tmp/gax/lib/python2.7/site-packages (from protobuf>=3.0.0b2->googleapis-common-protos>=1.1.0->gax-google-pubsub-v1)
Collecting enum34>=1.0.4 (from grpcio==0.13.1->google-gax==0.9.4->gax-google-pubsub-v1)
  Downloading enum34-1.1.6-py2-none-any.whl
Collecting futures>=2.2.0 (from grpcio==0.13.1->google-gax==0.9.4->gax-google-pubsub-v1)
  Downloading futures-3.0.5-py2-none-any.whl
Installing collected packages: six, protobuf, googleapis-common-protos, enum34, futures, grpcio, ply, httplib2, pyasn1, pyasn1-modules, rsa, oauth2client, google-gax, grpc-google-pubsub-v1, gax-google-pubsub-v1
  Running setup.py install for googleapis-common-protos ... done
  Running setup.py install for grpcio ... done
  Running setup.py install for ply ... done
  Running setup.py install for httplib2 ... done
  Running setup.py install for oauth2client ... done
  Running setup.py install for google-gax ... done
  Running setup.py install for grpc-google-pubsub-v1 ... done
  Running setup.py install for gax-google-pubsub-v1 ... done
Successfully installed enum34-1.1.6 futures-3.0.5 gax-google-pubsub-v1-1.0.16 google-gax-0.9.4 googleapis-common-protos-1.1.0 grpc-google-pubsub-v1-1.0.19 grpcio-0.13.1 httplib2-0.9.2 oauth2client-2.1.0 ply-3.8 protobuf-3.0.0b3 pyasn1-0.1.9 pyasn1-modules-0.0.8 rsa-3.4.2 six-1.10.0

[tseaver]

Likewise, https://pypi.python.org/pypi/grpc-google-pubsub-v1/1.0.19 is still present, and preferred by pip (although it looks recent enough to be correct).

[bjwatson]

@tseaver Could you try the following as a workaround:

pip install "grpc-google-pubsub-v1<1.0.0"
pip install "gax-google-pubsub-v1<1.0.0"

I'm running into some dependency errors on my workstation, but they might not be related (this is the first time I've directly tried the virtualenv style of installing these packages).

[tseaver]

That does get the 0.7.6 versions installed.

[bjwatson]

Cool. We'll get a firm decision on the version number, and then make any necessary updates.

[tseaver]

One note: if you decide to yank the 1.x.y releases from PyPI, those release numbers will be forever poisoned, as PyPI will not permit re-uploading a distribution that has been removed.

[bjwatson]

Agreed. I will track that we need to go to 1.0.20 or 1.1.0 when we finally do a GA release.

I have deleted all 1.x.y versions; 0.7.6 is the only version now.

I did not have time to work on the logging API today. I will try to get that done on Monday. Hopefully this at least half-unblocks you.

[tseaver]

@bjwatson the 0.7.6 release is a bit borked: it declares namespace packages which don't belong to it (gcloud, gcloud.pubsub), and writes files there which don't belong to gcloud-python. All code for the GAX wrappers should be located in `google.pubsub.v1' Can you please fix that and release a 0.7.7?

[bjwatson]

@tseaver Sure. I knew this changed before I joined, but didn't realize the context behind it. I'll immediately release a 0.7.7 as you requested to unblock you.

[bjwatson]

Done: https://pypi.python.org/pypi/gax-google-pubsub-v1

[bjwatson]

FYI, I've also updated the docs: http://pythonhosted.org/gax-google-pubsub-v1/index.html

[tseaver]

Thanks!

[tseaver]

@bjwatson I know you're on vacation until Friday, but I'm hoping to see the gax-google-logging-v2 release soonish.

[jmuk]

@geigerj might be able to help this?

[bjwatson]

@tseaver I'm actually in an all-day class, rather than on vacation, so I have a little time to spend in the evenings.

I just published version 0.7.8 of logging:

• https://pypi.python.org/pypi/grpc-google-logging-v2
• https://pypi.python.org/pypi/gax-google-logging-v2
• https://pythonhosted.org/gax-google-logging-v2/

Please let me know if there are any issues. Something I got a little stuck on is differences in how the toolchain on my workstation generates the grpc package vs. how it was generated for previous versions of pubsub. I didn't have to deal with this for pubsub, since I was basically re-publishing 1.0.19 as 0.7.x, but I did have to handle this for logging which has not previously been published.

I will also review the documentation more closely tomorrow, after I'm done with my class.

FYI @geigerj and @temiola.

[bjwatson]

@tseaver @daspecster Are you waiting on anything from me for resolving this issue? Please let me know if there's anything I need to fix in the generated grpc-google-* or gax-google-* packages.

[daspecster]

@bjwatson unless this is somehow related to https://github.com/GoogleCloudPlatform/gcloud-python/issues/1863, I think it's resolved.

WDYT @tseaver?

[tseaver]

I haven't actually tried to land the code yet which depends on gax-google-logging-v2, but the tarball on PyPI looks correct (it puts the generated files into google.logging.v2, rather than gcloud.logging).

[bjwatson]

Thanks.

[tseaver]

@bjwatson Where should I be reporting issues with gax-google-pubsub-v1?

[bjwatson]

Good question. I would suggest raising them as issues here, and tagging @tbetbetbe and I. The root cause for a pubsub issue could be in any one of the vKit repos (or several of them).