Protobuf: There's no 3.12.1 sdist on PyPI

[sunpoet]

Is it possible to release the 3.12.0 sdist on PyPI? It helps FreeBSD port devel/py-protobuf to download the source from PyPI instead of changing to github.com assets.
Thanks!

[hrw]

INFO:kolla.common.utils.openstack-base:ERROR: No matching distribution found for protobuf===3.12.0 (from -c /requirements/upper-constraints.txt (line 353))

Thank you for breaking all AArch64 CI builds.

Upload source!

[kamyar]

Can someone please fix this?

[hrw]

Seriously? Not even "deal with it" answer?

[kamyar]

After all this is google, let's hope they do not suddenly deprecate/cancel Protobuf. 🙏 😂

[hrw]

@kamyar maybe this would get Hadoop and others to move from protobuf 2.5 to something modern ;D

[atollk]

Windows builds are also broken for the same reason that @hrw posted above.

[sunpoet]

3.12.1 was released and there is still no sdist on PyPI.

[luizguilhermefr]

Any updates?

[bign8]

Looks like the last release with source distributions is 3.11.3 [Feb 3, 2020], followed by 3.12.0rc1 [May 4th, 2020] (at least according to pypi) which doesn't have an sdist. I didn't see anything in the diff https://github.com/protocolbuffers/protobuf/compare/v3.11.3...v3.12.0-rc1 that would manifest in this happening (not an expert in CMake or Bazel). And it doesn't look like it's the pypi servers issue, as some of the "latest" releases (at the time of this comment) have source distributions. Maybe something happened in the upload tool-chain over that 3 month gap (I would look, but I'm not a googler). :shrug:

[haberman]

Maybe something happened in the upload tool-chain over that 3 month gap (I would look, but I'm not a googler).

That is an excellent guess.

I released the 3.12.x series using our internal tooling. This was my first time doing a release, and I did not realize that an important part of the Python release had failed. Sorry about that.

I will look into how to get this working.

[phoenixuprising]

@haberman - it appears that not only the sdist is missing but also some of the bdist wheels. The specific bdist I'm seeing missing is the py2.py3-none-any wheel.

[haberman]

The script we are running for the release is here: https://github.com/protocolbuffers/protobuf/blob/master/python/release.sh

It is currently failing on line 83 I believe: https://github.com/protocolbuffers/protobuf/blob/master/python/release.sh#L83

For some reason uploading to the test instance of PyPI is failing with the following error, without even prompting me for a password:

Submitting dist/protobuf-3.12.2.tar.gz to https://test.pypi.org/legacy/
Upload failed (403): Invalid or non-existent authentication information. See https://test.pypi.org/help/#invalid-auth for details
error: Upload failed (403): Invalid or non-existent authentication information. See https://test.pypi.org/help/#invalid-auth for details

[haberman]

I remember debugging this for a while. Based on info I found from some searching, I created the following .pypirc but it didn't help:

[distutils]
index-servers =
    pypi
    testpypi

[pypi]
username: protobuf-packages

[testpypi]
username: protobuf-wheel-test

[haberman]

Unless someone has a quick fix for this, I can try just commenting it out to skip the test.pypi.org step of the release script.

[haberman]

Now I am getting the following crash inside distutils:

Creating tar archive
removing 'protobuf-3.12.2' (and everything under it)
running upload
Traceback (most recent call last):
  File "setup.py", line 285, in <module>
    ext_modules=ext_module_list,
  File "/usr/lib/python2.7/dist-packages/setuptools/__init__.py", line 145, in setup
    return distutils.core.setup(**attrs)
  File "/usr/lib/python2.7/distutils/core.py", line 151, in setup
    dist.run_commands()
  File "/usr/lib/python2.7/distutils/dist.py", line 953, in run_commands
    self.run_command(cmd)
  File "/usr/lib/python2.7/distutils/dist.py", line 972, in run_command
    cmd_obj.run()
  File "/usr/lib/python2.7/distutils/command/upload.py", line 62, in run
    self.upload_file(command, pyversion, filename)
  File "/usr/lib/python2.7/distutils/command/upload.py", line 137, in upload_file
    self.password)
TypeError: cannot concatenate 'str' and 'NoneType' objects

Some searching turned up this, maybe it's because my .pypirc specifies username but not password. `

[haberman]

I commented all lines out of my .pypirc. Now I am getting the same error I got from uploading to the test PyPI. This error seems to be coming from distutils. Does anyone know how to fix this? Do I need to put the password in the .pypirc?

Creating tar archive
removing 'protobuf-3.12.2' (and everything under it)
running upload
Submitting dist/protobuf-3.12.2.tar.gz to https://upload.pypi.org/legacy/
Upload failed (403): Invalid or non-existent authentication information. See https://pypi.org/help/#invalid-auth for details
error: Upload failed (403): Invalid or non-existent authentication information. See https://pypi.org/help/#invalid-auth for details

[di]

Hey @haberman, do you know if there's a reason it's attempting to upload with distutils? The recommended uploader these days is twine, the release script would become something like:

- python setup.py sdist upload -r https://test.pypi.org/legacy/
+ python setup.py sdist
+ twine upload -r testpypi -u protobuf-wheel-test dist/*

Which would not require a .pypirc and would prompt you for the password. I don't believe distuils supports prompts for username/password and just assumes everything is in .pypirc.

[haberman]

Thanks @di, that's very helpful. What would the equivalent twine lines be for these lines to push to the prod instance of PyPI?

https://github.com/protocolbuffers/protobuf/blob/master/python/release.sh#L104-L112

[di]

# Be sure to run build before sdist, because otherwise sdist will not include
# well-known types.
- python setup.py clean build sdist upload
+ python setup.py clean build sdist
+ twine upload -u protobuf-packages dist/* 
# Be sure to run clean before bdist_xxx, because otherwise bdist_xxx will
# include files you may not want in the package. E.g., if you have built
# and tested with --cpp_implemenation, bdist_xxx will include the _message.so
# file even when you no longer pass the --cpp_implemenation flag. See:
#   https://github.com/protocolbuffers/protobuf/issues/3042
- python setup.py clean build bdist_egg bdist_wheel upload
+ python setup.py clean build bdist_wheel
+ twine upload -u protobuf-packages dist/* 

(building .egg files with bdist_egg should be unnecessary)

[di]

Also, you'll probably want to pass --skip-existing to twine in all these invocations, especially if you're attempting to backfill source distributions for previous releases, otherwise it will likely complain that you're trying to upload distributions that already exist (if you run the entire release script).

[phoenixuprising]

@haberman - can you confirm that this will also resolve the missing bdist?

[haberman]

Thanks. Now I am getting:

Creating tar archive
removing 'protobuf-3.12.2' (and everything under it)
+ twine upload --skip-existing -r https://test.pypi.org/legacy/ -u protobuf-wheel-test dist/protobuf-3.12.2.tar.gz
InvalidConfiguration: Missing 'https://test.pypi.org/legacy/' section from the configuration file
or not a complete URL in --repository-url.
Maybe you have a out-dated '~/.pypirc' format?
more info: https://docs.python.org/distutils/packageindex.html#pypirc

[haberman]

@phoenixuprising I hope so, based on the twine commands above. But I know very little about the Python packaging ecosystem, so it's hard to say for sure.

[di]

It's -r testpypi, not -r https://test.pypi.org/legacy/ (the URL is built-in to twine).

[haberman]

The missing sdist and non-any wheel are now available for 3.12.2: https://pypi.org/project/protobuf/3.12.2/#files

Please let me know if any problems are still remaining.

[navels]

Works for me (Raspbian Linux). Thanks!

[atollk]

Looks like Windows is still missing for Python 3.8. Am I mistaken?

[softwarebloat]

The missing sdist and non-any wheel are now available for 3.12.2: https://pypi.org/project/protobuf/3.12.2/#files

Please let me know if any problems are still remaining.

Hi @haberman
thank you very much! but unfortunately i still have the problem

Collecting protobuf==3.12.1
Could not find a version that satisfies the requirement protobuf==3.12.1 (
from versions: 2.0.0b0, 2.0.3, 2.3.0, 2.4.1, 2.5.0, 2.6.0, 2.6.1, 3.0.0a2, 3
.0.0a3, 3.0.0b1, 3.0.0b1.post1, 3.0.0b1.post2, 3.0.0b2, 3.0.0b2.post1, 3.0.0
b2.post2, 3.0.0b3, 3.0.0b4, 3.0.0, 3.1.0, 3.1.0.post1, 3.2.0rc1, 3.2.0rc1.po
st1, 3.2.0rc2, 3.2.0, 3.3.0, 3.4.0, 3.5.0.post1, 3.5.1, 3.5.2, 3.5.2.post1,
3.6.0, 3.6.1, 3.7.0rc2, 3.7.0rc3, 3.7.0, 3.7.1, 3.8.0rc1, 3.8.0, 3.9.0rc1, 3
.9.0, 3.9.1, 3.9.2, 3.10.0rc1, 3.10.0, 3.11.0rc1, 3.11.0rc2, 3.11.0, 3.11.1,
3.11.2, 3.11.3)
No matching distribution found for protobuf==3.12.1

It happens to me using a linux alpine image. Not happening in my local machine (MacOS)

[hrw]

Thanks for sorting out 3.12.2 release.

Good side effect: I have learnt that OpenDev infrastructure has CI jobs to build wheels. So soon all jobs will speed up.

[hrw]

@haberman can you add "none" wheel and source for previous 3.12.* releases as well?

[sunpoet]

@haberman Thanks! 3.12.2 release works fine.

[phoenixuprising]

Thanks @haberman for fixing - seeing the none-any wheels now for 3.12.2. Still missing for 3.12.1 and 3.12.0 but I should be able to pin to >= 3.12.2 and be fine for my purposes.