Flatbuffers: What can I do to help make PyPi publications a reality?

[mikeholler]

Hey there,

Thanks for such a great tool. As I'm sure you're aware, the PyPi repository for flatbuffers is rather out of date. I would like to change this. What can I do to help you guys get builds published for each release?

We have some information in #4390 provided by @kbrose, but so far it looks like nobody has taken the initiative to make this process more continuous.

Some questions:

• What is the current state of the PyPi repository linked above?
• Does a maintainer of this project have direct control over it still?
• How have publications been made in the past?
• What are the complications with attempting to publish using the same methods from 2015 upon each new release?
• Are we still happy with the versioning scheme of what looks to be YEAR.MONTH.DAY.BUILD? I personally would suggest semantic versions that mirror the git-tagged releases.
• What can I do to help you get this back on track? I'm also willing to help on the Java side.

Happy Thanksgiving!

Mike

[rw]

Hi @mikeholler, I'm the Python maintainer. We've let releases lag, since 1) it seems users typically end up pulling from master, and 2) some changes should bump the Python package version even if they aren't a change to the FlatBuffers core code, and we have not explicitly decided how to handle that case. Ideally, we'd ship a new Python package at least as often as core FlatBuffers does.

To answer your questions:

• I have access to the PyPI project for FlatBuffers.
• Publications are totally manual: just uploading an sdist.
• Complications are that it's time-consuming.
• Semvar would be great.
• Do you know of a good way to automate this, even partially? This ties into a bigger project we've been working on to get complete continuous integration for all ports.

[mikeholler]

Hey @rw thanks for responding! I'm playing around with Travis on my fork and am making some progress towards publication when a new tag is pushed. I found a blocker for an issue I logged here: https://github.com/travis-ci/travis-ci/issues/8801

I'll give them a chance to respond, and if they don't get back in a reasonable amount of time I'll try to hack at it myself.

[xgdgsc]

It is confusing to see the github version number lower than pypi version. Glad to see this is going to be fixed soon!

[ahundt]

FYI I've seen other big projects that support multiple languages and use PyPi put setup.py in the root directory without issue. One example I can think of off the top of my head is https://github.com/bulletphysics/bullet3 and its pypi package pybullet.

[vishvananda]

What happened to this. Looks like there is potential for pypi ci/cd but it isn't enabled?

[Tinche]

Please do this, it's not that straightforward to depend on git and pip will yell at you anyway.

[tarehart]

I'm a bit new to pip, so just to make sure I understand--the current guidance is to use a github url in my setup.py dependency_links, and then tell all my clients to use --process-dependency-links when installing my package?

And the aforementioned pip yelling is because that flag is deprecated, so this strategy may break at some point? https://github.com/pypa/pip/issues/4187

That puts me in such a compromising position that I might consider copying the flatbuffers source into my own repo and that's pretty gross.

Would you consider doing a manual publication for each tagged release until you figure out your automation? Considering that you've only had one release so far in 2018, that doesn't strike me as unduly burdensome.

[aardappel]

@vishvananda We have some Python related CI in travis.yml, but it's not enabled because it currently fails (intermittently I think). If someone knows how to make it robust we can enable it.

@tarehart I can totally make a manual release.. if someone would spell out the incantations to me that would be appreciated, since I'm not that familiar with the Python ecosystem. Better yet if @rw or someone else can do it since I didn't even set up the original.

[tarehart]

Thanks, that would be great! I believe the instructions here are accurate and appropriate for your code: https://packaging.python.org/guides/distributing-packages-using-setuptools/#packaging-your-project

[Tinche]

Here are the steps, generally:

1. in the python directory, run python setup.py sdist. The current setup.py is set up to get the version from the environment variable 'VERSION', so set that beforehand. If you don't set it it will fallback to using the date. This will produce a 'dist/flatbuffers-1.9.tar.gz' source archive.
2. Upload this archive to PyPI. The best tool for this currently is Twine. (https://pypi.org/project/twine/) Install Twine into a virtualenv (or globally, if you're brave) and just run twine upload dist/flatbuffers-1.9.tar.gz.

You can do a simple smoke test into a virtualenv before uploading (and use this virtualenv to install twine).
In the python directory (or anywhere, but let's do it here), run python3 -m venv .venv. This creates a virtualenv in the .venv directory. Activate the virtualenv: . .venv/bin/activate (there are multiple activate scripts for different shells). Once activated, run pip install twine to install twine into the virtualenv, and pip install dist/flatbuffers-1.9.tar.gz to install the package you've just made. Then jump into the interactive interpreter and try importing the package:

$ python
>>> import flatbuffers
>>> (no error)

Once that's done, do the twine upload command, provide your credentials and you're done.

[aardappel]

@Tinche Thanks for the instructions!

Ok, produced the archive, and have a working twine. Made a pypy account (user: wvo). Now @rw just needs to wake up and make me a maintainer :)

[aardappel]

Ok, it has been uploaded.. but it still defaults to showing the 2015 version by default, because of the naming? There appears to be no way to change that.

[tarehart]

Yeah, it's probably because of the naming. I guess people who just do pip install flatbuffers won't get the latest version, but personally I don't mind because any sane client is going to specify the version anyway. Thanks for doing the release!

[Tinche]

You can hide the old versions I think. It's probably parsing the version string as a tuple of integers, and 2015 > 1.

[aardappel]

There are only options for delete (which would probably be a bad idea), not hide.

Would be good to fix this, as this could have a lot of people thinking the old versions are current, and then going thru the pain of finding out stuff doesn't work.

Kinda surprised pypi has no way to fix this.

[tarehart]

I'm getting some odd behavior when trying to install 1.9 via pip:

> pip install flatbuffers==1.9 --no-cache-dir
Collecting flatbuffers==1.9
  Downloading https://files.pythonhosted.org/packages/13/21/68e38fddb9271ae4f070bdd9315b44efbb37f46c3ec11bcbe56240578865/flatbuffers-1.9.tar.gz
  Requested flatbuffers==1.9 from https://files.pythonhosted.org/packages/13/21/68e38fddb9271ae4f070bdd9315b44efbb37f46c3ec11bcbe56240578865/flatbuffers-1.9.tar.gz#sha256=5f9f88ecac3d44909eada8614920b3765ff4315a1f6f0fec7898eb337902eae2, but installing version 20180617224558
Installing collected packages: flatbuffers
  Running setup.py install for flatbuffers ... done
Successfully installed flatbuffers-20180617224558

When using flatbuffers==1.9 as an install_requires in my own package, I get this error when installing: my-package 0.0.5 has requirement flatbuffers==1.9, but you'll have flatbuffers 20180617224558 which is incompatible.

This does not seem to be fatal, but I'm worried about hidden consequences in the future.

Is it possible something went awry with the versioning?

[aardappel]

@tarehart it seems PyPi can't really deal with how we changed the versioning scheme.

At this point it seems almost better to me to delete the old versions. That would cause people that depend on just flatbuffers (no version) to be silently upgraded though.. which may cause them subtle errors? @rw?

[Tinche]

No, the problem is the setup.py is set up to try to fish out the version from an environment variable (and from the date when that fails) when it's run, and it gets run when you install the default type of Python package (a source distribution, what we're uploading). That's simply how Python packages work.

There is another kind of Python package called a wheel. They are a little different in that no code gets run when you install from a wheel, just files get copied. (They can also be platform-specific and contain binary code for their platforms, that's how you can for example install numpy on Windows/macOS/Linux without a compiler. But that's beside the point.)

The downside of wheels is you need a slightly newer pip version to install them. But they have existed for a long time now so I don't think that's an issue any more. Would you like instructions on how to build a wheel (it's very similar)? We can have both wheels and sdists (.tar.gz) up on PyPI at the same time.

[aardappel]

@Tinche sure. Can future releases then be wheel-only? Does publishing a wheel do anything to correct the current mess?

[haroal]

To build a wheel, it is almost the same thing as building a source package like you did a few days ago: you just have to run python setup.py bdist_wheel instead of python setup.py sdist. It will create a .whl package in the dist/ folder. Then you just have to upload it like you did before. Be sure to have a VERSION="1.9" environment variable to create the wheel with the right version.

Be aware that wheels are dependent on the python version you use: if you create it using python 3, it will works on python 3 only (same thing for python 2). So if you want flatbuffers to be compatible with python 2 and python 3, you have to create a wheel using python 3 and then do the same thing using python 2.

You can release only wheels, but it is recommended by pypi to also release a source package to be compatible with old pip versions (that don't handle wheels).

You should always upload a source archive and provide built archives for the platforms your project is compatible with. In this case, our example package is compatible with Python on any platform so only one built distribution is needed.
Source: Python doc

Publishing a wheel won't do anything to correct the current mess because it will be added to the "1.9" version on the Pypi repo. I think the best way to clear it, is to download the old versions, remove them from pypi and reupload them with a 0.x version, in order to be sure that Pypi considers them as old packages. You should also add a message to indicate that they are equivalent to the 2015.x versions. This way, people who use them will have an error (only if they reinstall their dependencies, meaning that prod should continue working) but they will just have to set the version number to the new one to make their code working again, without forcing them to upgrade it.

[Tinche]

Sorry, I didn't have the time to respond until now.

Adding to what haroal wrote,

you will need to pip install wheel before using python setup.py bdist_wheel. (To install the actual library for making wheels.)

If you create a setup.cfg file next to setup.py and put this in it:

[bdist_wheel]
universal=1

you will create a universal wheel (both Py2 and Py3). Then you can upload just that.

I think uploading wheels will actually solve the version thing, but we should test it out beforehand. The current problem is that executing setup.py reads or generates the version when it's run, and source distributions work by running the setup.py both when building the distribution and when installing it. Wheels gets their version during building and put it metadata.json inside the archive.

Here's a way to test. Generate both a sdist and a wheel. Then remove the VERSION environment variable and install the actual package into a virtual environment. Then look at what the environment thinks what the version is.

> python3 -m venv .venv
> . .venv/bin/activate.fish
(.venv) > pip install dist/flatbuffers-1.9.tar.gz
(.venv) > pip list
DEPRECATION: The default format will switch to columns in the future. You can use --format=(legacy|columns) (or define a format=(legacy|columns) in your pip.conf under the [list] section) to disable this warning.
--- flatbuffers (20180622121247) ---
pip (9.0.3)
setuptools (39.0.1)

Now for the wheel:

> python3 -m venv .venv
> . .venv/bin/activate.fish
(.venv) > pip install dist/flatbuffers-1.9-py2.py3-none-any.whl
(.venv) > pip list
DEPRECATION: The default format will switch to columns in the future. You can use --format=(legacy|columns) (or define a format=(legacy|columns) in your pip.conf under the [list] section) to disable this warning.
--- flatbuffers (1.9) ---
pip (9.0.3)
setuptools (39.0.1)

[aardappel]

Thanks for the guidance, got as far as:

wvo@wvo ~/r/v/u/l/f/python> . .venv/bin/activate.fish
(.venv) wvo@wvo ~/r/v/u/l/f/python> pip install dist/flatbuffers-1.9.tar.gz
Processing ./dist/flatbuffers-1.9.tar.gz
Building wheels for collected packages: flatbuffers
  Running setup.py bdist_wheel for flatbuffers ... error
  Complete output from command python/.venv/bin/python3 -u -c "import setuptools, tokenize;__file__='/tmp/pip-b8a92e4t-build/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" bdist_wheel -d /tmp/tmpgyvmbmiipip-wheel- --python-tag cp35:
  VERSION environment variable not set, using datetime instead: 20180625194528
  usage: -c [global_opts] cmd1 [cmd1_opts] [cmd2 [cmd2_opts] ...]
     or: -c --help [cmd1 cmd2 ...]
     or: -c --help-commands
     or: -c cmd --help

  error: invalid command 'bdist_wheel'

[aardappel]

(and haven't uploaded the .whl yet)

[vishvananda]

did you see the comment about running pip install wheel? I think you need to run that to make the bdist_wheel command available

[aardappel]

@vishvananda yes, I got:

-$ pip install wheel
Requirement already satisfied: wheel in /usr/lib/python2.7/dist-packages (0.29.0)

I guess I also need to do this in the virtual env.. that wasn't obvious to me :)

[aardappel]

Ok, getting similar output to you, flatbuffers (20180625205540) for the sdist, flatbuffers (1.9) for the wheel. So should I upload the wheel?

[aardappel]

Ok, uploaded.

[aardappel]

To commemorate the 1.10.0 releases I am doing right now, I also deleted the old releases from 2015 that kept showing up on pypi as most recent: https://pypi.org/manage/project/flatbuffers/releases/

I know I am potentially breaking people, but it didn't seem like there were any other options.

[amiralia]

Are flatbuffer releases cross version compatible? E.g., can a flatbuffer written by version 1.10 be read by version 2015.*?

[aardappel]

@amiralia Yes, FlatBuffers are both forward and backwards compatible between the earliest releases and the latest versions, of all languages.

[stale[bot]]

This issue has been automatically marked as stale because it has not had activity for 1 year. It will be automatically closed if no further activity occurs. To keep it open, simply post a new comment. Maintainers will re-open on new activity. Thank you for your contributions.

[mikeholler]

It looks like you guys are publishing again, so time to close this.

@aardappel I will say I think deleting artifacts from PyPi is a horrible idea. That kind of action is how the leftpad incident occurred. The storage on PyPi is free, so was it just aesthetic reasons you deleted it? I really don't mind old and weird numbered versions being present in an artifact repository, in fact, I really would hope they stay there forever. I'm glad non of the software we use seems to be transitively affected by this decision (we wouldn't have had a direct problem with it, since we've moved far away from that version, and besides host our own lazy caches).