Warehouse: Allow git + ssh based uploads

Created on 16 Aug 2017 · 6Comments · Source: pypa/warehouse

Major Components

systems / infrastructure to allow pushing a git repository, rather than a collection of files
systems / infrastructure for ssh authentication

General Description

There's a lot of work required here, and likely bits of it would belong in

/ where-ever else, instead. But I figured I would start here.

My goal here is the ability to have a .pypirc file like so

[pypi]
[email protected]:pypi/upload

Which allows a upload command like

python setup.py git upload

And all the associated systems / infrastructure that would enable this functionality

Stretch Goals

This is a technical precursor that would open up a variety of pypi integrations, such as a "one click" "sync this repository with pypi" button on github that publishes the default branch. Or also an option where pypi auth is linked to github ssh keys, and github repo access, so being added as a github repo contributor allows pypi push access.

While integrations like that are clearly something the pypa org does not have direct control over, I think creating the technical infrastructure to allow it would be a huge boost for pip / python usage.

Sidenote

I personally know a few bundler and npm devs that could assist with implementation planning if no pypa members have the capacity for it

feature request

Source

lynncyrin

🎉1 👍1

All 6 comments

This is an interesting idea, although it's unlikely going to happen in the short term largely because I'm the only person and my main focus right now is getting feature parity in the rewrite of PyPI into the Warehouse code base. If someone else wanted to lead the effort on this, I'm more than happy to participate in the design work though!

I have a couple thoughts/concerns/questions about the idea in general though:

Currently there are solutions like TravisCI which can publish branches to PyPI automatically, essentially acting as a middleman that can translate between what Github authentication supports and what PyPI supports. What deficiencies exist in these existing solutions that we would be solving with something that is more built in?
To go from a VCS checkout to a sdist requires executing arbitrary code (currently whatever is in the setup.py, in the future this will have a different entrypoint). This means that any feature which allows someone to push a VCS and have PyPI generate the packages means we're suddenly going to be required to execute arbitrary, untrusted code. This is a a nontrivial amount of new, ongoing maintenance work unless there's some existing toolchain for running arbitrary untrusted code we can re-use.
TOFO for SSH is a pretty unsatisfying trust model for something like this... what, if any, mechanisms exist for allowing us pre-trust the SSH endpoints. The main thing I can think of is bundling a SSH key, but I feel like ideally if we're doing this, then we should just enable git push pypi.
To the above, why python setup.py git upload instead of git push pypi?

Those are my initial questions/concerns after reading this, and really the biggest hurdle I see is going to be the whole untrusted code bit.

dstufft on 16 Aug 2017

Yea I figured getting the shiny new https://pypi.org/ out was the highest priority for ya'll. I'm hoping I can fill out one section of this plan well enough to create a PR myself.

The main issue (if one could describe it as an issue) with Travis / Jenkins / custom setups is the developer setup time they require. That setup time is minuscule, but does still block some of the flashier "sync to pypi!" style platform integrations. That said, I'm not convinced about the value of this portion - unless a github / heroku / travisci engineer shows up in this issue.
How about a pure data file as an alternative to setup.py? Either built on top of Manifest.in or in a new file. I feel that would be less maintenance burden and security risk than setting up a system for executing setup.py.
(the SSH bit is where I'm weakest, so I'm going to drop it from the things I personally want to work on)
git push pypi is a tons better option than python setup.py git upload, yea!

So, to re-access, I'm dropping my personal scope down to

a pure data file format for python package info
git push pypi, and having that option only accept packages using the file from (1)

lynncyrin on 17 Aug 2017

Thanks for your suggestion, @lynncyrin!

You might already know: the folks working on Warehouse have gotten some limited funding via Mozilla's Open Source Support program to concentrate on improving and deploying Warehouse, and have kicked off work towards our development roadmap, focusing on getting it to the point where we can redirect pypi.python.org to pypi.org. Since this feature isn't something that the legacy site has, I've moved it to a future milestone.

Since, as you said, this feature is something that spans a few different codebases, you might also want to check out pypa/packaging-problems#60 and pypa/twine#216.

Is there any chance you will be at the PyCon sprints in Cleveland in May? Several people working on Python packaging and distribution will be sprinting there and I think it'd be a good time to work on stuff like this.

Thanks again for the issue and the discussion!

And @Mariatta, this is the issue I mentioned in reply to your tweet.