Twine: Recommended workflow for switching between multiple API tokens?

[nlhkabu]

Hi folks

I am working on adding instructions to the 'Add API token' page on PyPI (see https://github.com/pypa/warehouse/pull/6615). Currently, the instructions look like this:

API token for an entire PyPI account

We suggest that users create a .pypirc file in their home directory with the token details:

Token scoped to a PyPI project

We suggest that users can switch between multiple tokens with <code>twine --repository PROJECT_NAME</code>

Questions

1. Is this accurate?
2. Is this the recommended workflow?
3. Am I missing anything that it would be useful to document here?

Thanks :)

[sigmavirus24]

That looks accurate. I'd say that it's a hack but it's the first thing I thought of when I saw your question.

It would probably be good to have another way of managing these though instead of forcing folks to use --repository

[nlhkabu]

Ok, thanks @sigmavirus24. I will go ahead and get these instructions merged into Warehouse.

[bhrutledge]

@sigmavirus24 and @nlhkabu is it okay to close this? And/or should we open an issue for improving the UX for using Twine with API tokens?

[bhrutledge]

FYI @sigmavirus24 and @nlhkabu (and @di for good measure):

After trying this out for myself, I think the instructions might not be sufficient. Given this ~/.pypirc:

[pypi]
username = __token__

[testpypi]
username = __token__

[example-pkg]
repository = https://test.pypi.org/legacy/
username = __token__
password = pypi-...

I get this output:

$ twine upload --repository example-pkg dist/*
InvalidConfiguration: Missing 'example-pkg' section from the configuration file
or not a complete URL in --repository-url.
Maybe you have a out-dated '~/.pypirc' format?
more info: https://docs.python.org/distutils/packageindex.html#pypirc

It works if add I this to the top of~/.pypirc:

[distutils]
index-servers =
    pypi
    testpypi
    example-pkg

I discovered that twine.utils.get_config only uses values for defined index-servers, which defaults to ["pypi", "testpypi"].

https://github.com/pypa/twine/blob/330c0a238b271962e6eecc32ee3749b5a38e2f52/twine/utils.py#L78-L85

A quick search didn't uncover any issues on Warehouse, which surprised me. It seems like a quick addition to the relevant Warehouse docs, but seems to add to the case for improving Twine's handling of tokens.

[bhrutledge]

Related: https://github.com/pypa/twine/issues/565 - Not obvious how to use multiple project API tokens with keyring

[di]

While this works, I agree with @sigmavirus24 that it's a bit of a hack. I think it's probably worthwhile for us to think of a new and better way to do this in the .pypirc file (and maybe standardize the format while we're at it?)

Maybe something like this for repo-wide tokens:

[distutils]
index-servers =
    pypi
    other

[pypi]
token = pypi-ABC...

[other]
repository = http://example.com/pypi
token = pypi-DEF...

And this for project-specific tokens:

[distutils]
index-servers =
    pypi
    other

[other]
repository = http://example.com/pypi

[pypi:example-pkg]
token = pypi-ABC...

[other:example-pkg]
token = pypi-DEF...

[bhrutledge]

Thanks, @di. I wondered how much freedom we might have with the .pypirc format. I like how this removes the need for repository = https://upload.pypi.org/legacy/ in the config for project-scoped tokens.

FYI, there's an evolving discussion that started at https://github.com/pypa/packaging.python.org/issues/297#issuecomment-576409783, where @jaraco and @takluyver have proposed some ideas on how to make this work with keyring and flit.

[di]

I could be wrong, but I think these extra fields/sections would be ignored by any tool that doesn't know about them, so it should be fully backwards-compatible.

[bhrutledge]

Marked this and #565 as blocked until the discussion started at https://github.com/pypa/packaging.python.org/issues/297#issuecomment-576409783 is resolved.

[sigmavirus24]

The other thing to keep in mind is that the ini format is fairly flexible and projects like pytest and flake8 are able to do things like:

option =
   something: value
   something-else: another-value

So we could keep this super simple and have a:

tokens =
   *: pypi<general token>
   twine: pypi<project-token>
   ...

That keeps us from proliferating sections into the file, keeps the implementation simple, and is farily explicit without trying to hack around things. Then we can select the tokens out of there as necessary without doing "repository" based nonsense. Getting this to work with Keyring will be harder, as the discussion Brian linked indicates

[di]

@sigmavirus24 How would that handle the same project name with different tokens across multiple repos?

[sigmavirus24]

@di A more fully fledged example would be:

[distutils]
index-servers =
    pypi
    other
    another

[pypi]
tokens = 
   *: pypi-ABC...
   twine: pypi-BCD...
   etc: pypi-CDE...

[another]
repository = http://example.com/pypi
tokens = 
   *: pypi-DEF...
   twine: pypi-EFG...
   etc: pypi-FGH...

[other]
repository = http://test.example.com/pypi
tokens = 
    *: pypi-GHI...
    etc: pypi-HIJ...

[di]

Gotcha, so with that example, if the user just wanted to specify repo-wide tokens (which is probably the most common use case? maybe?) they'd have to write a minimum of:

[distutils]
index-servers =
    pypi
    other

[pypi]
tokens = 
   *: pypi-ABC...

[other]
repository = http://test.example.com/pypi
tokens = 
    *: pypi-GHI...

which feels clunky compared to:

[distutils]
index-servers =
    pypi
    other

[pypi]
token = pypi-ABC...

[other]
repository = http://example.com/pypi
token = pypi-DEF...

[bhrutledge]

From https://github.com/pypa/packaging.python.org/issues/297, it seems like the ideal scenario is that tokens are not stored in .pypirc, but rather in keyring. Does that influence the choice of format?

[sigmavirus24]

I think you're always going to have people who would prefer not to use keyring. And we'd have to go through a lengthy deprecation period to kill that off which isn't worth the effort frankly.

Gotcha, so with that example, if the user just wanted to specify repo-wide tokens (which is probably the most common use case? maybe?) they'd have to write a minimum of:

The fact that we don't know what the most common use-case is suggests that we don't know enough to continue pushing forward on making the right choice

[di]

As of right now, 60% of PyPI users who own/maintain at least package only have one package:

warehouse=> select count(*) from (select roles.user_id, count(roles.project_id) as
count from roles group by user_id) as projects;
 count
--------
 103550
(1 row)

warehouse=> select count(*) from (select roles.user_id, count(roles.project_id) as
count from roles group by user_id) as projects where projects.count = 1;
 count
-------
 62803
(1 row)

[bhrutledge]

Related (I think): I just put up a draft PR that documents the current spec of .pypirc: https://github.com/pypa/packaging.python.org/pull/734.

Also, we're talking a lot about .pypirc, but I'm curious how this would manifest in configuration via the command line and/or environment variables. For example, does this imply --token and TWINE_TOKEN? It seems friendlier for those to be aligned.

I'm wondering if it might be simpler to keep the current convention, but mitigate the hack by defaulting username to __token__, as suggested in https://github.com/pypa/twine/issues/561.

I think that would also require fewer updates to the docs in Twine, PPUG, and Warehouse.

[sigmavirus24]

As of right now, 60% of PyPI users who own/maintain at least package only have one package:

Is there a way to tell what % of that 60% use tokens at all?

Here's the point: If we go with token= then we're implicitly endorsing 1 global token that everyone should be using and thus we're going to calcify that use-case. Is the goal of project-scoped tokens not to (beyond enabling CI) to also encourage keeping tokens scoped appropriately to projects?

We should be meeting folks where they are today, certainly, but we should also be striving to push the community forward with the gentlest nudges we can think of.

[bhrutledge]

@sigmavirus24 I think @di's original suggestion handles project-scoped tokens via additional sections that inherit from the "base" section:

[pypi]
token = pypi-ABC...

[pypi:example-pkg]
token = pypi-DEF...

Another idea:

[pypi]
token = pypi-ABC...
project_tokens = 
    example-pkg: pypi-DEF...

That said, I'd still like to consider sticking with the current username/password hack, such that Twine's default configuration becomes:

[distutils]
index-servers =
    pypi
    testpypi

[pypi]
repository = https://upload.pypi.org/legacy/
username = __token__

[testpypi]
repository = https://test.pypi.org/legacy/
username = __token__

which I think leans more towards something like this:

[pypi]
password = pypi-ABC...

[pypi:example-pkg]
password = pypi-DEF...

[bhrutledge]

Building on my last suggestion, combined with my second proposal in https://github.com/pypa/twine/issues/565 and https://github.com/pypa/packaging.python.org/issues/297#issuecomment-581061098, I think this would allow CLI commands like:

$ keyring set pypi:example-pkg __token__
Password for '__token__' in 'pypi:example-pkg':

$ twine upload -r pypi:example-pkg dist/*

Aside: looking over those other issues makes me wonder if this is conversation should be moved to https://discuss.python.org/c/packaging/.

[di]

A few things on the password/token "hack":

1) I'd like us to imagine a world where using password in .pypirc is deprecated or discouraged, and I think that overloading password with the token value would make that hard or impossible.

2) I think that a default username of __token__ might produce some confusing behavior in some edge cases, for example consider the following:

```ini
[pypi]
usenrame = my_username
password = my_password
```

Twine now has no way to tell the user their username is "missing" because it happily falls back on the default of `__token__`, which fails authentication, even though the user has included the correct username/password (unless we want to start warning the user _every_ time Twine uses the default, which I think would get annoying for users who legitimately want to use tokens)

3) Using token over password allows the user to explicitly tell us "I expect to use tokens" which allows us to potentially do some preflight checking of this value (i.e. whether it's a valid token). Otherwise, we'd have to implicitly infer this through how the user is using the username field.

4) Having a clear delineation between what is a token and what is a password in configuration will help provide more context to whatever mechanism implements #362, which would allow an authentication provider to provide a different value based on what's being requested. E.g.:

```ini
[pypi]
password_provider = keyring

[other]
token_provider = some_other_module
```

(where we can probably just consider `keyring` the default password/token provider, but I'm including it here just as an example)