Warehouse: [Project-scoped API tokens] are too big for travis pypi deploy

[graingert]

Describe the bug

data too large - consider using travis encrypt-file or travis env set

Expected behavior
travis encrypt "pypi:token-here"
it prints a nice encrypted token

To Reproduce
$ travis encrypt "pypi:token-here"
data too large - consider using travis encrypt-file or travis env set

[webknjaz]

Interesting... Worked for me using a account-global token. Haven't tried it with a project-scoped one. Is it longer?

[graingert]

Mine was 195 characters

[webknjaz]

It looks like Travis CI limitation is 501 byte. You're probably doing something wrong. Are you using --com or --org?

$ dd if=/dev/urandom | LC_CTYPE=X tr -cd "[:alpha:][:digit:]" | head -c 256 | travis encrypt -r xxx/xxx --api-endpoint 'https://api.travis-ci.com/'
Please add the following to your .travis.yml file:

  secure: "VW5SHPo68Mz40zMlDJRwqgh5bsTzItdhDSAsK+Xozb64Z/4GOUJ++iWLFB7eqMA/pS9+FkuwUcBNBxB/DYCJ2jJRb3UxnF3JgBvsgNsi0FQAgvhzz3vKSfWZoTn9c1/jK8Ng1KmqJDkcwjC77EawUo8dbUz19Iw5eyES3803j/Q68luzO1IipZhQLrJRnCV0YhZTnCpoluOoePdv0GG/BU15PVMJIqn3B34fJk6mXrAfrhrjgsp2wt0XAj4e7cQEk80fZsM595mS6qgtLrNP4ZdIA6eSoYuaZ1oeK10wxuOz79wLkEv5LifdZaU5rEmn9grT+35B6Cekkk5rjotPQXgTKGByXWCQywvhYxpQU7zonZr3AeVgVMdFcFnNtxrDbN+dsyR6SwwyEUIBazxZyxN/ITTCHJLAad/1C9MPIY/SzZzNRYkwSbAjh+IxEcANm6Pitvf8UeP6bmII/61ovgPI0t3nxbtA2cnCaYeP6fbfqOBwdYc1ISBqR5w4pR85HBtZM7MI7qq+nME1i3UUG+j0MyixUIB10vOI1XnumxaQ2AqTOIoi3vm/4gDEay+S2YnWq7jMD4xCW6rQ+8b7lZWQhpRoig9rv6yb16oUOZsScCCwOkxbA4k33H2v+PbvBrFlvb8jKALYnaWqFlCU9WB4rIUGCjJ7NZcZ7I0h37U="

Pro Tip: You can add it automatically by running with --add.

$ dd if=/dev/urandom | LC_CTYPE=X tr -cd "[:alpha:][:digit:]" | head -c 502 | travis encrypt -r xxx/xxx --api-endpoint 'https://api.travis-ci.com/'
Outdated CLI version, run `gem install travis`.
data too large - consider using travis encrypt-file or travis env set

$ dd if=/dev/urandom | LC_CTYPE=X tr -cd "[:alpha:][:digit:]" | head -c 501 | travis encrypt -r xxx/xxx --api-endpoint 'https://api.travis-ci.com/'
Outdated CLI version, run `gem install travis`.
Please add the following to your .travis.yml file:

  secure: "t7P2dCpHE7x7nknQvUxGVSC5WFjTzDu2vnd59I26cG7jszeyEeE4J9sJctgwkHq5GGyiE24tZqsrfW4Nf6n3oS/S/KQRJb8zS3fSwnSlXDyrIDeH319/5i2qxolTZsXFKVkhNi9pzaAHCmFV2Id/VqjgFO9DegpuChObckJ8LGGhrAb7snRSwCtBAuAV0yE2sUsyhqwkKiglY4GSV/W+lswR+hxdkDlaqgA/uODSNWIMaFeuAUTwtLR1owpDyai5J9xTEk06PvYIWvOHvnu5Mcdck8rcNWyjUJiPGQEw2cFklH0JDuDSeAHULc/K3c0LMvXdN5HtPJMphuAH/7PKy4BOgIGIa40C9Hu4q1Tx9Y0jImOVUgywvWThFUBobxTsWLo0IFjRTe6wRfKR2/nzprnUiwQ1ouasynpF8/PDsuPczIm8kAUv+S9NM+7mLR+BFEg98f37wcxwYAM+nDqU8WkJjGxczoQybF0Xx1KW4QrFH7CrJfiQ4Urd7b6EQF58L5fD24T2SgIIA84r0bF0DWLUKvHMJifW8vRDPlThKXev3xwGti02EpUcnYATrWVOd5oJs1PBfirjp0zFRIE8GAjnBoVTQlAnG8JFYlvzJp+DdlB4NXY16VcIcyJ6kcWRUFDBKttJlmY7NnWxNuGQZ60kg11YjRGfhEju17KW6oQ="

Pro Tip: You can add it automatically by running with --add.

[webknjaz]

Workaround:

It's not very well advertised but you can actually use PYPI_PASSWORD env var.
DPL tool supports it as a fallback, see: https://github.com/travis-ci/dpl/blob/1077ce6/lib/dpl/provider/pypi.rb#L12.
You can set it as a secret var (which I think supports data of size up to 4096 bytes) on the Settings page of your repo @ travis-ci.com.

[brainwane]

Not sure whether we should consider this a documentation issue to address in https://github.com/travis-ci/docs-travis-ci-com/issues/2452 & #6211 or a thing we can/should address on the Warehouse side. @woodruffw ?

[graingert]

I'm using the travis snap if that helps anyone repeat?

[webknjaz]

@graingert I'm not sure what's packaged into that snap and don't have it myself :(
Did you notice any weird special chars in your token?

[woodruffw]

Hmm -- this probably falls under documentation, since we're already using the more compact ("V2") text serialization format for Macaroons. We could probably squish it further by using a binary format for caveats instead of JSON, but it'll still be a relatively long token.

I agree with @webknjaz that there might be something additional going on here: 195 characters is long, but well below the actual Travis CI limit.

[graingert]

No nothing weird, just the :

[brainwane]

@graingert Could I ask you to contact Travis CI support or their bug tracker and ask what they find, in case there's something else going on here?

[graingert]

I suspect this is a unique problem to me. I'm using travis set env now

[webknjaz]

@graingert I believe it's not safe to use that.

[webknjaz]

@graingert heads up: the issue seems to be a duplicate of https://github.com/pypa/warehouse/issues/6287

[jaraco]

This problem isn't unique to you. I've encountered it in a couple of projects myself. It seems some projects in Travis have a 128-byte key while others a 512-byte key (and maybe others with another length). The only recommended solution seems to be "don't use encrypted values in config".

[jaraco]

the issue seems to be a duplicate of #6287

I don't see how this is a duplicate of that issue. That issue discussed the escaping of values (@token and pypi:), both of which were changed to avoid those issues.

This issue is about how Travis config refuses to encrypt the value for certain projects, which was reported upstream in travis-ci/travis.rb#221, but the recommendation is to avoid using that feature.

[jaraco]

Although #6355 is a duplicate of this issue, the conversation is continuing there.

[webknjaz]

Well, it seemed to have the same roots back when I suggested that it's a duplicate ;)

[ssbarnea]

To be honest the pypi token looks like a joke made in order to test who would choke trying to use it. Its unresonable length breaks even the newer v3 travis deploy api, as documentd at https://docs.travis-ci.com/user/deployment-v2/providers/pypi/

Any attempt to encrypt that huge token would give you a data too large - consider using travis encrypt-file or travis env set