Dpl: deploy to PyPI is still broken due to sudo permissions

[ssbarnea]

It seems that the deployer still fails to work due to failure to install dependencies.

Example: https://travis-ci.com/pycontribs/jira/jobs/133196517#L615

Installing deploy dependencies
Successfully installed dpl-pypi-1.9.7
...
    Uninstalling pip-9.0.1:
Could not install packages due to an EnvironmentError: [Errno 13] Permission denied: '/usr/local/bin/pip'
Consider using the `--user` option or check the permissions.
Couldn't install pip, setuptools, twine or wheel.
failed to deploy

I found https://github.com/travis-ci/travis-ci/issues/7735 temporary workaround but this does not work either.

deploy:
  - provider: pypi
    ....
    edge:
      branch: twine-user-install

How can we publish to pypi using deploy? Another question that I have is how is it possible that such a simple use-case is not covered by internal testing? Clearly this is a bug in deployer code, one that even it fails to install.

[webknjaz]

Your workaround for you might be to pip install all those packages on before_deploy step. But in general you should use language: python.
Also, you might use a separate stage with deployment job in it.

[ssbarnea]

I managed to fix other issues dpl fixing branch but somehow every time i have a success I encounter something new and broken.

For example now the task fails after doing the deployment to pypi, apparently during deinstallation of dpl,... with a very weird error, one that makes me believe that stderr/stdout gets mingled between commands.

Deploying application
running sdist
[pbr] Writing ChangeLog
[pbr] Generating ChangeLog
[pbr] ChangeLog complete (0.0s)
[pbr] Generating AUTHORS
[pbr] AUTHORS complete (0.0s)
running egg_info
writing requirements to jira.egg-info/requires.txt
writing jira.egg-info/PKG-INFO
writing top-level names to jira.egg-info/top_level.txt
writing dependency_links to jira.egg-info/dependency_links.txt
writing entry points to jira.egg-info/entry_points.txt
writing pbr to jira.egg-info/pbr.json
[pbr] Processing SOURCES.txt
[pbr] In git context, generating filelist from git
warning: no previously-included files matching '*.pyc' found anywhere in distribution
reading manifest template 'MANIFEST.in'
writing manifest file 'jira.egg-info/SOURCES.txt'
running check
creating jira-2.0.0.0rc6
creating jira-2.0.0.0rc6/.github
creating jira-2.0.0.0rc6/.github/ISSUE_TEMPLATE
creating jira-2.0.0.0rc6/.idea
creating jira-2.0.0.0rc6/dist
creating jira-2.0.0.0rc6/docs
creating jira-2.0.0.0rc6/docs/_static
creating jira-2.0.0.0rc6/docs/extra
creating jira-2.0.0.0rc6/docs/templates
creating jira-2.0.0.0rc6/examples
creating jira-2.0.0.0rc6/hooks
creating jira-2.0.0.0rc6/jira
creating jira-2.0.0.0rc6/jira.egg-info
creating jira-2.0.0.0rc6/jira/utils
copying files to jira-2.0.0.0rc6...
copying .coveragerc -> jira-2.0.0.0rc6
copying .editorconfig -> jira-2.0.0.0rc6
copying .gitchangelog.rc -> jira-2.0.0.0rc6
copying .pre-commit-config.yaml -> jira-2.0.0.0rc6
copying .travis.yml -> jira-2.0.0.0rc6
copying .yamllint -> jira-2.0.0.0rc6
copying AUTHORS -> jira-2.0.0.0rc6
copying AUTHORS.rst -> jira-2.0.0.0rc6
copying ChangeLog -> jira-2.0.0.0rc6
copying LICENSE -> jira-2.0.0.0rc6
copying MANIFEST.in -> jira-2.0.0.0rc6
copying Makefile -> jira-2.0.0.0rc6
copying README.rst -> jira-2.0.0.0rc6
copying codecov.yml -> jira-2.0.0.0rc6
copying cspell.json -> jira-2.0.0.0rc6
copying package.json -> jira-2.0.0.0rc6
copying pytest.ini -> jira-2.0.0.0rc6
copying release.sh -> jira-2.0.0.0rc6
copying requirements-dev.txt -> jira-2.0.0.0rc6
copying requirements-opt.txt -> jira-2.0.0.0rc6
copying requirements.txt -> jira-2.0.0.0rc6
copying setup.cfg -> jira-2.0.0.0rc6
copying setup.py -> jira-2.0.0.0rc6
copying test.local -> jira-2.0.0.0rc6
copying tox.ini -> jira-2.0.0.0rc6
copying .github/ISSUE_TEMPLATE/bug_report.md -> jira-2.0.0.0rc6/.github/ISSUE_TEMPLATE
copying .github/ISSUE_TEMPLATE/feature_request.md -> jira-2.0.0.0rc6/.github/ISSUE_TEMPLATE
copying .idea/codeStyleSettings.xml -> jira-2.0.0.0rc6/.idea
copying dist/.gitignore -> jira-2.0.0.0rc6/dist
copying docs/Makefile -> jira-2.0.0.0rc6/docs
copying docs/advanced.rst -> jira-2.0.0.0rc6/docs
copying docs/api.rst -> jira-2.0.0.0rc6/docs
copying docs/conf.py -> jira-2.0.0.0rc6/docs
copying docs/contributing.rst -> jira-2.0.0.0rc6/docs
copying docs/examples.rst -> jira-2.0.0.0rc6/docs
copying docs/index.rst -> jira-2.0.0.0rc6/docs
copying docs/installation.rst -> jira-2.0.0.0rc6/docs
copying docs/jirashell.rst -> jira-2.0.0.0rc6/docs
copying docs/requirements-rtd.txt -> jira-2.0.0.0rc6/docs
copying docs/_static/.placeholder -> jira-2.0.0.0rc6/docs/_static
copying docs/_static/python-16.png -> jira-2.0.0.0rc6/docs/_static
copying docs/_static/python-256.png -> jira-2.0.0.0rc6/docs/_static
copying docs/_static/python-32.png -> jira-2.0.0.0rc6/docs/_static
copying docs/_static/python-64.png -> jira-2.0.0.0rc6/docs/_static
copying docs/extra/jira.xml -> jira-2.0.0.0rc6/docs/extra
copying docs/templates/.placeholder -> jira-2.0.0.0rc6/docs/templates
copying examples/basic_auth.py -> jira-2.0.0.0rc6/examples
copying examples/basic_use.py -> jira-2.0.0.0rc6/examples
copying examples/cookie_auth.py -> jira-2.0.0.0rc6/examples
copying examples/greenhopper.py -> jira-2.0.0.0rc6/examples
copying hooks/pre-commit -> jira-2.0.0.0rc6/hooks
copying jira/__init__.py -> jira-2.0.0.0rc6/jira
copying jira/client.py -> jira-2.0.0.0rc6/jira
copying jira/config.py -> jira-2.0.0.0rc6/jira
copying jira/exceptions.py -> jira-2.0.0.0rc6/jira
copying jira/jirashell.py -> jira-2.0.0.0rc6/jira
copying jira/resilientsession.py -> jira-2.0.0.0rc6/jira
copying jira/resources.py -> jira-2.0.0.0rc6/jira
copying jira.egg-info/PKG-INFO -> jira-2.0.0.0rc6/jira.egg-info
copying jira.egg-info/SOURCES.txt -> jira-2.0.0.0rc6/jira.egg-info
copying jira.egg-info/dependency_links.txt -> jira-2.0.0.0rc6/jira.egg-info
copying jira.egg-info/entry_points.txt -> jira-2.0.0.0rc6/jira.egg-info
copying jira.egg-info/not-zip-safe -> jira-2.0.0.0rc6/jira.egg-info
copying jira.egg-info/pbr.json -> jira-2.0.0.0rc6/jira.egg-info
copying jira.egg-info/requires.txt -> jira-2.0.0.0rc6/jira.egg-info
copying jira.egg-info/top_level.txt -> jira-2.0.0.0rc6/jira.egg-info
copying jira/utils/LICENSE -> jira-2.0.0.0rc6/jira/utils
copying jira/utils/__init__.py -> jira-2.0.0.0rc6/jira/utils
Writing jira-2.0.0.0rc6/setup.cfg
Creating tar archive
removing 'jira-2.0.0.0rc6' (and everything under it)
running bdist_wheel
running build
running build_py
installing to build/bdist.linux-x86_64/wheel
running install
[pbr] Writing ChangeLog
[pbr] Generating ChangeLog
[pbr] ChangeLog complete (0.0s)
[pbr] Generating AUTHORS
[pbr] AUTHORS complete (0.0s)
running install_lib
creating build/bdist.linux-x86_64/wheel
creating build/bdist.linux-x86_64/wheel/jira
copying build/lib/jira/__init__.py -> build/bdist.linux-x86_64/wheel/jira
copying build/lib/jira/client.py -> build/bdist.linux-x86_64/wheel/jira
copying build/lib/jira/config.py -> build/bdist.linux-x86_64/wheel/jira
copying build/lib/jira/exceptions.py -> build/bdist.linux-x86_64/wheel/jira
copying build/lib/jira/jirashell.py -> build/bdist.linux-x86_64/wheel/jira
copying build/lib/jira/resilientsession.py -> build/bdist.linux-x86_64/wheel/jira
copying build/lib/jira/resources.py -> build/bdist.linux-x86_64/wheel/jira
creating build/bdist.linux-x86_64/wheel/jira/utils
copying build/lib/jira/utils/__init__.py -> build/bdist.linux-x86_64/wheel/jira/utils
copying build/lib/jira/utils/LICENSE -> build/bdist.linux-x86_64/wheel/jira/utils
running install_egg_info
Copying jira.egg-info to build/bdist.linux-x86_64/wheel/jira-2.0.0.0rc6-py2.7.egg-info
running install_scripts
creating build/bdist.linux-x86_64/wheel/jira-2.0.0.0rc6.dist-info/WHEEL
creating '/home/travis/build/pycontribs/jira/dist/jira-2.0.0.0rc6-py2.py3-none-any.whl' and adding '.' to it
adding 'jira/__init__.py'
adding 'jira/client.py'
adding 'jira/config.py'
adding 'jira/exceptions.py'
adding 'jira/jirashell.py'
adding 'jira/resilientsession.py'
adding 'jira/resources.py'
adding 'jira/utils/LICENSE'
adding 'jira/utils/__init__.py'
adding 'jira-2.0.0.0rc6.dist-info/entry_points.txt'
adding 'jira-2.0.0.0rc6.dist-info/pbr.json'
adding 'jira-2.0.0.0rc6.dist-info/top_level.txt'
adding 'jira-2.0.0.0rc6.dist-info/WHEEL'
adding 'jira-2.0.0.0rc6.dist-info/METADATA'
adding 'jira-2.0.0.0rc6.dist-info/RECORD'
removing build/bdist.linux-x86_64/wheel
Uploading distributions to https://upload.pypi.org/legacy/
Uploading jira-2.0.0.0rc6-py2.py3-none-any.whl
100%|##########| 62.3k/62.3k [00:01<00:00, 48.4kB/s]
Uploading jira-2.0.0.0rc6.tar.gz
100%|##########| 97.2k/97.2k [00:01<00:00, 75.5kB/s]
dpl_1
$ rvm $(travis_internal_ruby) --fuzzy do ruby -S gem uninstall -aIx dpl
Removing dpl
Successfully uninstalled dpl-1.9.7
1.45s$ rvm $(travis_internal_ruby) --fuzzy do ruby -S gem install dpl
Successfully installed dpl-1.9.7
1 gem installed
invalid option "--file=dist/\nInstalled /home/travis/build/pycontribs/jira/.eggs/MarkupSafe-1.0-py2.7-linux-x86_64.egg\n[pbr] Generating ChangeLog\njira-2.0.0.0rc6.tar.gz"
failed to deploy

Full log at https://travis-ci.com/pycontribs/jira/jobs/133528638

[webknjaz]

@ssbarnea I think I saw some similar race conditions and it looks like the last error line is coming from the second provider you have (Releases). Also be aware that deployment can mutate state and PYPI provider normally cleans up generated dists on tear down.

[ssbarnea]

@webknjaz Indeed, it was releases and after many days I was finally able to release again. Shortly I am really disappointed about the user-experience with "dpl" which seems more of a big-PITA.

Using twine directly worked perfectly from first attempt. Also the github releases provider is buggy and is impossible to include release notes inside it. I am ready to bet the using github official tools would work, just like twine worked for pypi. I used only two dpl providers on my projects and both of them are deeply broken and not updated with bugs left open for years. This should be a reminder to myself that is better to avoid any CI specific features and use the official tools to perform the jobs... long live the good old bash! A wrapper is good only if is actively maintained, which is not the case for dpl.

[webknjaz]

Fair enough. Still, I normally prefer official CI tooling as it fits nicely and is better readable in configs.
For multi-CI it's, of course, a bit more tricky, so now I have some config-generator in mind to help with that, which I hope to start working on soon. It's also nice that you can make dpl run in other CIs as well.

[jayvdb]

@ssbarnea , please edit the issue title to be pypi instead of pypy ;-)

[stale[bot]]

Thanks for contributing to this issue. As it has been 90 days since the last activity, we are automatically closing the issue in 7 days. This is often because the request was already solved in some way and it just wasn't updated or it's no longer applicable. If that's not the case, please respond before the issue is closed, or open a new one after. We'll gladly take a look again! You can read more here: https://blog.travis-ci.com/2018-03-09-closing-old-issues

[ssbarnea]

The introduction if this stale bot drives me insane and makes me hate Travis for doing it. I wonder which manager decided that a brainless bot would be better than a human at filtering bugs, especially when we all know how much attention we got from the team, in general.

And before someone use “it is free” excuse, look at visual studio code project on github, again free and popular. Check how much time it takes for someone for a core to check a bug and so something about it. You will be amazed....

Bugs are getting stale because users are loosing hope!

[BanzaiMan]

@ssbarnea Sorry for the long delay. I also apologize for not reviewing https://github.com/travis-ci/dpl/pull/834 sooner. I will respond to that PR to discuss this further.

Thanks for your patience.

[gaieges]

Seeing this as well here: https://travis-ci.com/hCaptcha/hmt-escrow/builds/89551083

Is there a workaround for this?

[BanzaiMan]

@gaieges See https://github.com/travis-ci/dpl/pull/834#issuecomment-434036325 for a _possible_ fix.

[gaieges]

@BanzaiMan thanks! This seems to have got me past the install problem. I seem to have some others that I need to get through though.

[stale[bot]]

Thanks for contributing to this issue. As it has been 90 days since the last activity, we are automatically closing the issue. This is often because the request was already solved in some way and it just wasn't updated or it's no longer applicable. If that's not the case, please do feel free to either reopen this issue or open a new one. We'll gladly take a look again! You can read more here: https://blog.travis-ci.com/2018-03-09-closing-old-issues