Dpl: support github release :body

[mockitoguy]

Hey guys,

Thanks for awesome tooling! Please consider adding support for :body when creating release in github.

http://rdoc.info/gems/octokit/Octokit/Client/Releases#create_release-instance_method
https://github.com/travis-ci/dpl/blob/master/lib/dpl/provider/releases.rb#L74

Cheers :)

[Joshua-Anderson]

:+1: I can work on this in the next few days

[mockitoguy]

Awesome. Thanks :)

[Joshua-Anderson]

I'm not sure this is a good option anymore. Most of the time, we upload to a existing release which already has a body, so we can't set it.

However, if the creator uses a light git tag, it doesn't show up as a github release, so we create a new release. In this situation, we could potentially set the body.

I think that it would be really confusing for the user to have the body option that they specify not applied in some cases but occasionally in others. Secondly, you can easily edit the body yourself in the github ui.

[mockitoguy]

Oh I see.

My use case was continuous deployment. I don't really want to host downloadables with git releases. I just want some place to keep release notes. However, I can do it differently, not necessarily via Github releases.

Feel free to skip this ticket. Thanks for looking into it.

[terinjokes]

I, likewise, would love to have the ability to set the body. I have the body content on disk, but have no way to tell dpl to use it.

[abitrolly]

Travis allows to pass body to Octokit. The only problem is to figure out how to quote it for command line - see #657.

[valeriomazzeo]

@abitrolly I am having exactly the same issue, did you find a solution?

For me this worked:

body: "Test Body"

but I can't manage to expand a variable containing a proper changelog

[abitrolly]

@valeriomazzeo no solution. But perhaps the problem is in properly escaping newlines in expanded var.

[valeriomazzeo]

@abitrolly I made some progress with this and managed to get a successful upload:

before_deploy:
  - brew install jshon
  - export BODY=$(jshon -s "$(cat CHANGELOG.md)"))

deploy:
  provider: releases
  skip_cleanup: true
  api_key: ...
  body: ${BODY}

jhson properly escapes the markdown, I have tried setting that very same string in Octokit.rb and it gets uploaded properly and markdown show us correctly.

Although, when running from travis, markdown doesn't show up correctly, I am trying trimming the double quotes now:

export BODY=$(sed -e 's/^"//' -e 's/"$//' <<<$(jshon -s "$(cat CHANGELOG.md)"))

but if travis/dpl/releases doesn't add them properly when calling the client function, I am afraid it's not going to work.

[BanzaiMan]

While @valeriomazzeo's case appears to show that simple spaces may not be much of a problem, as the value specified for body is passed through to dpl and Octokit, the spaces (including newlines) could be a problem.

Escaping the body value may be able to get around this, I am not entirely certain what that would look like (both on the dpl command line, as well as on .travis.yml).

Perhaps we can have a separate option that reads the body parameter from a different source (e.g., a file).

[valeriomazzeo]

Trimming the double quotes works too, but the markdown doesn't show properly.

Using Octokit.rb directly, works as expected:

client.create_release("myuser/myrepo", "0.3.2", options = {:body => "# Change Log\n\n## [0.3.1](https:\/\/github.com\/myuser\/myrepo\/tree\/0.3.1) (2017-07-05)\n[Full Changelog](https:\/\/github.com\/myuser\/myrepo\/compare\/0.3.0...0.3.1)\n\n**Merged pull requests:**\n\n- Removed version log [\\#30](https:\/\/github.com\/myuser\/myrepo\/pull\/30) ([valeriomazzeo](https:\/\/github.com\/valeriomazzeo))\n- Fixed table names [\\#29](https:\/\/github.com\/myuser\/myrepo\/pull\/29)\n\n\n\n\\* *This Change Log was automatically generated by [github_changelog_generator](https:\/\/github.com\/skywinder\/Github-Changelog-Generator)*"})

Using travis:

If I edit the release, the markdown shows up escaped.

Note: screenshot and client command are not the same text

I am running out of ideas here...

is there a way to test dpl directly?

[valeriomazzeo]

I wrote a simple script to simulate bash / ruby interaction:

require 'octokit'

def foo(options={})
  client = Octokit::Client.new(:access_token => "youtokenhere")

  user = client.user
  user.login

  #puts options

  client.create_release("myuser/myrepo", "0.3.2", options = { :body => options[:body] })

  # this works too:
  # client.create_release("myuser/myrepo", "0.3.2", options.merge({:draft => true}))

end

foo(:body => ARGV[0])

executing the following produces the correct result:

export BODY=$(cat CHANGELOG.md)
ruby test.rb "${BODY}"

Any other variant doesn't work as expected:

ruby test.rb "hello\n\nworld" shows -> Hello\n\nworld

export BODY=$(cat CHANGELOG.md)
ruby test.rb ${BODY}

shows only the characters up to the first space

I see that the value is not wrapped in quotes, but I don't see how my script is any different than dpl:

invalid option "--body=# Change Log\n\n## [0.3.2](https://github.com/myuser/myrepo/tree/0.3.2) (2017-07-05)\n[Full Changelog](https://github.com/myuser/myrepo/compare/0.3.1...0.3.2)\n\n\n\n\\* *This Change Log was automatically generated by [github_changelog_generator](https://github.com/skywinder/Github-Changelog-Generator)*"
failed to deploy

[BanzaiMan]

@valeriomazzeo https://github.com/travis-ci/dpl/issues/155#issuecomment-313181155 the text shown indicates that \n was passed literally, rather than the newline which was intended. Could you try passing \\n instead? (Depending on the level of escaping, even more backslashes may be necessary.)

[valeriomazzeo]

@BanzaiMan if you see my last comment:

invalid option "--body=# Change Log\n\n## [0.3.2](https://github.com/myuser/myrepo/tree/0.3.2) (2017-07-05)\n[Full Changelog](https://github.com/myuser/myrepo/compare/0.3.1...0.3.2)\n\n\n\n\\* *This Change Log was automatically generated by [github_changelog_generator](https://github.com/skywinder/Github-Changelog-Generator)*"
failed to deploy

Perhaps there is something wrong going on between travis.yml and ruby dpl?
"${BODY}" doesn't seem to get expanded?

Update:

so... the error message comes from here:

https://github.com/travis-ci/dpl/blob/424c7a61c341f184b8aa66df178d4aca31455a6f/lib/dpl/cli.rb#L17

OPTION_PATTERN = /\A--([a-z][a-z_\-]*)(?:=(.+))?\z/
die("invalid option %p" % arg) unless match = OPTION_PATTERN.match(arg)

It turns out quotes are perfectly fine and it's all good, but we have been confused by the error message which is definitely a red herring.

The real problem is that regular expression that doesn't allow options to contains certain characters, in particular I believe the only problems are new lines.

[valeriomazzeo]

I had enough, just made my own: https://gist.github.com/valeriomazzeo/5491aee76f758f7352e2e6611ce87ec1

[adamvoss]

I'm not sure this is a good option anymore.

This is supported since #189. However there is an issue with how it handles new-line characters (appear as '\n' rather than the literal new line.) and possibly markdown is escaped (have not verified).

Most of the time, we upload to a existing release which already has a body, so we can't set it.

I suppose you have more aggregate statistics than me, but none of my projects or those contributed configuration behave this way. Maybe I am doing things wrong?

I think that it would be really confusing for the user to have the body option that they specify not applied in some cases but occasionally in others.

Agreed, could the description be over-written? I expect so, so then by default don't over-write but add a config option that enables overwrite?

Secondly, you can easily edit the body yourself in the github ui.

If I accept this as true, then I can "easily" manually execute build scripts and upload releases to GitHub :wink: . I agree not all use-cases will lend themselves to specifying the body via Travis, but those that do probably have reasons and benefits for doing so. For example, the one I am working on now puts the filesize and sha265 hash in the release body which is information that is required by the ecosystem that will be utilizing the released files.

EDIT: Of course, disconcertingly, I seem to be finding the hash calculated by travis is different than the one when I download from GH Releases...

[astorije]

Hey there, I see a lot of discussion in here, but as @Midnighter commented in https://github.com/travis-ci/travis-ci/issues/8568#issuecomment-336106310, I'm not sure if a solution was mentioned.

Essentially, what I'm trying to achieve is, whenever a tag gets built on Travis, extract the corresponding changelog entry from CHANGELOG.md, and create a GitHub release with that extracted Markdown content (for a practical example, see this changelog entry and this release, the release content is a copy-paste of the changelog entry).

In an ideal world, I would do something like the following, is this any possible?

before_deploy:
  - "export CHANGELOG_NAME=`cat CHANGELOG.md | ./extract_relevant_changelog_entry_name`"
  - "export CHANGELOG_BODY=`cat CHANGELOG.md | ./extract_relevant_changelog_entry_body`"

deploy:
  provider: releases
  skip_cleanup: true
  api_key: ...
  name: ${CHANGELOG_NAME}
  body: ${CHANGELOG_BODY}
  on:
    tags: true

Oh, question while I'm here: could you confirm that uploading a file is _not_ necessary? I.e. in the example above, I really just need to set name/body, but not upload any file.

[Midnighter]

I tried a number of different ways but I think the solution @valeriomazzeo brought forward is the only workable one. No matter how you enter the content it always gets garbled by the deploy script.

[stale[bot]]

Thanks for contributing to this issue. As it has been 90 days since the last activity, we are automatically closing the issue. This is often because the request was already solved in some way and it just wasn't updated or it's no longer applicable. If that's not the case, please do feel free to either reopen this issue or open a new one. We'll gladly take a look again! You can read more here: https://blog.travis-ci.com/2018-03-09-closing-old-issues

[Midnighter]

Before this issue disappears from the face of the earth. Are there any plans to change the GitHub releases deployment or not?

[benyanke]

I would also love to see a solution here. I would like to make my release notes the commit message for the final release commit, which would be very easy to generate, along with a bit of boilerplate on the bottom....this is, of course, impossible right now.

[christlang]

I would also love to see a solution here

[Jaskaranbir]

If this helps anyone, here's a simple bash script I am using to create automated releases: https://gist.github.com/Jaskaranbir/d5b065173b3a6f164e47a542472168c1

It uses github-changelog-generator to generate changelogs (changelogs are written to CHANGELOG.md, which I then read, convert to JSON, and make a Github-API call). Of course, modify and use as required.

[micheljung]

Please add this, sounds a like a low hanging fruit 🥝 :-)

[stale[bot]]

Thanks for contributing to this issue. As it has been 90 days since the last activity, we are automatically closing the issue. This is often because the request was already solved in some way and it just wasn't updated or it's no longer applicable. If that's not the case, please do feel free to either reopen this issue or open a new one. We'll gladly take a look again! You can read more here: https://blog.travis-ci.com/2018-03-09-closing-old-issues

[benyanke]

comment to disable stalebot

[BanzaiMan]

This _sort of_ works:

deploy:
  provider: releases
  body: |
    Title!
    more stuff
  on:
    all_branches: true
  edge:
    branch: "releases-body-escape"
  ⋮ # rest

The problem I see here is that the implementation at the moment relies on https://rubygems.org/gems/string_undump for String#undump, which could prove to be a problem if deployment runs with Ruby 2.5.0+ (in the future).

[Midnighter]

This sort of works

Does this allow for markdown or only the separation of title and the rest of the body?

[BanzaiMan]

Maybe. Not all markups have been tested.

[smaslennikov]

For me, even simply placing a body option errors out. Here's the commit, and here's the output:

'--file=./*rpm' '--file=./*deb' --file=INSTALL.md --file_glob --provider=releases --skip_cleanup --fold
invalid option "--body=## Please do read the commands you'll be executing! I'm not responsible for your mishaps.\n\nInstall on Ubuntu (tested on 16.04 and 18.04) by acquiring the DEBs and installing them altogether:\n\n```\nwget https://github.com/smaslennikov/packages/releases/download/v1.0.2/gpg_2.2.12-1_amd64.deb\nwget https://github.com/smaslennikov/packages/releases/download/v1.0.2/gpgme_1.12.0-1_amd64.deb\nwget https://github.com/smaslennikov/packages/releases/download/v1.0.2/libassuan_2.5.2-1_amd64.deb\nwget https://github.com/smaslennikov/packages/releases/download/v1.0.2/libgcrypt20_1.8.4-1_amd64.deb\nwget https://github.com/smaslennikov/packages/releases/download/v1.0.2/libgpg-error0_1.35-1_amd64.deb\nwget https://github.com/smaslennikov/packages/releases/download/v1.0.2/libksba_1.3.5-1_amd64.deb\nwget https://github.com/smaslennikov/packages/releases/download/v1.0.2/npth_1.6-1_amd64.deb\nwget https://github.com/smaslennikov/packages/releases/download/v1.0.2/ntbtls_0.1.2-1_amd64.deb\nwget https://github.com/smaslennikov/packages/releases/download/v1.0.2/pinentry_1.1.0-1_amd64.deb\nsudo dpkg -i *.deb\n```\n\nInstall on CentOS/RHEL (tested on CentOS 7) by acquiring the RPMs and installing them altogether:\n\n```\nwget https://github.com/smaslennikov/packages/releases/download/v1.0.2/gpg-2.2.12-1.x86_64.rpm\nwget https://github.com/smaslennikov/packages/releases/download/v1.0.2/gpgme-1.12.0-1.x86_64.rpm\nwget https://github.com/smaslennikov/packages/releases/download/v1.0.2/libassuan-2.5.2-1.x86_64.rpm\nwget https://github.com/smaslennikov/packages/releases/download/v1.0.2/libgcrypt20-1.8.4-1.x86_64.rpm\nwget https://github.com/smaslennikov/packages/releases/download/v1.0.2/libgpg-error-1.35-1.x86_64.rpm\nwget https://github.com/smaslennikov/packages/releases/download/v1.0.2/libksba-1.3.5-1.x86_64.rpm\nwget https://github.com/smaslennikov/packages/releases/download/v1.0.2/npth-1.6-1.x86_64.rpm\nwget https://github.com/smaslennikov/packages/releases/download/v1.0.2/ntbtls-0.1.2-1.x86_64.rpm\nwget https://github.com/smaslennikov/packages/releases/download/v1.0.2/pinentry-1.1.0-1.x86_64.rpm\nsudo yum install *.rpm\n```"
++result=1

I do realize this may be simply due to special characters in the string, but in reality, that issue should probably be somehow avoided.

[simbo1905]

we used the script by @Jaskaranbir with a slight enhancement to curl the last release to generate the changelog from. it needs a GITHUB_OAUTH_TOKEN set with repo write access.

[stale[bot]]

Thanks for contributing to this issue. As it has been 90 days since the last activity, we are automatically closing the issue. This is often because the request was already solved in some way and it just wasn't updated or it's no longer applicable. If that's not the case, please do feel free to either reopen this issue or open a new one. We'll gladly take a look again! You can read more here: https://blog.travis-ci.com/2018-03-09-closing-old-issues

[lindell]

I looked at the scripts here that would help with this problem but did also need file upload.
I created a tool to help with the releases and it might also be useful for other people :)

https://github.com/lindell/github-release-cli

.travis.yml:

before_deploy:
- curl https://github.com/lindell/github-release-cli/releases/download/1.1.0/github-releaser-travis -L --output github-releaser && chmod +x github-releaser
- export BODY=$(envsubst < ./CHANGELOG.md)
- export FILES=release-files/*
deploy:
  provider: script
  script: ./github-releaser -draft -verbose
  skip_cleanup: true
  on:
    tags: true

[svenfuchs]

I have added what I believe would be a potential solution to the issues discussed here in this PR: https://github.com/travis-ci/dpl/pull/1069

This renames --body to --release_notes (still keeping body as an alias), and adds --release_notes_file. If release_notes was given then this string will be posted as a body. If --release_notes_file was given then this file will be read, and its contents will be posted as a body.

It would be cool if some of you could give this a shot, or leave your thoughts on the PR.

I will close this issue here for the time being, but please feel free to re-open it if needed.