Toolkit: set-env truncates multiline strings

[andreasplesch]

This came up in the forum: https://github.community/t5/GitHub-Actions/set-output-Truncates-Multiline-Strings/m-p/52511/highlight/true#M8539

with a suggestion to post an issue here.

the enhancement
Please add an example to the docs, probably in https://github.com/actions/toolkit/blob/master/docs/commands.md, on how to best allow for multi-line value in set-env or set-output.

Code Snippet
The suggested solution in the forum is to manually escape newlines in the shell script with

REPORT="${REPORT//'%'/'%25'}"
REPORT="${REPORT//$'\n'/'%0A'}"
REPORT="${REPORT//$'\r'/'%0D'}"

potential bug
https://github.com/actions/toolkit/blob/master/packages/core/src/command.ts#L76
already should escape, so perhaps something else is not working.

193 seems related.

[andreasplesch]

405 also possibly related although multiline input should already be string.

[ericsciple]

Afaik there isn't a bug in the toolkit. When using the toolkit function, newlines are escaped.

When echo'ing the command manually (the community issue) the user must escape.

[andreasplesch]

Thanks, makes sense. Since the recommendation was to post an issue here, I thought workflow and toolkit were linked, eg. that workflow is using the toolkit. Where would be the appropriate place then to provide documentation about manual escaping ? It seems to be a recurring issue.

[ericsciple]

@thboop it sounds like the command docs are missing info about escaping:

• https://help.github.com/en/actions/reference/workflow-commands-for-github-actions
• https://github.com/actions/toolkit/blob/master/docs/commands.md (should this doc be deprecated now and link to dotcom doc instead?)

@andreasplesch in the community issue it looked like the user was echoing the command manually. It was a run step (using bash), so printing to STDOUT via echo ::set-output...

The function in the toolkit (npm module) does the escaping and then prints the formatted command ::set-output... to STDOUT. Typically the toolkit is used from a javascript action. Whereas in a run step (bash), need to format the command manually.

[andreasplesch]

https://help.github.com/en/actions/reference/workflow-commands-for-github-actions would be perfect for escaping info. I also noticed that problem matcher doc is missing there.

[staabm]

from a end-user perspectiv, its highly supprising that a line like echo "::set-output name=report::$report" does not support multi-line strings.

when setting the outout var based from a cli command like echo "::set-output name=report::$(ls -l)" one needs to waste a lot of time until recognizing it needs some kind of magic escaping.

would be great this could work out of the box

[andreasplesch]

The real solution presumably would be to not have to escape at all. Why should % and newlines be special ?

[ericsampson]

What? This seems nuts to expect a user to do the escaping manually.

[dfreilich]

Piling on here - I found this very surprising as well, as a user.

[jfreeland]

It gets more tricky if your other steps are using containers (_though for speed I would recommend trying to avoid using containers where at all possible_) but writing your output to a file (_e.g. in /tmp_) is another workaround. 🤷‍♂️

[ericsciple]

Unfortunately echo "::set-output name=report::$report" can't automatically support newlines.

The runner is reading stdout from the child process. The runner has to assume the value ends when it reaches the end of the line.

Other options would be something like, if the runner added a tool in the PATH to call instead which deals with escaping. For example, runner set-output --name report --value "$report"

Or a different approach would be if the runner supported a heredoc style when writing stdout commands. For example:

echo "::set-output name=report eof=EOF::"
echo "$report"
echo "EOF"

[wachunei]

Hello, I'm a little bit lost with what is actually happening here.

I'm trying to use the bash replacement but it seems like the output coming from a yarn script is already truncated to a the first line.

I have this code https://github.com/wachunei/directUC/actions/runs/200970194/workflow#L30

Do any of you know if there is a workaround to this that might work in my context? Thank you.

[andreasplesch]

If the first echo output after running the yarn script is unexpected, seeminlgy truncated to one line, it is perhaps an issue with the yarn script rather than with actions. Do the two lines work from an interactive bash shell ?

[wachunei]

I thought the same, so this morning I turned this into a bash script wrapping the yarn output and the issue is still happening.

In this echo output I'm getting the first line only https://github.com/wachunei/directUC/actions/runs/201452370/workflow#L32
(output: https://github.com/wachunei/directUC/runs/963981758?check_suite_focus=true#step:7:10)

But if I run this locally it seems like it is a single line output, this is also more confusing, are these chars ignored in the echo?

I'm mostly a front end developer, this is my first time trying to make automatic things like a release on tag push and add a body coming from a script to it, I really appreciate you taking your time to comment!

[andreasplesch]

I am noticing you are using zsh, not bash, interactively, and the the bash script is missing the the double quotes. You will need to be careful with the exact settings.

Is the interactive yarn output as expected ?

[wachunei]

With bash is the same output, without or with doublequotes (I added them though!). It is not as expected since I'm not passing the --starting-version option to it (I can take care of that later).

Update: this is the updated script with correct output locally

#!/usr/bin/env bash
OUTPUT="$(yarn run --silent auto-changelog --stdout -t ./.changelog/release-template.hbs $@)"
echo $OUTPUT

[andreasplesch]

bash$ OUTPUT="$(yarn run --silent auto-changelog --stdout -t ./.changelog/release-template.hbs $@)"
bash$ echo $OUTPUT

generates multi-line output ? You probably need double quotes around $OUTPUT to avoid word splitting.

[wachunei]

I added those, it generates multiline locally 👍🏼. In the runner I'm still getting the first line only https://github.com/wachunei/directUC/runs/964094234?check_suite_focus=true#step:7:11

local screenshot

[andreasplesch]

ok. Just try directly in the yaml wthout the release.sh:

text="$(yarn run --silent auto-changelog --stdout -t ./.changelog/release-template.hbs $@)"
echo "$text"

I do not know yarn enough but it seems strange to invoke yarn with another yarn call.

[andreasplesch]

Although, it seems ok to do that. But the outer yarn is missnig the stdout option.

[wachunei]

I do it because from the yaml I just run a yarn script, and the yarn script runs auto changelog (or the .sh that makes yarn run the auto changelog since this morning)

the --stdout is an auto-changelog option.

I'm getting the exact same result :(
https://github.com/wachunei/directUC/runs/964128634?check_suite_focus=true#step:8:9

[andreasplesch]

hm, right. Perhaps yarn is outputting to stderr ?
Try locally:

$yarn run --silent changelog:release --starting-version v1.0.1 > output
$cat output

[wachunei]

stdout is correctly redirected to output with expected multiple lines

[andreasplesch]

ok. Perhaps try that in the yaml ?

[ The question was if stdout is used in the first place but it seems to be. ]

[wachunei]

Same output, I thought setup-node might be installing yarn v2 but it is the same yarn version that I'm using locally

https://github.com/wachunei/directUC/runs/964164661?check_suite_focus=true#step:9:4

[happiness801]

Add me to the list of people who had to spend hours to figure out that multiple lines were being truncated. It was especially confusing for me because the first line of my output was blank so I was just getting nothing.

Additionally, using the escaping as suggested, my output is full of single-quotes for some reason.

Here's the first step (Bash):

        Tools/get-changed-fields.sh src/objects origin/${{ env.DIFF_BRANCH }} > changed-fields.out
        output="$(cat changed-fields.out)"
        output="${output//'%'/'%25'}"
        output="${output//$'\n'/'%0A'}"
        output="${output//$'\r'/'%0D'}"
        echo "::set-output name=changed-fields-output::$output"

Second step, using pullreminders/slack-action:

    - name: Changed Fields Slack Notification
      if: ${{ success() && steps.changed-fields-cache.outputs.cache-hit != 'true' }}
      env:
        SLACK_BOT_TOKEN: ${{ secrets.SLACK_BOT_TOKEN }}
      uses: pullreminders/slack-action@master
      with:
        args: '{\"channel\":\"${{ env.SUCCESS_SLACK_CHANNEL }}\",\"attachments\":[{\"title\":\"Changed Fields Report:\n${{ github.workflow }} #${{ github.run_number }}: ${{ env.BRANCH }} (by: ${{ github.actor }})\",\"title_link\":\"${{ github.server_url }}/${{ github.repository }}/runs/${{ github.run_id }}\",\"text\":\"```\n${{ steps.summarize-changed-fields.outputs.changed-fields-output }}\n```\"}]}'

The result I get in Slack is like this:

'
'Account '
'==========================='
''
' Fellowship_Certification_Required__c'
''

I cannot figure out where those single-quotes are coming from. They are not in the output file when assigned. Also, each line has multiple space characters for indentation, which is why we surround the output in ````` sets to achieve pre-formatted text. But the multiple spaces are reduced to one single space (an HTML like conversion??).

[happiness801]

I figured out a work-around for my use-case. I realized since I'm sending this to Slack via the 'args' argument, I needed newlines to be a literal '\' and 'n' in what was output to that command. And then I "fixed" the spaces by replacing space characters with a Unicode 'En Space' character:

        output="$(cat changed-fields.out)"
        output="${output//$'\n'/\\n}"
        output="${output// / }"     # replace regular space with 'En Space'
        echo "::set-output name=changed-fields-output::$output"

Which is resulting in the correct output:

===========================

  Fellowship_Certification_Required__c

       LABEL: Fellowship Certification REQUIRED!
        TYPE: Picklist

[thboop]

For set-env, you can now pipe that data to a file, rather than writing to stdout. This makes the newline scenario a little easier, see the curl example for Environment Files

[justAnotherDev]

i had the same issue as @happiness801 described here

'
'Account '
'==========================='
''
' Fellowship_Certification_Required__c'
''

I cannot figure out where those single-quotes are coming from.

the only solution that i found that works is to remove the single quotes from the accepted answer:

REPORT="${REPORT//'%'/'%25'}"
REPORT="${REPORT//$'n'/'%0A'}"
REPORT="${REPORT//$'\r'/'%0D'}"

REPORT ="${REPORT//'%'/%25}"
REPORT ="${REPORT//$'\n'/%0A}"
REPORT ="${REPORT//$'\r'/%0D}"

[ericsampson]

@thboop thanks - could we add a short note about set-env being deprecated to those docs? Otherwise people who see mentions of it via web search, or existing workflows, etc are going to be confused that there's no mention in that doc (and that GITHUB_ENV is the replacement). Thanks!!

[thboop]

set-env is now deprecated, i'm going to close out this issue.

[staabm]

set-env is now deprecated, i'm going to close out this issue.

what should be used instead?

[thboop]

The set-env command is now deprecated, and you should use the environment files

If you are using the toolkit core command exportVariable(name: string, val: string), it does this automatically if you are using v1.2.6 or later.

You can see a security advisory that prompted this here