Toolkit: Unable to create annotations

[xt0rted]

I have two actions that create annotations after running eslint & stylelint but I have yet to move them to v2. They run fine in v2 workflows though and annotations are created as you'd expect.

I'm trying to setup a new action that creates annotations just like my other actions but using the toolkit packages. I'm able to create a checks run using GITHUB_TOKEN and the github client, but I'm not able to call the update checks function with the annotations. I keep getting a 422 response back. Are there differences between v1 and v2 actions that would prevent this from working?

This call works:

const githubClient = new github.GitHub(core.getInput("repo-token", { required: true }));

const { data } = await githubClient.checks.create({
  ...github.context.repo,
  name: github.context.action,
  head_sha: github.context.sha,
  started_at: new Date().toISOString(),
});

But this call doesn't:

const githubClient = new github.GitHub(core.getInput("repo-token", { required: true }));

const { data} = await githubClient.checks.update({
  ...github.context.repo,
  check_run_id: id, // data.id from the first call
  completed_at: new Date().toISOString(),
  conclusion: conclusion,
  output,
  status: "completed",
});

[thboop]

Hi @xt0rted,

I'm having trouble reproducing this issue, I've outlined my steps below in case they are helpful!
Could you also provide an example output variable?
If possible, could you also provide a link to a run using your action, or a link to your action.

I tried the following below:

Here are the packages I am using:

    "@actions/core": "^1.1.0",
    "@actions/github": "^1.1.0"

Here is my action main.ts

import * as core from '@actions/core';
import * as github from '@actions/github';

async function run() {
  try {
    const githubClient = new github.GitHub(core.getInput("repo-token", { required: true }));
    const { data } = await githubClient.checks.create({
      ...github.context.repo,
      name: github.context.action,
      head_sha: github.context.sha,
      started_at: new Date().toISOString(),
    });
    console.log(JSON.stringify(data));
    const update = await githubClient.checks.update({
      ...github.context.repo,
      check_run_id: data.id, // data.id from the first call
      completed_at: new Date().toISOString(),
      conclusion: "success",
      output: 
      { 
        summary: "summary",
        title: "title",
        annotations: [
          {
            path: "./dummyfile.txt",
            end_line: 1,
            start_line: 1,
            annotation_level: "failure",
            message: "missing value"
          },
        ]
      },
      status: "completed",
    });
    console.log(JSON.stringify(update));
  } catch (error) {
    core.setFailed(error.message);
  }
}

run();

Here is how I invoke the action

name: test
on: push
jobs:
  build:
    runs-on: ubuntu-latest
    steps:
    - name: Test Action
      uses: bbq-beets/thboop-action@master
      with:
        repo-token: ${{secrets.GITHUB_TOKEN}}

[xt0rted]

@thboop I'll have to take a look at this again later, but the main difference I see between the two setups is yours isn't creating annotations in the update call. I'm able to create the checks run just fine, but when it comes to the annotations that's when the request fails.

[thboop]

@xt0rted
Sounds great, thank you!
I can confirm I can push annotations as well, I've updated the above code snippets.

[xt0rted]

@thboop this is embarrassing, the issue was on my end. I was setting the start_column for the annotation, but not the end_column. That's why the requests kept failing. The docs don't really mention this but after some trial and error that's what I narrowed it down to.

The issue I'm facing now is how to get the annotations to be associated with the job they ran in. In v1 I did this by using GITHUB_ACTION as the check name, but with v2 this gets set to self (or maybe still {user}{repo} if using one outside the current repo) and then another job shows in the left, and that's where the annotation results show.

[xt0rted]

After some more testing I'm seeing that if the workflow runs on push the annotations are created and shown, but if the workflow runs on pull_request the annotations are created but they don't get shown anywhere.

[thboop]

Hey @xt0rted
The checks API requires the head_sha, the sha provided by github.context.sha for pull_request builds is a merge sha, so these do not end up aligning.

I'm going to take this feedback to the team and see how we can improve this experience! Thanks for your feedback and quick responses on this item we really appreciate it!

For now, you can use access the head_sha for pull request builds via github.context.payload.pull_request.head.sha. I've confirmed that I can see the annotations for those builds.

I will also file an item for the 422 response code you were seeing and see how we can improve that messaging and error handling.

[bryanmacfarlane]

@thboop @xt0rted - OK to close this issue from a toolkit perspective? It seems like we figure it out but I want to make sure.

[xt0rted]

@bryanmacfarlane the annotations are working now, but I'm not sure how to associate them with the job that created them. This worked in v1 by using GITHUB_ACTION as the check name but not in v2 so far. This doesn't really feel like a toolkit issue, just lack of docs or server side functionality.

An example run can be seen here. The job self was created from the annotations in the ESLint step. When the action is in the repo it's running from the job is called self, if it's loaded from an external repo it's {user}{repo} or in this case xt0rtedeslint-action.

[thboop]

Hey @xt0rted ,
We are currently working on smoothly this experience out for you!

Our recommended flow for this is going to be using problem matchers, as these will automatically create issues on the run. We have a feature in development which will create annotations for issues on your run, providing the end to end functionality I believe you are trying to accomplish. This should allow you to more easily create annotations without having to manually use the sdk to create another check!

We have a ticket to update our documentation #56 which will cover how to do this.
I also want to cover in the docs on the toolkit more information about annotations and how users can create them from the toolkit, I've updated that issue to indicate that.

[xt0rted]

@thboop I'll give this a try and see how it works compared to the api approach.

[xt0rted]

@thboop I forgot to ask, when creating annotations with the api you can add action buttons to them. The UI shows them and reacts to clicking them, but I wasn't able to get that to trigger a workflow on check_run:requested_action. Is this something that's possible with actions right now?

[thboop]

@thboop I forgot to ask, when creating annotations with the api you can add action buttons to them. The UI shows them and reacts to clicking them, but I wasn't able to get that to trigger a workflow on check_run:requested_action. Is this something that's possible with actions right now?

@xt0rted , that doesn't seem intended. Other users recently reported similar issues with triggering workflows from apps. I'll pass along that you are seeing this issue as well to our engineering team. You can also reply in the community forums.

Just for clarity's sake, I do not think that issue is related to the toolkit.

[warrenbuckley]

Sorry to hijack this thread as I am interesting in using the Checks API for a linting tool as a GitHub Action like you are @xt0rted

With @xt0rted your example you have a new check run called self but to me this seems a super confusing UI to have a seperate tab to try and find the annotations.

Is the best way to create a brand new Check Run to do the annotations in or is it possible to use the same check run/suite that GitHub Actions is creating behind the scenes?

With something like

const checkList = await client.checks.listForRef({
  owner: github.context.repo.owner,
  repo: github.context.repo.repo,
  ref: github.context.ref
});

console.log('checkList data', checkList.data);

// The Check Run ID to use to update it later on
const checkRunId = checkList.data.check_runs[0].id;

Again this all seems like uncharted territory at the moment & would love some clarification please, to know what is the best/correct way to be creating/writing linters using GitHub Actions 😄

I would ❤️ if the GitHub Actions team ported/updated/created a new guide tutorial based on the Rubocop linter that is a GitHub App
https://developer.github.com/apps/quickstart-guides/creating-ci-tests-with-the-checks-api/

[thboop]

Hey @warrenbuckley

I am hopeful you are going to be really happy with the new annotation features once they are ready! They are currently being worked on!
In particular:

• We plan on having annotations attached to the current job/check to clear up the confusion you highlighted in your image above!
• Creating annotations as an action developer or a user creating action workflows will be much easier, not requiring you to use the checks api to get the current check or to update the current check.
• We will provide example actions for these features

That being said, this doesn't cover all the feedback you provided. Feel free to create a discussion on the community forum or submit a feedback ticket if you a way to get the current job or check via the checks api would be useful to you!

[bryanmacfarlane]

Closing per ^^ discussion. Let us know if there's something actionable in toolkit outside the items covered above and we can consider re-opening or another issue

[xt0rted]

@thboop I switched one of my JS actions over to use a problem matcher. It'd be helpful if there was a function in the toolkit to facilitate registering them (they seem to still use ## instead of :: based on the setup environment actions). The lines in the log are being highlighted as errors, but annotations aren't created. Is that part functional yet?

[thboop]

@thboop I switched one of my JS actions over to use a problem matcher. It'd be helpful if there was a function in the toolkit to facilitate registering them (they seem to still use ## instead of :: based on the setup environment actions). The lines in the log are being highlighted as errors, but annotations aren't created. Is that part functional yet?

@xt0rted , That feature is not quite ready yet, we will update the docs with more information when it is available!