Test-infra: Prow commands not working on github PR anymore, cannot /approve a PR

[aure-olli]

What happened:

When I created a test cluster on GKE with github gitops, little more than a month ago, the "create PR -> /approve" process worked perfectly. But now that I have created a new cluster, it doesn't work anymore. Any command (/approve, /pony, /honk) seems to be ignored.

The hooks are setup, and if I manually create the "approved" label (which should be done by Prow), then the PR is merged.

What you expected to happen:

I expect the normal behavior:

• open a PR
• type /approve in comment
• the "approved" label is created
• the PR is merged

How to reproduce it (as minimally and precisely as possible):

• Create a GKE cluster with default options:

jx create cluster gke --machine-type n1-standard-2 --max-num-nodes 2 --min-num-nodes 2 --region asia-southeast1-b

• Create a project

jx create quickstart --name=testapprove --language=go

• Create a new branch

cd testapprove
git checkout -b merge

• Edit something

nano main.go

• Push the new branch

git commit -am merge
git push -u origin merge

• Create a PR on github
• Post /approve comment
• Wait ... :(

Please provide links to example occurrences, if any:

I made my repository public for you to see: https://github.com/aure-olli/testapprove/pull/1
/approve, /pony and /goose were ignored.

Anything else we need to know?:

I thought it could be linked to the bot using my github account, but looks like the /pony command is not checking for this case, so at least that one should work.

github hook is working, since creating the "approved" label triggers the merge.

on deck, the tide status section /tide is empty, and the PR status /pr is not available (and bugs with a /pr-data.js 404 if I still try).

Thanks

[stevekuznetsov]

tide is not using webhooks but the search API to look for PRs to merge. It does seem that your webhook is not configured correctly. Did you update it to point at the new cluster? Does the GitHub UI show any errors? Could you paste your hook logs?

[aure-olli]

Hi

Thanks for your help

I can confirm that github hook works perfectly. Hook URL is the right one (same as the one described in kubectl get ingress). Events got systematically a 200 "Event received. Have a nice day." reply.

• last /approve event: https://github.com/aure-olli/testapprove/blob/logs/logs/52962780-0074-11ea-9ec3-4f3a31b39595.log

{
  "action": "created",
  [...]
  "comment": {
    "url": "https://api.github.com/repos/aure-olli/testapprove/issues/comments/550213958",
    [...]
    "user": {
      "login": "aure-olli",
      [...]
    },
    "created_at": "2019-11-06T09:03:39Z",
    "updated_at": "2019-11-06T09:03:39Z",
    "author_association": "OWNER",
    "body": "/approve"
  },
  "sender": {
    "login": "aure-olli",
    [...]
  }
}

• the /pony event: https://github.com/aure-olli/testapprove/blob/logs/logs/db13c880-0072-11ea-83ff-93f8f85494b2.log

{
  "action": "created",
  [...]
  "comment": {
    "url": "https://api.github.com/repos/aure-olli/testapprove/issues/comments/550209787",
    [...]
    "user": {
      "login": "aure-olli",
      [...]
    },
    "created_at": "2019-11-06T08:53:09Z",
    "updated_at": "2019-11-06T08:53:09Z",
    "author_association": "OWNER",
    "body": "/pony"
  },
  "sender": {
    "login": "aure-olli",
    [...]
  }
}

Here are both hook pods logs: https://github.com/aure-olli/testapprove/blob/logs/logs/hook-756489b884-g98br.log and https://github.com/aure-olli/testapprove/blob/logs/logs/hook-756489b884-ljpqd.log

The last log lines clearly show acknowledging the comments:

• last /approve

{
  "author":"aure-olli",
  "component":"hook",
  "event-GUID":"52962780-0074-11ea-9ec3-4f3a31b39595",
  "event-type":"issue_comment",
  "level":"info",
  "msg":"Issue comment created.",
  "org":"aure-olli",
  "pr":1,
  "repo":"testapprove",
  "time":"2019-11-06T09:03:40Z",
  "url":"https://github.com/aure-olli/testapprove/pull/1#issuecomment-550213958"
}

• /pony

{
  "author":"aure-olli",
  "component":"hook",
  "event-GUID":"db13c880-0072-11ea-83ff-93f8f85494b2",
  "event-type":"issue_comment",
  "level":"info",
  "msg":"Issue comment created.",
  "org":"aure-olli",
  "pr":1,
  "repo":"testapprove",
  "time":"2019-11-06T08:53:09Z",
  "url":"https://github.com/aure-olli/testapprove/pull/1#issuecomment-550209787"
}

However there is not reaction. In particular, the /pony event should either create an error log or a CreateComment log. Previous CreateComment logs show that the function is working fine. I Deduce that the handler of /pony event was never called.

Some other hook logs may be noteworthy, but nothing that explain the lack of /pony reaction.

• apparently there are problems with parsing OWNERS_ALIASES: https://github.com/aure-olli/testapprove/blob/master/OWNERS_ALIASES

{
  "base":"master",
  "client":"repoowners",
  "component":"hook",
  "error":"error unmarshaling JSON: while decoding JSON: json: cannot unmarshal array into Go struct field .aliases of type map[string][]string",
  "level":"error",
  "msg":"Failed to unmarshal aliases from \"/tmp/git701655555/OWNERS_ALIASES\". Using empty alias map.",
  "org":"aure-olli",
  "repo":"testapprove",
  "time":"2019-11-06T08:52:25Z"
}

• yep, environment-koalaband-staging has no OWNERS_ALIASES

{
  "base":"master",
  "client":"repoowners",
  "component":"hook",
  "error":"open /tmp/git508352919/OWNERS_ALIASES: no such file or directory",
  "level":"info",
  "msg":"No alias file exists at \"/tmp/git508352919/OWNERS_ALIASES\". Using empty alias map.",
  "org":"aure-olli",
  "repo":"environment-koalaband-staging",
  "time":"2019-11-06T08:47:14Z"
}

• This one appears 2 times, but right after the PR creation, so it seems normal

{
  "author":"aure-olli",
  "component":"hook",
  "event-GUID":"05b15180-0072-11ea-98c2-9cadc3067cfc",
  "event-type":"pull_request",
  "level":"warning",
  "msg":"Not enough reviewers found in OWNERS files for files touched by this PR. 0/2 reviewers found.",
  "org":"aure-olli",
  "plugin":"blunderbuss",
  "pr":1,
  "repo":"environment-koalaband-staging",
  "time":"2019-11-06T08:47:16Z",
  "url":"https://github.com/aure-olli/environment-koalaband-staging/pull/1"
}

• This one appears many times. aure-olli should logically be the "suggested approver". But it doesn't explain the /pony silence.

{
  "author":"aure-olli",
  "component":"hook",
  "event-GUID":"05b15180-0072-11ea-98c2-9cadc3067cfc",
  "event-type":"pull_request",
  "level":"warning",
  "msg":"Couldn't find/suggest approvers for each files. Unapproved: [\"\"]",
  "org":"aure-olli",
  "plugin":"approve",
  "pr":1,
  "repo":"environment-koalaband-staging","time":"2019-11-06T08:47:20Z",
  "url":"https://github.com/aure-olli/environment-koalaband-staging/pull/1"
}

My colleagues has the same problem while creating a test cluster: /approve is not working neither.

[nikhita]

@aure-olli Looks like the "bot account" is set up as your account itself. Prow ignores commands from the "bot account", so any comments that you make won't have any affect.

I just commented with an /approve and it seems to react. :)

Also.. do you have the pony and goose plugins enabled in plugins.yaml? Since the bot didn't comment with a pony/goose image, it looks like this hasn't been configured.

You'll also need to setup an unsplash api key for the goose plugin by the way:

https://github.com/kubernetes/test-infra/blob/34a66416254dcb4979a2ff88b8f6b072fa9af436/config/prow/plugins.yaml#L173-L174

[aure-olli]

Hi

Thanks for helping

It's hard to know what is activated in Jenkins-X deployment :thinking: But I see there are many configuration, so you must be right, /pony is not activated.

OK, so this is eventually due to the fact that the owner and the bot need to be 2 different accounts. However, it tickles me on 2 aspects:

• It was working little more than a month ago, and I didn't see any commit that changes that. Maybe Jenkins-X images are late though.

• This is the default deployment pipeline of Jenkins-X, so it gets highly counter-intuitive if it doesn't work straight ahead when testing it. It makes it hard to test with a free github account, as then need to create an account for the bot. Why this test is even necessary ? The regex seem already strong enough to not match the automatic messages.

Anyway, thank you, at least I now know how to avoid this problem.

[spiffxp]

/close
Please /reopen if further help is needed here

[k8s-ci-robot]

@spiffxp: Closing this issue.

In response to this:

/close
Please /reopen if further help is needed here

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.