Runner: Add functionality for allow self-hosted runner to protect workflow file and allow only execution only for collaborators for PRS

[MalloZup]

Description:

Hi all,
first of all I really like the github action runner and self-hosted runner.

It has however an issue with working for pull-request workflows.

I think the runner should have a way to be configurable to run only on Pull-requests from collaborator of X org or Repo.
The problem to be solved is to protect the github-workflow file and don't be changed by any arbitrary PR ( so there is no output redirection or other code executed)
it is a recurring topic in some forums, but there is no solution or any issue about this afaik. let me know, and ping me for any info .

best

[TingluoHuang]

@MalloZup sounds like some policy feature we can add on the service side even before sending the job to the runner.

@chrispat for feedback.

[MalloZup]

@TingluoHuang that could be awesome thx!

[plfiorini]

This would likely solve the security concerns of self-hosted runners

[MalloZup]

yes this is the goal :sun_with_face:

[chrispat]

@MalloZup I don't believe this makes sense as a runner feature but rather something that is enforced on the GitHub.com side. There are scenarios where customers want this type of feature for hosted runners and not just self-hosted.

[MalloZup]

@chrispat yes I agree with you partially.
I believe globally it make sense to have it on GIthub.com side.
However I think that also the runner can still have it own way to setup this.

Is like when developing a CLI command line.
Let assume the Github.com is our configuration file, from which the user setup option for our CLI. ( the CLI is the runner)

If an user use the CLI command line arguments, this overwrite the configuration file input (github.com).

In any ways, even if it you will implement globally you will need to instruct and implement the mechanism on the runner, afaik and correct me if I'm wrong.

Do you have a link where there is the Github.com UI source code where I can open such issue for the global part?

I think we could continue with this issue implementation on the runner side, since lot of user are depending on this to have a pragmatic/secure CI.

Personally the UI github.com implementation, from my perspective is the last part and imho it will take much more to implementation time then implementing such feature on the runner ( which is just a partial feature).
Having such feature on the runner side, is just a step imho that it will need to be done also if the options are passed from github.

Thanks for your answer and time ping me for any info or I can help.
:sunflower:

[chrispat]

@MalloZup Given the architecture of Actions the runner is not the best place to implement this feature. The runner is a relatively small part of the overall Action system and is designed to be a remote execution daemon for the Actions workflow.

In the case of GitHub hosted runners, self-hosted runners or runners that spin up on demand you waste a lot of time and money by spinning up a runner on a VM or in a container to make a simply policy check that should be done by the GitHub Actions workflow engine.

[MalloZup]

@chrispat do you know where I could open this issue for the github workflow engine?

Thank you in advance

[iagomelanias]

That feature would be extremely valuable to our team. 👍

We use Github Actions Runner labels to decide which infrastructure will be responsible for processing the job, some servers have more permissions than others. Here are some examples:

• There are runners which are responsible for provisioning resources using Terraform. It's possible to do literally anything inside of them.
• Some runners are responsible for deploying code to our stage environment.
• Some runners are responsible for deploying code to our prod environment.
• Other runners are responsible for only running tests using some cloud resources (storage, messaging, etc).

If a developer can change the labels and the jobs that are executed, all these security policies are thrown away. It would be possible to simply change the label of a test workflow to use a terraform runner and do anything there - delete a database, get access to any server or obtain any kind of sensitive data, like database passwords or private API keys. 💀

[MalloZup]

@chrispat or @TingluoHuang I could not find a place where I can move the issue in more appropriate place. If you have any feedback regarding this , it would be extremely welcome. thx!

[chrispat]

We don’t have a great place to track enhancements to the orchestration side of things as that is not open source.

We hope to have a public roadmap soon but the granularity of that may or may not account for these types of features.

[Penagwin]

We've been waiting for this feature since December. Our devs work out of forked repos, so we're unable to run tests on pull request because they're separate forks, even though everyone is apart of the organization that's private.

This means we have no feedback before merging stuff in.

[chrispat]

@Penagwin that is a different issue and it is something we are working on enabling. There will be some additional settings in your org and repos that you will be able to enable that will allow runs from forks of private repos.

[Penagwin]

@chrispat Is there an issue number I can track or an eta? I know covid threw a wrench in everything, but we've been waiting 8 months now...

[qinsoon]

We are eager to see this feature as well. I am also wondering if there is a workaround/hack for now to protect workflow files from unprivileged contributors. I found none... If anyone has a suggestion, I am happy to dig it deeper.

[MalloZup]

@qinsoon this feature could be implemented on the runner. SO yes it is possible to protect the workflow file without having to wait of unknown Feature coming from the github gui, which imho will not come really soon.

for reference take this comment:
https://github.com/actions/runner/issues/494#issuecomment-640346592

This is why I was also thinking that implementing it on the backend, is a more low-hanging fruit which will help the github action to become useful for people in short-term, without compromising the design of longterm ( taking this from gui).

So even if forking is always sub-optimal, I guess it would be ok to try it.

if anyone is willing to do this, create a LINK here so people can try it at least. :+1:

[MalloZup]

as I imagine the feature , it could be like this in the minimal way:

1) add a flag called like github_users="GINO;BAMBY;SUPERMAN"this flag is added to the action runner when started;
( if an user removal or addition, the admin of the worker need to restart the worker with new flag, but I guess this is not really time consuming)

only the user of github users can modify the workflow file.

The logic would be to have something https://github.com/actions/runner/issues/494#issuecomment-640346592 like stated on that comment.

I am not familiar to C# code and also maybe the runner itself would be need some developer documentation in order to explain its design and make it more accessible for the community to contribute, imho.

[chrispat]

These features can't be implemented in the runner in an efficient and secure way. We need to check these sorts of permissions before we allocate a VM. Also the runner is a hostile environment that can execute arbitrary code from an end user and really should not be considered for any sort of security boundary.

Unfortunately, there is no way for me to give you an issue to track for new features in the GitHub.com code base.

[MalloZup]

thx @chrispat fair enough. lets keep this issue open and maybe give update it here. enjoy.

[michhyun1]

We've been waiting for this feature since December. Our devs work out of forked repos, so we're unable to run tests on pull request because they're separate forks, even though everyone is apart of the organization that's private.

This means we have no feedback before merging stuff in.

I think this in itself needs to be its own issue. What's the point of a rep owner creating CI workflow github actions if that workflow doesn't run when an external customer submits a PR from a forked repo branch? I noticed the docs say 'private' base repo, but does it work for a public base repo?

Is there a GH issue for this that is currently being tracked?

[derekbelrose]

I was thinking the best way to handle this is to have a check box or something that requires CODEOWNER approval before running any webhooks in branch protections. This is a problem that exists not with just Github Actions but with Travis and CircleCI. A change to the respective workflows in a PR can trigger the workflow and they all use the PR as the basis for the workflow tasks.

So, I think being able to defer the triggering of webhooks until approval is a mitigation step. Make it off by default so not to break existing workflows.

[eladchen]

How about this:

An example:

Let's say .github/workflows/pull-request.yml exists, and the suggested checkbox above is checked, and
user John / DEV team is configured.

Workflow ".github/workflows/pull-request.yml" will not be executed if:

• A pull request is created based of a forked repository
• The pull request changed files include .github/workflows/pull-request.yml
• The pull request is not authored (triggered?) by user John / a member of DEV team.

This allows contributors to create pull requests, and have workflows provide them (and maintainers) with feedback on the changes introduced by the pull request, and, in the same time preventing untrusted parties from changing a workflow, thus preventing any security issues that might arise.

WDYT?

[derekbelrose]

@eladchen Im not sure how much work that is on the github end but that’s almost exactly what I need!

[eladchen]

I am not a 100% sure but I have a feeling a dedicated github app can be written to accommodate my suggestion, while waiting for a response/solution from the github folks.

IMO this would be a killer feature given many (and I mean many) people fail to recognise this is a potential security risk, especially open source ones.

[problame]

Also interested in this feature, pretty much as outlined above.

My use case: I maintain an open source project and want to automate building Debian packages, including pushing them to packagecloud. Right now, I can't do that with GitHub Actions because, to my understanding, any malicious PR could change the .yml file and steal the packagecloud credentials.

[allejo]

I think https://github.com/github/roadmap/issues/99 may be GitHub's solution to this problem?

[MalloZup]

@allejo thx for sharing! it cannot be considered a solution. I guess is okaish workaround,
the real solution would be policies on users/group/org

If my repository has 100 PRS I Need 100 manual operation to approve manually to run and then wait. Where in travis or similar, I just view the results.

But I think imho could be better then nothing

[MalloZup]

@allejo do you know what is the workflow? when the PR there is merged means the feature is ready? or how that roadmap works? tia

[derekbelrose]

@allejo that actually doesn’t fix this unless the setting to not trigger the workflow or to set a manual approval exists outside of the workflow. Any PR could change the workflow and it would override the existing rules, otherwise.

My needs require a workflow not to trigger if the workflow files have been changed in the PR or the push.

[dkurt]

I believe that there is a way to make Runner.Listener protect self-hoster runners using Secrets. From the description:

Secrets are not passed to workflows that are triggered by a pull request from a fork.

So repository owner can define a pair which can be verified at runner's side:

./config.sh --url https://github.com/username/repository --token *** --secret MY_SECRET:MY_VALUE

please take a look at this proposal here: https://github.com/actions/runner/issues/631

[DavidS]

@dkurt unless something changed since the last time I looked, nothing can stop the PR from removing that check from the workflow.

[dkurt]

@DavidS, But that would be a check on host side using the environment. Or do you mean there is a way to override GitHub's secret by PR's config? This way it won't work, right.

[DavidS]

@dkurt you're right, the secret won't be exposed. The problem here is that anyone who can post a PR can run arbitrary code on the self-hosted runner, which makes public repos with self-hosted runners on private hardware extremely dangerous.

[dkurt]

@DavidS, This is actually the problem of this topic. My proposal is to not run the code from PR from external users at all. The runner decides to run the workflow and it may have an option I called secret to match if we have an environment variable with specific name and value or not. According to GitHub docs,

Secrets are not passed to workflows that are triggered by a pull request from a fork.

So 1. We create a secret for specific repo 2. Start runner with that secret

[eladchen]

Hi Everyone,

I’ve put together a Github App that will trigger a workflow based on the changed files, and the actor behind the change.

I’m looking for a few early adopters to provide feedback before releasing it.
If you are interested please let me know, and I’ll setup a place for us to talk.

(Check out my last two comments for reference)

[timharris777]

@eladchen , I'm interested.

[derekbelrose]

I'd definitely be interested in testing this out.

[eladchen]

Hi @timharris777 @derekbelrose

I've created the below community in gitter for us to talk.
If anyone else is interested in hearing about the bot, please feel free to join.

https://gitter.im/github-protected-workflows/community

[timharris777]

Hi @eladchen, I already wrote a github app for our org that covers the functionality we need until github decides to cover this feature. Thanks for reaching back out though.

[eladchen]

How about this:

An example:

Let's say .github/workflows/pull-request.yml exists, and the suggested checkbox above is checked, and
user John / DEV team is configured.

Workflow ".github/workflows/pull-request.yml" will not be executed if:

• A pull request is created based of a forked repository
• The pull request changed files include .github/workflows/pull-request.yml
• The pull request is not authored (triggered?) by user John / a member of DEV team.

This allows contributors to create pull requests, and have workflows provide them (and maintainers) with feedback on the changes introduced by the pull request, and, in the same time preventing untrusted parties from changing a workflow, thus preventing any security issues that might arise.

WDYT?

Protected Workflows was released. Read this article on medium for details:

https://medium.com/@eladchen/protecting-your-github-workflows-b4359683bd69

[ashb]

To add to some of the requirements/ideas presented above: I work on apache/airflow, but due to how Apache Software Foundations manages repos we don't have admin access to the repos -- so it would be nice if we could _also_ set this at the runner level.

[ba32107]

Hi @eladchen, is your GitHub App open source by any chance? I'd be interested to see the implementation. I've spent a bit of time browsing the available webhooks, but it's not immediately obvious to me which webhook I could use to intercept workflow runs. There is workflow_dispatch, which is for manual workflow invocation (not very relevant for our use-case), and workflow_run, which, judging by the docs, doesn't seem to offer information about the current workflow being executed. If your App is not open source, could you share some pointers on how you managed to intercept workflow invocations?

Edit: it might be the case that the GitHub docs for the workflow_run are not complete or I'm missing something, but it seems strange that the payload doesn't contain any information about the workflow itself.

[ashb]

@ba32107 See https://github.com/potiuk/get-workflow-origin that one of my fellow committers on Apache Airflow created to get the "triggering run" for a PR

[ba32107]

Hi @ashb, thank you, I had a look at that project. Unfortunately it seems that this only works in the context of a workflow. If I understand correctly, the way this works is you need to explicitly pass ${{ github.event.workflow_run.id }} into this action.

According to the docs,, the github.event contains the webhook payload, which I assume in this case is workflow_run. However on the docs for workflow_run, I don't see any id field which would allow me to retrieve the workflow.

All that having been said, my use case is actually a bit different. I want to intercept workflow runs not in another workflow run, but in a GitHub App. To test this, I created a GitHub App and subscribed to all events. For some reason I can't get the workflow_run event to dispatch, no matter what I do. I can make the workflow_dispatch event fire by manually kicking off a workflow, but not workflow_run - so I am unable to examine the payload for myself.

Going one step further, let's assume that I am able to "catch" the currently running workflow in my App. How can I interact with that workflow run _before_ it starts? I want to be able to cancel it before it actually starts, or better yet, modify the workflow definition on the fly. This would be my ultimate goal. Does anyone know if this is possible?

It seems as if @eladchen's App does exactly what I described above, which is why I was wondering if his App is open source or not.

Edit: looks like the app is not open source - still, I'd be very interested to learn if my above use-case is possible to solve.

[edtan]

@ba32107 - It seems like the example in the docs for workflow_run does not contain all the fields. I just looked at some workflow_run events that my own app was receiving, and it has whole workflow_run and workflow sections. E.g.:

"workflow_run": {
    "id": 334440958,
    "node_id": "MDExOldvcmtmbG93UnVuMzM0NDQwOTU4",
    "head_branch": "master",
    "head_sha": "f57b60c6dae6f0a0734962014ec601c57e8dbf13",
    "run_number": 65,
    "event": "repository_dispatch",
    "status": "completed",
    "conclusion": "failure",
    "workflow_id": 2102793,
    "check_suite_id": 1416186754,
    "check_suite_node_id": "MDEwOkNoZWNrU3VpdGUxNDE2MTg2NzU0",

...

 "workflow": {
    "id": 2102793,
    "node_id": "MDg6V29ya2Zsb3cyMTAyNzkz",
    "name": "Deploy application",
    "path": ".github/workflows/deploy.yml",
    "state": "active",
    "created_at": "2020-08-05T03:16:17.000Z",
    "updated_at": "2020-08-05T04:27:12.000Z",
...

As you can probably tell from the example above, I'm starting the workflows above through a repository_dispatch event. I'm not sure why you're unable to receive workflow_run events.

Note that there is a corresponding event with a status of requested. So for your use case, I guess you can try to immediately cancel a workflow once you see that a workflow run has been requested.

[ba32107]

Thanks very much @edtan, this is very useful. I will have another look at receiving the events - I might have overlooked something. But this does sound promising! As for canceling it, I guess it comes down to timing: my request to cancel the workflow must reach GitHub _after_ the requested workflow event is dispatched, but _before_ the run actually starts. Relying on this timing does not sound ideal - I will have a look and report back.

[ashb]

Thanks very much @edtan, this is very useful. I will have another look at receiving the events - I might have overlooked something. But this does sound promising! As for canceling it, I guess it comes down to timing: my request to cancel the workflow must reach GitHub _after_ the requested workflow event is dispatched, but _before_ the run actually starts. Relying on this timing does not sound ideal - I will have a look and report back.

That's why I've gone for the approach of opening a pr against the runner - anything else seems to prone to race conditions. Not something you want for security controls

[adiroiban]

As mentioned by Chris , having the runner authenticate the workflow execution is a bit too late.

I guess that pull_request is the only trigger that allows external (non-write) collaborators.
I think workflow_dispatch required write permissions.

My suggestion is to allow running a custom workflow from a file that is local to the runner (or another protected repo), before running any PR workflow.

In this way, external collaborators can't overwrite the checks.

This assumes that 99% of the builds are from good people and will be approved anyway :)

The main protection is to prevent running random spam/botnet code from random github users.

[eladchen]

What's wrong with https://github.com/eladchen/protected-workflows ;P

[adiroiban]

:) Nice try @eladchen ... I did check it.

I am not sure that it can guarantee that the workflow is cancelled before any damaged is made.

A malicious user can push a workflow in which the first step is creating a quick daemon that runs in the background ... and even if you cancel the workflow, the process will still be there.

[eladchen]

You'll still enjoy the added value of the alerts sent by github when the workflow is cancelled.

[ashb]

As mentioned by Chris , having the runner authenticate the workflow execution is a bit too late.

@adiroiban Can you explain a bit more what you mean here?

[ashb]

What's wrong with https://github.com/eladchen/protected-workflows ;P

@eladchen That I can't verify how it works -- the app that powers it is not open source, so you are asking me to trust the security of my machines to you.

[eladchen]

@ashb Sorry you feel that way. Still, people use closed source software all the time.

Given enough installations, GitHub will audit the app, hopefully making you feel a little more comfortable using it.

[adiroiban]

@ashb I was saying that for MS/GitHub team restring a job/workflow at runner level is not that useful ... as by the time the job is executed on the runner, the MS/GitHub infrastructure has already started a new VM.

This needs to be implemented the GitHub action workflow dispatcher level and don't even bother talking with the runner.

---

But, it would help if we can have at least the option to not run workflows for PR from forks.
In a similar way in which we have protection for the secrets from fork.

If a PR will not trigger the workflow, we could then create an app that will listed for PR events and decide whether to dispatch the workflow or not.

[ba32107]

But, it would help if we can have at least the option to not run workflows for PR from forks.
In a similar way in which we have protection for the secrets from fork.

I think that's already the case, isn't it? Here's a screenshot of my private repo settings:

I did not disable this, it's off by default. The same can be applied to a public repo, if you move it to its own organization.

See also: https://github.blog/2020-08-03-github-actions-improvements-for-fork-and-pull-request-workflows/

[adiroiban]

Thanks @ba32107

My requirement is as follows:

• Have a similar feature for public repos
• Limit execution for forks only on self-hosted runners.

The use case is something like this.

There is an open source project with support for Linux, Windows and macOS ... but the project also supports Linux on ARM64.
The ARM64 runners are donated / maintained by a community member.
99% of the code will work on any OS. 99% of the PR are from core team members. Non-forks.
We still want to run tests on ARM64 on PR (as least for team members) to make sure ARM64 has no regression.

But maybe we just need to wait for GitHub Action to provide ARM64 runners and we will no longer need the self-hosted runners

[ashb]

@ashb I was saying that for MS/GitHub team restring a job/workflow at runner level is not that useful ... as by the time the job is executed on the runner, the MS/GitHub infrastructure has already started a new VM.

@adiroiban This issue is talking about _self hosted_ runners -- so GitHub isn't allocating any VMs.

@ba32107 That's something slightly different -- i.e. don't run actions for PRs from external contributors. Which is not what we (Apache Airflow team) want. We want some workflows (i.e. ones with privileged scopes, builds against main branch etc.) to run on our runners, but for third party PRs (i.e. all the contributions we get) we want to use the GitHub infrastructure to run on.

Also -- that only applies to private repos, of which apache/airflow is not

[dentarg]

I think that's already the case, isn't it? Here's a screenshot of my private repo settings:

I did not disable this, it's off by default. The same can be applied to a public repo, if you move it to its own organization.

See also: https://github.blog/2020-08-03-github-actions-improvements-for-fork-and-pull-request-workflows/

That setting is only available for private repos, not public repos.

[ba32107]

I believe the setting could work on public repos. If you create an organization, and move your public repo there, you can find this setting on the organization's setting page. I haven't tested if it works though.

Edit: never mind, the setting actually says:

These settings apply to private repositories. Repository administrators will only be able to change the settings that are enabled here.

So I was wrong.