Runner: The runner always in busy state even the job is finished.

[missedone]

Describe the bug
I have self-hosted deployment in kubernetes.
I had a deployment job ran failed, but the runner can't pickup new job, checked the runner logs see it's stuck at 2020-09-11 03:47:04Z: Running job: Deploy even I can see the job failed on Actions tab.

To Reproduce
Steps to reproduce the behavior:

1. don't know how to reproduce it

Expected behavior
the runner properly complete the job no matter success or failure, so the subsequent job won't be impacted.

Runner Version and Platform

Version of your runner?
2.273.1

OS of the machine running the runner? OSX/Windows/Linux/...
Linux

What's not working?

Please include error messages and screenshots.

Job Log Output

If applicable, include the relevant part of the job / step log output here. All sensitive information should already be masked out, but please double-check before pasting here.

2020-09-11 02:27:55Z: Running job: Build
2020-09-11 02:35:52Z: Job Build completed with result: Succeeded
2020-09-11 02:35:55Z: Running job: Build
2020-09-11 02:36:27Z: Job Build completed with result: Failed
2020-09-11 02:36:30Z: Running job: Build
2020-09-11 02:37:25Z: Job Build completed with result: Succeeded
2020-09-11 02:37:28Z: Running job: Build
2020-09-11 02:38:14Z: Job Build completed with result: Succeeded
2020-09-11 02:38:17Z: Running job: Deploy
2020-09-11 02:38:32Z: Job Deploy completed with result: Failed
2020-09-11 03:11:24Z: Running job: Build
2020-09-11 03:12:02Z: Job Build completed with result: Succeeded
2020-09-11 03:12:11Z: Running job: Deploy
2020-09-11 03:12:26Z: Job Deploy completed with result: Failed
2020-09-11 03:14:41Z: Running job: Create Release
2020-09-11 03:15:58Z: Job Create Release completed with result: Failed
2020-09-11 03:18:13Z: Running job: Build
2020-09-11 03:18:48Z: Job Build completed with result: Succeeded
2020-09-11 03:18:58Z: Running job: Deploy
2020-09-11 03:19:12Z: Job Deploy completed with result: Failed
2020-09-11 03:20:49Z: Running job: Create Release
2020-09-11 03:23:26Z: Job Create Release completed with result: Failed
2020-09-11 03:46:17Z: Running job: Build
2020-09-11 03:46:54Z: Job Build completed with result: Succeeded
2020-09-11 03:47:04Z: Running job: Deploy

Runner and Worker's Diagnostic Logs

If applicable, add relevant diagnostic log information. Logs are located in the runner's _diag folder. The runner logs are prefixed with Runner_ and the worker logs are prefixed with Worker_. Each job run correlates to a worker log. All sensitive information should already be masked out, but please double-check before pasting here.

[BrightRan]

A GitHub Community ticket that reports the similar issues: https://github.community/t/github-actions-workflow-stuck/132330

All the jobs run on the self-hosted runners in the workflow were completed successfully, but the workflow run keeps displaying as in progress with the yellow icon. And other queued jobs can't use the self-hosted runners because they are in busy.
Reconnect the self-hosted runners and restart the hosted machines does not work.

[donalffons]

Hi! I think I'm experiencing the same issue as described by @missedone and @BrightRan. My project is located here. It contains the full code, so it is possible just to clone or fork it if someone wants to attempt to reproduce the issue.

I can (and have been) using GH actions to successfully run the build for this project. The project has gotten bigger over the last weeks and a few commits ago, the GH actions stopped working. The master branch currently does not have any issues. The issues are present in the embind branch.

I experienced the same issue with self-hosted runners and runners provided by github. Currently, the project is configured to use self-hosted runners. When I run the project directly as a docker container, everything works well.

Here are two good commits to look in to:

Failing Commit d4b65ea

d4b65ea is a "full build" of the entire project. The build is currently stuck and not responsive in the GH actions website. Here is what I currently see on the website:

When I locally connect to the docker container, in which the build is running, using docker attach, here is what I see:

As of right now, the build has not been running for too long. I'll keep it running over night and I'm very confident that it will not return successfully. Will post updates here.

Succeeding Commit 353d5f4

353d5f4 is a reduced build of the project. The code is 99% the same - the only difference is the amount of "stuff" that is built by the Emscripten compiler. This build works fine.

Direct (i.e. non GH-action) builds work for both commits

There is a small convenience script build.sh in the parent directory, which simply builds a docker container and then runs it. This script works fine with both commits.

I would greatly appreciate help with this issue.

edit: The failing build got cancelled automatically in the end. However, I noticed that also the "direct" build is also behaving weirdly and might be the actual cause of the issue. The docker container hangs at the end of the build and never returns. So, this seems not to be related to gh-actions and instead seems to be more of a docker-related issue...

edit2: I now think that this article describes my particular problem and presents a solution: https://blog.phusion.nl/2015/01/20/docker-and-the-pid-1-zombie-reaping-problem/ . Not sure if that's the case for the creator of this issue, but might be worth checking out.