Why are our actions being stopped?

[io7m]

Select Topic Area

Question

Body

Hello!

We've been using GitHub Actions for years now to build an Android app:

https://github.com/ThePalaceProject/android-core

About a week ago, without any changes from us, all of our builds on the main branch started failing after about ten minutes. There's no error message other than:

The runner has received a shutdown signal. This can happen when the runner service is stopped, or a manually started runner is canceled.

1. We have no timeouts configured, so the actions aren't being killed by any timeout that we've set.
2. We have no other actions that are being triggered while these builds are running.
3. As far as I've been able to determine, we aren't running out of resources. Our Gradle builds run with a 4gb heap, while the runners are supposed to have 7gb of memory. We've tried adding varying degrees of swap space, but it makes no difference.

Is there some way that someone at GitHub can tell us why our builds are failing like this? It's impossible to debug when there's no useful error message. At this point, I can only assume our builds are being killed for taking excessive CPU time; dexing Java bytecode into Android bytecode is a very expensive process.

Somewhat strangely, we had one build run to completion earlier without being killed. We hadn't changed anything!

[appatalks]

👋 @io7m!

We value our community engagement and wanted to Thank you for reaching out!

I am wondering if a minor update to Ubuntu-latest image changed in-between the last successful run and when the build started failing, caused this build step trouble to surface 🤔

From the build logs, it does seem to be running into a runner shutdown signal:

##[debug]Skip Re-evaluate condition on runner shutdown.
System.Threading.CancellationToken.ThrowOperationCanceledException()

I am in agreement with your suspicion that this might be due to some sort of resource constraint that could be in play based off the runner getting that shutdown signal.

For example, in the debug archive download that can be grabbed from the Workflow run. I see some hints that some processes are being adjusted for memory utilization:

./runner-diagnostic-logs/Worker_20230828-113023-utc.log

[2023-08-28 11:30:28Z INFO ProcessInvokerWrapper] Updated oom_score_adj to 500 for PID: 1762.
[2023-08-28 11:30:29Z INFO ProcessInvokerWrapper] Updated oom_score_adj to 500 for PID: 1831.
[2023-08-28 11:30:34Z INFO ProcessInvokerWrapper] Updated oom_score_adj to 500 for PID: 1845
[2023-08-28 11:52:12Z INFO ProcessInvokerWrapper] Kill entire process tree since both cancel and terminate signal has been ignored by the target process.

Generally the standard configurations for GitHub hosted runners should do well for many uses, though as projects grow, become more complex, sometimes 2-cores just no longer cuts it and jumping into larger hardware might be the next natural solution.

However, before going that route, I noticed in the Android CI (Authenticated) Action, the swap is being lowered to 2G than the provisioned 4G of swap (prior to that I saw it was also attempted to being manually set to 16G from the commit history) that I checked in a previous run. The drives only come with 14G of SSD so I am understanding why that was reverted.

So I am thinking that playing with the swap is something I might try to get past this hurdle too, or if possible reduce the resource consumption of the build steps in some fashion.

On a quick side note, I found this community thread very insightful while studying this issue.

I hope this helps! Please let me know 🐧

-- @appatalks

[io7m]

We just upgraded to the smallest paid runner, and the build succeeded! It took roughly the same time that the unpaid runner took to fail, so I think we must have been right on the edge in terms of CPU usage.

We're going to be putting in some work to cut down on the amount of bytecode that needs to be processed, as that's where the build spends most of its time.

Thanks for the time!

[appatalks]

That's really great news to hear, Thank you for the update!

[axeluhl]

But then, what's the limit of CPU usage? I only found out about 6h total time limit. The way I read this, 6h means "wall time" and not CPU time, correct? So were you anywhere near those 6h total workflow duration, or else, what are CPU usage limits?

[io7m]

I think the only thing I could do would be to somehow log resource usage (cpu, memory, i/o) over an entire build and say "this is too much for the unpaid runner".

[io7m]

For what it's worth, I don't think it's all that important to know the limits. It's not like I can control how much CPU time or memory a given task takes. It would be good, however, if the system unambiguously made it clear that we'd exceeded internal limits.

[aubertc]

I had the same issue starting with the recent update of the Ubuntu 24.04 LTS runner image.

My hunch was also that I was exhausting resources, but the logs were not helpful.

The runner is using Makefile with the -j option that maximizes parallelism, but I guess this was creating too many threads at the same time. Limiting the number of jobs with -l 2.5 solved the issue.