Github Actions ARC - Ephemeral Runner Pod does not stand up

[juleszembik]

Why are you starting this discussion?

Question

What GitHub Actions topic or product is this about?

Actions Runner

Discussion Details

Hi everyone,

My team is currently experiencing an issue from the Github Actions server down time yesterday. Our runners were working fine until after the outage. A few of our runners are unable to stand up correctly. ARC controller is able to receive and send out a command to stand up a runner pod, however for some reason, the runner does not stand up even after rolling the specific controller for that pod and the GHA controller. Any ideas I can do to troubleshoot the runner?

Below is the log pulled from controller in charge of the specific missing runner.

2025-06-06T04:22:55Z  INFO  listener-app.worker.kubernetesworker  Compare  {"original": "{\"metadata\":{\"creationTimestamp\":null},\"spec\":{\"replicas\":-1,\"patchID\":-1,\"ephemeralRunnerSpec\":{\"metadata\":{\"creationTimestamp\":null},\"spec\":{\"containers\":null}}},\"status\":{\"currentReplicas\":0,\"pendingEphemeralRunners\":0,\"runningEphemeralRunners\":0,\"failedEphemeralRunners\":0}}", "patch": "{\"metadata\":{\"creationTimestamp\":null},\"spec\":{\"replicas\":2,\"patchID\":4,\"ephemeralRunnerSpec\":{\"metadata\":{\"creationTimestamp\":null},\"spec\":{\"containers\":null}}},\"status\":{\"currentReplicas\":0,\"pendingEphemeralRunners\":0,\"runningEphemeralRunners\":0,\"failedEphemeralRunners\":0}}"}
2025-06-06T04:22:55Z  INFO  listener-app.worker.kubernetesworker  Preparing EphemeralRunnerSet update  {"json": "{\"spec\":{\"patchID\":4,\"replicas\":2}}"}
2025-06-06T04:22:55Z  INFO  listener-app.worker.kubernetesworker  Ephemeral runner set scaled.  {"namespace": "runner-scale-set", "name": "runner-scale-set-7s8fb", "replicas": 2}
2025-06-06T04:22:55Z  INFO  listener-app.listener  Getting next message  {"lastMessageID": 100001268}```

[juleszembik]

Solved issue.

Ran kubectl get ephemeralrunner to show failed runners. Run kubectl delete ephemeralrunner <runnernames> to delete failed runners and release resources to get runners back up and running. Ran test workflow to verify working runners.