Confusion about correct use of ARC self-hosted runners

[sharvil10]

Why are you starting this discussion?

Question

What GitHub Actions topic or product is this about?

ARC (Actions Runner Controller)

Discussion Details

I have a question about how to correctly use the GitHub Actions Runner Controller to self-host runners.

Problem statement: Run entire CI pipeline on self-hosted Kubernetes cluster. We also have multiple teams so each team will have at least 1 CI pipeline in the same cluster.

Questions:

1. Note that we're not "deploying" an app or doing any CD in Kuberentes. But rather we're trying to run each stage/step of our CI pipeline in a Kubernetes pod. Which means the pods need to run and finish and move on to the next stage. This means we can't use the azure/k8s-deploy actions. Instead ARC is the better choice here. Is that assumption correct?
2. Assuming ARC is the way to go we could use the provided minimal images from ARC and scale runners in the runner-set. However, this works only for the pattern where we use the minimal image and then we install dependencies for builds in the CI pipeline code. However, we prefer to start from a custom image(which will also be built using a Kaniko docker-build stage in a seprate pipeline which will deploy the runner set.) with the environment setup correctly and just run the build commands. This helps us test the build image in the same CI as well and we don't have to change the Dockerfile instructions separately(basically the environment setup has only 1 source) . However, because of this each branch in the repo could have its own image if they change the Dockerfile. Which means that we need to create different runner-set deployments for each branch in each repo if you specify the image for the runner set? Is this the correct way to use ARC? Does Kubernetes mode provide solution to this?

Apologies in advanced if this is a very basic question.

[JonathanZawadke]

1. ARC vs. azure/k8s-deploy
   You're correct since you're not deploying apps to Kubernetes, but just want to run CI steps inside Kubernetes pods, ARC (Actions Runner Controller) is the better choice. azure/k8s-deploy is more for CD use cases, where you're pushing an app to a cluster. ARC lets you run GitHub Actions jobs in Kubernetes pods which fits your use case.

2. Custom runner images per branch
   Yes, ARC does support custom runner images, and using a pre-built image for your runners is a valid and efficient approach.

But you're right: if each branch can have its own runner image, then you would need a different RunnerDeployment (runner-set) per image, since each RunnerDeployment uses one image. ARC doesn’t support dynamically changing the image per job at runtime.

This can get complex if many branches have different images.

3. Alternative approach
   You might want to consider:

• Using one base image and installing some branch-specific tools during the job.
• Or, dynamically creating RunnerDeployments with automation (like a controller or script) only when needed.

Also: Kubernetes mode in ARC is mainly about how runners are run (as pods, via controllers), not about switching images dynamically.

[adityajethwa7]

Hi @sharvil10,

Your use case is quite common in enterprise environments. Let me address your questions with practical solutions:

1. ARC vs azure/k8s-deploy

You're absolutely correct. ARC is the appropriate choice for running CI workloads on Kubernetes. The azure/k8s-deploy action is designed for deploying applications to clusters, not for executing CI jobs within pods.

2. Dynamic Image Management Strategy

Instead of creating separate runner-sets for each branch, here's a more scalable approach:

# Use ARC's containerMode with dynamic image selection
apiVersion: actions.summerwind.dev/v1alpha1
kind: RunnerDeployment
metadata:
  name: dynamic-runner
spec:
  template:
    spec:
      containers:
      - name: runner
        image: ghcr.io/actions/actions-runner:latest  # Base runner
      - name: docker
        image: docker:dind  # For building images

3. Implement a Job-Level Image Strategy

Use GitHub Actions to dynamically build and use images within the job:

jobs:
  ci:
    runs-on: [self-hosted, kubernetes]
    steps:
      - uses: actions/checkout@v4
      
      - name: Build CI Image
        run: |
          # Build image specific to this branch/commit
          docker build -t ci-image:${{ github.sha }} .
      
      - name: Run Tests in Custom Image
        uses: docker://ci-image:${{ github.sha }}
        with:
          args: |
            make test

4. Kubernetes Jobs Mode (Recommended)

Consider using ARC's Kubernetes mode with ephemeral runners:

apiVersion: actions.summerwind.dev/v1alpha1
kind: RunnerDeployment
metadata:
  name: ephemeral-runner
spec:
  template:
    spec:
      ephemeral: true  # Runner terminates after job
      dockerEnabled: true
      dockerdWithinRunnerContainer: true

This allows you to:

• Use docker-in-docker to build images within jobs
• Avoid runner-set proliferation
• Maintain image flexibility per branch

5. Multi-Team Isolation Pattern

For multiple teams, implement namespace-based isolation:

# Create team-specific namespaces
kubectl create namespace team-a-runners
kubectl create namespace team-b-runners

# Deploy ARC per namespace with RBAC
helm install arc-team-a \
  --namespace team-a-runners \
  --set githubConfigUrl="https://github.com/org/team-a" \
  actions-runner-controller/actions-runner-controller

Alternative Architecture:

If branch-specific images are critical, consider a GitOps approach:

1. Use a separate repository for runner configurations
2. Implement a webhook/controller that creates RunnerDeployments on-demand
3. Auto-cleanup unused runners after inactivity

# Example controller logic
def on_workflow_requested(event):
    branch = event['ref']
    image = f"ghcr.io/org/ci-image:{branch}"
    
    # Create RunnerDeployment if not exists
    if not runner_exists(branch):
        create_runner_deployment(branch, image)
    
    # Set TTL for auto-cleanup
    set_runner_ttl(branch, "1h")

This approach scales better than static runner-sets per branch and maintains the flexibility you need.