Listeners are sometimes scheduled on Windows nodes

[macux]

Checks

• I've already read https://github.com/actions/actions-runner-controller/blob/master/TROUBLESHOOTING.md and I'm sure my issue is not covered in the troubleshooting guide.
• I'm not using a custom entrypoint in my runner image

Controller Version

0.4.0

Helm Chart Version

0.4.0

CertManager Version

N/a

Deployment Method

Helm

cert-manager installation

N/a

Checks

• This isn't a question or user support case (For Q&A and community support, go to Discussions. It might also be a good idea to contract with any of contributors and maintainers if your business is so critical and therefore you need priority support
• I've read releasenotes before submitting this issue and I'm sure it's not due to any recently-introduced backward-incompatible changes
• My actions-runner-controller version (v0.x.y) does support the feature
• I've already upgraded ARC (including the CRDs, see charts/actions-runner-controller/docs/UPGRADING.md for details) to the latest and it didn't fix the issue
• I've migrated to the workflow job webhook event (if you using webhook driven scaling)

Resource Definitions

# install arc
helm install arc `
    --namespace my-arc-namespace `
    --create-namespace `
    --set githubConfigSecret.github_token=$myPat `
    --set nodeSelector."kubernetes\.io/os"=linux `
    oci://ghcr.io/actions/actions-runner-controller-charts/gha-runner-scale-set-controller `
    --version 0.4.0

# linux-values.yaml contents (for linux based jobs):
template:
  spec:
    nodeSelector:
      kubernetes.io/os: linux

# windows-values.yaml contents (for windows based jobs):
template:
  spec:
    nodeSelector:
      kubernetes.io/os: windows
    containers:
    - name: runner
      image: location-of-my-windows-runner-container/my-windows-runner:latest

# scale sets are installed using the helm chart and one of the overrides above, e.g. for windows:
helm upgrade --install my-windows-runner-set `
    --namespace my-runners-namespace `
    --create-namespace `
    --set githubConfigUrl=$myGitHubUrl `
    --set githubConfigSecret.github_token=$myPat `
    --values windows-values.yaml `
    oci://ghcr.io/actions/actions-runner-controller-charts/gha-runner-scale-set `
    --version 0.4.0

To Reproduce

Using AKS:

# option 1:

1. Create an aks cluster which will have a linux based node pool for the control plane called "nodepool1"
2. Add a Windows node pool called "wpool" with a single Windows node
3. Install the ARC controller via the helm chart as shown above
4. Install a Windows runner scale set via helm chart as shown above
5. The scale set listener is scheduled on the Windows node created in step 2 and not on a linux node in "nodepool1" (despite its name - see below)

# option 2: As for #1 - but add a linux based user nodepool called "zzzpool" - same issue

# option 3: As for #1 - but add a linux based user nodepool called "aaapool" - this works, listeners are scheduled on this and Windows runners are correctly scheduled on the windows node pool

# option 4: As for #1 - but create the Windows node pool with zero nodes and auto scaling enabled - this also works but isn't a feasible workaround (and is obviously also slower when a Windows runner is required)

Note - if a listener pod needs to be recreated for any reason this same issue can occur.

Describe the bug

Described in the steps to reproduce above.

Describe the expected behavior

Listener pods are always scheduled on linux nodes regardless of the nodepool name.

Whole Controller Logs

-

Whole Runner Pod Logs

-

Additional Context

No response

[github-actions]

Hello! Thank you for filing an issue.

The maintainers will triage your issue shortly.

In the meantime, please take a look at the troubleshooting guide for bug reports.

If this is a feature request, please review our contribution guidelines.

[macux]

Small clarification on the above - it makes no difference if you are intending to install a Windows runner scale set (as described in step 4 above). The same issue occurs if you create a linux based runner scale set (e.g. using the linux-values.yaml also shown above).

Just did another test:

1. Created an AKS cluster
2. Added a Windows node pool named "wpool1" with 1 node - AKS created the node with name "akswpool1000000"
3. Installed a scale set with a nodeSelector as shown above in linux-values.yaml - the listener gets scheduled on the windows node
4. Deleted the listener pod and it gets recreated on the Windows node again
5. Added a linux node pool named "znixpool" with 1 node - AKS created the node with name "aks-znixpool-37486904-vmss000000"
6. Deleted the listener pod again - this now correctly schedules the pod on the node in znixpool1

So it seems the ordinal name of the node is relevant, not the name of the node pool - this is potentially a worse issue therefore as on AKS (as far as I know) you can't change the name of the node within a pool (not possible to use kubeadm on AKS?). Also, the only reason the above works is because AKS names the underlying managed scale sets differently for Windows & Linux:

Windows: aks<nodepool name>
Linux:   aks-<nodepool name>-<number>-vmss

And the AKS nodes just use the same names as above with a node ordinal suffix. In other words "aks-znixpool-..." comes before" akswpool1".

I think I can see where the necessary changes would be so I'll post another reply shortly detailing a potential PR.

Note - you could say what I've reported in this comment is "by design" as I'm effectively relying on AKS scheduling the listener on a control plane node, however my original test did show this also happening when linux user nodes were also present.

[macux]

Looks like the simplest PR here would be to use a nodeSelector in resourcebuilder.go where the PodSpec is created for a listener:

podSpec := corev1.PodSpec{
	ServiceAccountName: serviceAccount.Name,
	Containers: []corev1.Container{
		{
			Name:            autoscalingListenerContainerName,
			Image:           autoscalingListener.Spec.Image,
			Env:             listenerEnv,
			ImagePullPolicy: autoscalingListener.Spec.ImagePullPolicy,
			Command: []string{
				"/github-runnerscaleset-listener",
			},
		},
	},
	ImagePullSecrets: autoscalingListener.Spec.ImagePullSecrets,
	RestartPolicy:    corev1.RestartPolicyNever,
	NodeSelector:     map[string]string{
		corev1.LabelOSStable: corev1.Linux
	},
}

The above constants translate to: "kubernetes.io/os": "linux".

An extension of this could be to also define the node selector in the AutoscalingListenerSpec and then set its value in resourcebuilder.go but this seems superfluous....

@Link- could you let me know if this approach is acceptable and I will create a PR, thanks.

[macux]

One further update, I wanted to double check the original issue as my clarification post seemed to contradict it. This is partly true - my original test was using AKS with k8s v1.26.3 and my clarification post uses k8s v1.27.1 (currently in preview on AKS but due for GA this month).

I created a 1.26.3 cluster, ran my original test and it failed as described, then upgraded the cluster to 1.27.1, repeated the test and it worked. (My clarification post still fails as descried on 1.27.1, although this is less serious/likely).

So I'm now not 100% clear on whether there is a workaround using nodepool or node name and whilst it may seem to be fixed in 1.27.1 I think that the listener pod should be scheduled explicitly to avoid any of the above issues. This could be a blocker for us as we have to run Windows containers in our ARC cluster.

[antoine-axa]

Same issue here. I'm also using Kubernetes in version 1.26.3.
Testing ARC with an AKS cluster and it's working great with linux runners.

However I tested to deploy a new Windows node pool to host the Windows runners and when I deployed a new Linux runner scale set, the listener pod is automatically hosted on the Windows pool so it doesn't work.

To complete, I tested to remove the Windows node pool and redeploy a new Linux scale set and the listener pod is now schedule on the default node pool which is a system node pool.
So it would be great if we can have a way to specify the node pool for both the listener and the runners pods.

[mhuijgen]

I think the proper solution is indeed that the controller sets the NodeSelector to Linux for the listener pods it creates.

As a workaround in the mean time you can apply a NoSchedule taint to your windows nodepool in AKS (requires you to recreate the pool unfotunately) and add a toleration for this taint on any windows runner scale sets you have.

Example snippet from values file for scale set after creating the windows pool with taint
key: windows
value: only
effect: NoSchedule

template:
  spec:
    tolerations:
    - key: "windows"
      operator: "Equal"
      value: "only"
      effect: "NoSchedule"
    nodeSelector:
      kubernetes.io/os: windows

[macux]

Thanks @mhuijgen, that's actually what I've now done, I'd tested a taint previously and my notes say it didn't work but tried it again on Friday on 1.27.1 and it did work as required. I think I missed to also include the node selector = Windows in the values yaml for the windows scale set last time. I tested this with multiple scale sets including windows and Linux and they all worked as expected.

[nikola-jokic]

Hey, just to update you all on this issue: There is an ongoing work allowing modifications for a pod spec for the listener pod: #2758