@claude
The platform owner has asked you to evaluate these lessons from provisioning and onboarding a new k3s worker node as a CI runner, and update the relevant FuzeInfra docs (node provisioning runbook, cluster topology docs, ARC runner guide) where you deem applicable. Please consult Claude Opus 4.8 for the relevance decision on each item before making changes.
This is a companion to issue #221 (ARC runner onboarding per-repo guide). That issue covers what to do in each repo. This issue covers what to do at the cluster/infrastructure level when a new node is added.
Lesson 1: UFW on control plane must be updated for every new node (CRITICAL)
Already filed in #221 as a per-repo note — this is the infrastructure-level version.
What happened: fuzeinfra-ci-runner-1 joined the k3s cluster and its ARC scale set was registered. Runner pods started and showed 1/1 Running but appeared offline on GitHub. DNS resolution failed inside pods. Diagnosed via:
tcpdump -i flannel.1 on the runner node: DNS queries leaving via VXLAN ✓tcpdump -i eth0 on control plane: VXLAN packets arriving ✓ip -s link show flannel.1 on runner node: RX bytes = 0 (VXLAN responses never returned) ✗- Root cause: control plane UFW had no INPUT rule for UDP 8472 from the new node's IP
What belongs in the node provisioning runbook: After a node joins the k3s cluster, the control plane UFW must be updated:
# On control plane
NEW_NODE_IP=<new-node-external-ip>
NODE_NAME=<node-name>
ufw allow from $NEW_NODE_IP to any port 8472 proto udp comment "flannel vxlan from $NODE_NAME"
ufw allow from $NEW_NODE_IP to any port 6443 proto tcp comment "k3s api from $NODE_NAME"
ufw reload
This step is currently absent from any provisioning doc. It should be a mandatory, numbered step in the node provisioning runbook that cannot be skipped.
Lesson 2: New CI runner nodes need specific labels and taints before ARC runners will schedule
What happened: ARC scale set Helm values include a nodeSelector and toleration that must match the node. The node was added to the cluster but the labels/taints were not applied, so runner pods were Pending until this was discovered.
Required on every new CI runner node:
kubectl label node <node-name> fuzeinfra.io/pool=ci
kubectl taint node <node-name> fuzeinfra.io/ci=true:NoSchedule
What belongs in the provisioning runbook: Add these two commands as a required post-join step, alongside the UFW update.
Lesson 3: GitHub Actions minutes quota exhaustion is silent and looks like runner failure
What happened: Early in the runner onboarding, CI jobs were failing in 2-4 seconds with runner_name: "" and steps: [] and no logs. This looked like a runner configuration problem. The actual cause was GitHub Actions hosted-runner minutes quota exhausted for private repos (billed at 2× Ubuntu rate). The ARC runner onboarding was blocked by quota exhaustion, not misconfiguration.
Diagnosis: gh api orgs/<org>/settings/billing/actions shows minutes used vs included. When quota is exceeded, jobs are queued and immediately dropped with an empty run record.
What belongs in docs: Add a "Symptoms / Not the runner" section to the ARC runner troubleshooting guide noting that quota exhaustion looks identical to a broken runner and should be ruled out first before debugging infrastructure.
Lesson 4: VXLAN return path diagnosis — a quick 4-step checklist
What happened: Diagnosing why runner pods couldn't resolve DNS took ~2 hours because the diagnostic path wasn't documented. The symptom (DNS timeout inside pods, direct IP works) is a known failure mode for broken Flannel overlays.
Suggested addition to the cluster troubleshooting runbook:
Symptom: Pods can curl external IPs but DNS resolution fails.
Quick diagnosis:
1. ip -s link show flannel.1 # on the affected node — if RX=0, return path broken
2. tcpdump -i eth0 udp port 8472 # on control plane — do VXLAN packets arrive?
3. ufw status | grep 8472 # on control plane — is the source node IP allowed?
4. Fix: ufw allow from <node-ip> to any port 8472 proto udp
This 4-step path gets to root cause in under 5 minutes.
Lesson 5: ARC listener pod namespace vs runner pod namespace
What happened: Debugging was complicated by not knowing which namespace to look in for which pods. Listener pods are in arc-systems; runner pods are in arc-runners. The controller is also in arc-systems. This is documented in ARC docs but not in our internal runbook.
Suggested addition to ARC runner troubleshooting:
| What you're looking for |
Namespace |
Command |
| Scale set listener (per repo) |
arc-systems |
kubectl get pods -n arc-systems | grep <slug> |
| Runner pods (ephemeral) |
arc-runners |
kubectl get pods -n arc-runners -l app.kubernetes.io/name=<slug> |
| Controller |
arc-systems |
kubectl get pods -n arc-systems | grep controller |
STATE for this issue:
The platform owner requests Opus 4.8 evaluate each item for inclusion worthiness before any edits are committed.
@claude
The platform owner has asked you to evaluate these lessons from provisioning and onboarding a new k3s worker node as a CI runner, and update the relevant FuzeInfra docs (node provisioning runbook, cluster topology docs, ARC runner guide) where you deem applicable. Please consult Claude Opus 4.8 for the relevance decision on each item before making changes.
This is a companion to issue #221 (ARC runner onboarding per-repo guide). That issue covers what to do in each repo. This issue covers what to do at the cluster/infrastructure level when a new node is added.
Lesson 1: UFW on control plane must be updated for every new node (CRITICAL)
Already filed in #221 as a per-repo note — this is the infrastructure-level version.
What happened:
fuzeinfra-ci-runner-1joined the k3s cluster and its ARC scale set was registered. Runner pods started and showed1/1 Runningbut appearedofflineon GitHub. DNS resolution failed inside pods. Diagnosed via:tcpdump -i flannel.1on the runner node: DNS queries leaving via VXLAN ✓tcpdump -i eth0on control plane: VXLAN packets arriving ✓ip -s link show flannel.1on runner node: RX bytes = 0 (VXLAN responses never returned) ✗What belongs in the node provisioning runbook: After a node joins the k3s cluster, the control plane UFW must be updated:
This step is currently absent from any provisioning doc. It should be a mandatory, numbered step in the node provisioning runbook that cannot be skipped.
Lesson 2: New CI runner nodes need specific labels and taints before ARC runners will schedule
What happened: ARC scale set Helm values include a
nodeSelectorand toleration that must match the node. The node was added to the cluster but the labels/taints were not applied, so runner pods were Pending until this was discovered.Required on every new CI runner node:
What belongs in the provisioning runbook: Add these two commands as a required post-join step, alongside the UFW update.
Lesson 3: GitHub Actions minutes quota exhaustion is silent and looks like runner failure
What happened: Early in the runner onboarding, CI jobs were failing in 2-4 seconds with
runner_name: ""andsteps: []and no logs. This looked like a runner configuration problem. The actual cause was GitHub Actions hosted-runner minutes quota exhausted for private repos (billed at 2× Ubuntu rate). The ARC runner onboarding was blocked by quota exhaustion, not misconfiguration.Diagnosis:
gh api orgs/<org>/settings/billing/actionsshows minutes used vs included. When quota is exceeded, jobs are queued and immediately dropped with an empty run record.What belongs in docs: Add a "Symptoms / Not the runner" section to the ARC runner troubleshooting guide noting that quota exhaustion looks identical to a broken runner and should be ruled out first before debugging infrastructure.
Lesson 4: VXLAN return path diagnosis — a quick 4-step checklist
What happened: Diagnosing why runner pods couldn't resolve DNS took ~2 hours because the diagnostic path wasn't documented. The symptom (DNS timeout inside pods, direct IP works) is a known failure mode for broken Flannel overlays.
Suggested addition to the cluster troubleshooting runbook:
This 4-step path gets to root cause in under 5 minutes.
Lesson 5: ARC listener pod namespace vs runner pod namespace
What happened: Debugging was complicated by not knowing which namespace to look in for which pods. Listener pods are in
arc-systems; runner pods are inarc-runners. The controller is also inarc-systems. This is documented in ARC docs but not in our internal runbook.Suggested addition to ARC runner troubleshooting:
arc-systemskubectl get pods -n arc-systems | grep <slug>arc-runnerskubectl get pods -n arc-runners -l app.kubernetes.io/name=<slug>arc-systemskubectl get pods -n arc-systems | grep controllerSTATE for this issue:
fuzeinfra-ci-runner-1is the node that was provisioned and exhibited all these gapsThe platform owner requests Opus 4.8 evaluate each item for inclusion worthiness before any edits are committed.