GitHub Community
How are you doing metrics? #160451
-
|
Hi All I have since deployed ARC and have managed to get the self hosted runners up and running. We currently use Grafana internally so looking to see how I could use this with the runner deployment metrics Any help appreciated! |
Beta Was this translation helpful? Give feedback.
Replies: 14 comments 28 replies
-
|
You can use Prometheus to scrape metrics on action-controller-runner. Then you can connect Prometheus to Grafana The current version exposed following metrics.
In next release we will have following metrics too apart from above ones (PR already merged)
For some constraints we were using only the PercentBusy metric and not webhook metrics, we created a Grafana dashboard to track autoscaling metrics.
|
Beta Was this translation helpful? Give feedback.
-
|
@debugger24 Could you share the grafana json please |
Beta Was this translation helpful? Give feedback.
-
are these metrics usable if you're doing webhook based autoscaling, i.e. |
Beta Was this translation helpful? Give feedback.
-
|
@debugger24 can you provide the grafana dashboard json ? |
Beta Was this translation helpful? Give feedback.
-
|
If you could share the grafana dashboard json it would be very helpful , as it saves me half day 🫶 |
Beta Was this translation helpful? Give feedback.
-
|
Hey, I just published this Grafana dashboard. |
Beta Was this translation helpful? Give feedback.
-
|
@debugger24 that's great, thanks for the help! set { Did you have to do any additional Config to get this working? Installed Prometheus Stack : |
Beta Was this translation helpful? Give feedback.
-
Can you share more details what errors are you getting ? |
Beta Was this translation helpful? Give feedback.
-
|
Not Seeing any specific error message just not seeing it picked up under service discovery - however github-actions-exporter is being picked up also using serviceMonitor. |
Beta Was this translation helpful? Give feedback.
-
|
Actions Metrics Yaml Output Github Actions Exporter Yaml Output |
Beta Was this translation helpful? Give feedback.
-
|
Tried to Disable the Proxy but leave ServiceMonitor enabled - Still not being picked up |
Beta Was this translation helpful? Give feedback.
-
|
In my case,
Config I used was In https://prometheus-community.github.io/helm-charts/ In https://actions-runner-controller.github.io/actions-runner-controller metrics:
serviceAnnotations: {}
serviceMonitor: false
serviceMonitorLabels: {}
port: 8443
proxy:
enabled: false
image:
repository: quay.io/brancz/kube-rbac-proxy
tag: v0.13.0Disabled If you use Prometheus Operator, and enable |
Beta Was this translation helpful? Give feedback.
-
|
Strangely when trying to hit the endpoint I am getting Connection Refused ? However when I enable port forwarding it works |
Beta Was this translation helpful? Give feedback.
-
|
Please i need some help in getting all the metrics provided by ARC. have been added to the new release ( runner-controller 0.22.0 ). So i upgraded my controllers to the new version but still just seeing only the following metrics We do use CDK to deploy the Helm resources in Eks and my configuration is as follows Chart-Version: actions-runner-controller-0.22.0 Please can someone help me figure this out? |
Beta Was this translation helpful? Give feedback.
-
|
Just be aware that Helm doesn't update CRDs, if that is necessary to have the metrics you need to update manually the CRD |
Beta Was this translation helpful? Give feedback.
-
|
Hi @Moser-ss , can you please point me to a document on how to manually update the CRD ? I am also using the helm chart and only seeing half of the metrics coming up. |
Beta Was this translation helpful? Give feedback.
-
|
The CDRs are just normal Kubernetes resources so you can use kubectl apply and update CDR . The CDRs are located here charts/actions-runner-controller/crds Also the metrics server works with the input of the webhooks |
Beta Was this translation helpful? Give feedback.
-
|
Hi @Moser-ss Thank you for your reply. I have the metrics annotation as mentioned in the troubleshooting document, and the prometheus available at cluster level
But the workflow metrics alone not coming up in the grafana, which is the metric I am really after, so I am kind of stuck. can you please help ! 
|
Beta Was this translation helpful? Give feedback.
-
|
@Moser-ss , I use fluxcd for deployment , so added crds upgrade in the template itself, but still only workflow metrics are missing though. any idea what else I might be missing ? TIA!! |
Beta Was this translation helpful? Give feedback.
-
|
Running chart version 0.22.0 I also can't see any of the new metrics. I haven't upgraded, I've just added some values to my Prometheus and ARC Helm charts: kube-prometheus: ARC: Pods: The I can't see those new metrics on the |
Beta Was this translation helpful? Give feedback.
-
|
I have the exact same problem. The pods are up but no metrics coming through |
Beta Was this translation helpful? Give feedback.
-
|
Also this issue is open from another user: actions/actions-runner-controller#2799 I'm having a similar issue. I have ARC release update to the last version, CRDs are update also. But I just get 2 metrics
This is my configuration |
Beta Was this translation helpful? Give feedback.
-
|
This is the same for us. Everything on latest but only able to get these 2 metrics whilst using prometheus-community helm chart |
Beta Was this translation helpful? Give feedback.
-
|
Hi, I am using v.027.5 image version and though I have jobs in queue, I dont find see these metrics reflecting the same, I am looking for a metric on how many jobs are in queue, so I can setup alert on that, if the queue is too big. |
Beta Was this translation helpful? Give feedback.
-
|
I can also only see two 2 metrics with everything latest? Any idea? horizontalrunnerautoscaler_spec_max_replicas |
Beta Was this translation helpful? Give feedback.
-
|
Anyone looking for a dashboard that works with the new metrics should check out this blog: https://www.kenmuse.com/blog/enabling-github-arc-metrics/ I have found that the cardinality on the gha_* metrics can be insane because they include runner_name (pod name) and runner_id (job id) essentially making one series per data point which prevents even using To clean up your metrics try using a service monitor configured like this. You'll need two, one for the controller, one for the listener (bulk of metrics). This is the one for the listener: |
Beta Was this translation helpful? Give feedback.
-
|
Whelp update. It seems like runner_id/runner_name is a known issue and the devs haven't fixed it. For now you can use the subquery I wrote to actually graph the metrics but the growth of the number of time series is enough to make me drop the metric entirely until a fix is made. We may try to deploy a copy of the code that removes those labels. |
Beta Was this translation helpful? Give feedback.
-
|
@jameshounshell , the query seems to fit my usecase, but what I am really trying to achieve is to get the total minutes/seconds of execution time taken by all workflows in a single value for a selected time window, like the usage minutes in GitHub billing. do you think it is possible with current structure of metric? please share your thoughts? thanks! |
Beta Was this translation helpful? Give feedback.
-
|
I forked/edited the code to drop runner_id/runner_name and reduce the number of buckets. If you want to try running my version of the runner pull my repo and run this:
There's some silly checks that I couldn't figure out so I set the version to be the version number listed in the helm chart (currently ) otherwise the controller complains that the listener isn't the right version and won't deploy it. Add this to your values.yaml for the gha-runner-scale-set-controller helm chart |
Beta Was this translation helpful? Give feedback.
-
|
@rekha-prakash-maersk Here are the official prometheus docs for histogram manipulation https://prometheus.io/docs/practices/histograms/#count-and-sum-of-observations |
Beta Was this translation helpful? Give feedback.
-
|
Hi @jameshounshell , with new release of option to customise the bucket values, did you cut down from the default of 40 + to any reasonable smaller bucket values? if yes, can you please share it here for me to refer ? I looked at this default values but not sure if its right fit to this job execution time metric- https://pkg.go.dev/github.com/prometheus/client_golang/prometheus#pkg-variables |
Beta Was this translation helpful? Give feedback.
-
|
I noticed that @kenmuse had updated the sample Grafana dashboard and docs (thanks Ken!) at https://github.com/actions/actions-runner-controller/tree/master/docs/gha-runner-scale-set-controller/samples/grafana-dashboard , but I'm running into a roadblock with metrics for I'm assuming that there's some other scrape config that is adding that to the listener metrics? Anyone have any ideas here? |
Beta Was this translation helpful? Give feedback.
-
|
@nocturne1 - Good question. I'll try to add some additional documentation to cover that. TL;DR - You're spot on. It is coming from a scrape config. It's how many systems have configured the scrape config for Prometheus. It has a relabelling step that scrapes the labels for one or more Pods and includes those details. Longer versionThat value originates from the Prometheus scrape config. While many default configurations already have a job that scrapes labels for all of the pods, I'll provide a config example that's a bit more tuned to ARC. scrape_configs:
- job_name: arc-metrics
honor_labels: true
kubernetes_sd_configs:
- role: pod
namespaces:
names:
- arc-systems
relabel_configs:
- action: labelmap
regex: __meta_kubernetes_pod_label_(.+)The listeners (and runner pods) have labels like these: The kubernetes_sd_configs lets Prometheus scrape some details from the REST APIs. In the example above, the The scrape will return the labels as: The Hope that helps! You may also want to simplify the names provided for the pod name/namespace: You may find it helpful to review this documentation for deeper details. |
Beta Was this translation helpful? Give feedback.
-
|
@kenmuse Thank you for the explanation, and your invaluable contributions in providing a dashboard starter for the community. I actually in principle agree with the team's decision to not render coreos CRDs in the arc charts- it's not a standard and I strongly dislike having to install coreos CRDs when I don't run prometheus. Unfortunately coreos monitoring resources are a de-fact standard amongst community charts, and I'm equally irked by having to bring my own scrape config to these charts. I've used pod annotations to expose these metrics to my monitoring solution by way of pod discovery (via the prometheus.io annotations). I'm wondering if either:
My scrape config: relabel_configs:
- source_labels: [__meta_kubernetes_namespace]
target_label: namespace
- target_label: job
replacement: monitoring/annotations-discovery
- source_labels: [__meta_kubernetes_pod_label_app_kubernetes_io_name]
target_label: job
regex: (.+)
replacement: ${1}As an aside: I don't think any of this is yours/github's responsibility, and I think the k8s core team should look at a solution to help us not require coreos CRD installations to support these flows, but I'm sure they have other things to prioritise. |
Beta Was this translation helpful? Give feedback.
You can use Prometheus to scrape metrics on action-controller-runner.
Then you can connect Prometheus to Grafana
The current version exposed following metrics.
horizontalrunnerautoscaler_spec_min_replicashorizontalrunnerautoscaler_spec_max_replicashorizontalrunnerautoscaler_status_desired_replicasIn next release we will have following metrics too apart from above ones (PR already merged)
horizontalrunnerautoscaler_replicas_desiredhorizontalrunnerautoscaler_runnershorizontalrunnerautoscaler_runners_registeredhorizontalrunnerautoscaler_runners_busyhorizontalrunnerautoscaler_terminating_busyhorizontalrunnerautoscaler_necessary_replicashorizontalrunnerautoscaler_workflow_runs_comp…