Github action runners in non-privileged containres in k8s

[pujan14]

Hi everyone,

We are trying to setup environment where all containers are running in non-privileged mode and we would like to run github action runners in this environment and use them to build docker images, has anyone been able to accomplish this? If so I would like to know more about it.

[wherka-ama]

Hello Pujan,
this is a holly grail - to be able to build images and run the containers without the escalation of privileges.

To a degree that's feasible. You can build the images in rootless mode and without any extra priviledge. There are number of options here:

• jib
• kaniko
• podman/buildah
• rootlesskit
  ..
  You can read a nice summary here and also a good explanation on how it can be done with podman and jenkins on OpenShift. These would be good starters to grasp the context.

In regards to ARC, there is no so much a solution for that available out of the box I'm afraid. If you choose to run the docker in a side container, it's done in privileged mode by default. You can opt for a rootless, but then again it will be rootless, but still not quite unprivileged. If you choose to restrict yourself to unprivileged mode, you will not be able to create new (linux) namespaces which are essential for creating new containers. Docker In Docker for running containers will simply stop working.
If you are prepared to skip running docker actions and you will never need to run the containers inside your runner context, that's okay - you can try to plug the kaniko as a side container or use jib, which should work fine(the mileage may vary depending on the requirements).
I'm not offering the solution here as I'm biased by some specific constraints of our environment, but I can share with you that I was able to accomplish the part I described there. The only thing is, it was not sufficient. For us it was necessary to not only build, but also to run the images inside the ARC ecosystem and that had to be done with privileged mode. We have secured it though with other means to ensure we only get the permissions which are necessary for a given task.

I'd love to see some comments from the ARC team to clarify it a bit further as this is indeed a very important subject.

[jmrr]

Hi @wherka-ama, thanks for your answer, would you be able to share how you configured ARC (e.g. a sample of your selection of values.yaml) that at least helped you build an image using ARC? We're using self hosted runners set up with ARC in our microk8s cluster, and they work fine when running non-container-related CI pipelines (tests, etc.) however we still need to build a container image prior to the CD stage and we don't find good examples on how to achieve this... yet alone something out of the box (i.e. a Github Action)... and we've tried kaniko, buildah... to no avail

We believe that building container images is at the core of a kubernetes ecosystem (which ARC is part of) so if you can share your success story and we manage to make it work for our use case we'll contribute an example to the docs as we believe it's of paramount importance. Thanks!

[wherka-ama]

Hi @jmrr, thanks for your kind words. I'm a bit snowed under right now, but writing up on how to deal with this activity in a reasonably safe way is definitely something I would love to do in the near future!
I personally don't use microk8s setup, and I believe that's the context you would be interested the most. I can try to play with it and see if we can make it work there as well. If you're interested we can join the efforts and work on that part together - I guess I'm not too difficult to find on LinkedIn.
What I'm familiar with is the OpenShift and Kind and I can share some tips on that part fairly soon.

[ALL-SPACE-Anas]

Hi @wherka-ama have you managed to write up on your implementation of ARC anywhere? If yes, can you please link it here?

[wherka-ama]

@manas-suleman : are you interested in reading about the deployment of ARC on OpenShift? I could also write up how it can be done with Kind(on top of podman).
I believe we've managed to find a relatively good setup. Not perfect, but safe and quite efficient.
The only problem with writing it up is time. I can only spare some in 2-3 weeks from now.
If you don't hear anything from me, please send me a pinger here.

[Muzammil-Yernal]

@wherka-ama I am interested about reading the deployment of ARC on Openshift, can you please share some of your insights, I am currently facing a blocker building docker images as I cannot use DinD. Can you please advice on how it can be done using podman?. Thanks.