How to organize a composite CI workflow while preventing multiple checkouts of the same repository?

[vb-dontenwill]

Why are you starting this discussion?

Question

What GitHub Actions topic or product is this about?

Workflow Configuration

Discussion Details

We are migrating our TeamCity-based CI to GitHub Actions to keep everything in a single ecosystem.

We have a self-hosted Windows runner (needed for Delphi builds and code signing) and already built one Delphi project successfully in a workflow.

Challenge: Our monorepo contains ~20 Delphi projects (sharing source) and is about 15 GB. We want to run multiple builds/tests in parallel (different projects) on the same physical machine (multiple self-hosted runners or parallel jobs), but we do not want to perform a fresh 15 GB checkout per job because that is inefficient.

Goal: Replicate a fan-out style pipeline:

• Prepare / initial setup (single checkout)
• Parallel project builds (several independent jobs)
• Unit test execution as soon as they are built
• Final cleanup

(See diagram below.)

Question: What is the recommended GitHub Actions pattern to:

• Avoid repeated full clones in parallel jobs
• Still isolate builds logically (per job)

Any guidance or proven patterns would be appreciated.

[Khanmanan]

You can avoid multiple 15 GB checkouts by using a single initial checkout + artifact/cache sharing across jobs:

Option 1: Cache strategy

• Run actions/checkout@v4 only once in a "prepare" job.
• Use actions/cache to store the source directory.
• Subsequent jobs restore the cached source instead of re-checking out.

Option 2: Upload + download artifacts

• First job: checkout once and run setup.
• Upload the workspace as an artifact (actions/upload-artifact).
• Parallel jobs: download that artifact (actions/download-artifact).
• This avoids fresh clone each time.

Option 3: Matrix build

• Checkout once at the start of the workflow.
• Use a matrix job to fan out builds (different projects).
• Each job reuses the same code from cache/artifact.

This way:

• You isolate builds logically per job.
• You don’t waste time re-downloading 15 GB multiple times.
• It scales better for your ~20 Delphi projects.

👉 Example (simplified):

jobs:
  prepare:
    runs-on: self-hosted
    steps:
      - uses: actions/checkout@v4
      - uses: actions/upload-artifact@v4
        with:
          name: source
          path: .
  build:
    needs: prepare
    strategy:
      matrix:
        project: [proj1, proj2, proj3]
    runs-on: self-hosted
    steps:
      - uses: actions/download-artifact@v4
        with:
          name: source
      - run: build ${{ matrix.project }}
     
This pattern prevents repeated heavy checkouts but still keeps jobs isolated.

[vb-dontenwill]

We have implemented the CI using the jobs dependency with self-hosted runner.
We now have the problem that we can't get other runners (when using self-hosted, or generic "be" tag) to stop stealing the jobs that have been started by the runner who was used when the workflow was triggered. We now have to have an explicit "per-branch"-runner. This seems counterintuitive because the main branch runner will be much more active than the other 2. Is there a way to say: step 1 was executed by runner X, rest of the workflow MUST be executed by runner X?

This is how the action looks like in the currently:
/

name: CI (Continuous Integration)

on:
  push:
    branches:
      - test/github
  workflow_dispatch:
    inputs:
      branch:
        description: "Branch to build"
        type: string
        required: true

defaults:
  run:
    shell: powershell

jobs:
  prepare_repo:
    runs-on: be-main
    steps:
      - name: Checkout repository (automatic)
        uses: actions/checkout@v4

      - name: Prepare repository
        uses: ./.github/actions/prepare-repository

  build_projects:
    runs-on: be-main
    needs: prepare_repo
    strategy:
      matrix:
        include: 7 items
    steps:
      - name: Build <${{ matrix.name }} >
        uses: ./.github/actions/build-delphi-project

  build_unit_tests:
    runs-on: be-main
    needs: prepare_repo
    strategy:
      matrix:
        include: 9 items

    steps:
      - name: Build <${{ matrix.name }} >
        uses: ./.github/actions/build-delphi-project

  build_core_projects:
    runs-on: be-main
    needs: prepare_repo
    steps: sequential call of ./.github/actions/build-delphi-project for 7 projects

  run_db_unit_tests:
    runs-on: be-main
    needs: [build_projects, build_unit_tests]
    strategy:
      matrix:
        include: 6 items
    steps:
      - name: Run DB Unit Tests <${{ matrix.name }} >
        uses: ./.github/actions/run-db-unittest

  run_standalone_unit_tests:
    runs-on: be-main
    needs: [build_unit_tests]
    strategy:
      matrix:
        include: 2 items
    steps:
      - name: Run Standalone Unit Tests <${{ matrix.name }} >
        id: run_standalone_ut
        uses: ./.github/actions/run-standalone-unittest

  unittest_results:
    runs-on: be-main
    needs: [run_db_unit_tests, run_standalone_unit_tests]
    steps: ...

  build_dontenwill_components:
    runs-on: be-main
    needs: prepare_repo
    steps: sequential call of ./.github/actions/build-delphi-project 12 times

[vb-dontenwill]

We resolved the issue the following way:

We are still using multiple co-dependant jobs.
prepare_repo now uploads the actions needed for the run and outputs the actual github.workspace

prepare_repo:
  runs-on: be
    steps:
      - name: Checkout repository
        uses: actions/checkout@main
        with:
          clean: true
          submodules: true
          token: ...
      - name: Upload Run Actions
        uses: actions/upload-artifact@v4
        with:
          name: actions
          path: .github/actions
    outputs:
      workspace: ${{ github.workspace }}

every single job and sub-action (for the matrixes) downloads the actions and references github workspace by the output variable:

build_projects:
  ...
  steps:
    - name: Download Run Actions
        uses: actions/download-artifact@v4
        with:
          name: actions
          path: .github/actions
    - name: Build <${{ matrix.name }} >
      uses: ./.github/actions/build-delphi-project
      with:
        workspace: ${{ needs.prepare_repo.outputs.workspace }}

At the end of the process, I just call geekyeggo/delete-artifact@v5 on actions.
This exponentially sped up the build time. Maybe I'll make public action(s) to make this kind of behaviour more flexible in case other people need it.