Self-Hosted CI Runners: Security vs. Performance

The January 2023 CircleCI disclosure was the moment a lot of CISOs looked up from their dashboards and asked the question nobody wanted to answer: why are we shipping our source code, our build secrets, and our signing keys to somebody else’s cloud? Within a quarter, half the platform teams I talked to had a self-hosted runner migration on the roadmap. By the end of that year, a good chunk of them had discovered they hadn’t eliminated risk — they’d just relocated the blast radius inside the perimeter, next to the databases.

Here’s the uncomfortable thesis of this guide: self-hosted runners are not more secure than cloud-hosted runners. They are differently insecure. Cloud runners can’t reach your internal Artifactory, your staging database, or your Vault cluster. Yours can. That’s a design decision, not a bug, and it needs to be made consciously.

This is written for platform and DevSecOps engineers running private cloud — OpenStack, VMware, Proxmox, or plain bare metal — who are either standing up self-hosted runners or auditing the ones they inherited. It is not a GitHub Actions tutorial and it is not a GitLab install guide. It assumes you already know how to register a runner. It’s about making sure that runner doesn’t become the pivot point for your next incident.

TL;DR

• Self-hosting buys you data residency and egress savings — it does not buy you security by default. Ephemerality, network isolation, and OIDC are what buy you security.
• Ephemeral runners are non-negotiable. A persistent runner is a credentialed pet with 400 days of uptime and cached secrets nobody can enumerate.
• Default-deny egress from runner subnets. DNS filtering alone will get you owned via a hardcoded IP.
• Never store static cloud credentials on a runner. Use OIDC to assume short-lived roles, and inject secrets via Vault Agent or external-secrets.
• The performance tax of ephemerality is real (typically 45–90s cold start) and mostly solvable with pre-baked images and warm node pools.

Prerequisites

You’ll get the most out of this if you have:

• A working Kubernetes cluster (1.27+) for the ARC examples, or a comparable container orchestrator
• Familiarity with GitHub Actions or GitLab CI runner registration
• A private container registry (Harbor, Nexus, or an internal mirror) reachable from your runner subnet
• Administrative access to your cloud’s IAM to configure OIDC trust
• kubectl, helm, and nftables or iptables on the hosts you’re hardening

The Threat Model You Didn’t Write Down

Most teams self-host for one of two reasons: a compliance requirement about data residency, or a CFO who got tired of NAT gateway egress charges. Both are legitimate. Neither is a security control, and treating them as one is how you end up with a --privileged container mounting the host Docker socket because “the build needs it.”

What Self-Hosting Actually Buys You

Data residency is the real win — your source never leaves infrastructure you control, and you can prove it to an auditor. Egress cost reduction is the second real win, and it’s usually bigger than people expect once you’re running thousands of build-minutes a month. What it does not buy you is a smaller attack surface. You’ve replaced a vendor’s hardened multi-tenant environment with your own single-tenant environment that you now have to harden yourself, and you’ve connected it to your internal network.

The trap is what I call compliance theater: a team checks the “runners are self-hosted” box, then deploys a runner that runs as root, mounts the Docker socket, and has an IAM role with s3:*. Here’s what that looks like in practice — and every red flag is marked:

# docker-compose.yml — NAIVE self-hosted runner. Do not copy this.
services:
  runner:
    image: myorg/actions-runner:latest  # RED FLAG: mutable tag, no digest pin
    privileged: true                     # RED FLAG: full host device access
    user: root                           # RED FLAG: root in the container
    environment:
      RUNNER_TOKEN: ${RUNNER_TOKEN}      # RED FLAG: static, long-lived token
      AWS_ACCESS_KEY_ID: ${AWS_KEY}      # RED FLAG: static cloud creds on disk
      AWS_SECRET_ACCESS_KEY: ${AWS_SECRET}
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock  # RED FLAG: host Docker control
      - runner-work:/work                             # RED FLAG: persistent, never wiped
    restart: unless-stopped              # RED FLAG: the pet that never dies
volumes:
  runner-work: {}

Every one of those red flags is a lateral movement primitive. The Docker socket alone means any job can docker run -v /:/host and own the host. The static AWS keys mean the job can exfiltrate them and use them from anywhere on the internet. The persistent /work volume means job N+1 can read artifacts and caches from job N.

The New Attack Surface

A self-hosted runner is a persistent, credentialed host that sits inside your network boundary. That’s the whole problem. The supply chain doesn’t go away just because you self-host — actions/checkout@v4 is still third-party code executing in your environment, and if you’re not pinning it to a commit SHA, you’re trusting a mutable tag. The difference is that when it goes wrong, the malicious step now has a route to your internal services instead of being trapped in a vendor’s sandbox.

flowchart LR
    PR[Pull Request] -->|triggers| Runner[Self-Hosted Runner]
    Runner -->|can reach| SCM[Internal SCM API]
    Runner -->|can reach| Reg[Internal Registry]
    Runner -->|can reach| DB[(Staging DB)]
    Runner -->|can reach| Vault[Vault / Secrets]
    Runner -.->|default egress| Internet((Internet))
    subgraph Trust Boundary
        Runner
        SCM
        Reg
        DB
        Vault
    end

That dotted line to the internet is your exfiltration path. That solid line to DB is why cloud-hosted runners can’t hurt you the way self-hosted ones can.

Blast Radius Math

The single biggest security decision you will make is ephemeral versus persistent. A persistent runner accumulates state — cached credentials, installed packages, shell history, leftover tokens in /tmp. A shared pool of persistent runners means every job that lands on runner A inherits whatever job 3 ran last Tuesday. One compromised job poisons the pool.

Attack Vector	Cloud Runner	Self-Hosted Runner	Winner
Malicious step exfiltrates source	Vendor-scoped, detectable	Full internal network access	Cloud
Credential theft from runner	Ephemeral, short-lived	Persistent, cached, long-lived	Cloud
Lateral movement to internal DB	Not possible	Trivially possible if unfiltered	Cloud
Data residency violation	Structural risk	Eliminated by design	Self-Hosted
Egress cost at scale	Vendor markup	Your own NAT costs	Self-Hosted
Vendor supply chain compromise	Your blast radius	Your blast radius	Tie

Q: Isn’t self-hosting just more secure by default?

No. Self-hosting moves the trust boundary; it doesn’t remove it. You’ve traded a vendor’s multi-tenant isolation problem for your own single-tenant hardening problem, and you’ve added a network path from your build system to your production services. It’s more secure only if you actually implement ephemerality, network isolation, and short-lived credentials. Without those, it’s often worse than the cloud you left.

Ephemeral Runners Are Non-Negotiable

If you take one thing from this guide, take this: a runner that survives a single job is a liability. The persistent runner is the anti-pattern that keeps security teams up at night — the box that’s been up 400 days, has 12GB of cached credentials in /home/runner, and nobody can tell you what’s installed on it because the last three people who touched it left the company.

The Persistent Runner Anti-Pattern

Persistent runners fail in three predictable ways. First, cache poisoning: job A writes a tainted artifact to a shared cache path, job B consumes it, and now your release build has someone else’s payload. Second, credential leakage: environment variables, ~/.aws/credentials, and ~/.docker/config.json persist across jobs, so any job can read the previous job’s secrets. Third, drift: the runner slowly accumulates state until it’s a snowflake you can’t reproduce, and you can’t patch it without breaking a build.

Autoscaling with ARC

The reference architecture is the Actions Runner Controller (ARC) on Kubernetes, running ephemeral runners. ARC’s AutoscalingRunnerSet (the newer API, replacing the older RunnerDeployment) scales runner pods based on queued jobs, and each pod handles exactly one job before terminating.

apiVersion: actions.github.com/v1alpha1
kind: AutoscalingRunnerSet
metadata:
  name: arc-runners
  namespace: arc-runners
spec:
  githubConfigUrl: "https://github.com/myorg"
  githubConfigSecret: arc-github-app-secret
  minRunners: 0
  maxRunners: 50
  template:
    spec:
      ephemeral: true          # one job per pod, then terminate
      containers:
        - name: runner
          image: registry.internal/myorg/runner:24.09.0-ubuntu22.04
          resources:
            requests: { cpu: "2", memory: "4Gi" }
            limits:   { cpu: "4", memory: "8Gi" }
          securityContext:
            runAsNonRoot: true
            runAsUser: 1001
            allowPrivilegeEscalation: false
            readOnlyRootFilesystem: false   # runner needs /tmp writable
            capabilities: { drop: ["ALL"] }

The ephemeral: true flag is the only acceptable default. It tells ARC to register the runner as just-in-time and terminate the pod after a single job completes. No state survives. No cache poisoning between jobs. The securityContext block drops all Linux capabilities and runs as UID 1001 — if a job tries to escalate, it can’t.

The Performance Tax of Ephemerality

Let’s be honest about the cost. A warm persistent runner starts a job in roughly 2 seconds. An ephemeral runner has to pull the image, register with the SCM, and bootstrap the toolchain — typically 45 to 90 seconds cold, depending on image size and registry locality. At 10,000 jobs a month, that’s 7 to 15 hours of pure overhead.

Mitigations that actually work: pre-bake your runner image with toolchains pinned (no apt-get install at job time), keep a warm node pool so pods schedule immediately, and back your package caches with a real artifact store rather than actions/cache against the public backend.

# Pre-baked runner image — pinned tool versions, non-root, minimal surface.
FROM ubuntu:22.04

ARG RUNNER_VERSION=2.319.1
ARG NODE_VERSION=20.17.0

RUN apt-get update \
 && apt-get install -y --no-install-recommends \
      ca-certificates curl git jq unzip tar \
 && rm -rf /var/lib/apt/lists/*

# Non-root runner user
RUN useradd -m -u 1001 -s /bin/bash runner

# Pinned Node, no version managers at runtime
RUN curl -fsSL "https://nodejs.org/dist/v${NODE_VERSION}/node-v${NODE_VERSION}-linux-x64.tar.xz" \
      -o /tmp/node.tar.xz \
 && tar -xJf /tmp/node.tar.xz -C /usr/local --strip-components=1 \
 && rm /tmp/node.tar.xz

USER runner
WORKDIR /home/runner

RUN curl -fsSL "https://github.com/actions/runner/releases/download/v${RUNNER_VERSION}/actions-runner-linux-x64-${RUNNER_VERSION}.tar.gz" \
      -o /tmp/runner.tar.gz \
 && tar -xzf /tmp/runner.tar.gz \
 && rm /tmp/runner.tar.gz

ENTRYPOINT ["/home/runner/run.sh"]

This image installs nothing at job time. The toolchain is frozen into the layer, which means cold start is dominated by image pull, not apt-get. Pair it with a registry pull-through cache and cold start drops to the low end of that 45–90s range.

Dimension	Persistent	Ephemeral	Hybrid
Startup latency	~2s	45–90s cold	10–30s warm pool
Security posture	Weak — shared state	Strong — clean per job	Medium
Ops complexity	Low until drift bites	Medium (ARC, autoscaling)	High
Cost	Low idle, high risk	Higher idle if overprovisioned	Balanced

Q: How do I cache dependencies without a persistent runner?

Point your cache at an internal artifact store, not the runner’s filesystem. For GitHub Actions, configure actions/cache to use a self-hosted backend (an S3-compatible store like MinIO works), or skip the action entirely and use a registry pull-through cache for container layers and a remote cache for Bazel/Gradle. The rule is: caches live in a shared, access-controlled store, never on the runner. And scope caches per trust tier — a cache written by a fork PR must never be readable by a release build.

Network Isolation — The Layer Everyone Skips

If ephemerality limits how long an attacker has, network isolation limits what they can reach. This is the layer teams skip because it’s the one that breaks builds, and a broken build is a page at 2 AM. Do it anyway. A runner that can reach your production database is a runner that will eventually reach your production database.

Egress Filtering Is Not Optional

Default-deny egress from your runner subnet, then allowlist exactly what builds need: your SCM API, your registry, your artifact store, your package mirrors, and your secrets backend. DNS-based filtering alone is insufficient — an attacker who knows an IP can bypass your resolver entirely.

# nftables egress allowlist for a runner subnet (10.40.0.0/24)
# Default policy: drop. Only listed destinations pass.
table inet runner_egress {
  set allowed_v4 {
    type ipv4_addr
    flags interval
    elements = {
      10.10.0.0/24,      # internal SCM
      10.20.0.0/24,      # internal registry (Harbor)
      10.21.0.0/24,      # artifact store (MinIO)
      10.30.5.10,        # Vault
      140.82.112.0/20,   # GitHub API range — pin and review quarterly
      151.101.0.0/16     # package CDN — pin and review quarterly
    }
  }
  chain output {
    type filter hook output priority 0; policy drop;
    ip daddr @allowed_v4 accept
    ip daddr 10.40.0.0/24 accept   # intra-subnet (runner [-] node)
    ct state established,related accept
    counter log prefix "runner-egress-drop: " drop
  }
}

The log prefix on the drop rule is deliberate — you want every blocked egress attempt in your logs, because that’s how you detect a build step that’s phoning home.

Ingress: Stop Exposing the Runner

Runners should never accept inbound connections. ARC and the GitHub runner both use pull-based registration — the runner reaches out to the SCM, not the other way around. If you find a firewall rule allowing inbound to your runner subnet, someone added it “temporarily” to debug something and never removed it. Kill it.

Segmentation from Production

Put runners in their own VLAN, subnet, and Kubernetes namespace, and enforce it with NetworkPolicy. The runner namespace should have no route to your data tier. This is the single highest-value network control you can deploy.

# Default-deny ingress + egress for the ARC runner namespace.
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: runner-default-deny
  namespace: arc-runners
spec:
  podSelector: {}
  policyTypes: ["Ingress", "Egress"]
---
# Explicit egress allowlist for the same namespace.
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: runner-egress-allow
  namespace: arc-runners
spec:
  podSelector: {}
  policyTypes: ["Egress"]
  egress:
    - to:
        - namespaceSelector:
            matchLabels: { kubernetes.io/metadata.name: kube-system }
      ports: [{ port: 53, protocol: UDP }]   # DNS only
    - to:
        - ipBlock: { cidr: 10.20.0.0/24 }    # registry
      ports: [{ port: 443, protocol: TCP }]
    - to:
        - ipBlock: { cidr: 10.30.5.10/32 }   # Vault
      ports: [{ port: 8200, protocol: TCP }]

The first policy denies everything. The second punches holes for DNS, the registry, and Vault — nothing else. If your builds need more, add it explicitly and document why.

Approach	Complexity	Breakage Risk	Security Value
DNS filtering only	Low	Low	Low — trivially bypassed
Egress allowlist (L3/L4)	Medium	Medium	High
Kubernetes NetworkPolicy	Medium	Medium	High (in-cluster)
Full service mesh mTLS	High	High	High + identity

Q: My builds need to pull from the public internet — how do I filter that?

Don’t allow the public internet. Mirror what you need internally. Run a registry pull-through cache (Harbor or Nexus) for container images, an internal PyPI/npm/Maven mirror for packages, and allowlist only the mirror IPs. If a build genuinely needs a one-off external fetch, route it through an authenticated egress proxy you log, not a raw allowlist entry.

Secrets Management — The Part Where People Cry

Every secret you store on a runner is a secret a compromised job can read. GitHub and GitLab inject secrets into the job environment as environment variables — which means any step in the job, including a malicious one, can env and exfiltrate them. The correct answer is to store no static secrets on the runner at all.

Why Runner-Stored Secrets Are a Liability

The GITHUB_TOKEN is a good example of scope creep. It’s auto-generated per job, which is good, but its default permissions are often broader than the job needs. If a step can push to any branch, it can push a backdoor to your main branch. Set permissions: explicitly at the workflow level to the minimum the job requires.

Short-Lived Credentials via OIDC

The right pattern is OIDC federation: the runner presents a signed identity token to your cloud, which exchanges it for a short-lived role session. No static keys. No long-lived credentials. If the runner is compromised, the attacker gets a credential that expires in an hour and is scoped to a single role.

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "Federated": "arn:aws:iam::123456789012:oidc-provider/token.actions.githubusercontent.com"
      },
      "Action": "sts:AssumeRoleWithWebIdentity",
      "Condition": {
        "StringEquals": {
          "token.actions.githubusercontent.com:aud": "sts.amazonaws.com"
        },
        "StringLike": {
          "token.actions.githubusercontent.com:sub": "repo:myorg/myrepo:ref:refs/heads/main"
        }
      }
    }
  ]
}

This trust policy only allows the main branch of myorg/myrepo to assume the role. Pull request builds from forks get a different sub claim and are denied. That’s the fork-PR problem solved at the identity layer.

Secret Injection Without Persistence

Use Vault Agent Injector or the external-secrets operator to inject secrets into the job’s process environment at runtime, never to disk. Vault Agent can render secrets to a tmpfs mount that vanishes when the pod terminates.

# Vault policy — read-only, scoped to the exact path the job needs.
path "secret/data/ci/myrepo/deploy" {
  capabilities = ["read"]
}
path "secret/data/ci/myrepo/deploy/*" {
  capabilities = ["read"]
}

And the anti-pattern you must never write:

# DO NOT DO THIS. This writes a secret to disk where it persists.
echo "$DEPLOY_TOKEN" > /tmp/creds
curl -H "Authorization: Bearer $(cat /tmp/creds)" https://deploy.internal/api
rm /tmp/creds   # too late — the file may already be in a snapshot or cache

Read the secret directly into the process that needs it, or use a Vault sidecar that keeps it in memory. Never touch disk.

Mechanism	Rotation	Blast Radius if Runner Compromised	Setup Effort
Repo/org secrets	Manual	Full — all secrets readable	Low
Static cloud keys	Manual	Full — keys usable anywhere	Low
OIDC to cloud STS	Automatic, per-job	Scoped, expires in ~1h	Medium
Vault Agent sidecar	Automatic	Scoped, in-memory, TTL-bound	High

Q: Can’t I just use repository secrets?

You can, and for non-sensitive values it’s fine. But understand what you’re accepting: the secret is injected into the job environment, visible to every step, and static until you rotate it manually. For anything that grants access to production, use OIDC or Vault instead. Repository secrets are for a Slack webhook URL, not for your deploy credentials.

Performance Engineering for Self-Hosted Runners

Security without performance is a project that gets reverted. The goal is hardened runners that don’t make developers wait, because the moment builds get slow, someone will “temporarily” disable a NetworkPolicy to make it work.

Right-Sizing and Node Pools

Separate CPU-bound workloads (compilers, test runners) from I/O-bound ones (image builds, artifact uploads). A single giant runner that does everything wastes capacity and creates a noisy-neighbor problem. Run two node pools with different instance types, and let ARC schedule by label.

# AutoscalingRunnerSet with explicit requests/limits and a dedicated node pool.
apiVersion: actions.github.com/v1alpha1
kind: AutoscalingRunnerSet
metadata:
  name: arc-builders
  namespace: arc-runners
spec:
  githubConfigUrl: "https://github.com/myorg"
  githubConfigSecret: arc-github-app-secret
  minRunners: 2
  maxRunners: 40
  template:
    spec:
      ephemeral: true
      nodeSelector:
        workload: cpu-optimized
      containers:
        - name: runner
          image: registry.internal/myorg/runner:24.09.0-ubuntu22.04
          resources:
            requests: { cpu: "4", memory: "8Gi" }
            limits:   { cpu: "8", memory: "16Gi" }

The nodeSelector pins these runners to the CPU-optimized pool. Your image-build runners get a different label and a different pool. Don’t mix them.

Caching Strategy That Survives Ephemerality

Layer caching is your biggest win. A registry pull-through cache (Harbor proxy cache or Nexus) means the runner pulls base images from a local mirror instead of Docker Hub. For language builds, use remote caches: Bazel’s remote cache, Gradle’s build cache, or a self-hosted actions/cache backend on MinIO.

The caveat: caches are shared state, and shared state across trust boundaries is a poisoning vector. Scope caches by trust tier — a fork PR’s cache must be a separate namespace from your release build’s cache. Never let untrusted code write to a cache that trusted code reads.

Cost Modeling — Where Self-Hosting Wins and Loses

The honest TCO includes engineer time, idle capacity, and the egress you thought you’d save but didn’t because your registry mirror is still pulling from upstream. Below roughly 10,000 build-minutes a month, cloud-hosted runners are usually cheaper once you account for the engineer time to run ARC. Above 100,000, self-hosting almost always wins on raw compute cost — if you’ve done the autoscaling right.

Line Item	Cloud (10k min)	Self-Hosted (10k min)	Self-Hosted (1M min)
Compute	~$80	~$40 (idle included)	~$2,500
Egress	Vendor markup	Own NAT cost	Own NAT cost
Engineer time	~0	~0.2 FTE	~0.5 FTE
Registry/mirror	N/A	~$50	~$300
Effective total	~$80	~$1,800	~$8,000

The numbers above are illustrative ranges, not a quote — cloud pricing changes and your mileage will vary with instance types and utilization. The point is the shape: at low volume, engineer time dominates and self-hosting loses. At high volume, it wins decisively.

Q: Is self-hosting actually cheaper?

Only above a certain scale, and only if you’ve invested in autoscaling and pre-baked images. Below ~10k build-minutes a month, the engineer time to run ARC correctly usually exceeds the compute savings. Do the math with your own utilization numbers before you migrate.

Hardening the Runner Host

Even with ephemeral runners and network isolation, the host itself needs a baseline. This is the layer where you catch the things the higher layers miss.

OS-Level Baseline

Start from a minimal base image. No compilers unless the build needs them, no package managers at runtime, no SSH daemon. Run as non-root. Where possible, mount the root filesystem read-only and give the runner a writable tmpfs for /tmp and /home/runner. Apply the CIS benchmark subset that matters for runners: filesystem permissions, kernel hardening (kernel.unprivileged_userns_clone=0 unless you need it), and disabled unused modules.

Runtime Isolation

Containers are not a security boundary. Say it louder for the people in the back. A container escape is a well-worn path, and --privileged hands it to you on a platter. For untrusted PR builds — anything from a fork, anything from outside your org — use a stronger isolation runtime: gVisor, Kata Containers, or Firecracker microVMs.

# RuntimeClass for gVisor — schedules pods onto the runsc runtime.
apiVersion: node.k8s.io/v1
kind: RuntimeClass
metadata:
  name: gvisor
handler: runsc
---
# Reference it from the runner pod spec for untrusted jobs.
apiVersion: actions.github.com/v1alpha1
kind: AutoscalingRunnerSet
metadata:
  name: arc-untrusted
  namespace: arc-runners
spec:
  githubConfigUrl: "https://github.com/myorg"
  githubConfigSecret: arc-github-app-secret
  template:
    spec:
      ephemeral: true
      runtimeClassName: gvisor      # untrusted PR builds run here
      containers:
        - name: runner
          image: registry.internal/myorg/runner:24.09.0-ubuntu22.04
          securityContext:
            runAsNonRoot: true
            allowPrivilegeEscalation: false
            capabilities: { drop: ["ALL"] }

The runtimeClassName: gvisor routes this runner through gVisor’s user-space kernel, which intercepts syscalls and dramatically reduces the container escape surface. It costs performance — expect 10–30% overhead on syscall-heavy workloads — so only use it for untrusted builds.

Observability and Audit

Every job is an audit event. Log registration, job start, job stop, and network egress. Use Falco or a similar runtime security tool to detect suspicious behavior: a build step that curls an external host, a process that reads /etc/shadow, a shell spawned by a build tool that shouldn’t spawn shells.

# Falco rule — alert on outbound connections from runner pods to non-allowlisted hosts.
- rule: Runner Egress To Unknown Host
  desc: A CI runner pod connected to a host outside the allowlist.
  condition: >
    outbound and container
    and k8s.ns.name = "arc-runners"
    and not fd.sip in (10.20.0.0/24, 10.21.0.0/24, 10.30.5.10)
    and not fd.sport in (53, 443)
  output: >
    Runner egress to unknown host (pod=%container.name host=%fd.sip port=%fd.sport)
  priority: WARNING
  tags: [ci, runner, egress]

This rule fires when a runner connects to something outside your allowlist. Tune the CIDRs to match your environment. The point is that you’re alerted, not that you’re perfect.

Isolation Tech	Isolation Strength	Performance Overhead	Compatibility	Ops Burden
Standard container	Low	~0%	Full	Low
gVisor (runsc)	Medium-High	10–30%	High, some syscall gaps	Medium
Kata Containers	High	15–40%	High	High
Firecracker microVM	Very High	20–50%	Medium	Very High

Q: Do I really need gVisor for internal repos?

For trusted internal repos on a hardened ephemeral runner, standard containers are usually acceptable. For anything that runs code you don’t fully control — fork PRs, dependency bots, third-party contributors — use gVisor or Kata. The cost is a 10–30% slowdown on those jobs; the benefit is that a container escape doesn’t own your node.

Reference Architecture — Putting It Together

Here’s the opinionated stack I’d deploy today for a private-cloud shop running GitHub Actions at meaningful scale.

The Opinionated Stack

ARC on Kubernetes with ephemeral runners. OIDC to your cloud for all cloud access, no static keys. Vault for secrets, injected via sidecar. Default-deny NetworkPolicy with an explicit egress allowlist. gVisor RuntimeClass for untrusted PR builds. Pre-baked runner images with pinned toolchains. Harbor as a pull-through cache for both base images and your own runner images. Falco for runtime detection. Prometheus scraping ARC metrics for capacity planning.

What to Build First

You don’t need all of this on day one. The 80/20 rollout:

1. Ephemeral runners — switch to ephemeral: true and pre-baked images. This alone eliminates the persistent-runner class of incidents.
2. Network isolation — default-deny egress, allowlist the essentials. This limits what a compromised job can reach.
3. OIDC — replace static cloud keys with OIDC trust policies. This turns a permanent credential into an hour-long one.
4. Isolation runtime — add gVisor for untrusted PRs once the above is stable.

That sequence gets you 90% of the security value with a fraction of the operational complexity.

What We Deliberately Left Out

Multi-region runner federation is not worth it below a certain scale — the coordination complexity exceeds the latency benefit for most shops. Custom runner schedulers are a trap; ARC already does this well and is maintained by people who do it full-time. And don’t build your own secrets injection — Vault and external-secrets exist and are battle-tested.

Level	Characteristics
0 — Naive	Persistent runners, static cloud keys, no egress filtering, Docker socket mounted
1 — Ephemeral	ARC with ephemeral: true, pre-baked images, still no network isolation
2 — Segmented	Default-deny NetworkPolicy, egress allowlist, runners in dedicated namespace/VLAN
3 — Keyless	OIDC to cloud, Vault Agent injection, no static secrets on runners
4 — Hardened	gVisor for untrusted PRs, Falco detection, full audit logging, cache scoping per trust tier

Closing — The Uncomfortable Truth

Self-hosting moves the trust boundary. It does not remove it. The teams that win at this treat their runners as untrusted infrastructure — hostile territory that happens to run their builds — rather than as trusted pets that live inside the perimeter. The teams that lose treat the migration as a one-time project and go back to shipping features while the runner quietly accumulates credentials and network reach.

The action items, in imperative voice:

• Audit your current runners against the Level 0–4 maturity table above. Most teams are at Level 0 or 1 and don’t know it.
• Switch to ephemeral runners this quarter. If you do nothing else, do this.
• Enforce default-deny egress from runner subnets. Allowlist only what builds need, and log every drop.
• Replace static cloud credentials with OIDC. If a runner has an AWS access key on disk, that’s a finding.
• Add gVisor for untrusted PR builds. Fork PRs should never run on a runtime that shares a kernel with your node.
• Instrument for detection, not just prevention. You will miss something. Falco rules and egress logs are how you catch it.

Go audit your setup. The maturity table is your checklist. If you’re at Level 0, you have work to do — and the CircleCI breach is a reminder of exactly why.