GitLab Runners: Docker vs Kubernetes Executors Guide

Introduction

If you’re still using the shared GitLab SaaS runners for anything beyond a pet project, you’re gambling with your CI/CD pipeline’s stability and security. I’ve seen teams burn two days debugging a failed deploy only to realize a shared runner was blacklisted due to cryptomining activity from another tenant. Shared runners are a honeypot.

This guide is for senior sysadmins, platform engineers, and security engineers who need to decide between self-hosted Docker executors and Kubernetes executors for GitLab CI. After reading, you’ll know exactly which executor fits your team’s size, workload, and tolerance for YAML. You’ll also get config snippets that actually work in production — not the sanitized nonsense from official docs.

TL;DR

Prerequisites

A GitLab instance (self-managed or GitLab.com) with admin access to register runners.
For Docker executor: Docker Engine installed on a Linux host (preferably not your dev machine).
For Kubernetes executor: a Kubernetes cluster (v1.21+), kubectl configured, and a namespace for CI jobs.
Basic understanding of GitLab CI configuration (.gitlab-ci.yml).
A registry to store your custom helper images (optional but recommended).

Why Not the GitLab SaaS Runners? (The Obligatory Grumpy Take)

The cost/performance trap

Shared runners are fine for a 5-minute unit test. But when your pipeline needs 20 minutes of CPU-heavy compilation or integration tests, you’re paying in queue time and unpredictable IPs. GitLab’s free-tier shared runners throttle CPU after a few minutes and queue jobs behind thousands of others. I’ve seen builds take 45 minutes that ran in 8 on a self-hosted Docker runner.

# .gitlab-ci.yml - force your job to use a self-hosted runner by tagging it
build:
  tags:
    - docker-runner-prod
  script:
    - make

If you don’t add the tag, GitLab may fall back to shared runners. Always tag your jobs.

Security isolation theater

GitLab SaaS runners run your jobs inside Docker containers, but they share the host kernel. If you’re handling secrets (e.g., CI_JOB_TOKEN for API access) or building artifacts containing IP, that’s a shared trust boundary I wouldn’t cross. A teammate once had a shared runner expose the Docker socket through an environment variable — other tenants could have connected to it. Never happen again.

When you absolutely must self-host: compliance (SOC2, HIPAA), air-gapped networks, or you’re just paranoid enough to be safe.

Comparison table: GitLab SaaS runners vs self-hosted

Factor	GitLab SaaS Runner	Self-Hosted
Cost per job	Pay-as-you-go, but queue time is hidden cost	Fixed infrastructure cost
Security isolation	Shared kernel with untrusted tenants	Dedicated host or namespace
Performance	CPU throttling after minutes	Full resource control
Flexibility	Limited to Docker executor, no privileged mode	Any executor, custom images
Operational burden	None	You own the pain

The GitLab Runner Architecture – A Quick Refresher

Runner to coordinator job lifecycle

The runner polls the GitLab coordinator for jobs, pulls the executor (Docker, Kubernetes, shell), runs the job inside the executor’s isolation boundary, and sends logs/artifacts back. That’s it. The configuration lives in a single config.toml on the runner host (or for Kubernetes, in the runner’s ConfigMap).

# config.toml minimal structure for a Docker executor
concurrent = 4
check_interval = 3

[[runners]]
  name = "docker-runner-01"
  url = "https://gitlab.example.com/"
  token = "[GITLAB_RUNNER_TOKEN]"
  executor = "docker"
  [runners.docker]
    image = "alpine:latest"

Executor choices – the three you actually see in production

Shell executor: don’t. A junior admin once ran rm -rf / in CI because he wrote a cleanup script without changing directory. The shell executor executed it on the runner host. System restoration took 5 hours. Stick with containerized executors.
Docker executor: good isolation, simple, no orchestration overhead. Our workhorse.
Kubernetes executor: complex but powerful. Adds a Pod per job, integrates with cluster auto-scaling.

The critical difference: how each executor manages concurrency

Many people confuse concurrent (global limit on number of jobs the runner will execute simultaneously) with [runners].limit (max jobs this runner can take from the coordinator). If concurrent = 10 but limit = 2, only 2 jobs will be processed at a time. limit defaults to concurrent if unset. I’ve seen a Kubernetes executor with concurrent = 50 and no limit, causing 50 Pods to spawn and crash the namespace.

# setting concurrent and limit correctly
concurrent = 10

[[runners]]
  name = "docker-runner-prod"
  limit = 5   # only 5 parallel jobs, even though global concurrent is 10
  executor = "docker"
  ...

Docker Executor – The Reliable Workhorse (With Caveats)

What it does well – isolation without orchestration overhead

Simple: spin up a container per job, destroy it, repeat. No extra control plane needed. You can get away with a single bare-metal VM for months if your load is predictable.

# config.toml for Docker executor with common settings
[[runners]]
  name = "docker-prod"
  executor = "docker"
  [runners.docker]
    image = "ubuntu:20.04"
    privileged = false  # NEVER set this true unless you need DinD and fully understand the risk
    volumes = ["/cache"]
    shm_size = 256m     # avoid OOM mistakes
    allowed_images = ["registry.internal/*"]   # restrict images to your private registry

The allowed_images list prevents accidental pulls of large or malicious images. Set it.

The scalability cliff

On a single host, you’re limited by CPU cores, memory, and disk I/O. Adding more runners means adding more VMs or statically configured hosts. A 8-core, 32GB VM can run about 6 heavy jobs concurrently before thrashing swap.

# resource limits per runner in config.toml
[[runners]]
  name = "docker-prod"
  [runners.docker]
    cpus = "2.0"    # each job gets at most 2 CPU cores
    memory = "4g"   # each job gets at most 4 GB RAM

Set these limits or you’ll see OOM kills and erratic failures. Real numbers: we once had a job building a Go binary that consumed 10GB of RAM on a 8GB VM. Set limits, set alerts.

The dirty secrets – orphaned containers and image drift

If a runner crashes, containers can linger. That’s why you need cron jobs or docker system prune – but that kills caching too. The cleanup_interval setting in config.toml helps but doesn’t catch everything.

# config.toml cleanup settings
check_interval = 3
cleanup_interval = 300   # seconds between cleanup passes

War story: We had a runner with 50 orphaned containers after a kernel panic. The next week’s deploy used a stale helper image (we’d upgraded GitLab runner) and broke our builds for two hours. Always pin the helper_image version or use the same image tag as your GitLab runner version. Better yet, use a custom helper image that you control.

Comparison table: Docker executor vs Kubernetes executor – core differences

Aspect	Docker executor	Kubernetes executor
Isolation level	Container (kernel shared with host)	Pod (kernel shared with node, but namespace isolated)
Resource limits per job	vCPU/Memory in config.toml	CPU/Memory requests & limits in Pod spec
Scaling model	Manual: add VMs	Automatic: cluster auto-scaler and runner autoscaler
Setup complexity	Low (one docker run command)	High (Helm chart, RBAC, ingress)
Cache persistence	Host volume or S3	S3 (ephemeral node storage)
Cold start time	<1 second	30–60 seconds (Pod scheduling)

Kubernetes Executor – Power, But at What Cost?

The promises – auto-scaling, dynamic resource allocation, native scheduling

Kubernetes lets you treat each job as a Pod. That means no more feeding your hosts manually. But with great power comes YAML vomiting. The GitLab Runner Helm chart is the recommended way to deploy runner agents inside a K8s cluster.

# config.toml for Kubernetes executor (base config)
[[runners]]
  name = "k8s-runner-prod"
  executor = "kubernetes"
  [runners.kubernetes]
    namespace = "gitlab-ci"
    image = "ubuntu:20.04"
    cpu_limit = "2"
    memory_limit = "4Gi"
    service_account = "gitlab-runner"
    privileged = false

The hidden complexity – service accounts, namespaces, and `securityContext`

Every job pod runs under the runner’s service account. If you don’t lock it down, any CI job can run kubectl get secrets and dump all secrets in the namespace. Always create a dedicated service account with minimal RBAC.

# RBAC for gitlab-runner service account
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  namespace: gitlab-ci
  name: gitlab-runner
rules:
- apiGroups: [""]
  resources: ["pods"]
  verbs: ["get", "list", "create", "delete"]
- apiGroups: [""]
  resources: ["pods/log"]
  verbs: ["get"]
- apiGroups: [""]
  resources: ["pods/exec"]
  verbs: ["create"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  namespace: gitlab-ci
  name: gitlab-runner
subjects:
- kind: ServiceAccount
  name: gitlab-runner
  namespace: gitlab-ci
roleRef:
  kind: Role
  name: gitlab-runner
  apiGroup: rbac.authorization.k8s.io

War story: A developer put kubectl delete pod --all in a test script. The runner pod was evicted. So was the production deployment’s pod that happened to share the same namespace. They had a production outage for 10 minutes because they used the default service account. Separate namespaces and enforce RBAC.

The scalability promise – when it works and when it doesn’t

Kubernetes can scale the runner itself via the autoscaler and V2 runner (newer design), but that adds another layer of control loops. The real scaling power is from cluster auto-scaler adding nodes when Pods are pending. But cold start time for the first job can be 60 seconds as nodes spin up.

# Kubernetes runner pod spec for GPU jobs – nodeSelector example
[[runners]]
  name = "gpu-runner"
  executor = "kubernetes"
  [runners.kubernetes]
    namespace = "gitlab-ci-gpu"
    image = "nvidia/cuda:11.8-devel"
    cpu_limit = "4"
    memory_limit = "16Gi"
    [runners.kubernetes.pod_spec]
      nodeSelector = { "nvidia.com/gpu" = "true" }
      tolerations = [
        { key = "nvidia.com/gpu", operator = "Exists", effect = "NoSchedule" }
      ]

Real numbers: A cluster with 10 nodes (32GB each) can run 50+ jobs concurrently if properly sized, but only if you set concurrent and limit correctly — otherwise you get Pod evictions. I’ve seen a team set concurrent = 200 on a 3-node cluster. Nodes ran out of memory, Pods OOM killed, jobs failed intermittently with no clear error.

Comparison table: Docker vs Kubernetes – complexity and cost

Factor	Docker	Kubernetes
Initial setup time	30 minutes	2-3 hours (including RBAC, Helm chart tuning)
Scaling difficulty	Manual (add more VMs)	Automatic but requires cluster autoscaler config
Resource utilization	Good for stable load	Excellent for spiky load (bin packing)
Operational cost	Low (one box)	Higher (cluster management, etcd overhead)
Debugging	Easy (docker logs)	Harder (kubectl describe, pod logs, sometimes pod evicted before you see)

Docker-in-Docker (DinD) vs Kubernetes Pods – The Runner War Story

Why DinD is a necessary evil (and often a lazy one)

Building Docker images inside CI used to mean a docker:dind sidecar. It works, but you’re running an insecure daemon with --privileged. The GitLab runner must expose the Docker socket or run a DinD service that connects to a privileged daemon. That privilege escalates to the host if not careful.

# .gitlab-ci.yml using DinD service (not recommended)
build image:
  services:
    - docker:dind   # runs a Docker daemon inside a container
  variables:
    DOCKER_DRIVER: overlay2
  script:
    - docker build -t myapp .

The security escape vector: if the Docker daemon runs as root and the container has privileged access, an attacker can mount the host’s filesystem. I’ve seen a proof-of-concept that escapes a DinD container and modifies the host’s iptables.

Kubernetes-native image building – cleaner but not trivial

If you’re on Kubernetes executor, you can use Kaniko directly without DinD. No privileged containers, no socket mounts. Kaniko runs as a user-space container and builds images without root.

# .gitlab-ci.yml using Kaniko on Kubernetes executor
kaniko build:
  image: gcr.io/kaniko-project/executor:debug
  script:
    - /kaniko/executor --context=. --destination=registry.internal/myapp:$CI_COMMIT_SHORT_SHA --cache=true

Compare performance: DinD caching layers on the host’s Docker cache; Kaniko’s local cache is ephemeral (unless you attach a volume). You gain security but may lose cache hit rate. For many teams, the security gain outweighs the cache hit cost.

The “I just want it to work” trap

I’ve seen teams spend weeks on Kubernetes runner config only to revert to a simple Docker runner VM because their pipeline required --privileged for something stupid (e.g., mounting FUSE filesystems). Decision table:

Scenario	Recommendation
Building Docker images	Use Kaniko on K8s or Docker executor with DinD (if you trust the images).
Building with FUSE (e.g., AppImage)	Docker executor with `privileged = false` is often enough; if not, you need special node config.
need GPU scheduling	K8s executor with nodeSelector and tolerations.
Simple npm test	Docker executor on a cheap VM.

Operational Best Practices – Surviving Production

The Three Laws of Runner Configuration

1. Never run as root inside the container. Use user namespace remapping in Docker executor, or set securityContext.runAsNonRoot = true in Kubernetes executor.

2. Always set resource limits on both executors. A single job can DoS your cluster or host by consuming all CPU or memory.

3. Use allowed_images to restrict which images can be pulled. Otherwise, someone will run image: alpine:latest and then wget a cryptominer.

# allowed_images in config.toml for Docker executor
[[runners.docker]]
  allowed_images = ["registry.internal/*", "alpine:latest", "ubuntu:20.04"]

Monitoring runners like you mean it

If you aren’t logging runner API calls and job exit codes, you’re blind. GitLab runner exposes Prometheus metrics on port 9252 at /metrics. Scrape it.

# Prometheus scrape config snippet
scrape_configs:
  - job_name: 'gitlab-runner'
    static_configs:
      - targets: ['[runner-ip]:9252']

War story: We missed that one runner had a disk failure for three days because nobody watched systemd-analyze blame. The runner was still accepting jobs but writing logs to a failing disk — all job output was blank. We ended up rerunning 200+ jobs. Set disk usage alerts.

Handling cache like a senior engineer

The default S3 cache with key: "$CI_COMMIT_REF_SLUG" is fine until someone force-pushes and invalidates everything. Use cache_shared: true if you have multiple runners (Docker or K8s) to avoid duplicate cache uploads.

# runner-level cache configuration using MinIO
[runners.cache]
  Type = "s3"
  ServerAddress = "minio.internal:9000"
  AccessKey = "[id]"
  SecretKey = "[secret]"
  BucketName = "gitlab-cache"
  Insecure = false
  PathStyle = true
  CacheShared = true

Do not use NFS for cache with Docker executor — locking issues and stale files will cause intermittent failures. Use S3 or MinIO.

Common Mistakes and How to Avoid Them

Forgetting to set privileged = false – Many tutorials copy-paste a Docker executor config with privileged = true to support DinD. That gives full root access to the host inside the container. If you need DinD, run a dedicated runner for that use case and tag jobs accordingly.
Not pinning helper image version – GitLab runner auto-pulls the helper image matching the runner version. If you upgrade runner but a cached helper image stays, builds may fail. Either use helper_image = "gitlab/gitlab-runner-helper:x86_64-latest" (same as runner) or pin to a specific version.
Setting concurrent too high on Kubernetes executor – Each job creates a Pod. If you set concurrent = 100 but your cluster only has resources for 20 Pods, the remaining 80 will be pending until some finish. Set limit per runner to a sensible number.
Ignoring namespace quotas – In Kubernetes, set ResourceQuota on the namespace used by the runner to prevent job pods from exhausting all cluster resources.
Not tagging jobs – If you don’t tag jobs, GitLab may assign them to shared runners or any runner that matches no tags, leading to unpredictable execution environments.

FAQ

Q: Can I run Docker executor on Kubernetes as a pod? A: Yes, it’s called Docker-out-of-Docker, but you’ll need to mount the host’s Docker socket — defeating the whole point of Kubernetes isolation. Use Kubernetes executor instead.

Q: My job fails with “cannot connect to the Docker daemon” – what gives? A: You’re probably using the Docker executor without privilege and trying to run docker build. Switch to Kaniko or use the dind service with proper runner config.

Q: Should I use the GitLab Runner Helm chart? A: If you’re on Kubernetes, yes — it handles registration tokens, RBAC, and scaling. But read the chart values carefully; the defaults are often insecure (e.g., default service account has too many permissions).

Q: How many runners do I need per executor type? A: One runner per executor type (one Docker, one Kubernetes) is fine if you route by tags. Multiple runners of same type only if you need different environments (e.g., GPU vs general).

Q: Why are my cache hits so low with Kaniko? A: Kaniko’s local cache is ephemeral. To persist cache layers, you need to push to a remote registry or use a shared volume. Consider using --cache-repo=registry.internal/cache to share layers across builds.

The Verdict – When to Use Which (And Why Most of You Should Start with Docker)

Docker executor – the pragmatic choice for teams < 20

Simpler to set up, easier to troubleshoot, forgiving of mistakes. You can always migrate to Kubernetes later. Start with a single beefy VM running Docker runner with concurrent = 8. Add a second VM when you hit the limit.

Kubernetes executor – when you need elastic scaling + heavy resource control

If your pipelines need dynamic GPU allocation, spot instances, or fine-grained CPU/memory per job, Kubernetes is the only sane way. But only if you already have a Kubernetes operations team. Don’t spin up a cluster just for CI.

The hybrid approach – run both executors under the same coordinator

# config.toml with two runners – Docker and Kubernetes
[[runners]]
  name = "docker-runner"
  executor = "docker"
  tags = ["docker"]
  ...

[[runners]]
  name = "k8s-runner"
  executor = "kubernetes"
  tags = ["k8s"]
  ...

This lets you keep simple tasks on Docker and heavy/sensitive tasks on Kubernetes without rewriting pipelines.

Final comparison table: Executive summary

Factor	Docker	Kubernetes	Hybrid (both)
Setup time	Low	High	Medium (two configs)
Scaling	Manual	Automatic	Mixed
Security	Good (if not privileged)	Better (RBAC + namespaces)	Best of both
Cost	Low	Higher (cluster overhead)	Medium
Best for	Small teams, stable load	Large teams, spiky load	Complex pipelines

Closing war story: I once spent a month tuning a Kubernetes executor to handle 100 concurrent jobs. Then we found that 90% of our developers only needed a simple npm test that ran faster on a single Docker host. We scaled back to two VMs and saved $3k/month on nodes. Don’t overengineer.