GitLab Container Registry: Automated Cleanup and Retention

A client called me in a panic last spring. Their GitLab instance was melting — docker pull times had crept past 90 seconds, nightly backups were spilling past a 6-hour window, and the storage bill had quietly crossed $40,000 a year. The root cause wasn’t a runaway build. It was three years of CI pipelines pushing a fresh image tag on every single commit, to every branch, with nobody ever deleting one. Eight terabytes of registry data, and roughly 94% of it referenced tags nobody had pulled since 2023.

This guide is for the platform engineer, SRE, or DevSecOps practitioner who inherited that registry and now owns the cleanup problem. By the end you’ll understand GitLab’s three-layer registry model, know which API endpoints actually reclaim storage, be able to encode a retention policy that won’t delete your production images at 3am, and have a copy-pasteable reference implementation combining the Cleanup Policy API, a scheduled pipeline, and safe garbage collection.

TL;DR

• Deleting a tag does not free disk space until garbage collection runs — this is the #1 misconception.
• The Cleanup Policy API (PUT /projects/:id/registry/cleanup_policy) is declarative and cron-driven; use it as your baseline before writing any custom script.
• Always pair name_regex_delete with name_regex_keep, or you will eventually delete a release tag.
• Run GC with a maintenance window or in read-only mode; the -m flag deletes untagged manifests and is not reversible.
• Never run cleanup with a token you wouldn’t hand to an intern — scope it to a single project.

Prerequisites

• GitLab version: Cleanup Policy API requires GitLab 13.10+; the bulk tag-delete endpoint requires 14.x. Verify with gitlab-rake gitlab:env:info on self-managed, or check the version in the sidebar on GitLab.com. Feature details evolve — confirm against the official docs for your exact release.
• Permissions: Maintainer or Owner on the target project (or group) to set cleanup policies. Admin access on the GitLab host to run garbage collection.
• API access: A Personal, Project, or Group Access Token with the api scope (not read_api).
• Storage backend knowledge: You need to know whether your registry uses filesystem, S3, GCS, or Azure Blob — this changes your GC strategy.
• CLI tools: curl, jq, and (for GC) shell access to the GitLab host or the registry container.

Understanding GitLab’s Registry Architecture Before You Break It

You cannot safely delete anything from a container registry until you understand the three-layer model. GitLab’s registry implements the Docker Registry HTTP API V2, and that spec has a specific (and frequently misunderstood) relationship between repositories, manifests, and blobs. Get this wrong and you’ll orphan storage that GC won’t touch for weeks.

Repositories, Tags, and the Manifest Layering Problem

The layers are:

1. Repository — a namespaced name like mygroup/myproject/api. This is the top-level container.
2. Tag — a human-readable pointer (v1.2.3, latest, sha-abc123) that resolves to a manifest.
3. Manifest — a JSON document listing the config blob and layer blobs. Multi-arch images use a manifest list that points to per-architecture manifests.
4. Blob — the actual content-addressed layer data, stored once and shared across every manifest that references it.

Here’s the part that bites people: deleting a tag only removes the tag→manifest pointer. The manifest and its blobs remain until garbage collection runs. And because base layers (your python:3.12-slim, your alpine) are shared across hundreds of images, deleting a single tag often reclaims zero bytes. You delete 500 tags and the bucket shrinks by 2GB because everything shared the same Ubuntu base.

The Built-in Garbage Collection (and Why It’s Not Enough)

GitLab ships registry-garbage-collect (Omnibus) or registry garbage-collect (standalone). It walks the blob store and deletes blobs no longer referenced by any manifest. Critically:

• It does not delete tags. That’s your job, via the API or UI.
• It requires read-only mode (or a maintenance window) to avoid racing with in-flight pushes.
• The -m flag deletes untagged manifests — the manifests left behind after you delete their tags. Without -m, GC reclaims almost nothing in a typical cleanup scenario.

The dry run is your friend:

# Omnibus GitLab — dry run, no changes made
gitlab-ctl registry-garbage-collect -m

Wait — -m in a dry run is safe because nothing is deleted. But confirm the output lists what would be removed before you commit. If the numbers look wrong (e.g., it wants to delete a manifest you know is in use), stop and investigate.

Storage Backends That Change Your Strategy

The backend determines how GC behaves and whether you can cheat with lifecycle rules.

# /etc/gitlab/gitlab.rb — S3-backed registry configuration
registry['storage'] = {
  's3' => {
    'accesskey' => 'AKIAEXAMPLEKEY',
    'secretkey' => 'examplesecret',
    'bucket' => 'gitlab-registry-prod',
    'region' => 'us-east-1',
    'v4auth' => true,
    'rootdirectory' => 'registry'
  }
}

Run gitlab-ctl reconfigure after editing. The rootdirectory scoping matters: if you later add an S3 lifecycle rule, scope it to that prefix or you’ll nuke unrelated objects in the bucket.

Backend	Pros	Cons	GC Compatibility	Cost Notes
Filesystem	Simple, no network latency	Doesn’t scale past a few TB, single-host bottleneck	Native, fastest	Local disk cost only
S3	Scales infinitely, cheap at rest	Egress costs on pulls, eventual consistency edge cases	Native, slower on large stores	Storage + egress + API calls
GCS	Strong consistency, good tooling	Fewer GitLab tuning guides	Native	Similar to S3, watch egress
Azure Blob	Native Azure integration	Hot/cool tier gotchas	Native	Tier transitions can surprise you

Does GitLab automatically delete old images? No. GitLab runs no automatic deletion by default. You must configure a cleanup policy or run your own job. This is the single most common misunderstanding.

The Cleanup API — Your Primary Weapon

The GitLab UI’s delete button is a joke beyond a single image. It paginates badly, gives no bulk operations, and offers no preview. The API is the only serious tool for anything at scale.

Authentication and Scoping Your Token

Use a project access token with api scope for cleanup jobs. Not a group token, not a personal token, and absolutely not a root PAT.

Note the scope: read_api will let you list tags but not delete them. People waste hours on this. You need api.

A war story worth internalizing: an engineer I worked with ran a cleanup script against a group-level token. The script had a bug in its project-ID loop and deleted 4,000 tags across three other teams’ registries. The token’s blast radius was the group. A project token would have contained the damage to one repo.

Token Type	Scope	Blast Radius	Rotation	Best For
Personal (PAT)	All your access	Everything you can touch	Manual, easy to forget	Interactive debugging
Project	Single project	One project	Per-project, automatable	Cleanup jobs (recommended)
Group	All group projects	Every project in group	Manual	Rarely appropriate

The Endpoints That Actually Matter

# List all repositories in a project (paginate!)
curl --header "PRIVATE-TOKEN: $TOKEN" \
  "https://gitlab.example.com/api/v4/projects/$PROJECT_ID/registry/repositories?per_page=100"

# Delete a single tag
curl -X DELETE --header "PRIVATE-TOKEN: $TOKEN" \
  "https://gitlab.example.com/api/v4/projects/$PROJECT_ID/registry/repositories/$REPO_ID/tags/v0.1.0-old"

# Bulk delete tags matching a regex, KEEPING semver releases
curl -X DELETE --header "PRIVATE-TOKEN: $TOKEN" \
  "https://gitlab.example.com/api/v4/projects/$PROJECT_ID/registry/repositories/$REPO_ID/tags?name_regex_delete=^sha-.*&name_regex_keep=^v[0-9]+\\.[0-9]+\\.[0-9]+$"

The third call is the one people don’t know exists. The bulk endpoint accepts name_regex_delete and name_regex_keep as query parameters and processes all matching tags in one request. The name_regex_keep parameter is a safety net — anything matching it is excluded from deletion even if it also matches name_regex_delete.

Rate Limits, Pagination, and the 100-Tag Wall

Naive scripts break on large registries because they fetch one page and stop. GitLab caps per_page at 100. You must follow the Link header to walk to the next page.

import requests

def iter_all_tags(base_url, project_id, repo_id, token):
    """Yield every tag across all pages for a repository."""
    url = (f"{base_url}/api/v4/projects/{project_id}"
           f"/registry/repositories/{repo_id}/tags")
    headers = {"PRIVATE-TOKEN": token}
    params = {"per_page": 100, "page": 1}

    while url:
        resp = requests.get(url, headers=headers, params=params, timeout=30)
        resp.raise_for_status()
        for tag in resp.json():
            yield tag
        # Follow the Link header for rel="next", if present
        link = resp.headers.get("Link", "")
        next_url = None
        for part in link.split(","):
            if 'rel="next"' in part:
                next_url = part.split(";")[0].strip().strip("<>")
        url = next_url
        params = {}  # next_url already contains query params

This pattern is the difference between a script that works on 50 tags and one that works on 50,000.

What’s the difference between deleting a tag and deleting a repository? Deleting a tag removes a pointer; the underlying manifest and blobs persist until GC. Deleting a repository removes all tags and marks all associated manifests for deletion, but blobs shared with other repositories still survive until GC.

Building a Retention Policy That Survives Contact With Reality

Every retention policy looks great in a slide deck. The ones that survive encode business rules, not just age-based rules. Age alone will eventually delete something you needed.

The Four Tag Categories You Must Distinguish

1. Release/semver tags (v1.2.3) — keep forever, or a long TTL (2+ years). These are your rollback targets.
2. latest — keep exactly one, but define which one. Usually the most recent semver, not the most recent push.
3. Branch/commit tags (sha-abc123, main-a1b2c3) — aggressive TTL (7–30 days). These are CI artifacts with no long-term value.
4. Environment tags (staging, prod) — mutable pointers; keep N, but never delete the one currently deployed.

Rules That Won’t Bite You in the Ass

• Never delete the tag currently deployed to prod. Detect it by querying your orchestrator (Kubernetes image field, ECS task definition) and cross-referencing against the registry. If you can’t detect it, keep the last 5 prod-prefixed tags as a buffer.
• The “keep last N” pattern is a starting point, not a rule. N=3 fails if you deploy hotfixes faster than you tag releases. N=10 is safer for high-velocity teams.
• Protected tags are GitLab’s native mechanism for preventing deletion. Configure them for v* patterns. Limitation: they don’t protect against a cleanup policy that matches, because the policy runs with elevated privileges — verify this behavior in your GitLab version.
• Immutable tags via name_regex_keep in the cleanup policy is the reliable belt-and-suspenders.

The Cleanup Policy API (GitLab 13.10+)

This is the declarative, cron-driven mechanism you should reach for first.

# Set a cleanup policy: daily cadence, keep 10, delete anything older than 30d,
# but always keep semver release tags.
curl --request PUT \
  --header "PRIVATE-TOKEN: $TOKEN" \
  --data "cadence=1d" \
  --data "keep_n=10" \
  --data "older_than=30d" \
  --data "name_regex_keep=^v[0-9]+\\.[0-9]+\\.[0-9]+$" \
  --data "name_regex_delete=.*" \
  "https://gitlab.example.com/api/v4/projects/$PROJECT_ID/registry/cleanup_policy"

The semantics: GitLab runs a job every cadence interval. For each repository, it keeps the most recent keep_n tags, deletes tags matching name_regex_delete, but never deletes tags matching name_regex_keep. A tag is only deleted if it’s both older than older_than and not protected by keep_n or name_regex_keep.

Feature	Cleanup Policy API	Custom Cron Script
Setup effort	One API call	Script + hosting + auth
Maintenance	GitLab-managed	You own every bug
Failure modes	Rare, logged in GitLab	Silent, unless you alert
Observability	Built-in job history	You build it
Flexibility	Limited to policy schema	Unlimited
Group-level	Yes (as of GitLab 14.x, verify)	You implement it

How do I exclude release tags from cleanup? Use name_regex_keep=^v[0-9]+\.[0-9]+\.[0-9]+$. Anything matching is protected regardless of age. Can I apply a policy at the group level? Yes, in recent GitLab versions — check the docs for your release, as this evolved. Group policies cascade to projects that haven’t set their own.

Automating It — Scripts, Cron, and CI Pipelines

If your cleanup runs from someone’s laptop, it doesn’t run. Automation isn’t optional — it’s the whole point.

The Scheduled Pipeline Approach (Preferred)

Run cleanup as a GitLab CI scheduled pipeline. You get free authentication (the CI job token), centralized logs, and full audit history in the pipeline UI.

# .gitlab-ci.yml — scheduled registry cleanup
stages:
  - maintenance

cleanup:registry:
  stage: maintenance
  image: python:3.12-slim
  rules:
    - if: $CI_PIPELINE_SOURCE == "schedule"
  variables:
    DRY_RUN: "false"
  before_script:
    - pip install --quiet requests
  script:
    - python scripts/registry-cleanup.py --dry-run=$DRY_RUN

Create the schedule under CI/CD → Schedules in the project, set a cron expression like 0 3 * * * (3am daily), and set DRY_RUN=true on the first run to validate.

The External Cron Fallback

When you have no choice — multi-instance registries, self-managed quirks, or a policy that needs cross-project logic — use external cron with a lock.

# /etc/cron.d/registry-cleanup — runs at 3:15am, locked against overlap
15 3 * * * root flock -n /var/lock/registry-cleanup.lock /usr/local/bin/registry-cleanup.sh >> /var/log/registry-cleanup.log 2>&1

The flock -n fails immediately if another run holds the lock, preventing overlapping runs from double-deleting or corrupting state. Log to a file you actually rotate.

Observability — Logging What You Delete

Log before you delete, not after. If a delete call fails partway through, your log is the only record of what was attempted.

import json, logging, sys, time

logging.basicConfig(stream=sys.stdout, level=logging.INFO,
                    format="%(message)s")
log = logging.getLogger("registry-cleanup")

def delete_tag(session, base_url, project_id, repo_id, tag, dry_run):
    record = {
        "ts": time.time(),
        "action": "delete_tag",
        "project_id": project_id,
        "repo_id": repo_id,
        "tag": tag,
        "dry_run": dry_run,
    }
    log.info(json.dumps(record))  # log FIRST
    if dry_run:
        return
    url = (f"{base_url}/api/v4/projects/{project_id}"
           f"/registry/repositories/{repo_id}/tags/{tag}")
    session.delete(url, timeout=30).raise_for_status()

Structured JSON logs feed cleanly into Loki, Elastic, or CloudWatch. Your compliance team will ask for an audit trail of what was deleted and when — this is it.

The Garbage Collection Gap — Where Storage Actually Gets Reclaimed

Deleting tags frees almost nothing until GC runs. This is where most teams fail: they run a beautiful cleanup policy, check the bucket size, see no change, and conclude the whole thing is broken.

When and How to Run GC

# Omnibus GitLab — delete untagged manifests and reclaim their blobs
gitlab-ctl registry-garbage-collect -m

# Standalone registry (container or binary install)
/opt/registry/bin/registry garbage-collect \
  --delete-untagged=true \
  /etc/docker/registry/config.yml

The -m / --delete-untagged=true flag is what actually reclaims space. Without it, GC only removes blobs unreferenced by any manifest — and since your deleted tags left their manifests behind, nothing gets freed. With it, GC walks untagged manifests and removes them plus their exclusive blobs.

GC Mode	What It Deletes	Risk	When To Use
Default (no -m)	Blobs unreferenced by any manifest	Low	Rarely useful post-cleanup
-m / --delete-untagged	Untagged manifests + their exclusive blobs	Medium — irreversible	After every tag cleanup

Scheduling GC Without Breaking CI

GC requires the registry to be read-only or stopped to avoid races with pushes. On Omnibus:

# Enter read-only mode, run GC, return to read-write
gitlab-ctl registry-garbage-collect -m

The Omnibus wrapper handles the read-only toggle automatically in recent versions — but it does briefly block pushes. Schedule it at 3am, not 3pm. For Kubernetes/Helm deployments, consult the chart’s maintenance values; the mechanism differs and details shift between chart versions.

Verifying Reclamation

# Before and after on filesystem backend
du -sh /var/opt/gitlab/gitlab-rails/shared/registry

For S3, check bucket metrics in CloudWatch or aws s3 ls --summarize --recursive. If the size doesn’t drop, you’re almost certainly looking at shared layers — a 500-tag cleanup that frees 2GB is normal when everything shares the same base image. It’s not a failed GC.

Why didn’t my storage shrink after cleanup? Either GC hasn’t run (deleting tags doesn’t free blobs), or the freed blobs were shared layers still referenced by other manifests. Run GC with -m and re-check.

Common Mistakes and How to Avoid Them

1. Deleting tags and expecting instant space reclamation. Tags are pointers; blobs are storage. Fix: always schedule GC after cleanup, and understand shared layers.

2. Using name_regex_delete=.* without name_regex_keep. You will delete every tag in the repository, including your releases. Fix: always pair the two, and test with a dry run first.

3. Running GC during business hours. In-flight pushes race with GC and can corrupt the registry index. Fix: maintenance window, or use the read-only mode wrapper.

4. Scoping the cleanup token too broadly. A group or personal token turns a script bug into a multi-team incident. Fix: project access tokens only, rotated quarterly.

5. Enabling S3 lifecycle rules on the registry bucket. A “delete after 30 days” rule will delete blob data the registry still references, corrupting images. Fix: never apply lifecycle rules to a registry bucket, or scope them to a prefix you don’t use.

FAQ

Q: Does GitLab automatically delete old images? No. GitLab runs no automatic deletion by default. You must configure a cleanup policy via the API or UI, or run your own scheduled job. Without it, every tag ever pushed lives forever.

Q: What’s the difference between deleting a tag and deleting a repository? Deleting a tag removes a single pointer to a manifest. Deleting a repository removes all tags and marks all its manifests for deletion, but blobs shared with other repositories survive until GC. Neither frees storage immediately.

Q: How do I exclude release tags from cleanup? Set name_regex_keep=^v[0-9]+\.[0-9]+\.[0-9]+$ in your cleanup policy. Anything matching is protected from deletion regardless of age or the keep_n count. Test the regex against your actual tag list before deploying.

Q: Can I run cleanup as a scheduled pipeline? Yes, and you should. Use rules: - if: $CI_PIPELINE_SOURCE == "schedule" in your job, create the schedule in the GitLab UI, and run with DRY_RUN=true first to validate. This beats external cron for auth, logging, and auditability.

Q: Why didn’t my storage shrink after cleanup? Two reasons: GC hasn’t run (tag deletion doesn’t free blobs), or the freed blobs were shared layers still referenced by other manifests. Run GC with -m and re-check bucket size. A small drop after a large tag cleanup is normal.

Q: Can I apply a cleanup policy at the group level? Yes, in recent GitLab versions — group-level policies cascade to projects that haven’t defined their own. Verify availability for your exact release, as this capability evolved across 14.x and later.

Key Takeaways

Key Takeaways

• Policy API first. Set a declarative cleanup policy before writing any custom script — it’s less code, fewer bugs, and better observability.
• GC is not optional. Deleting tags reclaims nothing until you run garbage collection with -m. Schedule it in a maintenance window.
• Scope your token to one project. A project access token with api scope contains the blast radius of any script bug. Never use a group or root token for cleanup.

Copy the reference implementation, adapt the regex to your tag conventions, run it in dry-run mode for a week, then ship it. Your storage bill — and your 3am pager — will thank you.