Traefik 502 Bad Gateway: Docker Reverse Proxy Fixes

2:47 AM. PagerDuty fires. “Production site returning errors.” You fumble for the laptop, hit the URL, and get that familiar white page: 502 Bad Gateway. Traefik is up. Traefik is healthy. Traefik is cheerfully proxying every request into a void and returning 502s with the calm confidence of a service that believes it’s doing exactly what you asked.

Here’s the thing nobody tells you at 2:47 AM: Traefik 502s are almost never Traefik’s fault. They’re a symptom of a broken assumption somewhere in your stack — a label that doesn’t say what you think it says, a network that Traefik isn’t actually attached to, a backend that hasn’t bound its socket yet, or a middleware chain that’s silently short-circuiting. Traefik is just the messenger, and you keep shooting the messenger.

This guide is for sysadmins, platform engineers, and DevSecOps folks running Docker + Traefik in production. By the end, you’ll have a mental model for why 502s happen and a five-command triage flow that turns “restart it and pray” into an actual diagnostic process. We’ll break a working config on purpose, watch it fail, and fix it properly.

TL;DR

• A 502 from Traefik means the upstream connection failed or returned garbage — check the backend, not Traefik, first.
• The three most common root causes: missing loadbalancer.server.port label, Traefik not on the backend’s Docker network, and startup race conditions.
• Docker’s HEALTHCHECK is not honored by Traefik’s Docker provider for routing decisions (Swarm is different).
• Build the five-command triage into your on-call runbook. Alert on 5xx rate, not just container liveness.

Prerequisites

Before we start breaking things, you need:

• Docker Engine 24.x or newer (any recent version works; check docker version)
• Docker Compose v2 (docker compose, not the legacy docker-compose binary)
• Traefik v3.x — as of this writing, v3 is current. Pin the exact patch version in prod; :latest is how you get surprise 502s on a Tuesday.
• curl, openssl, and jq installed on the host
• Root or docker-group access on the host

What a 502 Actually Means (And Why Traefik Isn’t Lying to You)

Traefik 502s get blamed on Traefik because Traefik is the thing returning the 502. But the HTTP semantics are unambiguous, and reading them correctly saves you an hour of grep’ing the wrong logs.

• 502 Bad Gateway — Traefik reached the upstream (or tried to) and either the TCP connection failed, the connection was reset, or the upstream returned a malformed response. The backend is broken or unreachable.
• 504 Gateway Timeout — Traefik connected but the upstream didn’t respond within the timeout window. The backend is alive but slow or hung.
• 503 Service Unavailable — Traefik has no healthy backend to route to at all. Usually a load balancer with zero servers, or all servers marked down.

Traefik’s Docker provider defaults to 502 for provider-level failures because from its perspective, “I have a router, I have a service, but I can’t establish a connection to the server” is a gateway error. It’s technically correct, which is the best kind of correct and the worst kind of helpful at 2 AM.

Traefik’s request lifecycle in 30 seconds

Every request walks this path:

Client → EntryPoint → Router (matches rule) → Middleware chain → Service → LoadBalancer → Server (container IP:port)

Each stage fails differently. A router that doesn’t match gives you a 404. A middleware that rejects gives you whatever that middleware returns (401, 403, redirect). A service with no reachable server gives you a 502. Knowing which stage failed tells you which config to look at.

Here’s a minimal known-good baseline. Save this — we’ll break it deliberately later.

# docker-compose.yml — known-good baseline
services:
  traefik:
    image: traefik:v3.1
    command:
      - "--providers.docker=true"
      - "--providers.docker.exposedbydefault=false"
      - "--entrypoints.web.address=:80"
    ports:
      - "80:80"
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock:ro
    networks:
      - proxy

  whoami:
    image: traefik/whoami:v1.10
    labels:
      - "traefik.enable=true"
      - "traefik.http.routers.whoami.rule=Host(`whoami.example.com`)"
      - "traefik.http.routers.whoami.entrypoints=web"
      - "traefik.http.services.whoami.loadbalancer.server.port=80"
    networks:
      - proxy

networks:
  proxy:
    name: proxy

Note three things: exposedbydefault=false (so you must opt in), the socket mounted :ro, and both services on the same named network. Every one of those is a 502 waiting to happen if you get it wrong.

Error code	Likely cause	First diagnostic command
502	Backend unreachable, wrong port, connection refused	docker exec traefik wget -qO- http://backend:PORT/
502	Backend not on Traefik’s network	docker network inspect proxy
502	Middleware (forwardauth) backend down	docker logs traefik --tail 200
504	Backend slow/hung, timeout too low	curl -v -w "%\{time_total\}" http://backend:PORT/
503	No healthy servers in load balancer	Check traefik.http.services.X.loadbalancer.healthcheck

Actionable takeaway

Never debug a 502 by touching Traefik first. The backend is the suspect until proven otherwise. Traefik is a router, not a magician.

The Docker Provider — Where Traefik Gets Its Truth

Traefik’s Docker provider watches the Docker socket, reads container labels, and builds routers and services dynamically. There is no config file listing your backends — the labels are the config. This is powerful and it’s also why a typo in a label produces a 502 instead of a config validation error.

Mounting the socket: :ro is not enough

Everyone mounts /var/run/docker.sock:ro and calls it a day. The :ro flag makes the file read-only inside the container — it does not restrict what the Docker API lets you do. Anyone who can talk to that socket has effective root on the host. They can start privileged containers, mount the host filesystem, and exfiltrate every secret on the box.

The correct pattern is a socket proxy that filters the API surface:

# docker-compose.yml — socket proxy pattern
services:
  socket-proxy:
    image: tecnativa/docker-socket-proxy:0.2
    environment:
      CONTAINERS: 1        # Traefik needs to list/inspect containers
      SERVICES: 0
      TASKS: 0
      NETWORKS: 1
      POST: 0              # block all write operations
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock:ro
    networks:
      - proxy

  traefik:
    image: traefik:v3.1
    command:
      - "--providers.docker.endpoint=tcp://socket-proxy:2375"
      - "--providers.docker.exposedbydefault=false"
    networks:
      - proxy

The proxy exposes only the read endpoints Traefik actually needs and blocks everything else. Most tutorials skip this because it’s one more container. Do it anyway — the alternative is that a compromised Traefik container owns your host.

Labels are a contract, not decoration

The single most common 502 in the wild comes from a missing port label. If you don’t declare traefik.http.services.X.loadbalancer.server.port, Traefik guesses — it picks the first exposed port it finds on the container. If your image exposes 8080 and 3000, and Traefik picks 8080, you get a 502 because nothing is listening there.

# BROKEN — missing server.port, Traefik guesses wrong
labels:
  - "traefik.enable=true"
  - "traefik.http.routers.app.rule=Host(`app.example.com`)"
  # no loadbalancer.server.port — Traefik picks first EXPOSE

# CORRECT — explicit port, no guessing
labels:
  - "traefik.enable=true"
  - "traefik.http.routers.app.rule=Host(`app.example.com`)"
  - "traefik.http.routers.app.entrypoints=web"
  - "traefik.http.services.app.loadbalancer.server.port=3000"

The “it worked yesterday” version of this bug happens when someone rebuilds an image and the order of EXPOSE directives changes, or a new port gets added. Traefik silently re-picks, and your 502 appears out of nowhere.

Network Topology — The Silent Killer

Traefik can only reach containers it shares a Docker network with. This sounds obvious. It is violated constantly. The default bridge network doesn’t do DNS by container name, and if Traefik is on proxy while your app is on default, Traefik resolves the app name to nothing and 502s.

# docker-compose.yml — explicit network attachment
services:
  traefik:
    image: traefik:v3.1
    networks:
      - proxy

  api:
    image: myapp:1.4.2
    networks:
      - proxy          # MUST be on the same network as Traefik
      - backend        # internal-only network for DB access

  postgres:
    image: postgres:16
    networks:
      - backend        # DB is NOT on proxy — correct

networks:
  proxy:
    name: proxy
  backend:
    name: backend
    internal: true     # no outbound internet for the DB tier

Here’s what a broken topology looks like when you inspect it:

$ docker network inspect proxy --format '{{range .Containers}}{{.Name}} {{end}}'
traefik api
# If `api` is missing from this list, Traefik cannot reach it. Full stop.

The host.docker.internal trap

On Docker Desktop (Mac/Windows), host.docker.internal resolves to the host. On Linux, it doesn’t exist unless you add extra_hosts: ["host.docker.internal:host-gateway"]. Teams develop on Macs, deploy to Linux, and wonder why their backend that “worked locally” 502s in prod. Don’t route through the host — put both containers on the same network and use the service name.

Swarm and overlay networks

If you’re on Docker Swarm, the Docker provider doesn’t see Swarm services the way it sees standalone containers. You need the Swarm provider (--providers.swarm=true) and overlay networks change DNS behavior. This is a different beast — read the docs for the current version before assuming your Compose labels port over.

Network mode	Traefik can reach backend?	Notes
Same custom bridge	Yes	Container-name DNS works. This is what you want.
Default bridge only	No	No DNS by name. Traefik resolves to nothing.
Traefik on host network	Only via published ports	Loses container DNS; use host.docker.internal hacks. Avoid.
Swarm overlay	Yes, with Swarm provider	Docker provider won’t see Swarm tasks.
Different custom networks	No	The classic “recreated container on new network” bug.

War story: A team spent a week blaming Traefik’s “flaky load balancer.” Every deploy, roughly one in five requests 502’d. The cause: docker compose up -d recreated the app container, which landed on a fresh network instance, while Traefik stayed attached to the old one. Traefik’s Docker provider kept the stale IP for a few seconds and routed into the void. The fix was a stable named network, not a Traefik restart.

Health Checks, Readiness, and the Lie of “Container is Up”

A running container is not a ready container. Docker’s HEALTHCHECK tells Docker whether the app is healthy. Traefik’s Docker provider — non-Swarm — does not use that information for routing. It routes to any running container with matching labels. This is the single biggest source of “why is it 502ing when docker ps says healthy?”

# docker-compose.yml — healthcheck + depends_on
services:
  api:
    image: myapp:1.4.2
    healthcheck:
      test: ["CMD", "wget", "-qO-", "http://localhost:3000/health"]
      interval: 5s
      timeout: 3s
      retries: 5
      start_period: 10s
    networks:
      - proxy

  worker:
    image: myworker:1.4.2
    depends_on:
      api:
        condition: service_healthy   # wait for API to actually be ready
    networks:
      - proxy

start_period is the key field people forget. It tells Docker to ignore failures during the first N seconds while the app boots. Without it, your healthcheck fails during startup and Docker marks the container unhealthy before the app ever binds its socket.

For Traefik-side health checking (Swarm provider, or explicit load balancer health checks), you configure it via labels:

labels:
  - "traefik.http.services.api.loadbalancer.healthcheck.path=/health"
  - "traefik.http.services.api.loadbalancer.healthcheck.interval=5s"
  - "traefik.http.services.api.loadbalancer.healthcheck.timeout=3s"

This makes Traefik itself poll the backend and remove unhealthy servers from rotation. On the Docker provider this is opt-in; on Swarm it’s more integrated. Check the docs for your provider’s current behavior.

The startup race: App container starts, Traefik sees the labels, routes a request, and the app hasn’t bound port 3000 yet. Connection refused. 502. This is normal for the first few seconds after a deploy. The fix is start_period plus Traefik health checks, or accepting a brief 502 window during deploys — which is exactly what a rolling deploy strategy is for.

TLS, SNI, and the 502s That Aren’t Really 502s

Sometimes the browser says 502 and the actual failure is a TLS handshake that never completed. Browsers are terrible at distinguishing “connection reset” from “gateway error” when the TLS layer is involved. Use curl -v and openssl s_client — they tell you the truth.

# Confirm the TLS handshake and SNI
$ openssl s_client -connect app.example.com:443 -servername app.example.com [/dev/null 2]&1 | head -20
# Look for: "Verify return code: 0 (ok)" and the correct CN/SAN.
# If you see "no peer certificate available" or a CN mismatch, it's TLS, not a 502.

The websecure vs web entrypoint confusion is a copy-paste classic. If your router is defined on websecure but the client hits http://, Traefik has no router matching that entrypoint and returns a 404 — or, if a redirect middleware is misconfigured, a 502.

# traefik.yml — static config with proper entrypoints and ACME
entryPoints:
  web:
    address: ":80"
    http:
      redirections:
        entryPoint:
          to: websecure
          scheme: https
  websecure:
    address: ":443"

certificatesResolvers:
  letsencrypt:
    acme:
      email: ops@example.com
      storage: /acme/acme.json
      httpChallenge:
        entryPoint: web

# Router labels — note entrypoints match the static config
labels:
  - "traefik.enable=true"
  - "traefik.http.routers.app.rule=Host(`app.example.com`)"
  - "traefik.http.routers.app.entrypoints=websecure"
  - "traefik.http.routers.app.tls=true"
  - "traefik.http.routers.app.tls.certresolver=letsencrypt"
  - "traefik.http.services.app.loadbalancer.server.port=3000"

Browser symptom	Actual layer	Confirm with
502 on HTTPS, 200 on HTTP	TLS/SNI mismatch	openssl s_client -servername host
502 + cert warning	ACME failed, self-signed fallback	docker logs traefik | grep acme
502 immediately, no TLS	Router on wrong entrypoint	curl -v http://host
404 on HTTPS	No router for websecure	Check entrypoints= label

Opinionated callout

Stop setting insecureSkipVerify: true in your Traefik config to “make TLS work.” You’re not fixing a bug, you’re disabling certificate validation on a production reverse proxy. That’s a security incident with a deployment pipeline, not a workaround.

Middlewares — The Order of Operations Will Bite You

Traefik evaluates middlewares in the order they’re declared. This is a one-line fact with catastrophic consequences. If you put redirectregex before auth, an attacker can craft a URL that matches the redirect and bypasses auth entirely. If you put stripPrefix before auth, your auth service sees the wrong path and rejects everything — or worse, allows everything.

# CORRECT middleware order: auth first, then transformation
labels:
  - "traefik.http.routers.app.middlewares=auth@file,stripprefix@file,ratelimit@file"
  # Evaluation: auth → stripprefix → ratelimit
  # Auth sees the ORIGINAL path. Transformation happens after authorization.

# WRONG — auth runs after prefix stripping, sees wrong path
labels:
  - "traefik.http.routers.app.middlewares=stripprefix@file,auth@file"

The forwardauth 502

This one trips up every team eventually. You protect a route with forwardauth, pointing at an auth service. The auth service goes down. Traefik tries to reach it, fails, and returns 502 — not 401, not 403. From an alerting perspective this is a problem: your “auth failures” dashboard looks fine while your “5xx rate” dashboard is on fire.

# forwardauth middleware — note the response headers you need to forward
labels:
  - "traefik.http.middlewares.auth.forwardauth.address=http://auth-service:4181/verify"
  - "traefik.http.middlewares.auth.forwardauth.trustForwardHeader=true"
  - "traefik.http.middlewares.auth.forwardauth.authResponseHeaders=X-User-Id,X-User-Roles"

trustForwardHeader=true tells Traefik to trust X-Forwarded-* headers from upstream. Only enable this if Traefik is the first proxy in the chain — otherwise you’re letting clients spoof their own source IP. authResponseHeaders copies headers from the auth service’s response onto the forwarded request, so your backend can read the authenticated user identity.

The distinction matters for alerting: 502 from forwardauth means the auth service is down, not that a user was rejected. Alert on it separately.

A Repeatable Diagnostic Flow (Stop Guessing)

Here’s the five-command triage. Run them in order. Stop when you find the failure.

#!/usr/bin/env bash
# traefik-triage.sh — five-command 502 triage
set -euo pipefail

BACKEND="api"          # container name
PORT="3000"            # the port Traefik should be hitting
NET="proxy"            # the shared network

echo "== 1. Is the backend running? =="
docker ps --filter "name=${BACKEND}" --format 'table {{.Names}}\t{{.Status}}\t{{.Ports}}'

echo "== 2. Is Traefik on the same network? =="
docker network inspect "${NET}" --format '{{range .Containers}}{{.Name}} {{end}}'

echo "== 3. Can Traefik reach the backend? =="
docker exec traefik wget -qO- --timeout=3 "http://${BACKEND}:${PORT}/health" || echo "UNREACHABLE"

echo "== 4. What does Traefik say? =="
docker logs traefik --tail 200 2>&1 | grep -iE "error|502|dial|refused" | tail -20

echo "== 5. What does the client see? =="
curl -sv "https://app.example.com/" -o /dev/null 2>&1 | grep -E "< HTTP|error|refused"

Step 3 is the money maker. If Traefik can’t wget the backend from inside its own container, the problem is networking or the port — not routing, not TLS, not middleware. If it can, the problem is upstream of the connection.

Reading access logs like an adult

Enable structured access logs so you can actually query them:

# traefik.yml — access log config
accessLog:
  format: json
  fields:
    defaultMode: drop
    names:
      ClientHost: keep
      RequestPath: keep
      DownstreamStatus: keep
      OriginStatus: keep
      Duration: keep
      ServiceName: keep
      RouterName: keep

The critical field is OriginStatus — the status code the backend returned. DownstreamStatus is what Traefik sent to the client. If OriginStatus is 200 but DownstreamStatus is 502, the backend responded but Traefik couldn’t relay it (rare, usually a connection reset mid-response). If OriginStatus is 0 or absent, Traefik never got a response at all — pure connection failure. That’s your 502.

# Find every 502 and see which router/service produced it
docker logs traefik 2>&1 | jq -c 'select(.DownstreamStatus == 502) | {RouterName, ServiceName, RequestPath, OriginStatus}'

Symptom	Command	Healthy result
Backend down	docker ps	Container Up with expected ports
Network mismatch	docker network inspect proxy	Both traefik and backend listed
Connection refused	docker exec traefik wget ...	HTTP 200 with health body
Routing/label bug	docker logs traefik | grep error	No “no available server” errors
Client-side failure	curl -sv https://host/	< HTTP/2 200

Actionable takeaway

Paste the five-command triage into your on-call runbook today. The next 502 will take you four minutes instead of forty. Future you will buy present you a beer.

Hardening and Avoiding the Next 502

Production checklist

• Socket proxy, not raw socket mount. The :ro flag is not a security boundary.
• Explicit server.port on every router. No guessing, ever.
• Health checks on every backend, with start_period sized to your slowest cold start.
• Access logs shipped to your SIEM. JSON format, OriginStatus preserved.
• Alert on 5xx rate, not container liveness. A container can be “up” and still 502’ing everything.
• Pin the Traefik image version. traefik:v3.1.2, not traefik:latest.

What to never do

• traefik.enable=true on every container by default. Opt in explicitly; exposedbydefault=false is the correct default.
• Exposing the Traefik dashboard publicly without auth. It leaks your entire routing topology. Use basicauth middleware plus an IP allowlist, or don’t expose it at all.
• :latest on the Traefik image in prod. A minor version bump can change provider behavior and you’ll be debugging at 3 AM.
• insecureSkipVerify: true anywhere. Ever.

Common Mistakes and How to Avoid Them

1. Missing loadbalancer.server.port. Traefik guesses the first exposed port. It guesses wrong. Fix: declare the port on every service label, no exceptions.

2. Traefik and backend on different networks. The container was recreated and landed elsewhere. Fix: use a stable named network and attach both services explicitly.

3. Assuming Docker HEALTHCHECK affects routing. It doesn’t, on the Docker provider. Fix: use Traefik load balancer health checks or the Swarm provider.

4. Middleware order causing auth bypass. redirectregex before auth is a security hole. Fix: auth always first, transformations after.

5. Debug log left on in prod. --log.level=DEBUG writes every routing decision to disk and tanks throughput under load. Fix: INFO in prod, DEBUG only during an incident, then revert.

FAQ

Q: Why does my service work with docker run but 502 behind Traefik? Because docker run doesn’t care about networks or labels, and Traefik does. When you run standalone, the container binds its port and you hit it directly. Behind Traefik, the container must be on Traefik’s network, must have traefik.enable=true, and must declare loadbalancer.server.port. Any of those missing produces a 502 while docker run keeps working fine.

Q: Traefik returns 502 for the first 10 seconds after deploy — is that normal? Yes, and it’s fixable. The app container starts before its framework has bound the socket, so Traefik routes to a port with nothing listening. Add a healthcheck with start_period and a Traefik load balancer health check so Traefik waits for the backend to be ready before adding it to rotation.

Q: Why does my auth-protected route return 502 instead of 401? Because the forwardauth middleware couldn’t reach your auth service. Traefik treats “auth service unreachable” as a gateway failure, not an authorization failure. Check that the auth service is up and on the same network, and alert on 502s from auth routes separately from normal 401s.

Q: Should I run Traefik in Docker or as a host binary? Docker, in almost every case. You get the provider integration, dynamic config from labels, and the same deployment model as your apps. The exception is if you need Traefik to bind privileged ports on a host where you can’t run Docker as root — rare. The socket-proxy pattern mitigates the main security concern.

Q: Does Traefik respect Docker HEALTHCHECK? On the Docker provider (non-Swarm), no. Traefik routes to any running container with matching labels regardless of Docker’s health status. On Swarm, health checking is more integrated. If you need Traefik to skip unhealthy backends on the Docker provider, configure loadbalancer.healthcheck.* labels explicitly.

Q: Why did my 502 start after docker-compose up recreated a container? Because the recreated container landed on a new network instance while Traefik stayed on the old one, or Traefik’s provider cache held a stale IP for a few seconds. Fix: use a stable named network (networks: proxy: name: proxy) so recreation doesn’t create a new network, and give Traefik a moment before you assume it’s broken.

Key Takeaways

• A 502 is a symptom of a broken backend assumption — check Docker networking and labels before touching Traefik config.
• Declare loadbalancer.server.port on every service; Traefik’s port guessing is the #1 cause of “it worked yesterday.”
• Docker HEALTHCHECK does not influence Traefik routing on the Docker provider — use Traefik health checks or depends_on with service_healthy.
• Middleware order is security-relevant: auth before transformation, always.
• Build the five-command triage into your runbook and alert on 5xx rate, not container liveness.

The 502 is not a disease. It’s a symptom, and symptoms have causes. Once you internalize that Traefik is a router that faithfully reports what it can’t reach, the mysticism evaporates — and you stop restarting containers hoping the problem goes away. The fix is a mental model and a runbook, not a Stack Overflow copy-paste you don’t understand.

The next 2 AM page is coming. It might not even be a 502. But when it is, you’ll be back in bed in ten minutes instead of ninety, and your team will wonder what changed. Nothing changed except that you stopped guessing.

The full working docker-compose.yml, the traefik.yml static config, and the triage script are in the companion repo — link it in your on-call doc and move on with your life.