Git Bisect: Binary Search for Regressions That Saves Hours

A caching layer nobody touched in six weeks quietly tripled p99 latency last Tuesday. The grafana dashboard lit up at 03:11. By 04:00 the on-call had ruled out the obvious suspects — no deploy in the last 18 hours, no config change, no infra incident. By 05:30 they had checked out four commits by hand, rebuilt, redeployed to staging, and confirmed each one “looks fine.” By 07:00, exhausted, they gave up and rolled back a release from three weeks prior that probably wasn’t the cause. The regression was introduced 47 commits back by a subtle interaction between a header rename and a reverse proxy cache rule. git blame pointed at neither commit. git log -p on the cache module showed 400 diffs, none individually suspicious.

This is the shape of the problem git bisect solves. It turns a linear walk through N commits into a binary search over log₂(N). Forty-seven commits becomes 6 tests. Two hundred and fifty-six commits becomes 8. If you can express the bug as a shell predicate that returns 0 or 1, you never hunt a regression by hand again. This guide is for engineers who can already git log and git blame but reach for a manual checkout loop when things break. By the end you’ll be writing git bisect run scripts, wiring bisect into CI on demand, and knowing exactly which failure modes produce confident garbage — and how to catch them.

TL;DR

• Bisect is binary search over the commit DAG. If it takes N manual tests to find a regression, bisect finds it in roughly log₂(N).
• git bisect run [script] is the only version worth doing more than once a year. Manual bisect is a teaching tool.
• Exit codes are the entire interface: 0 good, 1–127 bad, 125 skip, 128+ abort.
• Non-monotonic bugs and flaky tests are the two failure modes that produce wrong answers, not just slow ones. Harden the predicate.
• CI images clone shallow by default. git fetch --unshallow before you try to bisect there.

Prerequisites

• Git 2.30 or newer (bisect itself is ancient, but --first-parent and term-renaming behavior stabilized around then — check git --version).
• A repository with real history. Shallow clones will not work.
• A reproducible way to run the failure. If you can’t script the check, bisect cannot help you.
• For CI integration: a container image with the toolchain and enough disk to hold the object store.

The 3 AM Problem: Someone Else’s Commit Broke Your Service

The hard part of regression hunting is not the algorithm. It is that the bug is usually emergent — a behavior that arises from the combination of two or three innocuous commits that merged over the course of a sprint. git blame shows you who last touched a line, not who changed the sequence of calls that made the line slow. git log -p path/to/module shows you a wall of diffs where every individual change looks fine in isolation.

The manual anti-pattern — checkout, rebuild, redeploy to staging, retest, repeat — is the worst possible way to spend four hours. It is O(N) and it is usually wrong at 3 AM because humans get sloppy by commit 12.

The math is the entire argument:

Commits between good and bad	Linear worst case	Bisect worst case	Bisect typical
8	8	3	2–3
16	16	4	3–4
64	64	6	5–6
256	256	8	7–8
1024	1024	10	9–10

Bisect turns an N-step problem into a log₂(N)-step problem. Everything else in this guide — scripts, CI jobs, term pairs, replay — is ergonomics layered on top of that one fact.

How Bisect Actually Works

Git bisect requires two markers: a commit where the bug is not present (“good”) and a commit where it is (“bad”). It then picks a midpoint on the commit graph, checks it out, and asks you — or your script — which side it belongs to. Each answer halves the search space.

The subtlety is what “good” and “bad” mean. They are not judgments about code quality. They are the output of a predicate — a function that returns true or false for a given commit. The predicate is a test. It can be a unit test, an integration test, a five-minute load test, a kernel boot, or a manual curl against a staging deploy. Bisect does not care. It only cares that the predicate is deterministic.

The other subtlety: the commit graph is not a list. Bisect walks a DAG and, on merge-heavy repositories, prefers commits that have the most reachable descendants — that’s how it guarantees logarithmic behavior. --first-parent restricts the walk to the mainline, which is faster to reason about but will miss regressions introduced on a branch and exposed by a merge.

# The idiomatic setup: bad marker, good anchor, done.
git bisect start
git bisect bad HEAD
git bisect good v2.4.18

# Sample output
Bisecting: 23 revisions left to test after this (roughly 5 steps)
[a3f9c1e2d4b8] refactor: extract cache control into middleware

After you supply the markers, git checks out a midpoint. You run the predicate, then issue git bisect good / git bisect bad / git bisect skip. When the range collapses to a single commit, git prints the culprit. Easiest way to inspect what happened later: git bisect log before you reset.

Approach	When to reach for it
Manual checkout loop	Never, once you’ve learned bisect.
git bisect interactive	First time in a repo, or when the predicate is genuinely manual.
git bisect run [script]	Default for anything you’ll do more than once.
git log --ancestry-path	Traversal of a known merge topology, not a search.

Do I need to tag my releases to use bisect? No. Any commit hash works as an anchor. HEAD~50, v2.4.18~20, main@\{yesterday\}, or the SHA you copied from a Slack thread are all fine. The only requirement is that the anchor is actually on the correct side of the change — which is the topic of the next section.

Your First Bisect: The Manual Way

You’ll do this once. Maybe twice. After that, automation wins.

The single most important thing is picking a good anchor. “It worked in staging last month” is not an anchor — it’s a vibe. If you cannot name the commit where the predicate definitively passed, you are already guessing. A useful trick when the bug is behavioral rather than hard-failing is to rename the terms up front:

# Rename good/bad to working/broken — resists muscle-memory mistakes at 4 AM.
git bisect start --term-new=broken --term-old=working
git bisect broken HEAD
git bisect working v2.4.18

A full manual session looks like this. Annotated for the shape, not to be copy-pasted verbatim:

$ git bisect start
$ git bisect bad HEAD
$ git bisect good v2.4.18
Bisecting: 23 revisions left to test after this (roughly 5 steps)
[a3f9c1e2] refactor: extract cache control into middleware

# run your test here — build, deploy, curl, whatever
$ ./scripts/check-latency.sh
FAIL

$ git bisect bad
Bisecting: 11 revisions left to test after this (roughly 4 steps)
[7e2b1a09] feat: add X-Cache-Key header

$ ./scripts/check-latency.sh
PASS

$ git bisect good
# ... and so on, until ...

$ git bisect good
4c8b2a11f3 is the first bad commit
commit 4c8b2a11f3...
    fix: normalize content-type header sent to upstream

Two things to know about cleanup. First, if a commit in the middle of the range doesn’t build at all, do not mark it good or bad — that is a lie that poisons the search. Use git bisect skip. Skipping one commit is free. Skipping four in a row degrades bisect to linear because it runs out of clean midpoints.

Second: when you’re done, you must get back to a branch. Detached HEAD after bisect is the number one complaint about the tool, and the fix is one command.

# When you're done — always.
git bisect reset

# Or jump straight back to a specific branch/ref.
git bisect reset main

# If you forgot and you're stuck in detached HEAD:
git checkout -

Invocation	What it does	When to use it
git bisect start [bad] [good]	Start with markers in one shot	When you already know both ends
git bisect start then markers	Interactive start	When you want to inspect the range first
git bisect start --first-parent	Restrict walk to mainline	Trunk-heavy repos, no long-lived branches
git bisect start --term-new=X --term-old=Y	Rename terms	Behavioral or perf regressions

git bisect run: Let the Machine Do the Tedious Part

The entire value of git bisect run is that it eliminates the human from the loop. Once your predicate is a script, bisect becomes a background job that you start and walk away from.

The interface is exit codes. That is it.

Exit code	Bisect interpretation	Common mistake
0	good	Using true as a no-op and returning 0 by accident
1–127	bad	Returning 127 for “command not found” — that’s the shell’s job
125	skip	Forgetting to use this when the build fails
128+	abort bisect	Shell dying from a signal; set -e will land here silently

The trap is that most shells return the last command’s exit code. If your script runs git clean -fdx && make build && pytest -k test_latency and the make build step fails, the script exits 2, bisect marks the commit bad, and the search is poisoned before you’ve even started. You want a script that distinguishes “predicate failed” from “environment broke.” That means explicit exit 125 on build failures.

A minimal bisect-run.sh that has survived production:

#!/usr/bin/env bash
set -u  # NOT set -e: we handle errors explicitly to control exit codes

# Clean untracked build artifacts from the previous iteration.
git clean -fdx > /dev/null 2>&1 || exit 125

# If the build fails, this commit is untestable — skip it, do not mark bad.
if ! ./gradlew --quiet assembleDebug > /tmp/build.log 2>&1; then
  echo "build failed on $(git rev-parse --short HEAD)" >&2
  exit 125
fi

# Run the narrowest test that reproduces the regression.
# Here: a single JUnit class. Adjust to your stack.
if ./gradlew --quiet test --tests 'com.example.LatencyRegressionTest' > /tmp/test.log 2>&1; then
  exit 0    # predicate passed = good
else
  exit 1    # predicate failed = bad
fi

Run it:

git bisect start
git bisect bad HEAD
git bisect good v2.4.18
git bisect run ./scripts/bisect-run.sh

The equivalent inline invocation, without a script file, for one-off checks:

git bisect run bash -c 'cargo build --release && cargo test --test regression::page_render'

When the toolchain is heavy or needs isolation — Rust, C++, or anything with system dependencies — wrap the predicate in a container so each iteration runs in a known-good state:

git bisect run docker run --rm -v "$PWD:/src" -w /src \
  ghcr.io/yourorg/ci-image:latest ./scripts/bisect-run.sh

Time budget reality check. A predicate that takes 20 minutes per iteration, over 8 iterations, is 2h40m. A predicate that takes 90 seconds is 12 minutes. Optimizing the predicate — narrowing the test, trimming the build, caching dependencies — matters more than any git flag you will ever learn.

Can I bisect against a database migration or infra change? Yes, but the predicate must be self-contained per iteration. Snapshot the schema before bisect, restore it between iterations, and make the predicate responsible for provisioning the correct migration state. Do not assume a persistent database survives across checkouts.

War Stories

Intermittent latency regression. Symptom: p99 elevated 20% during load tests only. The predicate was a k6 script — five minutes of traffic, fail if p99 crosses a threshold. Because that predicate is inherently noisy, we marked flakes as skip rather than bad, and accepted a few more iterations. The culprit turned out to be a Content-Type header change that omitted a charset parameter, which broke a downstream cache’s key matching. No test in the suite caught it; only the load profile did.

// scripts/load-check.js — run with k6, exit 1 if p99 > 400ms
import http from 'k6/http';
import { check } from 'k6';

export const options = {
  vus: 50,
  duration: '5m',
  thresholds: { http_req_duration: ['p(99)<400'] },
};

export default function () {
  check(http.get('https://staging.example.com/api/items'), {
    'status 200': (r) => r.status === 200,
  });
}

Kernel/firmware regression. This is the canonical bisect use case — Linux kernel development has used it for two decades. Predicate: make -j$(nproc) && ./run-qemu-test.sh. The killer is compile time. A full kernel build plus QEMU boot is 30+ minutes per iteration; even at 8 iterations that’s half a workday. It is still worth doing, because the alternative is days of manual patching.

The “works on my machine” case. A commit that passes every local test but fails in production because of config drift — an env var set only in prod, a feature flag flipped in the dashboard, a reverse-proxy rule that only exists behind the load balancer. If the predicate does not reproduce the production environment, bisect will happily converge on the wrong answer with total confidence. The mitigation is boring