A CI Gate That Quizzes You on Your Own Code

The Problem

Code review catches bugs. It does not catch the author who doesn’t understand their own diff. You’ve seen it: a PR lands, something breaks two weeks later, and the person who wrote it can’t explain why the change was there in the first place. The code worked, the tests passed, nobody actually understood it.

I wanted a gate that checks comprehension, not correctness. Before you merge into main, you answer a few questions about your own diff. Get them right and you merge. Hand-wave and you don’t.

So I built a Comprehension Gate: a required CI check that quizzes you on your PR before letting it through.

The Shape

The whole thing runs on GitHub Actions and Claude. Two phases:

Generate. When you open a PR against main, the workflow computes the diff, hands it to Claude, and asks for up to three questions that probe load-bearing understanding — why the change exists, what breaks if you revert it, where the data flows. Claude posts those questions as a PR comment and sets the comprehension-gate status to failure. The gate is red until you answer.

Grade. You reply with a comment starting with /answer. That fires the workflow again. Claude reads your answers against the questions and the diff, and decides whether each one shows real understanding or just restates the diff. Every question has to pass. If they do, the status flips to success and you can merge. If not, you get hints — not answers — and unlimited retries.

That’s it. Open PR, get quizzed, answer, merge.

One Workflow, Many Repos

We run this across six repos. I did not want six copies of the logic drifting apart.

So the real workflow lives once as a reusable workflow in a shared repo. Every other repo gets a ~20-line caller that just points at it:

jobs:
  gate:
    uses: my-org/shared-workflows/.github/workflows/comprehension-gate.yml@main
    secrets: inherit

Change the gate once, every repo picks it up. The callers exist only to forward secrets and declare the permissions GitHub requires.

I also leaned on Claude Code’s --json-schema flag instead of writing a parser. The action returns structured output matching a schema I hand it, so the questions come back as {questions: [{id, prompt}]} and the grade comes back as {pass, results, summary}. No regex, no scraping a chat transcript. The model fills a contract.

Fail Closed

The default state of the gate is failure. It only goes green when Claude explicitly passes you. If the API call errors, if the model returns garbage, if anything goes sideways, the status stays red and the gate sets failure with a note to re-run or escape-hatch out. You can never accidentally merge because the check silently didn’t run.

The escape hatches are deliberate:

• Bots skip automatically. Image-bump PRs from a PAT don’t need to understand anything.
• Docs and lockfiles skip. A diff that’s all *.md and bun.lock isn’t worth a quiz.
• A skip-comprehension label auto-passes anything, for the times you just need to ship.
• Admins bypass branch protection entirely.

That last one I toggled on and off a few times. Disabling admin bypass during setup was a mistake — it created a chicken-and-egg where I couldn’t push the fix that made the gate work because the gate was blocking it. Re-enabling admin bypass (enforce_admins: false on the classic-protected repos, a bypass_actors entry on the frontend ruleset) gave me the escape valve to land fixes directly while everyone else still goes through the gate.

Takeaway

The gate doesn’t measure whether your code is good. Plenty of other checks do that. It measures whether you can explain your own change before it becomes everyone’s problem. The cost is three questions and a comment. The payoff is that the person merging into main has, at minimum, articulated why their diff exists.

Most of the design effort went into the unglamorous guarantees: fail closed, one source of truth, the right escape hatches. The quiz is the fun part. The plumbing is what makes it trustworthy.