A new study making the rounds claims GPT-5 outperforms federal judges at legal reasoning. The paper tested the AI against the same hypothetical scenarios given to 31 U.S. federal judges in a prior experiment, and GPT-5 made fewer "errors"—meaning it deviated less from established legal doctrine.

Headlines practically write themselves: AI Better Than Human Judges! Time to replace the bench with servers!

Not so fast.

The Consistency Problem

Here's what the study actually measured: consistency with the letter of the law. GPT-5 gave uniform answers across scenarios. The judges? They gave different answers depending on context, circumstances, and the specific facts presented.

The researchers define an "error" as any departure from doctrine. But as the paper itself acknowledges, "such departures may not always reflect true lawlessness. In particular, when the applicable doctrine is a standard, judges may be exercising the discretion the standard affords."

Translation: judges are supposed to deviate from rigid rules when circumstances warrant. That's literally their job.

The Judgment in Judgment

Consider a scenario that surfaced in the Hacker News discussion: a teenager takes explicit photos of themselves and sends them to their partner. Under the strict letter of child pornography laws, that teenager is simultaneously the perpetrator and victim of their own crime. They could face felony charges and sex offender registration.

Judges routinely throw out these cases. Not because they're lawless, but because applying the statute mechanically would create an absurd result—criminalizing a teenager for victimizing themselves.

GPT-5 would presumably apply the law as written. Consistent? Yes. Just? Absolutely not.

The Wealth Gap in Justice

There's another uncomfortable truth buried in this debate. When wealthy defendants get different outcomes than poor ones, we typically blame corruption. But often what money buys is better lawyering—attorneys who can present the nuances that trigger judicial discretion.

The public defender with 200 active cases doesn't have time to craft the narrative that helps a judge see why the letter of the law shouldn't apply here. The $800/hour white-shoe lawyer does.

If we replace judges with AI that mechanically applies doctrine, we don't eliminate this disparity. We eliminate the escape valve that lets justice occasionally triumph over law.

The Legitimacy Question

Legal AI advocates often pitch consistency as inherently desirable. But consistency with what, exactly?

Laws are written by humans, which means they're written with agendas. When legislators can't pass their preferred policy directly, they sometimes achieve it indirectly through overbroad statutes that technically cover unintended behavior. Judges applying discretion serve as a check on this—they can refuse to extend a statute beyond its obvious purpose.

An AI trained on legal doctrine and case law will inherit whatever biases exist in that corpus. But unlike a human judge, it can't step back and ask whether applying this rule to this case makes sense in the broader context of justice.

What This Means for Founders

If you're building legal tech, this study is a useful data point but not a green light. The places where AI excels in legal reasoning—document review, research, first-pass analysis—are precisely the tasks where consistency matters and discretion doesn't.

The moment you move toward decision-making, you enter territory where "fewer errors" might actually mean "worse outcomes." A legal AI that never deviates from doctrine might be worse than one that occasionally makes mistakes in the name of equity.

There's also a practical consideration: courts and regulators are watching. The legal profession is notoriously slow to adopt new technology, but it's lightning-fast at regulating perceived threats to its authority. Any legal AI product that positions itself as a replacement for human judgment—rather than a support tool—is inviting regulatory attention.

The Real Takeaway

GPT-5 can reason about law with impressive consistency. But law isn't just reasoning—it's judging. It's looking at a human being and deciding whether the rule written on paper should apply to their specific situation.

Sometimes the answer is no. Sometimes a teenager shouldn't be prosecuted for taking photos of themselves. Sometimes a first-time offender deserves mercy. Sometimes the letter of the law would create injustice.

These decisions require something AI fundamentally lacks: the ability to say "this is technically correct but morally wrong."

That's not a bug in human judges. It's the feature.

Until AI can reliably tell us when to break its own rules, the bench is safe. And honestly? That's probably how it should be.