LegalForecastBench

This benchmark gives models motion to dismiss briefs from real federal cases and asks the model to predict how the judge will resolve each issue in the motion.

Why this benchmark matters

Testing models’ ability to predict motion outcomes is valuable because it is an objectively verifiable, high-level judgment task. In real-world litigation, predicting how a judge is likely to react to a contemplated litigation strategy or a potential motion is usually performed by the most senior and highly paid lawyers on a litigation team.

Unlike many legal work-product tasks, moreover, the outcome is an externally observable judicial disposition. I operationalize that disposition as a binary outcome: whether the motion’s first written disposition fully dismisses a given claim against a given defendant. This approach could be more reliable and objective than rubric-based approaches like Harvey’s Legal Agent Benchmark (LAB).

This benchmark is also contamination resistant by design. It evaluates models’ ability to predict judicial decisions made after their documented training-data cutoffs, reducing the risk that the models have simply memorized the decision in pre-training. I’ve built a pipeline that ingests new cases, so that as models’ training cutoffs advance, I can create a new version that uses the most recent decisions at that point in time.

One open question is whether prediction is a good proxy for the models’ pure legal reasoning ability. In principle, a judge’s leanings—pro-plaintiff or pro-defendant—could influence the outcome in some cases, so testing prediction with the judge’s identity present (as I do) in part tests the model’s ability to identify the leanings of the particular judge who decided that case.

But my hypothesis is that this will not predominate over legal reasoning on this kind of benchmark. Most garden-variety litigation matters are straightforward enough that the outcome likely does not turn on the judge’s judicial philosophy, so the most effective way to predict the outcome of most cases will simply be to be better at analyzing and understanding the facts and applicable law, based on the same written submissions a real judge would use to decide the case.

Moreover, even to the extent that the benchmark tests prediction (rather than indirectly testing legal reasoning ability), the fact is that predicting how a real judge might react to a real fact pattern is far more valuable in real-world use than the mere ability to reason in the abstract about the academically “correct” legal answer. The experience and judgment to understand what factors impact real-world outcomes is much more economically useful than abstract academic legal reasoning.