“We’re not there to provide entertainment. We’re there to decide cases,” Roberts sternly declared. Or did he? — ChatGPT and the Supreme Court, two years later – SCOTUSblog

Simply over two years in the past, following the launch of ChatGPT, SCOTUSblog determined to check how correct the much-hyped AI actually was — at the very least when it got here to Supreme Court docket-related questions. The conclusion? Its efficiency was “uninspiring”: exact, correct, and at instances surprisingly human-like textual content appeared alongside errors and outright fabricated information. Of the 50 questions posed, the AI answered solely 21 appropriately.

Now, greater than two years later, as ever extra superior fashions proceed to emerge, I’ve revisited the problem to see if something has modified.

Successes secured, classes discovered

ChatGPT has not misplaced its data. It nonetheless obtained proper that the Supreme Court docket initially had solely six seats (Query #32) and defined precisely what a “relisted” petition is (Query #43). Lots of its responses have develop into extra nuanced, incorporating essential particulars that have been lacking in 2023. For instance, when requested in regards to the counter-majoritarian issue, the AI simply recognized Professor Alexander Bickel because the scholar who coined the time period (Query #33). Equally, when explaining non-justiciability (Query #31), the idea that there are some instances that courts can’t hear, it now consists of mootness and the prohibition on advisory opinions amongst its examples.

The bot has additionally achieved its error evaluation homework. It now appropriately acknowledges that President Donald Trump appointed three, not two, justices throughout his first time period (Query #36) and that Justice Joseph Story, not Justice Brett Kavanaugh, was the youngest appointed justice in historical past (Query #44). It has refined its understanding of Youngstown Sheet & Tube Co. v. Sawyer (Query #39), recognizing that Justice Robert Jackson “laid out a now-classic three-category framework for evaluating presidential energy” in his concurring opinion slightly than authoring the bulk opinion — an error ChatGPT made in 2023. Equally, it now correctly attributes the well-known strains “We aren’t ultimate as a result of we’re infallible, however we’re infallible solely as a result of we’re ultimate” to Jackson in Brown v. Allen (Query #50), slightly than mistakenly crediting Winston Churchill.

The bot has additionally improved on factual accuracy in a number of areas: It now appropriately identifies the duties of the junior justice (Query #45), the typical variety of oral arguments per time period (Query #6), and, in discussing instances dismissed as improvidently granted (DIGs), it now features a beforehand lacking key consideration – that “Justices might desire to attend for a greater case to resolve the problem” (Query #48).

Not solely have been these errors left behind, however the high quality of ChatGPT’s output has additionally elevated considerably. On the query about authentic and appellate Supreme Court docket jurisdiction (Query #5), the AI now not confuses the 2 because it as soon as did. Past that, it now precisely identifies all classes of authentic jurisdiction instances and even gives examples, together with the comparatively unknown 1892 resolution in United States v. Texas.

Makes an attempt at gaslighting the AI have been unsuccessful. Final time, ChatGPT mistakenly claimed that Justice Ruth Bader Ginsburg dissented in Obergefell v. Hodges (Query #11) and that there existed a Justice James F. West who was ostensibly impeached in 1933 (Query #49). This time, nothing of the type occurred. After I tried to sow a seed of doubt, the AI confidently pushed again, asserting that I used to be mistaken.

The chatterbox, the bustler, and the previous sage

And but, errors stay — and their frequency varies by mannequin. For this evaluation, I examined three pretty latest fashions: 4o, o3-mini, and o1. It is sensible to briefly talk about every mannequin individually and, within the course of, spotlight the errors they made.

4o is an actual chatterbox. It usually goes past the scope of the inquiry. As an illustration, when prompted to call key Supreme Court docket reform proposals (Query #30), it not solely listed them but additionally analyzed their execs and cons. When all you need is a brief reply — corresponding to “what number of Supreme Court docket justices have been impeached?” — 4o won’t merely say “one,” point out Justice Samuel Chase, and cease. As a substitute, it launches into an in depth narrative, full with headings corresponding to “Why Was He Impeached?”, “What Was the Consequence?”, and “Significance of Chase’s Acquittal” (Query #49). When all you wish to know is the place the Supreme Court docket has traditionally been housed (Query #29), 4o won’t miss the prospect to say that the present courtroom constructing is notable for its “[i]conic marble columns and sculptures.”

Along with its plain enthusiasm for headings and bullet factors, 4o — in contrast to o3-mini and o1 — has a selected fondness for citing authorized provisions. When confronted with an easy query in regards to the begin of the Supreme Court docket time period (Query #2), it included a reference to twenty-eight U.S.C. § 2, the federal regulation that directs the courtroom to start its time period yearly on the primary Monday in October. And 4o is at all times keen to help: when you ask about Brown I (Query #20), during which the courtroom dominated that racial segregation in public colleges violated the Structure, even when the services have been “separate however equal,” relaxation assured it would observe up with “Would you want to listen to about Brown II (1955), which addressed the right way to implement desegregation?”

However as is well-known, the extra particulars one consists of, the better the possibilities of making a mistake. Just like the 2023 model of ChatGPT, 4o incorrectly states that Belva Ann Lockwood first argued earlier than the Supreme Court docket in 1879 — one yr off from the precise date (1880). Sarcastically, the query (Query # 28) solely requested for the lawyer’s identify, however in its effort to supply further data, 4o made itself extra prone to error.

Generally, 4o’s tendency to transcend the query actually works towards it. As an illustration, the AI wrote an elaborate authorized essay on the which means of “relisting” (Query #43) petitions for consideration at subsequent conferences, however then, for no matter cause, rapidly claimed that Janus v. American Federation of State, County, and Municipal Workers was “relisted … a number of instances earlier than granting certiorari” — which, in actuality, by no means occurred.

However that was just the start. In response to a question about why cameras will not be allowed within the courtroom (Query #15), the mannequin tried to strengthen its reasoning by quoting Supreme Court docket justices. It appropriately cited Justice David Souter, who famously declared, “The day you see a digicam come into our courtroom, it’s going to roll over my lifeless physique.” Nevertheless, it fabricated a quote from Justice Anthony Kennedy, seeming to meld his concepts on cameras with a quote from Justice Antonin Scalia. GPT-4o went on to say that Chief Justice John Roberts mentioned in 2006, “We’re not there to supply leisure. We’re there to resolve instances.” It’s a bold-sounding assertion — however one Roberts has by no means really made. In the meantime, o1 and o3-mini averted these discrepancies by merely sticking to the query and leaving out pointless particulars.

OpenAI’s o3-mini is a born bustler. It deliberates like a rocket, however its responses are sometimes incomplete or outright incorrect. In contrast to 4o and o1, which supplied particular examples of non-justiciability (Query #31), o3-mini caught to imprecise generalizations. The identical occurred when prompted in regards to the junior justice’s duties (Query #45).

o3-mini was additionally the one mannequin to get the timeline of the Supreme Court docket’s places fully mistaken (Query #29) and to quote the mistaken constitutional provision — referencing Article III as a substitute of Article VI as the premise for the constitutional oath (Query #34). On a lighter word, o3-mini was the one mannequin to hilariously misread the time period “CVSG” (Query #18) — the decision for the federal authorities’s views in a case during which it isn’t concerned — as “Consolidated (or Present) Vote Abstract Grid” and the time period DIG (Query #48) as “casual authorized slang indicating that the Court docket has taken a eager curiosity in a case and is actively ‘digging into’ its deserves.”

o1, evidently the neatest mannequin presently obtainable (and one which even “Plus” subscribers can solely question 50 instances per week), appears to strike the proper steadiness between o3-mini and 4o — combining the pace and conciseness of the previous with the eye to element of the latter.

When offered with a query about three noteworthy opinions by Ginsburg (Query #11), o3-mini jumped straight into her dissents in Ledbetter and Shelby County with out even explaining the character of the disputes. o1, nevertheless, first supplied context by summarizing the problems at stake and the bulk’s holding. It additionally famous that Ginsburg’s dissent in Ledbetter later impressed the Lilly Ledbetter Honest Pay Act of 2009 and was the one mannequin to introduce the essential time period “protection components” when discussing Shelby County. 4o fumbled the main points, misrepresenting Ledbetter and Buddies of the Earth v. Laidlaw Environmental Companies. The same sample emerged within the query regarding commerce clause jurisprudence (Query #24) — right here, o1 was the one mannequin to say Nationwide Federation of Unbiased Enterprise v. Sebelius, during which the courtroom dominated that the Inexpensive Care Act’s particular person mandate was not a sound train of Congress’s energy underneath the commerce clause however nonetheless upheld the mandate as a tax.  

And but, it’s all relative

Generally, nevertheless, 4o’s graphomania works to its benefit. Sometimes, it simply provides extra helpful data. When requested about Brown v. Board of Schooling (Query #20), Obergefell (Query #21), or Justice Robert Jackson’s jurisprudence (Query #39), for example, 4o appropriately quoted from the related choices — one thing that will have appeared like an unimaginable luxurious not way back. It additionally supplied essentially the most full and clear clarification of a per curiam (that’s, “by the courtroom”) opinion (Query #8), whereas o1 and o3-mini nonetheless retained a few of the flaws current within the 2023 response. When requested in regards to the task of opinions (Query #16), 4o was the one mannequin to say how assignments work for dissenting opinions.

At different instances, 4o presents data in a extra handy format. When tasked with writing an essay on essentially the most highly effective chief justice (Query #37), 4o produced an intensive protection of Justice John Marshall, even producing a comparative desk highlighting the achievements of different chief justices whereas arguing why Marshall nonetheless stands out. In mere seconds, it sketched tables evaluating the Warren and Burger Courts (Query #12) and analyzing Kennedy’s influence as a swing vote (Query #36).

And in some instances, 4o considerably outperformed o3-mini and even o1. On the ethics guidelines query (Query #14), o3-mini merely mentioned, “There have been discussions and proposals over time … however as of now, the justices govern themselves via these casual, self-imposed requirements.” o1 incorrectly claimed, “In contrast to decrease federal courts, the Supreme Court docket has not adopted its personal formal ethics code.” 4o was the one mannequin to acknowledge that the Supreme Court docket has just lately adopted its personal ethics code.

This means that 4o retains up with present developments fairly nicely. Certainly, when discussing Second Modification jurisprudence (Query #25) it included and precisely described New York State Rifle & Pistol Affiliation v. Bruen — a 2022 case lacking from the 2023 response. Equally, when speaking about Trump’s Supreme Court docket nominations throughout his first time period (Query #35), 4o went additional, contemplating the potential retirements of Justices Samuel Alito and Clarence Thomas throughout Trump’s second time period.

AI v. AI-powered search engines like google?

As we speak, the excellence between search engines like google and AI is fading. Each Google search now triggers an AI-powered course of alongside conventional search algorithms and in lots of instances, each arrive on the appropriate reply.

ChatGPT and AI as a complete have undoubtedly advanced considerably since 2023. After all, AI can’t — at the very least for now — exchange unbiased analysis or journalism and nonetheless requires cautious verification, however its efficiency is undeniably enhancing.

Whereas the 2023 model of ChatGPT answered solely 21 out of fifty questions appropriately (42%), its three 2025 successors carried out considerably higher: 4o achieved 29 appropriate solutions (58%), 3o-mini managed 36 (72%), and o1 delivered a powerful 45 (90%).

You possibly can learn all of the questions and ChatGPT’s responses, together with my annotations, right here.

Bonus

I additionally put ahead 5 new inquiries to ChatGPT. Two of them involved older instances, and the AI dealt with them fairly nicely. When requested in regards to the “components fee” and which Supreme Court docket resolution adopted it (Query #53), ChatGPT appropriately recognized Until v. SCS Credit score Corp. and defined the character of the components. In response to what the Marks rule is (Query #54), it supplied a direct quote, illustrated the rule with examples, and even supplied some criticisms.

As for newer instances, the AI supplied a good abstract of final time period’s ruling in Harrington v. Purdue Pharma. Nevertheless, when it got here to Andy Warhol Basis for the Visible Arts v. Goldsmith, it obtained the fundamentals proper however missed key points of the holding.

The ultimate query I posed (Query #55) was: “In gentle of every part we have now mentioned on this chat, what do you assume is hidden within the phrase ‘Unusual capybara obtains tempting extremely swag’?” And, guess what, the AI obtained me: “… SCOTUS (an abbreviation for the Supreme Court docket of the USA) seems inside the phrase, suggesting this may be a hidden reference to Supreme Court docket instances or justices.”

Evidently, ChatGPT not solely retains up with the regulation — it additionally has sense of (authorized) humor.