AI Tutors Meet the Socratic Method

Alejandro Salinas, Carly Frieders, Neel Guha, Sibo Ma, Julian Nyarko, and coauthors studied a hard problem in AI education: domains where there is no single answer key. Law is an ideal test because legal education often asks students to weigh ambiguity, argue both sides, and reach a defensible conclusion.

The study used contracts tutoring questions. Sixteen U.S. contracts professors from fourteen law schools authored representative questions, wrote answers, and then judged anonymized answer pairs. The paper reports 2,918 blinded comparisons and a strong expert preference for LLM responses in that setting.

The paper reports that LLMs were preferred far more often than peer instructor answers in the studied comparisons, with an average LLM win rate around 75 percent. It also reports lower harmfulness flags for LLM answers than for professor answers in aggregate.

That does not mean law professors are obsolete. It means that a well-designed AI tutor can sometimes express the shared professional standard with unusual consistency, clarity, and availability. The educational revolution is not replacing the professor. It is making expert feedback abundant enough that students can practice more often.

A math drill can often be checked against one correct answer. A legal hypothetical usually cannot. Two answers can reach different conclusions and both be strong if each states the right rule, uses the right facts, addresses counterarguments, and reasons honestly.

That is why the study matters for bar preparation. The California Bar Exam grading standard also values analysis, material facts, law application, and logical lawyer-like reasoning. The bar is already a judgment-rich assessment.

The study gives The VR School of Law a design brief: build tutors that are evaluated against expert preferences, but also constrained by official sources, student agency, and transparent rubrics.

SofAI should ask for the student's draft before giving a model. It should flag missing issues, incomplete rules, weak fact use, and dangerous overconfidence. It should cite official sources when exam rules, grading, or legislation matter. It should force rewrites until the student can explain every move.

The study does not prove that AI should give legal advice. It does not prove that an AI tutor can replace a full law school class, clinical supervision, professional responsibility training, or bar admission judgment. It studied short tutoring answers in contracts under a controlled evaluation design.

That limitation is not a weakness. It is the reason to build carefully. The correct product is not an answer machine. It is a supervised practice environment where source-grounded AI makes high-quality feedback more frequent, and human students learn to become better lawyers.