Scientific American: fake AI case law is testing lawyers’ duty to verify

When a lawyer puts a nonexistent case into a filing, it is no longer a cute error from the age of new tools. It is a failure in the chain of responsibility. Scientific American points to a continuing trend of attorneys being caught citing cases invented by artificial intelligence. The uncomfortable part is how simple the issue is: models can produce text that looks legally convincing, but that does not make it true.

In legal work, form carries weight. A case name, citation, year, court and procedural trail are not decoration. They are pointers to verifiable authority. If a model generates a case that does not exist, and a lawyer files it without checking, the court receives fiction with a professional signature. That is not a technical edge case. It is a clear example of automation amplifying an old weakness: the temptation to skip the boring, manual and essential part of the job.

The problem is sharper because AI does not fail like a conventional search engine. It often does not say “no result.” It writes a sentence that sounds like a result. Law is especially vulnerable to that behavior because legal language already has dense structure, rigid syntax and an authoritative tone. A generative system can imitate that surface well enough to mislead a user who is looking for speed rather than proof. So the real question is not whether AI tools are “good” or “bad.” The question is who performs final verification.

Professional rules leave little room for excuses. The American Bar Association’s Model Rules of Professional Conduct emphasize duties of competence, diligence and candor toward tribunals. In federal civil litigation, Rule 11 requires filings to have legal and factual grounding. An AI system can be an assistant, but it cannot hold the license, absorb the sanction or supply professional judgment.

The weakest point, then, is not only model hallucination. It is process hallucination: the belief that text that looks orderly can be treated as verified. That matters in firms and legal departments operating under deadline, cost and productivity pressure. If AI is used for drafts, summaries or research leads, there must be a bright line between “working material” and “ready for court.” Without that line, legal practice gets a machine for producing confident errors.

There is also a broader civic layer. Courts depend on trust that parties and their representatives are not inserting invented authorities into the record. When that happens, the damage is not limited to one lawyer’s reputation. Each incident wastes judicial time, complicates proceedings and signals that basic verification can be delegated to a tool with no concept of accountability. Even official warnings about generative systems’ limits, such as OpenAI’s note on model knowledge limitations, mean little if practitioners ignore them in daily work.

The conclusion is blunt but necessary: AI in law cannot replace citation hygiene. If a lawyer cannot open the case, confirm the citation and explain why it belongs in the argument, the case does not belong in the filing. The legal profession can adopt new tools, but it cannot outsource its duty to verify.