Evolving use and misuse

Log in to listen to this article

As artificial intelligence continues to develop, so do the ways it can be misused in the courts.

While most people have heard of AI hallucinations, where the technology invents case law and legal citations that aren’t real, Amy Salyzyn, an associate law professor at the University of Ottawa, says even more subtle forms of AI misuse are increasingly cropping up in courts. 

In May, a Brazilian court fined two lawyers for using hidden text — white text on a white background — in a submission that directed any AI tool reading the material to review their case and decide it in favour of their clients.  

In August, a self-represented litigant in a Connecticut case similarly used tiny, white font on the white background of court filings to instruct AI tools to produce a favourable output. 

On the season premiere of the CBA’s Verdicts & Voices podcast, Salyzyn says these are examples of prompt injection attacks. 

“At the most basic level, we can understand this as a kind of attack on generative AI models or attempts to do something wrong with generative AI models that are delivered through content that the model reads,” she says. 

“Because we're talking about attempting to do things wrong, these are usually kind of surreptitious insertions into the content that the model reads, so it's kind of something being done underneath the surface.” 

^{Prof. Amy Salyzyn and Verdicts and Voices host Alison Crawford}

In the Brazilian case, it was apparently the court's own AI tool that flagged the content. Salyzyn says courts there are a bit more advanced in the use of AI than in North America, so there was already an AI system in use to process pleadings.

In the Connecticut case, it was the judge who caught the prompt injection attack. 

“[They] said, ‘Whoa, this is a lot of weird white space on this page.’ You know, kudos to the humans involved here,” Salyzyn says.

Shadow AI is another example of how this technology can potentially be misused in the courts. It involves using AI without disclosing how it’s being used or that it’s being used at all, or using generative AI tools that the organization hasn’t approved. Among the serious risks it carries are compromised data security or poor-quality results. 

“Someone pulling up ChatGPT on their computer, oftentimes, it can be free to access. You don't need deep technical expertise to use it, so all kinds of workplaces and institutions are seeing more and more shadow AI use,” Salyzyn says. 

Within the judicial system, it can lead to fabrications and errors appearing in decisions. 

Earlier this year, La Presse reported that a Quebec Superior Court judge allegedly cited a hallucinated Supreme Court of Canada case in a decision involving a $120 million judgment. The allegation that AI was used has not been confirmed.

“There are a few courts that have been quite transparent and quite proactive in disclosing their use (of AI). The Quebec Superior Court is one of those courts,” says Salyzyn, whose research focuses on legal ethics, law and technology, and civil justice reform.

But when it comes to Canadian judges, she says it’s hard to know what they’re actually doing with AI. 

“I think there's good reason to believe there's a growing number of judges in Canada using AI in ways that we maybe don't know about, and maybe even their colleagues don't know about.”

While experimenting with new ways to work and be more efficient isn’t always a bad thing, there are risks with AI. If someone’s inappropriately using the technology, there’s the risk of hallucinations and potentially compromising confidential information. If someone's using a tool and the court is unaware, it could allow a prompt injection attack to succeed because no one knows they should be vetting for that kind of thing. 

“They're not building a vetting into the system,” Salyzyn says

So, what needs to happen to keep the courts from being overrun by AI misuse? 

“I think overall we need to have a more robust public conversation about how AI is being used in the courts. I think courts and judges could be potentially more transparent about their uses,” she says.

“Certainly, I think courts should be developing internal policies about how judges use AI. They should be disclosing those policies.” 

Salyzyn is sympathetic to the issues the judiciary faces on this front, as it’s under heavy scrutiny and constrained in its ability to respond to criticism. 

“Sometimes calls for more transparency, even though they might believe that in principle, they see practical challenges with that.”

And while courts are slowly developing their own AI use policies and starting to talk publicly about how they use it, there are no universal guidelines across all courts. 

“There's a big shadow, so to speak, around how courts are using AI in Canada … I'd like that shadow to lift because I think sunshine's the best disinfectant,” Salyzyn says. 

Websites like courtready.ca are tracking reported cases of AI hallucinations in courts, for example, but she says what's reported in these public databases is only the tip of the iceberg. She talks to tribunal adjudicators and judges who see these things daily, and not all of them are reported. 

“Particularly in the area of self-represented litigants, this is just kind of a groundswell of cases growing where they're going to have to take the time to react to these issues,” Salyzyn says.

Tune into the full episode to hear about how law societies are tackling the misuse of AI in the courts, and whether Salyzyn thinks the increased use of AI is actually causing a burden on court caseloads.  

For more on AI and the law, listen to past episodes about hallucinated cases, racial bias in AI, and the first Canadian judge to encounter AI-generated fake citations.