Kudos to the humans

Log in to listen to this article

As artificial intelligence keeps evolving, so do the ways of using and misusing it in the courts. To kick off our new season, University of Ottawa expert Amy Salyzyn joins the show to explain everything from prompt injection attacks to shadow AI, and how the human beings who work in Canada’s justice system are handling the challenge.

For more on AI and the law, listen to past episodes about hallucinated cases, racial bias in AI, and the first Canadian judge to encounter AI-generated fake citations.

Also, check out AI in Practice, a monthly CBA webinar series that examines AI in the Canadian legal context.

Verdicts & Voices is a legal current affairs podcast presented by the Canadian Bar Association. With her retinue of expert guests, host Alison Crawford keeps listeners up to date on news, views, and stories about the law and the justice system in Canada.

Views expressed are not necessarily those of the CBA.

Transcript

Welcome back to a new season of Verdicts & Voices. This is the place you need to be if you're interested in legal news and views. Verdicts & Voices is a weekly podcast, and it's brought to you by the Canadian Bar Association. And I'm your host, Alison Crawford.

There is a lot to catch up on since our season finale in June, and we're planning to tackle the issues of cash bail in Ontario, how lawyers are dealing with the new tort of intimate partner violence, along with a tee-up of the Supreme Court of Canada's fall session in its new building.

Our focus today is artificial intelligence and how people's use of AI continues to evolve and affect the administration of justice. I probably should have said there how people use and misuse AI. Joining me in the studio here to share her thoughts and expertise on this topic is Amy Salyzyn. She's one of Canada's leading experts on AI in the justice system, and she's an associate law prof at the University of Ottawa. And that's where she holds the Dean's Research Professorship in Technology and Justice Futures. Welcome back to the podcast, Amy.

Amy Salyzyn

Thanks so much for having me. It sounds like you have great season on tap. Excited to kick it off with you.

Alison

I'm happy that you're here to kick it off too. So, you were last on the show about eighteen months ago. I think I was just learning about generative AI at that point, so it was all new to me. But since then, of course, I've learned a few new terms, and there have been a few new instances of AI in our courts. And one of them I learned about this July is “prompt injection attack.” So, is that something you could explain to me and tell us how it's turning up in the courts?

Amy Salyzyn

Yeah, sure. I guess, I mean, eighteen months in AI world is kind of eighteen years or maybe eighteen hundred years, I'm not sure. So, you're right, a lot's happened. And you asked me about one term, prompt injection attack. That's a term that can mean a lot of different things. We could potentially describe it at a very technical level. For our audience here, I'm not gonna do that. I'm gonna try and keep it simple.

So, kind of at the most basic level, we can understand this as kind of attacks on generative AI models or attempts to do something wrong with generative AI models that are delivered through content that the model reads. And because we're talking about attempting to do things wrong, these are usually kind of surreptitious insertions into the content that the model reads. So, it's kind of something being done underneath the surface.

That's a simple explanation. I think if I give you maybe some real-life examples of what's happened with the courts.

Alison

Yeah. Because this is obviously something that can't happen by accident.

Amy Salyzyn

Yeah, exactly, exactly. And so, for a while, when I was talking to judges or lawyers, I'd mentioned this thing of prompt injection attacks. It was kind of a hypothetical thing that, you know, might go wrong, could go wrong. And what we've actually seen in the last several months is two real instances of what seem to be prompt injection attacks on courts.

The first one that was we were made aware of was in May 2026. It was reported that a Brazilian court fined two lawyers for using hidden text. So essentially using kind of white fonts on a white background, nothing too overly complex.

Alison

Well, what did it say?

Amy Salyzyn

Yeah, and so essentially, they included that in the submissions and directed any AI tool that was going to be reading the material to basically look at the case superficially and try to decide the case in the favor of their clients. And this was caught by the court. Apparently, it was the court's own AI tool that flagged the content.

Alison

So, the courts are using AI already to detect…? Like, how did --

Amy Salyzyn

Yeah, exactly. So essentially, you know, as a general matter, courts in Brazil are a bit more advanced in their AI use than courts in North America. And so, there was already an AI system being used to process pleadings, look at pleadings, things like that. And potentially the lawyers that were involved were aware of that. And so they had this white text again on a white background that contained these, kind of, hidden instructions to try and game the AI system to decide the case in their favor. The court did catch that, very sternly imposed a fine. And there's also, as far as I understand, ongoing disciplinary proceedings.

So that was one case. We weren't sure how much we'd see this happen again. In August, this happened in an American court actually. And it was reported in August that in a Connecticut case there was a self-represented litigant this time. And again, in his court filings, there was information contained in a white text, apparently a very tiny font size, so maybe two-point font. And it instructed quite similarly the AI tool to, among other things, produce a favorable output.

Alison

Yeah. And I read about this one, and I think here, it wasn't AI detected it. It was the judge, I think, who said, like, whoa, this is a lot of weird white space on this page.

Amy Salyzyn

Yeah. I mean, you know, kudos to the humans involved here. In this case, it wouldn't have been that the court was using AI itself. And so even if it was, even if the human hadn't caught this, it wouldn't have been fed into an AI system. But still clearly the litigant was thinking maybe an AI system is used. I'm going to try and potentially get it to decide the case in my favour.

I should mention, I as far as I know, in both cases, the people involved have denied any wrongdoing. They've kind of had alternative explanations.

Alison

Like what?

Amy Salyzyn

From what I understand from the reporting in the case in Brazil, the indication from the lawyers was they weren't trying to manipulate the court but prevent maybe opposing parties from misusing AI. It doesn't seem that that explanation was accepted. My understanding is the self-rep was saying he was simply trying to, you know, audit or assess if AI tools were being used. In both cases it seemed like these alternative explanations weren't really accepted by the decision makers involved.

Alison

Yeah. Now earlier this year, like the last time we talked, we were talking about lawyers who had used it and who had been caught in the first two instances. And of course, there have been hundreds since then. But earlier this year, I was totally blown away when I read about a decision where a Quebec Superior Court judge cited a hallucinated case at the Supreme Court of Canada, Crawford v. Crawford, not involving me at all. But have you… Is there any more information about what's happened since then?

Amy Salyzyn

Yeah. I mean, when I read that case like you, I was pretty troubled. From what's reported, it looks like there was kind of multiple errors in the decision. So, this wasn't one hallucinated case. And it seemed like the types of errors were the type we normally see with AI-generated material. We don't exactly know what's happened yet. The case is under appeal. As far as I know, that appeal is still outstanding. It's a pretty complex underlying case involving lots of different kinds of tangential proceedings involving a lot of money. And so, we may see that appeal come forth. There's some interim motions being made. And last I checked with lawyers in Quebec, they haven't heard anything in terms of the case being got to appeal yet.

Alison

Yeah, and do we know whether any complaints were made to the Canadian Judicial Council?

Amy Salyzyn

I haven't heard anything, you know, official on that. We wouldn't necessarily know a complaint's made until it proceeds, of course. And it's likely, you know, if the Canadian Judicial Council was involved, they would probably want to wait until the underlying proceeding kind of made its way through the courts so there's more information. So, you know, everybody's keeping their eyes on that one. I think I think we should be.

Alison

Yeah, absolutely. I guess that would be an example of what they call that shadow AI, because you've written about that recently for Slaw, talking about this as being a growing problem because we don't know, there's no way of knowing how courts are using AI in what they do every day. What do we know about judges and how they're using generative AI?

Amy Salyzyn

Yeah, yeah. So, maybe for the listeners, that term “shadow AI” is just a way to kind of encapsulate anybody who's using AI that's not officially approved by their workplace or institution. And of course, that's quite easy to do in some cases because you know, someone pulling up ChatGPT on their computer, oftentimes it can be free to access. You don't need deep technical expertise to use it. And so, kind of, all kinds of workplaces and institutions are seeing more and more shadow AI use occurring.

In terms of what Canadian judges are actually doing with AI, like you said, it's a bit hard to know. There's a few courts that have been, you know, quite transparent and quite proactive in disclosing their use. The Quebec Superior Court's one of those courts, in fact. They've had a pilot project that they've reported on. But I think there's good reason to believe there's kind of a growing number of judges in Canada, you know, using AI in ways that we maybe don't know about, and maybe even their colleagues don't know about.

You know, it may be because they want to, you know, experiment with ways to become more efficient. It's not necessarily a bad thing for people to try out new ways to work. But with this kind of use, there can be risks. If the individual is using AI in a more inappropriate way, you have those risks of hallucination, maybe risks with confidential information. Maybe, you know, if someone's using an AI tool and the court doesn't know, there could be risks of that kind of prompt injection attack being successful because no one knows that they should be vetting for that kind of thing, they're not building a vetting into the system.

Alison

So what do you think needs to happen?

Amy Salyzyn

I think overall we need to have a more robust public conversation about how AI is being used in the courts. I think courts and judges could be potentially more transparent about their uses. Obviously, you know, there's things like deliberative secrecy and there's questions about what level of detail. But you know, certainly I think courts should be developing internal policies about how judges use AI. They should be disclosing those policies. And we should be having a collective conversation about what uses seem appropriate in terms of administering justice.

Alison

Yeah. Is that something that they're, you know, is that happening? Is that starting to happen? I mean, you wrote an article and I'm wondering if you got any feedback on that.

Amy Salyzyn

Yeah, I mean, I'm sympathetic to the judiciary. I think they're under a lot of scrutiny and they have limited ability to respond to criticism sometimes. And so, I think, sometimes, calls for more transparency, even though they might believe that in principle, they see practical challenges with that. I think we're seeing kind of, you know, slowly, different courts coming out with their policies and different courts talking publicly about their uses, but we don't have anything that's kind of consistent and universal and so I still think there's a big shadow, so to speak around how courts are using AI in Canada. And I think I'd like that shout out to lift, because I think, you know, sunshine's the best disinfectant, so to speak. And I think we can have a productive public conversation about it. It's not to chastise anybody or to kind of discipline anybody, but it's just, let's talk about this collectively.

Alison

Yeah, so you know, as I mentioned off the top of this interview, the last time we talked on air about this, you know, there had been two lawyers who had cited AI in their documents and they were hallucinations. So, in July, the Law Society of Ontario fined and suspended the licence temporarily of one of those lawyers. So, what do you think of those consequences for her actions?

Amy Salyzyn

Yeah, yeah. So that lawyer was suspended six months and fined $10,000. You know, a pretty significant penalty to impose on a lawyer to have to leave your practice for that long, that's a lot of money. I think an important part of that particular case to point out is that it wasn't just AI use itself. There was also dishonesty involved, unfortunately. And it was dishonesty both to the court initially and also to the Law Society. And that's a really serious type of misbehaviour on the part of a lawyer. It's really important that they are forthcoming to courts, are forthcoming to law societies.

In terms of the appropriateness of the penalty, I know that was jointly proposed by the parties. We don't have a lot of, kind of, case law on this, but I think looking at it, it does seem proportionate in the sense that this was something very serious that was done. And you know, one considers the, you know, the initial dishonesty as well. It wasn't only a mistake. There were other things involved.

Alison

Yeah. Are other law societies sort of working in the same way in terms of consequences for lawyers who are caught?

Amy

There isn't much out there. And I think maybe one theme of today's discussion is about transparency. So, you know, one thing about law societies is that we only necessarily hear of how they interact with lawyers when something reaches a public hearing. And so a lot of law societies have other mechanisms in terms of private reprimands or having discussions with lawyers. We don't know how much of that's going on out there, kind of the things brewing beneath the surface. In terms of published decisions, I note just last week there was a New Brunswick lawyer who was sanctioned by the New Brunswick Law Society. I don't know if you saw that.

Alison

I didn't.

Amy Salyzyn

Yeah. Misused generative AI in a brief that he had submitted to a New Brunswick provincial court. And I think it involved, you know, multiple hallucinated cases as well. That lawyer did agree to plead guilty to the complaints against him and consented to a number of sanctions. He was suspended for two weeks, fined $10,000, paid some costs as well. Interestingly, he also has to do some continuing education. So, he must complete four and a half hours of CPD. And then also there is a requirement that he review all of his files that are actively before the courts and make sure this problem doesn't occur in those files. And so, I think law societies are trying to think creatively and robustly about what they do in these situations.

Alison

Well, that's interesting.

Amy Salyzyn

And I think, you know, by the nature of what we're seeing, we’ll probably see more of these cases build. There's lots of lawyers and lots of court cases, and at least in some of them, you know, some of them may be making this misstep of filing problematic materials.

Alison

And I've checked that courtready.ca website and I've noticed there lots of self-reps are the parties who have been caught with that as well. So, and that's, that's challenging.

Amy Salyzyn

Yeah, exactly. And I think, I mean, if you look at it, about… I don't have the right numbers exactly, but 75% of the cases or more are self-represented litigants with, you know, a smaller proportion being lawyers. I'd also note that what's reported in these public databases is the tip of the iceberg. Because I talk to tribunal adjudicators, I talk to judges, and some of them are seeing these things on a daily basis and not all of them get reported. And particularly in the area of self-represented litigants, there’s this kind of groundswell of cases growing where they're gonna have to take the time to react to these issues coming forward.

Alison

Yeah, I've heard about that in immigration.

Amy Salyzyn

Exactly.

Alison

Okay. So, you know, do we have any information at this point about the effect of AI on caseloads at the courts? Like, is this having an effect on how many cases they’re seeing?

Amy Salyzyn

I haven't seen any formal statistics published in Canada, but I've seen in other jurisdictions, particularly at some tribunals, a large proportion of cases increasing. I've heard from regulators as well, places like law societies but also other types of regulators, the caseloads are increasing. One thing also we're seeing in some contexts is the complexity and length of what people are submitting is increasing as well, particularly if there aren't page limits, particularly if it is self-represented litigants. So, you know, people who are adjudicating in those contexts have to spend the time to maybe go through, you know, 300 pages to see, you know, separate the wheat from the chaff to figure out what's real, what’s not real. And so it is creating a burden in some contexts. I think it is differentially impacted, but certainly in some contexts.

Alison

Oh my gosh, that sounds like a nightmare to me. Because of course there's been a lot of writing about how judicial decisions are becoming longer but it doesn't help when all the submissions are longer as well.

Amy Salyzyn

Yeah.

Alison

Now, recently, I've had the opportunity to speak with a number of family law lawyers, and one thing that they have raised as concerns that they're having is about AI and how it might impact the kind of work they're doing, and sort of, like, the potential use of AI to alter digital evidence such as photos and videos and transcripts. Is that something you know much about?

Amy Salyzyn

Yeah, maybe it's kind of that deep fake phenomenon. Some people call it that. And so, you know, now with the tools we have, it's increasingly easy to access powerful technology that can depict, you know, what seems to be real people, real events, real material, but it happens to be totally synthetic, totally generated. And there's a worry that that might start, you know, seeping into what we're seeing in the courts. As far as I know, I've seen a couple of cases in the United States where deep fake material that's been submitted by litigants has been detected. I haven't heard about it being, you know, a groundswell of a problem in Canada. But just because it's not happening now doesn't mean it can’t happen. I think it's a really challenging problem. What I am seeing in Canada, despite the fact that maybe people aren't using these tools, is increasing allegations that maybe the other side's deep-faking evidence. So, sometimes that can be kind of done for bad faith purposes to try and disrupt proceedings, saying that video of me stealing something's all fake. And so then the courts and decision makers have to spend the time trying to go through that. I also think, you know, increasingly, the public, maybe even in good faith, are starting to distrust their own eyes because they're told they can't. They're seeing these powerful deep-fake materials. And so, I think even trust in the courts as fact-finding arbiters may be shaken by the fact that this technology is out there, and people may be increasingly worried. You know, do we have systems that can separate fact from fiction?

Alison

Do we? Do courts?

Amy Salyzyn

I mean, I have a lot of faith in our courts. When we talk about evidence, it depends on the context, but there’s lots of other ways you can try and authenticate a piece of evidence through other testimony or through, kind of, the chain of custody. So, it really depends on what we're talking about. And I think our courts still have robust systems of evidence. But I think, you know, one challenge is – we talked about courts already being burdened – that they have to start, kind of, adjudicating these trials within a trial about: is this evidence generated? Someone may not have the ability to hire an expert. So, we may need to develop new systems to deal with those allegations, those concerns, in a much more efficient way. If we start seeing, kind of, more and more of those concerns pop up.

Alison

Yeah. Well, I haven't seen a limit yet to what people want to do with AI, so I think that's something to keep an eye on. Well, look, it is always a pleasure to talk to you, Amy. Thank you for coming into the studio and joining me today.

Alison

Thank you very much.

Alison

Amy Salyzyn is an associate law prof at the University of Ottawa, where she focuses on legal ethics, law and technology, and civil justice reform. In June, the Law Society of Ontario recognized her contributions to the law with a medal.

Well, it's great to be back in the saddle for another season of Verdicts & Voices. If you have any good ideas for an upcoming episode, please get in touch over LinkedIn. This podcast is produced by the Canadian Bar Association and I'm your host, Alison Crawford.