Cloud Hosting

Audio of this conversation is available via your favorite podcast service.

One of the most urgent debates in tech policy at the moment concerns encrypted communications. At issue in proposed legislation, such as the UKs Online Safety Bill or the EARN It Act put forward in the US Senate, is whether such laws break the privacy promise of end to end encryption by requiring that content moderation mechanisms like client-side scanning. But to what extent are such moderation techniques legal under existing laws that limit the monitoring and interception of communications?

Todays guest is James Grimmelmann, a legal scholar with a computer science background who along with Charles Duan recently conducted a review of various moderation technologies to determine how they might hold up in under US federal communication privacy regimes including the Wiretap Act, the Stored Communications Act, and the Communications Assistance for Law Enforcement Act (CALEA). The conversation touches on how technologies like server side and client side scanning work, the extent to which the law may fail to accommodate or even contemplate such technologies, and where the encryption debate is headed as these technologies advance.

What follows is a lightly edited transcript of the discussion.

Justin Hendrix:

James, Im happy to have you back on the podcast this time to talk about a paper that I believe is still in the works, Content Moderation on End to End Encrypted Systems: A Legal Analysis with your co author, Charles Duan.

I would love to just get you to, in your own words, say why it is you chose at this moment to set out to write this piece of work.

James Grimmelmann:

So this comes out of work that some of my colleagues at Cornell Tech have been doing. Tom Ristenpart, whos a computer scientist, and his group have been working on, lets call it online safety.

With the technologies people use now, so one branch of their work, which has been very influential, deals with securing peoples devices in cases that involve intimate partner abuse. Those are cases where the threats are literally coming from inside the house and the abusers may have access to peoples devices in ways that traditional security models didnt include.

Another major strand that Tom and his team have been working on has to do with abuse prevention in end to end encrypted systems. So encrypted messaging is where the message is scrambled in a way so that nobody besides the sender and recipient can read it. Well, if youre sending that message through a server, through email or through a messaging system like Facebook Messenger or WhatsApp or Signal.

Then the question arises, is the message encrypted on its way from you to the Facebook servers and then from Facebook servers to its recipient, or is it encrypted in a way that not even Facebook can read it? If its encrypted in a way that only you and the person youre sending it to can read it, and Facebook sees it as just an equally random string of gibberish, thats called end to end encryption.

And this has been promoted as an important privacy preserving technology, especially against government agencies and law enforcement that might try to surveil communications or have the big platforms do it for them. A challenge, however With end to end encrypted messaging is that it can be a vector for abuse.

If the platform cant scan its contents, it cant look for spam or scams or harassment. Somebody who sends you harassing messages through Facebook Messenger, youll receive it. But Facebooks detectors wont know it. And If you try to report it to Facebook, then Facebook doesnt have direct evidence of its own.

This was actually received through its platform. Its open to potential false reports of abusive messaging. And so in that context, Tom and other computer scientists have been trying to find techniques to mitigate abuse. How can you report abusive messages to a platform? Or if youre a member of a group that uses encrypted communications for all members of the group, and some platforms do now have, and encrypted group chats, how can you and the other participants say, so and so is being a jerk in our community. We dont want further messages from them. And so theres this broad heading of computer science work on abused mitigation in end to end encrypted communications.

Long background on a bunch of computer science stuff, I am here as the law talking guy. So my postdoc Charles and I he, like me, has a background in computer science as well as law have been working with the computer scientists on the legal angles to this.

And in particular, Charles and I have been asking, do these abuse prevention mechanisms comply with communications privacy law? There are laws that prohibit wiretapping or unauthorized disclosure of stored electronic communications. Do these techniques for preventing abusive communications comply with the various legal rules that aim to preserve privacy?

Because in many ways, it would be a really perverse result if people using a technology designed to preserve their privacy cant also use a technology that makes those messaging safe because they would be held to have violated each others privacy. Something very backwards about that result, but our communications privacy laws are so old that it takes a full legal analysis to be certain that this is safe to do.

So our draft, which is very long, goes through a lot of those legal details.

Justin Hendrix:

So I want to get into some of the questions that you pose, including some of the normative questions that you kind of address towards the end of the paper, which pertain to news of the moment questions around. The Online Safety Bill in the UK, for instance, and the fight over encryption thats happening there, et cetera.

I do want to give the paper its due and go through what youve tried to do methodically on some level. But I do want to start perhaps with that last point you just made, which is this idea that these technologies, encrypted messaging apps, are a different generation of communications technology that the law didnt anticipate.

Is that broadly true in your view?

James Grimmelmann:

Thats probably true. Our communications privacy laws were written. Literally, with previous generations of technology in mind, its called wiretapping because this applies to wire communications, which is a telegraph or a telephone that has a physical wire running and ultimately from one person to the other.

And we still use that terminology. And there are still a lot of assumptions from older technologies baked into how the laws are written and the concepts that they use.

Justin Hendrix:

So lets talk about the spate of laws that you looked at here. You looked at the Wire Tap Act; the Stored Communications Act; Pen Registers and Trap and Trace Devices; the Computer Fraud and Abuse Act; and the Communications Assistance for Law Enforcement Act CALEA, as some folks will know it. Are there other laws that perhaps youll have to look at in the final analysis?

James Grimmelmann:

So weve been taking this paper around to conferences, and we got excellent feedback that we also need to address mandatory reporting laws around child sexual abuse material. Because those two impose certain obligations on telecommunications providers or possibly participants when they become aware of certain kinds of material and so moderation techniques that could make them aware of those materials definitely trigger the obligations of those laws. I think its ultimately the five you mentioned plus the CSAM laws.

Justin Hendrix:

So lets talk about the moderation approaches. And maybe it would be helpful for us to just go through them one by one. And in your words, if you can offer a description of what these technologies are.

James Grimmelmann:

Okay, lets start with message franking, which is really a technique designed to address the kind of scenario I mentioned to you before. Youre using an end to end encrypted messaging system, and somebody sends you something abusive.

Pictures of their genitals, repeated messages saying, I hope you die, something that you really dont want to receive. And the technical challenge that its trying to solve is how do you make this reportable to the platform so the platform can help you without undermining the privacy guarantees of end to end encrypted messaging in the first place.

And the solution, which is incredibly ingenious technically, is to allow for a kind of verified reporting in which the recipient of a message can send a report to the platform that is provably based upon actual messages. The recipient cant forge the message and say, Oh, this person sent me this abuse of content, when they didnt actually send it.

So the sender is locked in. They are committed to anything that they send. And if the recipient decides its abusive, they can report it at the same time. The platform should learn nothing. Unless the recipient actually chooses to make a report unless and until that person says, I didnt want to receive this.

This violates platform policies. The platform should have to tell nothing about the message at all. And it turns out that by basically putting a couple of well designed electronic signatures on each message, you can design a system that does this. Its called message franking. The idea being like you frank a message with a stamp, and rubber stamp, you know, carries all the information the platform recipient will later need in case of an abuse report.

And Im lumping forward tracing together with message franking because its basically an extension of it. In forward tracing, if a message is reported as abusive, The platform can trace it back not to the person who sent that specific message, but to everybody before them in a chain if it was forwarded, and that might be relevant.

If a message gets forwarded to somebody and says, this is actually like illegal material that I did not want to be involved with. The platform can then run it back to the original sender who introduced it to the network, which could be useful in rooting out somebody who is using it for abusive purposes.

So basically, its a clever application of cryptographic techniques that have been invented in this millennium after all of the communications privacy laws we discussed were drafted.

Justin Hendrix:

And which of the encrypted messaging apps that folks are familiar with at the moment are using this technique?

James Grimmelmann:

So its basically research stage. Facebook is the one that is leading the way in terms of developing this technology. Facebook was one of the their research arm was one of the original creators of one of the original message franking proposals. So theyre the one that has invested the most in making this workable.

Justin Hendrix:

And of course, Facebook intends to make its Messenger encrypted by the end of the year, its promised. So perhaps its interested in doing so alongside the introduction of technologies like this. Lets talk about whether this comports with the various laws and frameworks that youve assessed. How does it stand up when you look back at the statute?

James Grimmelmann:

So this is an answer Ill probably give you repeatedly, which is, we think its okay, but were less certain about that than we would like to be.

So lets take the wiretap act. The wiretap act, as you might expect, prohibits intercepting electronic communications in a way that lets you learn their contents. And the classic case here is like the literal wiretap plugging into a phone cable. Or also connecting to a network box and just grabbing a copy of somebodys incoming email in flight as it arrives.

And it might seem like, well, theres no interception here because only when theres an actual abuse report made to the platform does the platform learn the contents of a message, but its not quite that clean because the definition of contents in the Wiretap Act is quite broad. The statute defines it as any information concerning the substance purport or meaning of a communication.

And theres a non frivolous argument that this little franking tag, the little stamp that the platform gets applied to each message actually does contain some information about the substance of the message. It does allow the platform to verify the messages authenticity, and there are courts that have expressed at least doubt about whether this kind of metadata verifies a messages contents is in fact itself also contents. And if you go down that road, you wind up then asking a whole bunch of other statutory questions under the Wiretap Act. Does the participation of the platform in applying the franking tag to a message as it gets sent through from sender to recipient, Is that an interception under the statute again textually a hard question, and then perhaps most interestingly, and this was one really opens up a thorny set of issues.

Should we think about the participants in this communication as having consented to this process. Should the sender of the message be able to say, Wait a minute. I didnt consent to all of this cryptographic mumbo jumbo that you did when I sent a message. I did not consent to the steps necessary to verify me as the sender. I thought I was using a completely encrypted end to end messaging system. I did not agree to any of this.

And from one perspective, this is a bad argument for a person sending abusive messages to make. But from another, they do have a point that this does not completely comport with the way that end to end encrypted messaging is used in the broad public discourse.

If you think of it as meaning no one besides you and the recipient can ever learn anything about your message, then this is a small inroads on the privacy guarantees of E2EE.

Justin Hendrix:

So were going to come back to that last comment I think more than once as we go through this and perhaps well address it in the summary conversation as well because I think you might be able to say that about each of these things.

But next you go to server side automated content scanning. A lot of folks like to toss out this phrase, homomorphic. Encryption. I liked the somewhat artful description you have of this technique where the server learns nothing. Ill read it.

Imagine a blindfolded chef wearing thick mittens who follows instructions to take things out of a box, chop them up, put them in the oven for an hour at three 50 degrees, and then put it back in the box. This chef can roast vegetables for you, but doesnt learn whether you were roasting potatoes or parsnips. Its a pretty good description, I suppose, of how this is supposed to work, technically.

Lets talk first, perhaps, about whether this technology works at all.

James Grimmelmann:

So, homomorphic encryption is another one of these really interesting modern developments in cryptography.

The idea is that you can perform a computation on some data without learning anything about the data. And this seems like a kind of pointless thing to do if its just you working with your own data. But if you have some untrusted party who has a lot of Processing capacity and you want them to do some work for you.

Its actually quite valuable. Like if the chef can run an efficient enough kitchen, we might all hand off our vegetables to them to do this for us. And in particular, homomorphic encryption could be used to scan content for matching against certain kinds of. Like CSAM, Child Sexual Abuse Material registries, or certain kinds of spam detection, without letting the person doing the scanning know that it has been scanned in that way.

And you might think, well, whats the point then? Well, you can modify the message being transmitted. To flag it for the recipient so that before you open that picture of somebodys genitals, you might get a warning saying the attached image appears to be of somebodys genitals. Do you wish to proceed? And that would actually be a meaningful anti abuse factor that the server does this matching against a complicated model for you.

You dont have to have the whole huge database of these pictures on your device, and you might not be in a position to do it yourself easily. The platform can do this to help warn people about the messages that theyre receiving.

Justin Hendrix:

Is this a legal technology, at least according to the laws that you reviewed?

James Grimmelmann:

Again, we think its legal, but were not as certain as we would like to be. Take the wiretap attack analysis. The platform can do things that manipulate the message. Once again, were in that world of asking, is it receiving contents? Here, the argument against liability depends, I think, on some of the exceptions to Wiretap Act liability that the Act includes in it.

So, for example, the Wiretap Act has this exception for the ordinary course of business. In which platforms can inspect messages part of their ordinary operations and platforms routinely do spam detection and antivirus scanning on our message attachments already. So this seems to fit within the class of things that they already do.

The analysis under the other statutes is also pretty good. One of the nice things about this kind of encryption is that platforms dont retain any information once they do the processing. They send it out, it leaves their system. That means that they are not retaining the kinds of stored communications that could trigger the Stored Communications Act.

Thank you. We like it. We would like this to be legal. We think it is. We dont have 100% certainty.

Justin Hendrix:

And is it the case, based on your review, that this technology is still fragile, still unlikely to work at scale?

James Grimmelmann:

Its not scalable currently. Ordinary computation is fast. Applying and removing encryption is reasonably fast.

Homomorphic encryption is kind of slow. The work you have to do in order to compile your computation down into the kind of thing you can do blindfolded with mittens on makes it a lot less efficient. Its not surprising. Anything you do wearing thick, heavy gloves is going to be a lot less effective because you cant feel what youre doing.

And so its not a scale worthy technology yet, but its impossible enough than it might be that its worth thinking in advance about its legality.

Justin Hendrix:

So next well talk about what is, you know, perhaps the most discussed potential form of content moderation for encrypted. Messaging apps these days, client side automated content scanning.

Of course, Apple proposed one such system. Apparently the UK Home Office is funding the development of prototypes in this space, perhaps in anticipation of the potential passage of the Online Safety Bill there. How does client side scanning work? Do you have another cooking metaphor that could explain this one to us?

James Grimmelmann:

And not quite as elegantly client side scanning is really you have the client that you are using to send messages. So the Facebook Messenger app or the Signal app or Apples messaging app would perform some kind of computation, some check of your content on the device before it sent or when its received, and the scanning then can flag either for the user or for some external authority, whether it matches against some database of concerning communications.

Justin Hendrix:

And is it legal?

James Grimmelmann:

This gets really complicated, in part because of the diversity of these systems. There are a lot of different architectures. Some of them involve trying to scan against databases without revealing to the client whats in the database. Because if you figure if a database is a prohibited content, you cant just give everybody a complete copy of the things youre not supposed to have.

And also because they involve communications, that is, if Im trying to query what Ive got on my device against some database of things. It may involve sending a comp, a digest of what Ive got out to the network and back. And does that process constitute an interception? This brings us back to the same kinds of questions we asked when we were doing message franking.

Have I, as the user of this app, consented to have my data scanned in this way? And possibly to have some flag about its status being sent to the third party whos providing this app. Again, this is a hard question. I dont think you can answer it fully on the technical side. You cant just say, well because this app works this way and you ran the app you consented to it.

That same argument would say you consented to spy on your phone. But you also cant Just say, well, I didnt want this. So its, theres no consent at some point. People have to know how the software theyve been chosen to run. Its been explained to them works, or we have, you know, serious, you know, computer law violations.

Every time anybody is surprised by an app feature. So its going to be very fact dependent in a slightly uncomfortable way.

Justin Hendrix:

Youve mentioned theres some variability in terms of how these client side scanning schemes work. Are there versions of client side? scanning that you are more comfortable with than others?

Are there those that youve seen that you would regard as, you know, potentially spyware or very concerning from a privacy standpoint and ones that perhaps, I guess, are a little more responsible?

James Grimmelmann:

I mean, the obvious dividing line here is a client side app. That reports the results out to a third party versus one that merely reports it to parties to the communication.

That is, I might very well as a recipient want to have had the senders device do a client side scan and have a cryptographic certification that it didnt include stuff in this abusive database. I could see that, and if thats not revealed to anybody outside the communication, it seems reasonably privacy friendly.

If its scanning against the government provided database of terrorist supporting content, or the kinds of safety concerns that the UK Home Office would like to be monitoring for, thats a bigger intrusion on privacy. Now, it may be that the particular things on this list are particularly concerning, but you get into the fact that this is scanning your messaging for reporting out to the government, and you get into serious questions about the transparency of the process by which things are added to that database.

And so you really cant assess the privacy implications without having a larger conversation about the institutional setting.

Here is the original post:
Content Moderation, Encryption, and the Law - Tech Policy Press