Thank you so much. So the work that I'm gonna share is collaboration with Amy Hassanoff, who's also on the call, Anna Gibson, and Sidja Chow, who's also on the call. Do you see my slides?

Yes. Okay. So I'll I'll just get started. I'm not gonna talk a lot, maybe about fifteen to twenty minutes, but I would love to spend more time having a discussion with you all and hearing what you think. So the things that I'm gonna talk about today are different kinds of online harms. Like hate speech, targeted harassment, just thinking about the different ways that people get harmed by other people online in a very broad sense. And one of the main methods that we're gonna use is, speculative design or thinking about alternative ways that the future could work. Again, this is work with Amy Hassonoff, Anna Gibson, Sijit Chao, and Shaogoon Jabber. So I'm gonna start with, an investigation that happened in early two thousand seventeen. An An investigative journalist uncovered a private Facebook group that was called Marines United. Where they found that hundreds of veterans and active duty marines were circulating nude and invasive photos of other military service women without their knowledge. They found dozens of Google Drive folders linked to the Facebook page that included whole dossiers of these women including their names, their military branches, their nude photos, screenshots of their social media accounts, and all sorts of other invasive things. Now this, scandal in particular stood out because it was a form of harassment and abuse that was extremely coordinated and the scale was the lot. But it's just an example of all of the ways that social media is extremely toxic and can hurt a lot of people. In fact, there was a peer research, center study in 02/2017, so the same year, that showed that forty one percent of Americans have personally been subject to abusive behavior online. And one in five of them have been subject to particularly severe forms of sexual harassment, stalking, revenge porn, physical threats, or sustained harassment over time. So following, the investigative journalists report, Facebook quickly shut down that Facebook group. But similar ones began cropping up on the platform. So we had Marines United two point zero, three point zero, all of these other groups. In theory, almost all of the social media platforms ban sexually explicit photos. Particularly if they are flagged as nonconsensual. So if a victim sees the images of themselves on Facebook, they can report it and most of the time it will get taken down. But that doesn't keep the images from being shared on private groups like these ones where they're less likely to be reported because everyone's in on what's going on. So as a result of mounting pressures from advocacy groups for victims of sexual assault, Facebook promised that they were gonna take a more active role into stopping these forms of harassment. The result, which Facebook began rolling out in April 2017 and is now a partnership with nine organizations across eight different countries. Is interesting because and I highlight it because it's one of the most proactive efforts that any of the social media companies have done to address any kind of online abuse. And yet despite the amount of effort and money that has gone into this project, I'm gonna argue that it's actually doomed to fail. But it is a revealing failure, because it shows us the points, or points to the fundamental limitations in the way that social media companies themselves think about, the problem of online harm and the ways that they encourage the rest of us to think about it as well. So I'm gonna talk a little bit more about what this is. So Facebook's process for addressing revenge porn and other nonconsensual sexual images requires that the victim upload their images to the company where a specially trained employee reviews the image and then creates a digital fingerprint of it. So they don't keep the image itself, they create a fingerprint of it. Then you can have image matching software so that if that same image gets uploaded again, the the software will, automatically detect it and remove it. But there are a few major shortcomings to this. The first is that it assumes that the victim has access to the images that they don't want shared. It requires that they trust Facebook with extremely sensitive, content of their own. And also the image matching software is remarkably easy to fool. So people have shown that changing the background or changing small things about the image can fool the software because again, it's not keeping the image itself, it's keeping a fingerprint of it. But perhaps most importantly in this whole process, it takes control away from the victim. So it assumes that all of the victim might possibly want is for these images to be detected and automatically removed. So the victim never learns whether this ever happened or not and they also have no proof in case they want to take further action. In fact, routinely, lawyers have pointed out that when they send preservation of evidence letters to social media companies, around things like revenge porn, they routinely get ignored and the content gets taken down anyway. Even though the victim is explicitly asking to keep it because they want it as evidence. So I'm gonna argue that at the most basic level, this process like a lot of the other attempts at addressing online harm So far it's not because the algorithms are not good enough but because there's a crucial misrepresentation of what the problem is. So social media companies have created this represent this problem statement that encompasses a wide range of online harms. All the things from violating photos to violent and threatening posts to Nazi symbolism, all as a problem of content moderation. And as a result of this problem, as the result of this representation of the problem, the thing the result the response has to be removing the posts that the platform seems sees as against the rules or toxic or whatever they call it. And occasionally, banning the user who posted the content. So this is the story of content moderation. It actually started because social media companies wanted to minimize spam on their platform so they started taking some content out. And then it sort of grew to encompass the response to all the different sorts of online harm that could happen. But I'm gonna argue that this way of thinking about it, thinking about the content as what is causing the harm and taking the content away as what we're supposed to do Is extremely limiting and it's almost entirely unable to address the harm that is caused to the victim and what they actually need. So in order to think differently, we brought in a different framework that exists called restorative justice. So restorative justice is coming out of some abolitionist traditions, some efforts to imagine alternatives to the criminal justice system which sort of similar to the content moderation system tries to identify people who are acting against the rules and punishing them. So restorative justice is a model that has been established as an alternative. Instead of thinking about violations and harms as violating laws, they think about violations as things that create obligations. So the central obligation of restorative justice when someone harms someone else is to right the wrong. What restorative justice attempts to do is center the survivors or victims of harm, go through a process of understanding what their needs are and how those needs might be met. There's an alternative model called transformative justice that seeks to say, okay, if restorative justice is looking at individual harms and trying to address them, what about when those harms are the result of large scale systemic problems? So transformative justice asks us to think about those underlying issues that exist and how might we collectively address them so that individual harms don't keep happening. So the primary tool for restorative justice is communication. And practitioners have come up with all of these structured modes of communication like bringing people into a circle to discuss the harm, creating a list of things that everyone's promising that they're gonna go and do. Everyone signing it, then having follow-up meetings. So basically making the primary tool of action from being punishment in the criminal justice system or removing the content in the content moderation system into the primary tool of action being communication. So here's a little table that I made, just to show again how we're shifting when we think about it from our sort of justice framework, how that fundamentally shifts what questions we ask. So when we come to the problem with a content moderation framework, we might ask, what content has been reported? Is the content against the rules that the platform has set? And should the content be removed, demoted, flagged, or ignored? But if we look at the same kinds of harms from a restorative justice framework, it will completely change what questions we ask. So we might ask, who has been hurt? What are their needs? And whose obligation is it to meet those needs? And of course there would be different obligations on different people. So the person who has caused the harm will have some obligations, the platform will have some obligations because they created the conditions for the harm to happen and potentially to continue happening. So we started doing some participatory design sessions where we brought in restorative and transformative justice practitioners and activists, content moderation from sites like Reddit, and we had interviews and three online workshops where the goal was to try to envision what taking that approach to online harms would look like. Of course the kinds of processes that practitioners have already created work really well in places like schools where most people know each other, it's a tight knit community and there are some mechanisms for holding people accountable. But it was really difficult to think about what would taking those same questions and those same frameworks look like if applied to an online space where a lot of people are anonymous, people might just make another account, it's really hard to hold someone accountable. So through these workshops, we actually gave our participants scenarios of real online harms that we had found on the internet and we asked them to think through how they would address them. I'm not gonna go into too much detail on it, but I'm just gonna go through some of the things that came out of one of those scenarios which was actually a revenge porn scenario. So again, our part our participants started asking who has been heard, what are their needs, and whose obligation is it to meet those needs. And here are some of the obligations that they came up for the platform. So one of them was to assign a trained case worker. So, in this case this would be someone who would help the person support them, provide validation, more importantly maybe share information about what they can do and what the different paths are that they can take. It talks about being trauma aware because someone who has been, sexually harmed is in a place of trauma and reliving that trauma can be much more harmful, than not. So in particular, some of the things around the Facebook tool that asks the person to upload the images again can just recreate that trauma. So they were very they were insistent that people who have gone through trainings to be trauma aware to be the people who support these people who have been harmed and even create the structures for it. Another was to support the harmer to take accountability. Because unlike the criminal justice system for instance that just assumes that someone who has done wrong needs to be punished and set aside from society, restorative justice actually attempts to provide support for someone who has done harm, to understand what harm they've done and to hold themselves accountable and to strive to right the wrong. So instead of deleting the content and assuming that it didn't happen or just banning that person from the community, where possible our participants really wanted to reach out to them if possible and try to engage them in a way to understand what they've done and how it has harmed other people. A really important one was stopping the continuation of the harm And that could mean placing limitations on posting, it could mean different kinds of things that could be done immediately to stop the harm from continuing to happen. And the last one was creating structure for continued accountability. They talked about things like Facebook groups should maybe have a process for dealing with these things, where if someone is harmed they know who to go to, or there's a point person, and then there's a process that everyone is informed of when they join the group and then everyone knows what to do when things like this happen. So, to wrap up, I talked about what would it look like if we took restorative or transformative justice approaches to addressing different kinds of online harm that happened. And in no way am I suggesting that this is gonna be easy or straightforward but maybe it shouldn't be. Maybe online harm is extremely complicated and it will take a lot of time and energy for us to make progress in it. Another thing that I wanted to point out is that we found that it takes a huge amount of labor and time to effectively address effectively address online harm. So one question would might be, why would social media companies spend that much money on this? And I don't really have a good answer for that. I think that they probably won't. But that in the current conditions because they're just not incentivized to do that. But there are different things that we can think about like if we know what works then maybe we could push them towards doing that. Or maybe having a platform as big as Facebook encompassing all of these different countries is just impossible for them to effectively address online harm. And maybe we should be thinking about alternatives that are maybe smaller or more managed by the people who who participate in them and to, so that they can go through these processes where harm doesn't keep happening on these platforms. And finally, if one thing that we can say is that if platforms like Facebook can't in their current structures effectively address and stop harm from happening, should they exist? And I think that that's a question that we should be thinking about more. And finally I wanted to just point out that I, in no way am I saying that we should just sit back for platforms to come and solve these problems that are definitely social and cultural problems that have existed from way before these platforms even did. And as we work to address these, we should be thinking about, how do we transform our own cultures so that things like sexual harm don't keep happening. And I I just wanna point out a quote that I really like from Maryam Kaba who is an organizer and educator and prison abolitionist. She says, I'm actively working towards abolition, which means I'm trying to create the necessary conditions to ensure the possibility of a world without prisons. So I instead of thinking about taking the really narrow problem of content moderation and how do I how how do I create alternatives for it, I think that we should be working towards a world where we don't need a structure like that and thinking about how do we get there. So that's all I had to say and I'm really really looking forward to the conversation. Thank you so much for listening.

Yeah. I was wondering right from the beginning how you scale it and in a sense you kind of addressed it by saying that you kind of don't know how how to do that. And I'm still wondering, like, you know, if you have many you know, I hate Facebook so it's easy to convince us get rid of Facebook, but we won't get rid of Facebook. And having many small alternatives to that is not going to solve the problem because it will be just as wild as it is right now. So I think that in the scale which we in which we are operating, you can't just say, well, let's do something. I mean we have to think about a way that can be scaled.

Yes. Exactly. And one of the things that I think about with that is when we create schools we know that every classroom needs a teacher and we know that there are all these people who need to work there, in order for this social configuration to work. And I think that we haven't thought about the amount of labor and work that it goes into making online social spaces where harm doesn't keep happening. So I think we need to fundamentally question that. And one of the reasons why I I talked about the scale of Facebook wasn't really that I think making it smaller will reduce the workload. I think it'll be the same workload but I think that just thinking about a singular platform encompassing all these different cultures and all of these different countries where harm happens in really different ways and needs to be addressed in really different ways. I don't think that a singular platform or singular approach is going to work. So those are slightly different arguments. I do think that we need way more money and labor spent on addressing online harm than we're doing right now. Even the ways that thinking about the ways that content moderators are treated right now with a very low pay and with no support. I think that that's just a recipe for failure and there's no way we're gonna address online harm without addressing that question of labor. But the other question, the question of can this be done at scale? I would argue, no. It can't. And we should stop trying to make it work at scale.

So in this case, you're just giving up on it? So if we have a lot of sleazy sites that post whatever they want and we're not trying to stop it, then it's how is it going to help us? There are some nice places where that are well managed. How is it going to help the victim?

So would it help if so if you're saying that there are some people that are trying to do harm on purpose, does it help if other people who aren't trying to do harm on purpose are protected from them? I would say yes. So these are two different questions and then so there you can think about harm that happens that where you can make the harm better, you can talk to the harmer and repair the harm, and harm that happens that is on purpose and is happening because maybe the harmer is in a position of power. And I think that those two should be addressed differently. I don't think that I think in the second case where someone is doing harm on purpose and are in a position of power, I think that in those cases the, in that case, and according to restorative justice in that case the number one priority is protecting the people who are being harmed from that harm. And I think that that's a good thing. I also want to not be the sole speaker so I want to invite Amy and Sajja as well and everyone else to respond as well.

I I don't think, just if I if if I may, I mean, I I don't think that that I think that the choices here are a little false. You know, and I'm thinking of, for instance, the way in which Facebook is already you know, which is the example others have raised, is already sorting people into communities as a response to this, you know, deemphasizing the news feed and and raising the emphasis of groups, you know, that that even the platform, this highly scaled platform is turning to pockets of smaller scale in order to address some of these very issues. And I could see very much, like, restorative justice practices restorative justice practices essentially occurring at the Facebook group level, and being kind of part of the part of the moderator or the the kind of, leadership team toolkit, rather than rather than the question either being we all go to our little Mastodon instance, or we or we are all billions of people on Facebook. I mean, the the the big platforms are already looking to smaller scale as a response

within larger scale context.

Yes. And I think part of the reason for that from from Facebook's perspective is that it is impossible to moderate, a platform this big with the set of rules that they've kept trying to moderate with. Like, this is allowed, this is not allowed, and it just doesn't work. And so I I agree with Nathan that even Facebook is pushing more towards having its Facebook group admins do more of the moderation work. And I definitely don't think that it's a zero one thing. You can think about trainings for instance for those moderators which encompass some more of the traditions of restorative justice or its philosophies and practices. Or you could think about even the wording when someone does get banned from say Twitter. Like, you could think about how it that is framed and what what it what it serves. Like, do you word it to that person as saying, you broke the rules and therefore you are punished? Or do you try to explain to them, maybe this is how you harm someone and, like, try to help them understand it and try to make try to put them in a path where if they want to, they can try to repair the harm. So there are small things you can do, and then there are big things you can do. But I agree with Amy that one of the things that we're definitely pushing back against is, scale as a necessary thing. And meaning the way that Amy framed it was perfect. It's like something that can be 10 x with zero extra cost. And I don't think that when thinking about something as serious and as harmful as the kinds of harms that we're talking about that really impact people's lives, we that should be a necessary criteria for how we address them.

Please.

I can just ask it. I think it kind of fits a little bit with some of the other questions like Samantha's question might actually freeze what I was getting at better. But basically, like, what are the existing examples of platforms that might, like, use this framework to construct community norms? And then my question was also around, like, is there a way to kind of bridge that gap between this like community managed, decentralized, transformative framework we're going for? Like we're, you know, with the abolition of police, some folks say defunding the police might be a way to kind of get to abolition while taking shorter term steps. Is there something like that that you see might fit into this framework of transformative justice that is sort of taking us in that direction over time?

I I think that's a really good question. And one of the ways that I think about change more generally is I I try to think about what the what the end goal is or what we would hopefully have, and then what are the steps that we can take towards it. And the steps that we take are never gonna be perfect and they're always gonna have problems. And probably the end state that we imagine is also gonna have problems that we don't foresee, but we have to do something. So I think in terms of what steps do we take, Sijia's work has been a lot around that. I didn't get to talk about it today, but maybe I'll give her the space to talk more about what she's been thinking about in terms of communities that can actually try to do this right now.

I think I was generally wondering, you know, if we're trying to move in this direction that's been outlined, which I think is great, and we're starting where we are, like how do we first learn from communities that are already maybe doing this in online spaces as you already mentioned? And then also think about, like the analogy of kind of defunding the police on the way to abolition. Like, think about something like that in this context.

I have some immediate reactions to that. I think the question of mass harassment is definitely interesting. It was actually one of the scenarios that we gave, the participants in our workshops, and it was really interesting that when so there is this question of, I think, what whether the person who has done the harm can reasonably reasonably be assumed to be in community with the person who was harmed. And whether or not that was true, the reactions were really different. So when they were reasonably in community and they were somewhat known, the reaction was more about trying to bring that person in and explaining to them the harm that they've done and trying to reintegrate them into the community and trying to repair the harm. So moving from a place of horrible thing has happened in this community with the engagement of the whole community to a place where now we understand what that harm was, now we've tried to repair it and now we're in a better place because of it, because of the work that we've done. But when the person was not in community like when we had the scenario of a journalist who was getting mass harassed on Twitter, the reaction went towards protecting the person who was harmed. And so how can the community, help this person to protect them from the continuation of this harm. I I keep going back to another quote by Maryam Kaba. I'm just I really like her. Which was about, she says to stop saying that we should hold people accountable because we can never hold someone accountable. They can only hold themselves accountable. So, you can think about what are conditions that reasonably someone can we can help someone hold themselves accountable. And what are conditions where we can't do that right now and then our reaction can be how do we protect the people who are being harmed right now? So thinking about all of the things that we can do at this moment and what do we prioritize and what do we try to do? I also wanna give Amy and suggest space to respond as well.

Yes. Thank you. Well, first, I'm I'm incredibly grateful. In the in the community I've been publishing in. There's a a very healthy moderation literature, and I've been trying to build the case for a governance or institutional literature, and making that case has been a challenge because it it feels to this audience pretty redundant. And, you know, I felt like there's a case I can make that moderation doesn't cut it, and you just made it crystal clear. I'm, really grateful for that. It's a brilliant argument. Of course, I don't know if my you know, I'm I'm trying to replace content moderation with platform governance that could be susceptible. I'm not trying to replace it with restorative justice. It's possible the platform governance, you know, is a is a satisfying umbrella term that that restorative justice could fit in, or it's possible I fall into the same trap. I'm just reproducing the problem. It's just something I have to think about. Regardless, I'm very grateful for your critique. I did wanna ask a question just to get a better sense of the of the different parts of the system you're you're exploring and proposing. So there's a point so I so I can file a report. That doesn't I'm not instantly within your system recognized as a victim. I assume there's some process I have to go through to officially kind of become a victim, in your in your eyes in the sense of restore of triggering the restorative sort of, process. I wanted to know the first, like, where's where is that line? Do does content have to violate terms of service for me to be recognized by you as a victim and not just as a report filer? And regardless, if there are people you're you're turning back who don't enter the sort of victim stage, what do you do to stay out of the role of sort of being accused of becoming the same marginalizer that you're trying to reform in the sense of, do you is my question clear?

Somewhat. It's a complicated question for sure.

Oh.

Oh, this is all wonderful, so keep going

if If you if you

have something to say. Yeah.

But may may I first just kinda respond with my, understanding of your answer? That you've you've done a lot. You've accomplished a lot. You're you're working really hard and and having great thoughts. But there's one the but you haven't solved the problem that fact finding is necessary, that it's important to, you know, believe women, that there's an inherent conflict between performing a fact finding process and believing women. And and that's sort of a area for future research. Is that a fair representation of

Mariam Cava again.

Who says Who says that people expect sometimes abolitionists to have a complete answer. They don't, neither does the prison system. And so it's important to think about these as extremely complex things that no one is gonna have an answer for right away, but we can take steps. And if you think about what harms are currently happening because of the systems that we have, then it becomes really necessary to think about alternatives. So I completely agree with you. I actually really like the, online, the governance framework because I think that any kind of social system requires governance and calling it what it is and thinking about it in that way is better than assuming that it doesn't exist or the platform is neutral or whatever. And I think that's one of the things that restorative justice as a framework gives us is thinking about different people and what their obligations are. So the community has obligations that are different from the platforms obligations. And in the Twitter journalist example, that I think that the platform definitely has an obligation to stop this harm from continuing to happen. So what would that look like? What would that look like in a governance framework? I think that's really important. And I don't think there's gonna be a singular system that is gonna solve all these problems. We're gonna have to think about the multitude of actors, what their obligations are and how can they act better basically. I also wanted to point out really quickly that the Twitter journalist example is not an anomaly. There's very, there's documented evidence that on groups like four chan, they actually teach each other how to circumvent these algorithms for content moderation. So it's very, it's deliberate. It's it's they're very sophisticated at doing it and they will consistently do harm in ways that don't technically break the content laws. And I don't think that there's gonna be an an way around that that doesn't involve people being involved in it.

If I may say something, all algorithms require complex one may require people in the middle. I think that the question that was raised that is interesting is to what level you can judge if a governance can evolve such that it's going to reduce those cases and how to make a system that allows you to evolve governance because money is always going to be limited. And so you can always say somebody should solve it for me, but nobody will solve it for you. It's a it's a matter of getting good governance.

Yeah. I agree.

No. I was doing the opposite.

Oh, yeah. I I don't know if I need to ask, but I I just wanna register my gratitude for this presentation. And I think it's just such a, such a refreshing thing. So often we rely on these like weirdly authoritarian tools of silencing and exile for for for content moderation, and it's just so necessary to bring in other tools. So thank you for the work you're doing. I really look forward to seeing how this develops.

Thank you so much for inviting us. This is such a great conversation. I learned so much. And I'm I'm really grateful for the opportunity.

And Yeah. If you could send a a link to it, I'll share with everyone else. Awesome.

I've already invited all of you.

Oh, cool. Thank you, Nathan.

I'll stop the recording now. Thank you all. Oh, wait. We have one more thing to do.

two, one.

Yay. Thank you. Thank you. Bye. Thank you.

Did you get all the questions from the chat? You're muted.

Yeah. I'm working on it. It's the formatting is weird. Oh, give me a sec. Your questions are great. Actually, man, this is good.

That was fantastic.

I post it.

I didn't do anything pretty. I I'll just count people to, like, repaste their question from there.

Sounds good. Thank you all.

Great work organizing and running it. Thank you. Mhmm.

Bye bye.