Metagov Salehi
Metagovernance Seminar Archive | 2025-10-20 | Unknown
Speaker 1: All right. So maybe I'll just start with introducing our speaker. I'm super excited that we have, Nilu Far Salehi with us today. So Doctor. Salehi is an assistant professor at the school of information at UC Berkeley who does some really really great research on social computing, participatory and critical design, and human centered AI. And her research group was recently awarded an...
Top Keywords
- restorative 0.018
- restorative justice 0.018
- harm 0.015
- justice 0.015
- victim 0.010
- scale 0.007
- content 0.007
- facebook 0.007
- harmed 0.006
- moderation 0.005
- online 0.005
- online harm 0.005
Transcript
Speaker 1
0:00 – 0:00
All right. So maybe I'll just start with introducing our speaker. I'm super excited that we have, Nilu Far Salehi with us today. So Doctor. Salehi is an assistant professor at the school of information at UC Berkeley who does some really really great research on social computing, participatory and critical design, and human centered AI. And her research group was recently awarded an NSF grant to study how a restorative justice approach to social media moderation and online governance might be implemented. So I think we're really lucky to have her, and I think one of her students joined us, who's working on the project. Yes. Yeah. Is also here. So we're really lucky to have both of them here today, and I'll just hand it off to you now. Thanks again for coming.
Speaker 2
0:15 – 0:15
Thank you so much. So the work that I'm gonna share is collaboration with Amy Hassanoff, who's also on the call, Anna Gibson, and Sidja Chow, who's also on the call. Do you see my slides?
Speaker 3
0:30 – 0:30
Yes.
Speaker 2
0:45 – 0:45
Yes. Okay. So I'll I'll just get started. I'm not gonna talk a lot, maybe about fifteen to twenty minutes, but I would love to spend more time having a discussion with you all and hearing what you think. So the things that I'm gonna talk about today are different kinds of online harms. Like hate speech, targeted harassment, just thinking about the different ways that people get harmed by other people online in a very broad sense. And one of the main methods that we're gonna use is, speculative design or thinking about alternative ways that the future could work. Again, this is work with Amy Hassonoff, Anna Gibson, Sijit Chao, and Shaogoon Jabber. So I'm gonna start with, an investigation that happened in early two thousand seventeen. An An investigative journalist uncovered a private Facebook group that was called Marines United. Where they found that hundreds of veterans and active duty marines were circulating nude and invasive photos of other military service women without their knowledge. They found dozens of Google Drive folders linked to the Facebook page that included whole dossiers of these women including their names, their military branches, their nude photos, screenshots of their social media accounts, and all sorts of other invasive things. Now this, scandal in particular stood out because it was a form of harassment and abuse that was extremely coordinated and the scale was the lot. But it's just an example of all of the ways that social media is extremely toxic and can hurt a lot of people. In fact, there was a peer research, center study in 02/2017, so the same year, that showed that forty one percent of Americans have personally been subject to abusive behavior online. And one in five of them have been subject to particularly severe forms of sexual harassment, stalking, revenge porn, physical threats, or sustained harassment over time. So following, the investigative journalists report, Facebook quickly shut down that Facebook group. But similar ones began cropping up on the platform. So we had Marines United two point zero, three point zero, all of these other groups. In theory, almost all of the social media platforms ban sexually explicit photos. Particularly if they are flagged as nonconsensual. So if a victim sees the images of themselves on Facebook, they can report it and most of the time it will get taken down. But that doesn't keep the images from being shared on private groups like these ones where they're less likely to be reported because everyone's in on what's going on. So as a result of mounting pressures from advocacy groups for victims of sexual assault, Facebook promised that they were gonna take a more active role into stopping these forms of harassment. The result, which Facebook began rolling out in April 2017 and is now a partnership with nine organizations across eight different countries. Is interesting because and I highlight it because it's one of the most proactive efforts that any of the social media companies have done to address any kind of online abuse. And yet despite the amount of effort and money that has gone into this project, I'm gonna argue that it's actually doomed to fail. But it is a revealing failure, because it shows us the points, or points to the fundamental limitations in the way that social media companies themselves think about, the problem of online harm and the ways that they encourage the rest of us to think about it as well. So I'm gonna talk a little bit more about what this is. So Facebook's process for addressing revenge porn and other nonconsensual sexual images requires that the victim upload their images to the company where a specially trained employee reviews the image and then creates a digital fingerprint of it. So they don't keep the image itself, they create a fingerprint of it. Then you can have image matching software so that if that same image gets uploaded again, the the software will, automatically detect it and remove it. But there are a few major shortcomings to this. The first is that it assumes that the victim has access to the images that they don't want shared. It requires that they trust Facebook with extremely sensitive, content of their own. And also the image matching software is remarkably easy to fool. So people have shown that changing the background or changing small things about the image can fool the software because again, it's not keeping the image itself, it's keeping a fingerprint of it. But perhaps most importantly in this whole process, it takes control away from the victim. So it assumes that all of the victim might possibly want is for these images to be detected and automatically removed. So the victim never learns whether this ever happened or not and they also have no proof in case they want to take further action. In fact, routinely, lawyers have pointed out that when they send preservation of evidence letters to social media companies, around things like revenge porn, they routinely get ignored and the content gets taken down anyway. Even though the victim is explicitly asking to keep it because they want it as evidence. So I'm gonna argue that at the most basic level, this process like a lot of the other attempts at addressing online harm So far it's not because the algorithms are not good enough but because there's a crucial misrepresentation of what the problem is. So social media companies have created this represent this problem statement that encompasses a wide range of online harms. All the things from violating photos to violent and threatening posts to Nazi symbolism, all as a problem of content moderation. And as a result of this problem, as the result of this representation of the problem, the thing the result the response has to be removing the posts that the platform seems sees as against the rules or toxic or whatever they call it. And occasionally, banning the user who posted the content. So this is the story of content moderation. It actually started because social media companies wanted to minimize spam on their platform so they started taking some content out. And then it sort of grew to encompass the response to all the different sorts of online harm that could happen. But I'm gonna argue that this way of thinking about it, thinking about the content as what is causing the harm and taking the content away as what we're supposed to do Is extremely limiting and it's almost entirely unable to address the harm that is caused to the victim and what they actually need. So in order to think differently, we brought in a different framework that exists called restorative justice. So restorative justice is coming out of some abolitionist traditions, some efforts to imagine alternatives to the criminal justice system which sort of similar to the content moderation system tries to identify people who are acting against the rules and punishing them. So restorative justice is a model that has been established as an alternative. Instead of thinking about violations and harms as violating laws, they think about violations as things that create obligations. So the central obligation of restorative justice when someone harms someone else is to right the wrong. What restorative justice attempts to do is center the survivors or victims of harm, go through a process of understanding what their needs are and how those needs might be met. There's an alternative model called transformative justice that seeks to say, okay, if restorative justice is looking at individual harms and trying to address them, what about when those harms are the result of large scale systemic problems? So transformative justice asks us to think about those underlying issues that exist and how might we collectively address them so that individual harms don't keep happening. So the primary tool for restorative justice is communication. And practitioners have come up with all of these structured modes of communication like bringing people into a circle to discuss the harm, creating a list of things that everyone's promising that they're gonna go and do. Everyone signing it, then having follow-up meetings. So basically making the primary tool of action from being punishment in the criminal justice system or removing the content in the content moderation system into the primary tool of action being communication. So here's a little table that I made, just to show again how we're shifting when we think about it from our sort of justice framework, how that fundamentally shifts what questions we ask. So when we come to the problem with a content moderation framework, we might ask, what content has been reported? Is the content against the rules that the platform has set? And should the content be removed, demoted, flagged, or ignored? But if we look at the same kinds of harms from a restorative justice framework, it will completely change what questions we ask. So we might ask, who has been hurt? What are their needs? And whose obligation is it to meet those needs? And of course there would be different obligations on different people. So the person who has caused the harm will have some obligations, the platform will have some obligations because they created the conditions for the harm to happen and potentially to continue happening. So we started doing some participatory design sessions where we brought in restorative and transformative justice practitioners and activists, content moderation from sites like Reddit, and we had interviews and three online workshops where the goal was to try to envision what taking that approach to online harms would look like. Of course the kinds of processes that practitioners have already created work really well in places like schools where most people know each other, it's a tight knit community and there are some mechanisms for holding people accountable. But it was really difficult to think about what would taking those same questions and those same frameworks look like if applied to an online space where a lot of people are anonymous, people might just make another account, it's really hard to hold someone accountable. So through these workshops, we actually gave our participants scenarios of real online harms that we had found on the internet and we asked them to think through how they would address them. I'm not gonna go into too much detail on it, but I'm just gonna go through some of the things that came out of one of those scenarios which was actually a revenge porn scenario. So again, our part our participants started asking who has been heard, what are their needs, and whose obligation is it to meet those needs. And here are some of the obligations that they came up for the platform. So one of them was to assign a trained case worker. So, in this case this would be someone who would help the person support them, provide validation, more importantly maybe share information about what they can do and what the different paths are that they can take. It talks about being trauma aware because someone who has been, sexually harmed is in a place of trauma and reliving that trauma can be much more harmful, than not. So in particular, some of the things around the Facebook tool that asks the person to upload the images again can just recreate that trauma. So they were very they were insistent that people who have gone through trainings to be trauma aware to be the people who support these people who have been harmed and even create the structures for it. Another was to support the harmer to take accountability. Because unlike the criminal justice system for instance that just assumes that someone who has done wrong needs to be punished and set aside from society, restorative justice actually attempts to provide support for someone who has done harm, to understand what harm they've done and to hold themselves accountable and to strive to right the wrong. So instead of deleting the content and assuming that it didn't happen or just banning that person from the community, where possible our participants really wanted to reach out to them if possible and try to engage them in a way to understand what they've done and how it has harmed other people. A really important one was stopping the continuation of the harm And that could mean placing limitations on posting, it could mean different kinds of things that could be done immediately to stop the harm from continuing to happen. And the last one was creating structure for continued accountability. They talked about things like Facebook groups should maybe have a process for dealing with these things, where if someone is harmed they know who to go to, or there's a point person, and then there's a process that everyone is informed of when they join the group and then everyone knows what to do when things like this happen. So, to wrap up, I talked about what would it look like if we took restorative or transformative justice approaches to addressing different kinds of online harm that happened. And in no way am I suggesting that this is gonna be easy or straightforward but maybe it shouldn't be. Maybe online harm is extremely complicated and it will take a lot of time and energy for us to make progress in it. Another thing that I wanted to point out is that we found that it takes a huge amount of labor and time to effectively address effectively address online harm. So one question would might be, why would social media companies spend that much money on this? And I don't really have a good answer for that. I think that they probably won't. But that in the current conditions because they're just not incentivized to do that. But there are different things that we can think about like if we know what works then maybe we could push them towards doing that. Or maybe having a platform as big as Facebook encompassing all of these different countries is just impossible for them to effectively address online harm. And maybe we should be thinking about alternatives that are maybe smaller or more managed by the people who who participate in them and to, so that they can go through these processes where harm doesn't keep happening on these platforms. And finally, if one thing that we can say is that if platforms like Facebook can't in their current structures effectively address and stop harm from happening, should they exist? And I think that that's a question that we should be thinking about more. And finally I wanted to just point out that I, in no way am I saying that we should just sit back for platforms to come and solve these problems that are definitely social and cultural problems that have existed from way before these platforms even did. And as we work to address these, we should be thinking about, how do we transform our own cultures so that things like sexual harm don't keep happening. And I I just wanna point out a quote that I really like from Maryam Kaba who is an organizer and educator and prison abolitionist. She says, I'm actively working towards abolition, which means I'm trying to create the necessary conditions to ensure the possibility of a world without prisons. So I instead of thinking about taking the really narrow problem of content moderation and how do I how how do I create alternatives for it, I think that we should be working towards a world where we don't need a structure like that and thinking about how do we get there. So that's all I had to say and I'm really really looking forward to the conversation. Thank you so much for listening.
Speaker 1
1:00 – 1:00
Awesome that was really really great, cool to hear. There are a few questions in the chat, so maybe we'll kind of like go through those first and then have people, like, kinda just jump in and see where the conversation goes. Yeah. And then, Ofer, I think your question was first.
Speaker 4
1:15 – 1:15
Yeah. I was wondering right from the beginning how you scale it and in a sense you kind of addressed it by saying that you kind of don't know how how to do that. And I'm still wondering, like, you know, if you have many you know, I hate Facebook so it's easy to convince us get rid of Facebook, but we won't get rid of Facebook. And having many small alternatives to that is not going to solve the problem because it will be just as wild as it is right now. So I think that in the scale which we in which we are operating, you can't just say, well, let's do something. I mean we have to think about a way that can be scaled.
Speaker 2
1:30 – 1:30
Yes. Exactly. And one of the things that I think about with that is when we create schools we know that every classroom needs a teacher and we know that there are all these people who need to work there, in order for this social configuration to work. And I think that we haven't thought about the amount of labor and work that it goes into making online social spaces where harm doesn't keep happening. So I think we need to fundamentally question that. And one of the reasons why I I talked about the scale of Facebook wasn't really that I think making it smaller will reduce the workload. I think it'll be the same workload but I think that just thinking about a singular platform encompassing all these different cultures and all of these different countries where harm happens in really different ways and needs to be addressed in really different ways. I don't think that a singular platform or singular approach is going to work. So those are slightly different arguments. I do think that we need way more money and labor spent on addressing online harm than we're doing right now. Even the ways that thinking about the ways that content moderators are treated right now with a very low pay and with no support. I think that that's just a recipe for failure and there's no way we're gonna address online harm without addressing that question of labor. But the other question, the question of can this be done at scale? I would argue, no. It can't. And we should stop trying to make it work at scale.
Speaker 4
1:45 – 1:45
So in this case, you're just giving up on it? So if we have a lot of sleazy sites that post whatever they want and we're not trying to stop it, then it's how is it going to help us? There are some nice places where that are well managed. How is it going to help the victim?
Speaker 2
2:00 – 2:00
So would it help if so if you're saying that there are some people that are trying to do harm on purpose, does it help if other people who aren't trying to do harm on purpose are protected from them? I would say yes. So these are two different questions and then so there you can think about harm that happens that where you can make the harm better, you can talk to the harmer and repair the harm, and harm that happens that is on purpose and is happening because maybe the harmer is in a position of power. And I think that those two should be addressed differently. I don't think that I think in the second case where someone is doing harm on purpose and are in a position of power, I think that in those cases the, in that case, and according to restorative justice in that case the number one priority is protecting the people who are being harmed from that harm. And I think that that's a good thing. I also want to not be the sole speaker so I want to invite Amy and Sajja as well and everyone else to respond as well.
Speaker 5
2:15 – 2:15
I can just add that the oh, sorry. Go ahead.
Speaker 1
2:30 – 2:30
No, I was just wondering if you had anything to
Speaker 5
2:45 – 2:45
say. Okay. Zoom, yes. It's always fun. No, I mean, Nilfara, that was amazing. Thank you for summarizing all that so wonderfully. I I just just talking about the scale question, it's something that we are the collaborators, we've all thought about and talked about a lot of, like, how do we deal with this, question that comes up every time we talk about this project, which is, but does it scale? How does it scale? Can you not scale it? What if you can't scale it? And I don't think that we've necessarily got, like, a satisfying answer to that other than this sort of, like, maybe it can't scale and maybe it shouldn't. And then the question becomes, well, if we can't if Facebook, like, is inevitably large, does it still need to can it if it still is existing, right, then how do you apply these things at scale? And I think one of the things that we talked about that that question asks, at least within, like, a tech company discourse is, can we replicate this without further cost in terms of labor and resources? And we can just, like, create some kind of code or bot or algorithm, and then we can just, like, multiply it and scale it up by 10 x, a thousand x, a million x, and it doesn't cost anything extra, really. And I think the way that we've been just thinking about that is, like, it can scale in that, like, transforming how we think about justice needs to, in a sense, scale and spread across the population for us to achieve the sort of radical changes that we are dreaming, you know, dreaming about. But can you do it without cost? No. Like, obvious definitely not. It it's it's costly. But scale usually when we're asked it by people in the tech industry means, can we just 1,000,000 exit for free? No, we cannot. That's the answer. I have a lot of feelings about scale.
Speaker 3
3:00 – 3:00
Granted that it may not scale for free, I have sort of lost the thread here as to whether you do see a way forward or whether you're just saying, it didn't work, give up.
Speaker 6
3:15 – 3:15
I I don't think, just if I if if I may, I mean, I I don't think that that I think that the choices here are a little false. You know, and I'm thinking of, for instance, the way in which Facebook is already you know, which is the example others have raised, is already sorting people into communities as a response to this, you know, deemphasizing the news feed and and raising the emphasis of groups, you know, that that even the platform, this highly scaled platform is turning to pockets of smaller scale in order to address some of these very issues. And I could see very much, like, restorative justice practices restorative justice practices essentially occurring at the Facebook group level, and being kind of part of the part of the moderator or the the kind of, leadership team toolkit, rather than rather than the question either being we all go to our little Mastodon instance, or we or we are all billions of people on Facebook. I mean, the the the big platforms are already looking to smaller scale as a response
Speaker 1
3:30 – 3:30
to this challenge
Speaker 6
3:45 – 3:45
within larger scale context.
Speaker 2
4:00 – 4:00
Yes. And I think part of the reason for that from from Facebook's perspective is that it is impossible to moderate, a platform this big with the set of rules that they've kept trying to moderate with. Like, this is allowed, this is not allowed, and it just doesn't work. And so I I agree with Nathan that even Facebook is pushing more towards having its Facebook group admins do more of the moderation work. And I definitely don't think that it's a zero one thing. You can think about trainings for instance for those moderators which encompass some more of the traditions of restorative justice or its philosophies and practices. Or you could think about even the wording when someone does get banned from say Twitter. Like, you could think about how it that is framed and what what it what it serves. Like, do you word it to that person as saying, you broke the rules and therefore you are punished? Or do you try to explain to them, maybe this is how you harm someone and, like, try to help them understand it and try to make try to put them in a path where if they want to, they can try to repair the harm. So there are small things you can do, and then there are big things you can do. But I agree with Amy that one of the things that we're definitely pushing back against is, scale as a necessary thing. And meaning the way that Amy framed it was perfect. It's like something that can be 10 x with zero extra cost. And I don't think that when thinking about something as serious and as harmful as the kinds of harms that we're talking about that really impact people's lives, we that should be a necessary criteria for how we address them.
Speaker 7
4:15 – 4:15
To add on Nilofar.
Speaker 2
4:30 – 4:30
Please.
Speaker 7
4:45 – 4:45
Sorry. I'm on my mask. So, so I can talk about how we want to implement restorative justice and see how those conditions are difficult or not difficult to scale. I think one thing is institutional buying. For example, Facebook, how can they put ad work into that? And another thing is labor. One is the labor for training. If you want people to do restorative justice, you have to teach people about the concept and also the labor of implementing. So and also there's the change in people's notion of justice, which is we can also call that culture. So if you want to get people to adapt to this model, you have to get victims and offender self harm, have a willingness to join. So for both of those things, for all of those things, institutional buy in and labor and changing people's notion of justice, we can see that it's very difficult to scale. But I would say that in the long term, if people can change gradually, it is possible that we can implement restorative justice in more and more communities. It's just because the world that we are living in, we think it's not possible to scale it. But for example, I've interviewed someone who have I've interviewed someone who have, experiences with recorded data. It's gonna be done. It's just one quick. So they will say that, so I've done it in a real life practice. Why can't why can't I do it online? So if people have the right if people have the mindset of using resources as this, it's very easy for them to do so.
Speaker 1
5:00 – 5:00
There are actually a lot of questions in the chat now, and I I would love to continue this discussion of scale, but I also wanna make sure we get to some of the other questions. I think Divya's question was next.
Speaker 8
5:15 – 5:15
I can just ask it. I think it kind of fits a little bit with some of the other questions like Samantha's question might actually freeze what I was getting at better. But basically, like, what are the existing examples of platforms that might, like, use this framework to construct community norms? And then my question was also around, like, is there a way to kind of bridge that gap between this like community managed, decentralized, transformative framework we're going for? Like we're, you know, with the abolition of police, some folks say defunding the police might be a way to kind of get to abolition while taking shorter term steps. Is there something like that that you see might fit into this framework of transformative justice that is sort of taking us in that direction over time?
Speaker 2
5:30 – 5:30
I I think that's a really good question. And one of the ways that I think about change more generally is I I try to think about what the what the end goal is or what we would hopefully have, and then what are the steps that we can take towards it. And the steps that we take are never gonna be perfect and they're always gonna have problems. And probably the end state that we imagine is also gonna have problems that we don't foresee, but we have to do something. So I think in terms of what steps do we take, Sijia's work has been a lot around that. I didn't get to talk about it today, but maybe I'll give her the space to talk more about what she's been thinking about in terms of communities that can actually try to do this right now.
Speaker 9
5:45 – 5:45
Thanks.
Speaker 7
6:00 – 6:00
So Nilofar and Amy's more work is more around commercial content moderation on space Facebook. My work is about using restorative justice in gaming communities, which include, a platform called Discord, where the moderators are volunteers. So we are thinking about, doing restorative justice, in those communities where the volunteer moderators can be, possible source of, the facilitators of those processes. And so to answer the question more specific, so can can you repeat the question a little bit?
Speaker 8
6:15 – 6:15
I think I was generally wondering, you know, if we're trying to move in this direction that's been outlined, which I think is great, and we're starting where we are, like how do we first learn from communities that are already maybe doing this in online spaces as you already mentioned? And then also think about, like the analogy of kind of defunding the police on the way to abolition. Like, think about something like that in this context.
Speaker 7
6:30 – 6:30
Sure. I think in those communities I investigated, there are some existing elements of restorative justice that we can borrow. For example, the volunteer moderators like to give offenders second chances. So they will first mute or warn people and then give permanent, temporary events before they give permanent events. So it seems like they would like to make effort so that the offenders get chances to change. While they have the labor, we think what they need to change is is their notion of justice because as one facilitator, one restorative justice practitioners told me, what matters is not how many chances you give people, but what you do in between to change them. Because the no matter the punishment or even the explanations the moderators give them, It's usually around rules. For example, you shouldn't do this because you have violated the rules, or if you do that again, you are out. So those are not the learning that offenders necessarily need, but instead, they need to learn the impact of their actions. And, in restorative justice, those learning usually come from, listening to the victims who share, or even the facilitators can tell them, what are some impact. So just by changing the language, for example, just by changing the language, how moderators communicate with the offenders, it's not necessarily additional labor. It's just a change of waste of thinking. So instead of saying you shouldn't do this because you are violating the rules, you can say this is the potential impact that you have created, and the community is willing to help you to change through those several ways. So this is one possible element of restorative justice. And also, the moderators will actually take time. Some moderators will actually take time to talk with the victims and offenders, and to ask them what has happened before making a decision. The difference with restorative justice is that the moderator will try to oversee will try to see what has happened and make a final decision. Well, restorative justice believes that victims and offenders have their own story, and we should look at the holistic story and give the agency to victim and let them decide, what has happened. So I would say that, again, the framework is there. It's just how the final decision is made and how people think who should make the final decision. So those things are different. So it's also related to the, question of scale and labor. I think the moderators, the volunteer moderators that I, interviewed, they want to help offenders to change. They want to make a better community, and they are also making effort to do it. It's just that people right now, they only know punitive justice. They only know the current model. This is the way they are solving the problem. Right now, we want to provide them with an alternative and with the, similarly, similar amount of work, they can achieve something else. We're not sure whether it will work or not, but whether, but at least we're providing them an alternative option so that they can try to give offenders a chance in another way first before they decide to give people punishment.
Speaker 1
6:45 – 6:45
Yeah. Kind of relating to that point, if I could pitch in with, like, a follow-up question. I'm kind of there's a question in the chat by Jenny later on asking about, I think you've been talking a lot about, like, kind of it seems like it sounds like they're one on one to one cases where, like, the person who's been harmed and then the person who's doing the harming, like, are interacting one on one. So, Jenny's question, was really great. It was about, like, when what happens when there's, like, collective or, like, anonymous sets of harmer. So, like, a army of Reddit trollers or something like that. Sorry to kidnap your question, Jenny.
Speaker 9
7:00 – 7:00
It's all good. Yeah. Well, I was also I was wondering if you ever worked with Tracy Cho who was working on the Block Party app. I just follow her on Twitter, but I know because she's been working in this space. In her attempts to talk about this app that is supposed to solve online harassment, I think she's been harassed a lot by Reddit. And it's it's it's so it's not almost not worth trying to get individual people in the room. And then there's also an interesting reference. A Reply All podcast has an episode called the Snapchat Thief, and these, like, two radio commentators bring in, someone who had their Snapchat handle stolen and had compromising photos, like, stolen and and put him in put them in the same room as these, like, basically teenage Bitcoin OG handle thieves. And the conversation was extremely interesting to hear. So, I I, like, love this idea. It's super interesting research, so thanks.
Speaker 2
7:15 – 7:15
I have some immediate reactions to that. I think the question of mass harassment is definitely interesting. It was actually one of the scenarios that we gave, the participants in our workshops, and it was really interesting that when so there is this question of, I think, what whether the person who has done the harm can reasonably reasonably be assumed to be in community with the person who was harmed. And whether or not that was true, the reactions were really different. So when they were reasonably in community and they were somewhat known, the reaction was more about trying to bring that person in and explaining to them the harm that they've done and trying to reintegrate them into the community and trying to repair the harm. So moving from a place of horrible thing has happened in this community with the engagement of the whole community to a place where now we understand what that harm was, now we've tried to repair it and now we're in a better place because of it, because of the work that we've done. But when the person was not in community like when we had the scenario of a journalist who was getting mass harassed on Twitter, the reaction went towards protecting the person who was harmed. And so how can the community, help this person to protect them from the continuation of this harm. I I keep going back to another quote by Maryam Kaba. I'm just I really like her. Which was about, she says to stop saying that we should hold people accountable because we can never hold someone accountable. They can only hold themselves accountable. So, you can think about what are conditions that reasonably someone can we can help someone hold themselves accountable. And what are conditions where we can't do that right now and then our reaction can be how do we protect the people who are being harmed right now? So thinking about all of the things that we can do at this moment and what do we prioritize and what do we try to do? I also wanna give Amy and suggest space to respond as well.
Speaker 5
7:30 – 7:30
Yeah. No. That's that's what I was thinking too. I would only just add that one of the frameworks that we drew on in sort of thinking about this and that actually a lot of our participants sort of independently came up with was something called victim's interests from like a criminal justice theorist called Kathleen Daley. Maybe it's Dally. Anyway, someone can correct me if they know. But anyway, it kind of outlines that there are sort of six or seven basic interests that a survivor or a victim of a harm typically has, and this is across different harms. And they're pretty obvious things like to be safe from further harm, for validation, for various things, and only a certain a small number a certain percentage of those interests are having anything to do with the person who harmed them. It's often so much more about their community validating that harm happened and saying that was a bad thing that happened to you, and it was not your fault. It was the other person's fault, and we are on your side, essentially. Like, we support you. And so there was so much that our participants told us about, especially the ones from the restorative and transformative justice worlds, but also the moderators as well that were in those sort of mass attack scenarios. Sometimes you can't figure out who the harmer is because they're hidden behind different ways of concealing their identity, but there is still a community of people that are that know the victim or survivor, and they can still help meet a lot of their needs. Not all of them, but a lot of them that don't have anything to do with the person who did harm. And what was just so interesting, I think, to us in doing this project is how much of content moderation meets zero victim needs and only just is like, we're gonna now remove this image, and that's it, or we're gonna remove this post and that's it. You haven't validated the victim. You haven't supported the victim. You haven't told them that you understand what, you know, how they're feeling or that they their their anger is you know normal or expected. There's like no aspect that's been met of that, but in fact what's interesting in reading also the restorative and especially the transformative justice literature is sometimes they'll have a process go through without involving the harmer at all. It will only involve a harmed party, and the person who's doing the harm either they're in a position of power, or they won't engage because no one is able to reach them, like emotionally reach them, maybe not physically, but in the sort of case of these, you know, collective coordinated coordinated attacks, I guess collective as well. But I think what's what these approaches teach us is to really focus on the the person who has been harmed rather than the person who's doing the harm.
Speaker 1
7:45 – 7:45
If there are no follow ups to that, maybe I'll move along the Zoom chat question queue. I think, Seth, you are next.
Speaker 10
8:00 – 8:00
Yes. Thank you. Well, first, I'm I'm incredibly grateful. In the in the community I've been publishing in. There's a a very healthy moderation literature, and I've been trying to build the case for a governance or institutional literature, and making that case has been a challenge because it it feels to this audience pretty redundant. And, you know, I felt like there's a case I can make that moderation doesn't cut it, and you just made it crystal clear. I'm, really grateful for that. It's a brilliant argument. Of course, I don't know if my you know, I'm I'm trying to replace content moderation with platform governance that could be susceptible. I'm not trying to replace it with restorative justice. It's possible the platform governance, you know, is a is a satisfying umbrella term that that restorative justice could fit in, or it's possible I fall into the same trap. I'm just reproducing the problem. It's just something I have to think about. Regardless, I'm very grateful for your critique. I did wanna ask a question just to get a better sense of the of the different parts of the system you're you're exploring and proposing. So there's a point so I so I can file a report. That doesn't I'm not instantly within your system recognized as a victim. I assume there's some process I have to go through to officially kind of become a victim, in your in your eyes in the sense of restore of triggering the restorative sort of, process. I wanted to know the first, like, where's where is that line? Do does content have to violate terms of service for me to be recognized by you as a victim and not just as a report filer? And regardless, if there are people you're you're turning back who don't enter the sort of victim stage, what do you do to stay out of the role of sort of being accused of becoming the same marginalizer that you're trying to reform in the sense of, do you is my question clear?
Speaker 2
8:15 – 8:15
Somewhat. It's a complicated question for sure.
Speaker 5
8:30 – 8:30
Yeah. I mean, I I think I can address part of it maybe. So one thing we learned from the restorative justice world that that already exists is that typically these processes don't happen unless the the person who has who says I've been victimized, I've been harmed, that, like, the person who's facilitating the process, we've already decided that that harm has happened. So if you read the restorative justice literature, one thing that people will say is that this this process this framework doesn't actually include a fact finding process. And so that's kind of external as far as I can tell so far in my reading. That appears to be prior to and external to the restorative justice process. So the question how to decide, like, who is a victim, I think it's a really complicated one. And one of the things that we came up against or that we talked about a lot in the project as well was to sort of we decided for the current thing we're working on to bracket off things that appear to be conflict or that we we don't know if it's conflict or if there is a particular victim. So in this I can tell you that in the scenarios we gave to our participants, we didn't have questions for them to answer that involved fact finding about, like, is this person actually a victim or not? We we sort of presented them scenarios in which, like, it's already clear. Like, we already know. These are not borderline edge cases. I mean, I hope this is, like, future projects for us or someone else of, like, how do you actually distinguish between a conflict and harm? Do you do two separate processes, one from each perspective? Do you have conflict mediation? I mean, there's a whole world of online dispute resolution, which probably some of you know a lot about that we are totally kinda separate from at this point. That seems to be more going in different directions, although I'm sure there's overlap that we don't know about yet. So actually, if anyone knows of interesting literature that combines sort of social justice and transformation stuff with
Speaker 10
8:45 – 8:45
Oh.
Speaker 5
9:00 – 9:00
The online dispute resolution world, I would love to know where the intersection is because I'm just starting to learn about that. But I think this this question of, like, does a piece so that was the first one part that I perceived in your question. Another part was, is a piece of content harming? And then how do we decide whether that content is harming you or whether it's the person that's harming you, and then what do you do about that sort of?
Speaker 10
9:15 – 9:15
Oh, this is all wonderful, so keep going
Speaker 4
9:30 – 9:30
if If you if you
Speaker 10
9:45 – 9:45
have something to say. Yeah.
Speaker 5
10:00 – 10:00
Oh, I mean, I just I think, like, it's it's a part that we have we but I have not figured out yet. Maybe, Neil, for our CGI have something else to add. But in terms of, like, how do you draw that distinction between someone who is upset by seeing content versus someone who is being harmed by one or more specific individuals in a more directed way? So I think, like, one of our scenarios, in was based on a real thing that happened with the Twitter journalist, the the journalist being harassed on Twitter. And so, the scenario was a bunch of, a Twitter mob was sending them pictures of ovens because the the journalist was Jewish, and so the ovens were obviously anti semitic comment. And they didn't violate the terms of service, and I think this has actually happened. Right? It didn't violate the terms of service because it was just a picture of an oven. Right? So in that case, I think you you need, like, a human, right, to look at the whole case and say, it's a journalist who's Jewish, who's getting pictures of ovens. This should violate something because it's clearly, like, threatening and abusive. But if someone just puts up a picture of an oven, like, on Twitter, there's no algorithm in the world that's gonna be like, oh, ovens. That's bad. Like, we can't have pictures of ovens. So it's it's this question of, I think I don't I'm afraid to bring it back to scale, but I I do think that it's a question that's like this group, like, demonstrates that, like, a piece of content that bothers you, you have to figure out you you have to, like, get further into that question of, like, is it directed at you? Is it just something you don't like? Like, you kind of need I think you need a human to sort of sort that out. And in in an ideal world, like, I think there'd be human resources too, not necessarily a human resource department, but the humans that are resources that can help you figure out, how to proceed with that. I don't know. Maybe, Forrest, do you have more to say?
Speaker 10
10:15 – 10:15
But may may I first just kinda respond with my, understanding of your answer? That you've you've done a lot. You've accomplished a lot. You're you're working really hard and and having great thoughts. But there's one the but you haven't solved the problem that fact finding is necessary, that it's important to, you know, believe women, that there's an inherent conflict between performing a fact finding process and believing women. And and that's sort of a area for future research. Is that a fair representation of
Speaker 5
10:30 – 10:30
What do you think, Nilfarr? Have we do we I don't think we have an answer from that yet. Right? I What do you do?
Speaker 2
10:45 – 10:45
Mariam Cava again.
Speaker 5
11:00 – 11:00
Please do.
Speaker 2
11:15 – 11:15
Who says Who says that people expect sometimes abolitionists to have a complete answer. They don't, neither does the prison system. And so it's important to think about these as extremely complex things that no one is gonna have an answer for right away, but we can take steps. And if you think about what harms are currently happening because of the systems that we have, then it becomes really necessary to think about alternatives. So I completely agree with you. I actually really like the, online, the governance framework because I think that any kind of social system requires governance and calling it what it is and thinking about it in that way is better than assuming that it doesn't exist or the platform is neutral or whatever. And I think that's one of the things that restorative justice as a framework gives us is thinking about different people and what their obligations are. So the community has obligations that are different from the platforms obligations. And in the Twitter journalist example, that I think that the platform definitely has an obligation to stop this harm from continuing to happen. So what would that look like? What would that look like in a governance framework? I think that's really important. And I don't think there's gonna be a singular system that is gonna solve all these problems. We're gonna have to think about the multitude of actors, what their obligations are and how can they act better basically. I also wanted to point out really quickly that the Twitter journalist example is not an anomaly. There's very, there's documented evidence that on groups like four chan, they actually teach each other how to circumvent these algorithms for content moderation. So it's very, it's deliberate. It's it's they're very sophisticated at doing it and they will consistently do harm in ways that don't technically break the content laws. And I don't think that there's gonna be an an way around that that doesn't involve people being involved in it.
Speaker 4
11:30 – 11:30
If I may say something, all algorithms require complex one may require people in the middle. I think that the question that was raised that is interesting is to what level you can judge if a governance can evolve such that it's going to reduce those cases and how to make a system that allows you to evolve governance because money is always going to be limited. And so you can always say somebody should solve it for me, but nobody will solve it for you. It's a it's a matter of getting good governance.
Speaker 2
11:45 – 11:45
Yeah. I agree.
Speaker 1
12:00 – 12:00
There are about three minutes left. So maybe we'll squeeze in, like, one more question, and then the conversation can continue via various channels. So I think, Lane, your question was next.
Speaker 11
12:15 – 12:15
I'm actually okay. I think it's a little bit less interesting than some of the other questions. I did notice Seth passing the baton to Ofer. I'm I'm happy to do the same if he's still got a question. Oh, sorry. He said
Speaker 10
12:30 – 12:30
No. I was doing the opposite.
Speaker 11
12:45 – 12:45
I I I defer to the next question in the queue.
Speaker 1
13:00 – 13:00
Okay. Nathan, I think that was your question.
Speaker 6
13:15 – 13:15
Oh, yeah. I I don't know if I need to ask, but I I just wanna register my gratitude for this presentation. And I think it's just such a, such a refreshing thing. So often we rely on these like weirdly authoritarian tools of silencing and exile for for for content moderation, and it's just so necessary to bring in other tools. So thank you for the work you're doing. I really look forward to seeing how this develops.
Speaker 2
13:30 – 13:30
Thank you so much for inviting us. This is such a great conversation. I learned so much. And I'm I'm really grateful for the opportunity.
Speaker 1
13:45 – 13:45
I'm also super super grateful you could join us today and your also your collaborators Amy and Shijia, it was really great to have like three people kind of like pitching with different perspectives because they made the conversation really really lively. So thank you to all of you for coming to talk with us about this, it was really really awesome for me to hear more about it. So it kind of, so since we only have a minute left I think it kind of makes sense to like, move the conversation maybe, like, asynchronously. I can, like, copy and paste some of the questions that are still in the queue into the Slack or something. I don't know, Neil, or Amy, and Suja, if you want to join the Slack, you are totally, totally welcome to.
Speaker 2
14:00 – 14:00
And Yeah. If you could send a a link to it, I'll share with everyone else. Awesome.
Speaker 6
14:15 – 14:15
I've already invited all of you.
Speaker 2
14:30 – 14:30
Oh, cool. Thank you, Nathan.
Speaker 7
14:45 – 14:45
Yeah. Awesome. Okay.
Speaker 6
15:00 – 15:00
I'll stop the recording now. Thank you all. Oh, wait. We have one more thing to do.
Speaker 1
15:15 – 15:15
Oh, the applause. Yes. Thank you to our speakers. So should we should we do, like, a turn your mics on and do a little clappy clap?
Speaker 7
15:30 – 15:30
K. Three,
Speaker 6
15:45 – 15:45
two, one.
Speaker 2
16:00 – 16:00
Yay. Thank you. Thank you. Bye. Thank you.
Speaker 6
16:15 – 16:15
Did you get all the questions from the chat? You're muted.
Speaker 1
16:30 – 16:30
I think a new one just popped up. So let me just double check I have everything.
Speaker 10
16:45 – 16:45
Yeah. I'm working on it. It's the formatting is weird. Oh, give me a sec. Your questions are great. Actually, man, this is good.
Speaker 1
17:00 – 17:00
Conversation was there are so many questions, but I didn't wanna interrupt the conversation. So
Speaker 6
17:15 – 17:15
That was fantastic.
Speaker 1
17:30 – 17:30
For a while, and then I'll interrupt. Okay.
Speaker 10
17:45 – 17:45
I post it.
Speaker 1
18:00 – 18:00
Oh, you post it? Thanks, Dan.
Speaker 10
18:15 – 18:15
I didn't do anything pretty. I I'll just count people to, like, repaste their question from there.
Speaker 2
18:30 – 18:30
Sounds good. Thank you all.
Speaker 10
18:45 – 18:45
Great work organizing and running it. Thank you. Mhmm.
Speaker 7
19:00 – 19:00
Okay.
Speaker 2
19:15 – 19:15
Bye bye.
Speaker 5
19:30 – 19:30
Bye.