Thanks for the the really kind and maybe overly generous introduction. I'm only about five years into the space and I'm I'm really focused on volunteer community moderation. So the the Reddits, the Discords, Facebook groups, Reddit Reddit singular, not credits, plural. But, yeah, I'm I'm gonna talk a little bit today about some stuff that that actually just came up yesterday. So this is very fresh and for better or for worse, you're all the the first people to to hear about this. So apologies if if this is a little rough, and I would love for you to critique me and challenge me and and all of that. So I do have a few slides. So I guess I'll I'll share a little bit. Does this okay. So the title here is cooperative responsibility and content moderation, And I'm gonna get to that, and I'm gonna explain to you what cooperative responsibility is, but I've got some other stuff to cover first. So my work has really been, as I mentioned, on volunteer reliant content moderation, which is how I'm I'm phrasing it as of yesterday. And that's gotten a fair bit more attention recently, particularly as Facebook groups have been in the news, as WallStreetBets has been in the news. I I was talking to a reporter last week who kinda opened with a question like, does volunteer community moderation work? Which is kind of a a big question, but it's one that people are asking right now because it's just not something that has been discussed in the in the news media very much. So the platforms I'm looking at are these. Of course, Wikipedia is a little bit different, and the symbol you may not recognize is Archive of Our Own, which is a really interesting story. It's a a fan fiction hosting platform, but there's some good literature about that from Casey Fiesler and and some others. But my my TLDR or, I guess, too long maybe read are these three papers. This is like the the essential Joseph. And the first two of these are based on a lot of interviews with community moderators, and I'll just briefly summarize them. The third one is what I would call kind of an aggressive literature review, and I'm not gonna talk about that today, but it's an opinion piece combo with lit review of of forty years of moderation research cut down by 30% at the request of reviewers. So imagine all the things that you think I should have said were originally in there. So the interview stuff briefly is based on a number of interviews that I did with moderators, specifically Twitch, Reddit, and Facebook groups. If I were to do this over again, I would add Discord. And some of the Reddit and Twitch moderators talked about Discord, but that was the group. And 23 follow ups after two years with those same moderators. It's about a 40% reinterview rate. The first thing that came out of that work, and this is basically the the core of that first paper, is what are the processes for community moderation specifically across these three platforms and and I did a bit of comparison. And each of these steps in the paper had a bunch of subthemes and variants, but this is a process oriented paper. These are the things that moderators do. This is what they grow go through. These are the processes. And as I was doing this work, I ran into this wonderfully amazing interesting quote from a moderator, which is this one. This is a Reddit moderator. I see myself more as a gardener kind of mod. I'm very active planting new posts and also removing the weeds, and and this is like a a super evocative quote. And immediately like, this is not the way we think of mod volunteer moderators a lot, but when you read it, you can kind of immediately imagine what that means. So I wanted to do more with metaphors, and that was the second project that I did in this line. So looking at metaphors that moderators use to self describe volunteer community moderators to understand how they see their own roles, what they use to make decisions, and maybe in the future what types of tools could help each type of moderator. And the the TLDR on that paper is this figure. And these are the I'm forgetting the math right now. 22 different metaphors and macro categories of metaphors for community moderators, all ones they use to describe themselves. So you can see some fairly familiar ones on here. If you've spent time on Reddit, you've probably seen more of the governing and maybe managing categories. Other platforms do things a little bit differently. There's a little bit of comparison in the paper between different platforms, but I like this figure especially because I think it's also pretty evocative. And when moderators see it, they tend to identify with it, be able to to self describe based on some of these categories. And, also, when people who make tools see it, they can pretty immediately start to think about, oh, like, what what if I did build a tool for a gardener moderator or a a teacher moderator? And that's been super fun and rewarding for me and and Stevie. I did this work with Stevie Chancellor who I think many of you heard from a couple of weeks ago. It's been super rewarding for both of us to see. But the main thing I wanna talk about today is what's missing from these, which is what about relationships between volunteer moderators and companies. So between, subreddit moderators and Reddit the company, Facebook groups moderators, and Facebook the company, etcetera. And that wasn't something that I initially covered really in each of those papers. So we've heard and I'm gonna come back to this a little bit later, but just a preview. We've heard some ethical critiques from major scholars in in moderation research about volunteer community moderation, reasons why they think it or the version they understand of it is unethical. And here are some my high level summaries of of theirs their arguments. So they may not think these are entirely fair, but here they are. First, it's a way for companies to shirk their responsibilities, and I I've actually heard this one quite a lot even from people on on trust and safety teams. Relying on paid on unpaid labor is exploitative in a in a labor sense. Volunteer moderators, and this is something that I've definitely heard in my interviews, can be exposed to trauma inducing content, and they don't have the same support mechanisms that paid moderators would have. Though even even those ones are not necessarily adequate. This one, which well, you can say what you will about this. Some of you probably have opinions, but volunteer moderators are less accurate than paid moderators is the assertion. And finally, basically, bad people can become volunteer moderators and and facilitate problematic communities. Now I I disagree with parts of each of these in a number of ways. And, actually, yesterday, I started writing a thing about why, But I just wanna to mention these as kind of a a foundation for the the cooperative responsibility bit. So cooperative responsibility and content moderation. So when we think about ethics in content moderation, one of the main things that people tend to talk about is stuff like transparency and accountability. And this is from the Santa Clara principles, which is like a kind of a a widely known framework for the ethics of moderation. It's specifically targeted at companies. So evaluating companies based on how they how transparent they are in in moderation decisions, for example. But there's not really a fantastic framework for specifically evaluating the relationship between companies and moderators, though I know at least a couple of people here have published in spaces that are pretty directly adjacent to that. So I have found and and, yes, I'm actually gonna talk mostly here about somebody else's idea. This framework, which wasn't originally intended for content moderation, but I think it fits very well. And that's the idea this is from a paper by Helberger, Pearson, and Powell. Platforms and users need to agree on the appropriate division of labor with regard to managing responsibility for for their role in public space. And the examples they gave in this paper were about relationships between, for example, Uber drivers and Uber the company. So not quite content moderation, but I think it does fit pretty well. So these are the the stakeholders that they talked about mostly, but I'm kind of adapting it to look more like this and focusing less on governments and advertisers for now. So they don't provide in the paper, like, a set of criteria to evaluate whether a company is doing this well, but they do give four steps to getting there. So, here are my summaries of those steps. So number one, collective definition of essential public values. So users, moderators, and companies collectively kind of agree on the core values of the space. This isn't core values of of human society. This is core values of of Reddit, for example. Second, the different stakeholders, companies, users, moderators accept that they have a role and sort of, agree on what those roles are. Third, development of a process of deliberation, so cooperative deliberation specifically. And then last, translations of the outcomes of that deliberation into what they call terms and technologies, which I'll talk about just a little more later. And I'm gonna give some brief not very good examples of how this is being done already focusing on Reddit because Reddit isn't doing all of these things fantastically well, but I think maybe the examples will help nudge our direction to talk about other ways companies are doing this well or not. These are kind of a little bit utopian in these steps are a little bit utopian in their focus, and I don't expect that this is something that companies will see and say, oh, yeah. We should totally do that. We'll get right on it. But I think it is a good framework for action. So to start, collectively defining essential public values on Reddit. Reddit has a content policy, which is developed by Reddit, but, to some extent informed by feedback from users. And we see that in, sort of ongoing critiques from users and moderators in in threads where the company talks about this. We saw, I think it was last June, a lot of protests and temporary subreddit blackouts critiquing Reddit's handling of hate speech, so some input there. The more user driven content policy, which is also not collaborative, but collaborative between users in the platform quite so much, but also it is at least recognized on the site, is Retikit, which is sort of a user driven mini unofficial content policy, which is kind of the core values of Reddit according to users. So please do and also some please don'ts. So, basically, these are, at their core, maybe collectively ish defined public values. And a couple of quotes here that I'm not I'm not gonna go into depth, but just some examples from from interviews about moderator's thoughts on these. So second, divide divided division of labor, collectively defined. This one actually is sort of directly written into the content policy. Basically, communities may have their own rules, and Reddit provides tools to support moderators in enforcing those rules. So a fairly clear ish division of labor, though, certainly, there are disagreements about where those lines are. And so just a couple of examples here of reasons where moderators kind of break their autonomy to to seek help. So that is ideally a collaborative process where on issues where one group can't immediately solve everything, they seek help. And we see right at the company trying in some ways to do this in the other direction, working with subreddit moderators to help clean up or fix problematic subreddits, which doesn't always work, certainly, but in some cases, they seem to try. The multi stakeholder process of deliberation isn't something that I think companies have really implemented particularly well. The closest we kind of see well, a bunch of quotes I have a bunch of quotes from moderators on Reddit about how they think Reddit hasn't done this particularly well. Inconsistency in rule application, lots of talk about Reddit's failures with regard to The Donald. And in fact, Steve Huffman, the Reddit CEO, agreeing with many of those critiques saying that they they had handled it poorly, but still the the process for deliberation on that front wasn't collaborative, really. And finally, this piece, translation of outcomes into terms and technologies, and that's something that Reddit certainly has had some difficulties with, though also some successes, but I'm gonna highlight the difficulties. Some of you may have been paying attention with the the kind of the fiasco that was the the one day deployment of the start chatting feature, which was a chat room that was to be embedded in subreddits that was going to be centrally moderated rather than moderated by subreddits or subreddit moderators. And moderators kind of rioted, and the feature was retracted almost immediately with an apology. So that outcome probably would have ended up better if it had been the result of a more collaborative, deliberative process. And here's the here's the the apology that resulted from that or one quote from it. So I'm a little over my fifteen minutes already, so I'm gonna be mindful in wrapping this up quickly. Few takeaways from these principles. This framework, I think, has potential for analyzing how platforms engage with different categories of user stakeholders. There are a lot of challenges in evaluating how well this works in terms of getting access to data, but also in defining these these steps or the criteria for evaluation. And it is a little bit utopian to think that for profit platforms and users could or will collaborate in these ways, but I don't think the fact that it's utopian is a reason not to push for it. So I'm gonna return just briefly to these critiques to remind you, and I would love to talk more about your thoughts on these. I'm guessing some of you have many as I do. And also just also, oops, a reminder of these, and and we could also I'd I'd love to talk more about examples of these and whether you think these steps are reasonable or interesting or realistic. So with that, thank you very much, and I'm enthusiastic to talk with you.

Yeah. Sorry. I haven't been reading.

Oh, no. No. We do questions at the end. I'm sorry, Joseph, to be off the video. Let's see. And I'm sorry. I don't want to monopolize the discussion, so I'll just pick one question for now. I've been puzzling about how Wikipedia fits in your framework. I think it's pretty well tried, so it's understandable you're not focusing on it. But that as a result, I don't know where it fits in your system. Is it representative? Is it an exception? It I mean, how where does it fit in the framework?

Yeah. It's a little bit outside, I think, what was intended by the the authors and my adaptation, I think the framework was meant to apply to collaborations between users and other stakeholders and and for profit platforms.

I think there are problems Oh, specifically for profit. Okay.

Yes. Specifically with for profit platforms in the loop, those stakeholders could be nonprofit entities and in many cases are. I think it's interesting to try to apply it. I'll I'll actually, I'll switch to that slide. I think it's interesting to try to apply it to Wikipedia to think about who these stakeholders are. I think this can apply to most any platform with multiple stakeholders, but it wasn't the original intent the framework to do it without a for profit component in the loop, if that makes sense. So I think you could take a look at Wikipedia like this, but that wasn't the initial focus.

And I guess that has a big influence on how general the framework is. Mhmm. Sure. Thanks.

That's a really good question. I think this isn't a multi stakeholder process of deliberation is unrealistic for application to every moderation related decision. So you you certainly wouldn't imagine it happening on the level of comments that Subreddit moderators are are moderating for the most part. So I don't need a panel of people to decide whether a spam or phishing link should be removed. But I think for larger decisions and more far reaching decisions, you can imagine, multi stakeholder processes of deliberation both within subreddits in kind of more participatory or democratic conversations in in those cases, maybe or maybe not actually including the platforms in those discussions. But on the on the level of The Donald, I think the one of the main issues in that case was Reddit moderators thought and eventually Steve Huffman agreed that the platform hadn't moved fast enough. And I think one of the reasons for that was a disagreement about the ethical responsibilities of Reddit as a platform. Moderators thought that the platform had a responsibility to step in a little bit more aggressively than than creditors company initially thought. So I think reaching that value set could have been the outcome of a multi stakeholder process. And keep in mind that faster in that scenario, It took Reddit years to to ban well, quarantine first and then ban The Donald. So a faster process could could have cut it down from four years to two. And I think in that scenario, a multi stakeholder process is definitely viable to to discuss and arrive at a conclusion about something like that over the course of a couple of years. Does that sort of answer what you were getting at?

I guess it was three years to quarantine. But yeah. Sorry. Go ahead.

That's a a really interesting question and one I think is kind of just starting to appear or really reappear because a lot of content moderation literature is revisiting old stuff whether intentionally or not to talk about the evaluation of what researchers are calling complex sociotechnical systems, but basically humans and algorithms working together. So one example of that would be to look on Reddit at kind of a a content moderation pipeline from the the very beginning of absolutely everything that's posted to Reddit to looking at the different steps in that pipeline, what gets removed or flagged or temporarily withheld at each step of that pipeline. And to a large extent, researchers can't see all of that without I mean, some view into what Reddit, the company, is doing. We can, with support from moderators, get a better sense of that from mod logs. And I some I know some some mod logs are public. But I think it would be an interesting project to do to try to do a full pipeline audit like that, and look at as best you can, trying to identify the flaws and biases in each step. So I guess that is one way to evaluate community moderation. I guess some of that also comes into play in the ethical critiques and particularly the the accuracy point, which I disagree with the premise of. The the person whose article I read that brought up this point said that, basically, paid, you know, company employed content moderators get it wrong 10% of the time. How much worse must that be when volunteer moderators do it? And I think we have to be very careful about just saying that there is truth in moderation decisions and being clear about where we're getting the definitions for truth when we do that. So if we're auditing, overall accuracy is maybe not the best metric and considering other metrics like bias in general. And, I mean, I I could talk for hours about what bias means in content moderation, but that's a start. In terms of tools, I think just more complete logs would be an ideal starting point. And we can get some of that. As I mentioned, some of it is public. It would be great to have some more access to datasets from Discord, for example, to look at moderation that happens on Discord servers. And I think that as a starting point would be fantastic. I don't have a an immediate answer for what tools beyond that would be good, but it would be a fun question to think about.

Yeah. I think this actually kinda dovetails with the question that Jenny, posted after mine. But and this is kind of related to a couple of the points like the the essential public values thing. So I did I did one paper a couple years ago looking or actually I guess this is a 2020 paper, a 2020 CHI paper looking at, Wikipedia stakeholder values for algorithms

Mhmm.

And algorithm development on Wikipedia. I know that one. Yeah. And so that's, like, that's, like, one example of a way that we can arrive at values is having a researcher like me or you go and talk to people and do a months long process of, like, you know, going through grounded theory, checking out, you know, what what what surfaces from this process. But that's extremely intensive and something that, you know, isn't likely to happen that often with each specific thing that gets that gets released. And so I'm just I'm I'm very curious about what you think, like, how how technically can we do a better job of understanding values or understanding or or, like, you know, incorporating these multi stakeholder processes, like, specifically mechanisms and things like that. How much have you thought about that?

Yeah. I've thought about it a fair bit in terms of evaluating the ways companies do that. And there's a lot of different angles. So there's deliberation about a bunch of different things. Part of that is deliberation about values that go into codes of conduct, but another angle is also deliberation about what tools moderators need and and how those can be designed. For example, the the chat feature, the start chatting feature on Reddit would have benefited, I think, from a a more sophisticated multi stakeholder process of deliberation. But those, I think, are things that can be approved through pretty classic participatory design methods and and more broadly user centered design.

Right.

The, I think, more messy, academically interesting and complicated processes are kind of the the more platform wide value decisions. And I'll mention the Facebook oversight board as one model for that that I'm sure you many of you have opinions about, whether that is really multi stakeholder. Well, anyway, I think another interesting case is Twitch's advisory board. I don't know how many of you know much about Twitch, but, it's a a live streaming platform, based around, mostly games. And they have an advisory board that I think last I checked had eight members, four of whom are prominent streamers and four of whom are sort of in the academic slash nonprofit space. And they don't do things the same way that the oversight board does. They don't hear cases. They have inputted to policies. And one of the reasons that they don't hear cases is because cases often involve very private situations where one person, for example, has been kind of targeted for harassment by somebody else over a long period of time and they want that not made public. So I've heard critiques of the transparency of those processes. But in making those critiques, I think we have to be careful about thinking about whether transparency is the right metric in quite the same way. So if Twitch were publishing the outcomes of those cases, there are ways where that could be harmful to the people involved. So in that case, it's kind of harder or more complicated to imagine what a multi stakeholder process of deliberation about the actual cases would look like because of the the confidential, private, potentially harmful, if released, information involved. But I think the the way that Twitch has done it so far for better or for worse, and and a lot of people, including myself, have some critiques, has been to involve that advisory board and some outside stakeholders as well in the deliberation processes surrounding policymaking. So when they're considering a new policy, they will get feedback, basically. And that is a form of this, for better or for worse. I think the the right form will vary right in quotes form will vary across platforms given the different circumstances. So I think there isn't one right way to do it, but we can critique each platform's approach case by case. Hopefully, that that starts to get your question. I'm gonna just scroll up a a ways to look at Jenny's question too. Yeah. I think I don't know if you wanna add or elaborate on that, Jenny, but I could answer what you what you've typed.

Can you restate it first?

Yeah. Sure. That's a a great and really fun question. I and I actually was just talking with Estelle a few days ago about the splintering, which I personally I've observed in Facebook groups, particularly leftist Facebook groups, which I will brief briefly summarize as a not radical leftist. Radical leftist seem to all hate each other. So they they have trouble maintaining groups for a long time. But I think one thing we notice and you you any of you who are connected with the policy kit project and and that sphere sphere, Amy Zhang at UW and and now some other folks have probably thought about or heard discussion of the failures of participatory governance in online communities, which you could see the splintering of communities as one form of. But to to summarize super briefly, one thing that happens in these communities is that moderators are kinda fixed in that they're selected and and it's pretty hard to oust them. And they tend to be representative of a core group of users, usually early users in the space. The the kind of the the founders and the people that the founders agreed with and sometimes some other folks. But what this process means is that often they have difficulty adapting to cultural changes in the makeup of the group. So when a lot of new people join who may have different values than the moderators, those people don't end up with representation in the moderation team, and that can cause conflicts and and fracturing. So the processes intra group for democratic deliberation or multi stakeholder processes tend, on the technical side, just simply not to be there. They're not supported by the platform. Moderators have to create them and be willing to let themselves get voted out. And that takes kind of a lot. It's pretty rare to see that happening organically. So I think it's fair to say to your question directly, it doesn't happen very often at the start of community creation or throughout community lifetimes. And I think that's a place and lots of people are working on this. Lots of great people are working on this where where platforms could be better designed to to facilitate that, which kind of partially answers the next question from John. Does anybody do this well?

No.

There there are people that do it. I mean, in my to my standards, no. There are platforms and groups that do it better. The the more, collaborative open projects that build their own spaces and their own governments, I think tend to do this a little bit better than the platforms or or the the ones that succeed to do this a little bit better than the

Where's your where's your standard coming from, Joseph? The non corporate space or you're like your own communities, you know, Slack and listserv or sort of a relative to what?

Do you mean my standard for these communities?

Yeah. Who who's doing it right? No one to my standard.

My standard is an an arbitrarily high bar set based on these principles and interviews with moderators and a long history of observations of failures, I guess. I don't have a great standard to set, but I'd say I have a lot of experience of complaining when things fail.

Okay. Well, so my my the I mean, call me call me too pragmatic, but, you know, since I guess any any governance style you develop is gonna have serious upsides and serious downsides. Mhmm. And, so there's always gonna be something to criticize. Am I being too forgiving? Or, have you, or, you know, is it are you kind of just going I I guess I don't know my question. Yeah. I I Are we in the same place there?

Yeah. Totally in terms of the the broader statement, they all have upsides and downsides. And I guess that's kind of to your next question in the chat. I'm sorry. I'm going a little bit slowly and talking a lot. So there's a lot of questions that I'm seeing new messages appearing. So I'll try to get to as much as I can. But yeah, one of the broader points that I that I want to make kind of as a researcher is that I think a lot of the problems of the Facebook traditional method, for example, to approaching content moderation, which is pretty heavily top down and groups moderators are kind of there but not tremendously supported. I think a lot of the failures the the visible failures of that model can be addressed at least in part by a better division of labor between the platform and users, and, of course, more collaborative process and more support for users in that. And that's I could talk for a very long time about exactly what that means, But I would say if I had to summarize an answer to to your question, all models have their failures. But if we take the best of each and try to put them together, we can hopefully arrive at something that has a less impossible task. We'll say.

Thank you.

Sorry. Amy is next on the list.

Yeah. The that's that's a great question. The best example of a company that is, in my opinion, doing this well is Discord. But, unfortunately, I can't explain how because it's confidential, ironically. But they've done some really interesting things with including users in the loop on handling major moderation issues that come up. I'm trying to I'm trying to think of this is not a helpful answer because I can't say enough to to explain what I mean. Yeah. I think ideally the the model would be upfront also related to decisions and outcomes and also in tool development, though it kind of looks different in each of those cases. But I think there is room for cooperation in all of those. And I guess one part of that which I haven't really talked about, but I think is very important is platforms supporting tool developers. So a lot of the moderation features on Reddit and especially Discord are user developed. They're not actually part of the platform so those some of them now are. But it's really valuable and I think important when platforms help and support developers because often the developers are more in tune with the needs of their communities than than the the, engineers at the companies. So, I would I know there are logistical and political challenges with this, but I would love to see Facebook do a better job at well, do kind of any job at supporting users who develop tools to to moderate Facebook groups. Right now that well, they could do a better job. So that's that's a very brief answer. There's a lot more to say about that question, but I'm cognizant of time. And yeah.

Thank you, Joseph. Thank you all.

Yeah. Super fun to be here. I love the chance to talk about this in in more depth with some folks who are really thoughtful in the area. I can stick around for