Yes. I'm okay with clarifying questions during the talk as long as they're not going to take, you know, several minutes because I find that tends to derail a good talk.

Alright. I'm going to share my screen. Can I get a thumbs up to make sure I'm not showing you a super embarrassing Spotify playlist and it's actually the slides? Okay. Awesome. You think I'm joking, but I've been in a, like, a really big, like, alt nineties, like, binge the last week and a half. So hi everybody. I'm Stevie. I just started, at the University of Minnesota as an assistant professor, and I'm gonna talk to you about some work that's, complimentary to, my work on human centered machine learning. And this specifically focuses on online volunteer moderators and how that intersects with Reddit and other several large scale communities. So a little bit polite request. The second half of this talk is more formative than the first half of the talk. So in the first half of the talk, I'll just be talking to you about a paper that's been published in New Media and Society. So you can quote that, go read it, do all you want with that work. The second part of the talk is current work that I'm working on now, and so I would appreciate if you don't quote that yet for conclusions or findings. I promise it's gonna get submitted in the next couple of months, and so you'll be able to have a field day with that once it's out, but I wanted to bring you some of the like really cool fresh research we've been working on about this topic. So as Sungyeong hinted, I do research typically on human centered machine learning. And the area that I do research in on this topic is dangerous mental health behaviors in social media. Now human centered machine learning is kind of big broad field that people have been pitching and there's a bunch of money going into centers at universities and in other places. Human centered AI is like all the rage within ML and fixing bias and all of this. The way I think about doing it is kind of a two pronged approach. I like to build ML systems that can predict when, people are going to engage in or post about dangerous mental health behaviors on social media data. This primarily focuses on suicide, pro eating disorder behaviors, and self injury. But what I also care about on that side as well is what happens when you start applying AI systems to build these things on platforms, and what happens to the meta issues surrounding those kinds of AI systems. So issues around ethics and bias, around governance, and around especially, medical conditions and interventions. I care a lot about these things because I want these algorithms to be deployed in a way that centers the perspectives of the people that they affect. So the reason that I'm actually interested in moderation and have been since, the kind of start of my research in this topic is because social media companies, and I'm sure you're not surprised, have really complex relationships with people who engage in and use their platforms to discuss mental illness, specifically focusing on these dangerous mental illness behaviors that people will talk about. Now when I say dangerous mental illness behaviors, some people are talking about struggling with something like suicidal thoughts. Some people will discuss and describe how to engage in behaviors that might lead to outcomes related to dangerous mental illness, ideations that they're having. And so in short, this content ranges across the spectrum of actively encouraging injury or threatening to harm themselves to support communities that are trying to stabilize people like suicide crisis communities. Social media doesn't do a great job at handling the nuances between those kinds of communities and basically has this, like, litany of banning and kind of quarantining strategies that it uses in dealing with this content. This is just, screenshots I pulled in the last couple of days from Reddit, Instagram, and Tumblr, which are some of the biggest communities that support mental illness, support spaces. And their way of handling this kind of varies by the content, but in short, most of it gets put behind these interstitials that you can see on the right where Instagram is asking this person, this is on a search of hashtag suicide, Are you sure you wanna see its content? Do you need help? Reddit tends to go down the route of either banning or quarantine content. Most of the time it's quarantined or most of the time it's banned. This is r slash pro ed or pro eating disorder community that was banned about eighteen months ago. And then Tumblr replaces all of the image content with this lovely little image of saying that this has been violated or removed from violating Tumblr's content guidelines. And so there's a wide strategy of how communities deal with mental illness content. And I really got interested in this because if I'm building AI systems that can predict when this content appears, moderators are immediately interested in trying to use that to decrease their loads and build systems that can help them get through content faster. And this is actually what I saw when I was working at Tumblr as an intern, in 2017. So pre Tumblr messing up their community guidelines, like, this is back when Tumblr was much more of a free space. The Tumblr moderators were actually really interested in working with me in helping them get through their queue faster. At the time, Tumblr moderators were between two and six weeks behind in getting through reports that had come through their system. And the pro eating disorder content that we focused on in particular, the mod team told us that they were really upset about having to look at this content on top of having to look at graphic violence, beheadings, copyright content that the pro eating stuff, like, really, really kind of stung, in special ways because of how graphic and disturbing it was. And so while I was at Tumblr, and there's a paper about this that I published, we'd worked on trying to come up with ways to help alleviate some of their, content moderation woes. Now the system wasn't actually adopted behind the scenes because Tumblr did a major change to its community guidelines, after I had left and the system had actually gotten kind of deployed in the back end and tested, my stuff never got implemented. But we spent a lot of time talking to the mod staff about how they wanted to balance precision and recall issues in developing and deploying AI systems. And then I got more interested in this idea of who's behind content moderation when I started working as the head mod of a Reddit community. Now I say that this community is about medium sized, medium large. It's about 400,000 subscribers right now. We get somewhere between 50 and a 100 posts a day. And I was working as the head mod over about five or six other mods. I did this for about, four or five years. And I got really interested in this question because I started noticing the ways and the models that my different moderators would use in describing the way that they wanted to take stuff down. And the kinds of things that rules changes. And so I got really excited sort of from both of these opportunities about thinking about how you can visualize and use the, labor as a model for understanding moderation. Now to computer scientists that look at the moderation as volunteer work instead of labor. And look at open source communities and, like, this free, like, software movement and all of this labor and this effort that gets put in and says, Stevie, if you start calling this stuff labor, we're gonna have to quantify its value and then people will want payment. Well, okay. I can see a couple of you smirking, right? Because this is a little silly. We've had this issue with volunteer moderators since the beginning of the Internet, right? AOL settled with, volunteer air quotes moderators in the February for giving them kickbacks and not giving them workers protections on the expectations, for moderating some of its chat communities. This kind of tension between volunteer and work that produces value for a company has been a key sticking point in moderation studies. And I think in the last five or six years with the publication of Robert's book and Gillespie's book and a bunch of other, kind of papers that have pulled this labor, argument up to the forefront, this framing of moderation as labor has actually been pretty common or common within the last, I'd say, five or six years within the scholarship. And that's why we wanted to start looking at it now given that there's a bunch of awesome opportunities to start thinking about it this way. I'm gonna take a pause here and take a look at the chat. Do do do could Tumblr or Instagram be held liable for banning groups that could help people avoid suicide or harm? It's a really, really good question. Can I table that one because it's a thorny question about responsibility until the end? Because the question is sorta, kinda, but socially, no. What resources do these platform provided have warnings or frictions redirect you? Good question. So, these platform warnings tend to, send people to hotlines. For suicide, it's the National Suicide Prevention hotline or the equivalent hotline within their country of origin, and for eating disorders, it sends them to the National Eating Disorder Association hotline. The reality is, like, most people who see these interstitials don't actually engage with them because they think they're stupid. They're like, right now, I don't wanna talk to a trade therapist who might call the cops on me for a wellness check. I just wanna talk to my friends about how I'm struggling with my binging behaviors currently. And so interstitials are almost always ignored and found to be pretty burdensome for people who participate in these communities. Volunteer work isn't labor. Yeah. That's some of the takes that I've gotten when I've kind of pitched this argument to other people. They're like, but free labor is an important backbone of the Internet. And I'm like, well, come on. We we we can we can think a little more broadly about the way that labor contributes to a safe and functioning Internet. Regarding Tumblr, what does it mean for content to be quarantined? Tumblr doesn't quarantine content specific well, okay. As of 2017, they didn't quarantine content specifically because they didn't have the quarantine model that Reddit had set up. Reddit obviously engages in quarantine more aggressively. They will also shadow ban. I don't know if Tumblr would shadow ban as well. Alright. I'm going to move on to talking about the next piece in the talk, and I'll take a quick peek at, questions in a couple of minutes. So, obviously, moderator oh, I forgot to advance you to that side, or maybe I clicked backwards. I'm not sure. So anyways, moderation is labor. How can we actually understand this, like, a little bit more intuitively? And how can we actually talk to people and understand the practices that moderators are taking in engaging with labor? Because of the communities that I'm mostly interested in, I got interested after working with Tumblr and after being a Reddit head mod myself about volunteer content moderation because there's no direct ties of money to the support of commercial content mod or there's no direct ties of money to their labor exclusively in most of these communities, excluding people who get kickbacks or other kinds of bonuses, in their platforms. So I'm gonna talk to you about two studies. One of which has already been published and the other one is current work. About two ways that we wanted to conceptualize the value of what moderators do and how this ties to our notions of what labor is. The first one is about how labor is conceptualized through the social roles that the moderators themselves use to explain what they do and why they do it on volunteer, in volunteer moderation situations and online communities. And the second one ties a little more explicitly to this notion of value that I mentioned, which is tracking and understanding the actual time and value that moderators generate using mod logs for Reddit communities. So the first one that I want to talk about, is this idea about social roles. And this paper that's, I did with Joseph Siri, Jeff Kaufmann, and myself, and it's out in Numenian society, that tries to understand the ways and the metaphors that people use to describe the practices that they take within their moderation strategies. Now the reason that we look to metaphors rather than other more explicit like, oh, I really like to do these actions, is because there's already been work on the actions that research that moderators take. But there's a lot less work in kind of the conceptual strategies, the values, or the meta kind of values and approaches that moderators take to making their decisions. I see someone mentioned that Joseph is on our schedule for next month, which means I won't totally steal his thunder on this talk. I keep promise I will keep it short. Joseph is also wonderful, and you should definitely hire him if you're looking for tenure track faculty positions. So anyways, we looked at the moderate the the ways that metaphors kind of capture people's approaches and values in moderation. So what we did, we have 79 interviews, 56 people, and these people are all moderators on a variety of different communities across Facebook groups, Twitch streams, and Reddit. This is across a bunch of different styles of communities too. Some that are more based on discussion, image sharing, gaming obviously on Twitch. And so there's a bunch of different topical focus focuses and interest related relevant for these moderators themselves. We then went through and looked for the the metaphors and similes that people use to describe the ways that they moderate. So when people used another term like I'm a filter for content, we labeled that as a metaphor because a person literally is not a sieve that catches the content that goes through, right? And so then we organized the metaphors we found, which was I think over two or 300 different metaphors into 20 or so different categories and five themes that we found in this community. So what we found is these kind of five broad topics: nurturing and supporting communities, people who oversee and facilitate fighting, managing, and governing and regulating communities. Now I obviously don't have time to get through each of these specifically, and so I'm going to show you a couple examples that I think are interesting, especially as it relates to this interest I've had about using labor as a metaphor to understand volunteer moderators and what that kind of a framing can help us understand for the value that they produce. I see one question. Lane asks, how do you define what a moderator is? Is it a narrow definition or broad definition? Yes. So the the role of moderator or community manager depends on the community. What we did is we, included, moderators in formal roles within each of those communities. So Twitch chat moderators that are in formal roles that can remove and ban and and remove and engage with content, Reddit mods in the official role position, as well as moderators for Facebook groups. So we didn't include informal moderation that could be done through social, navigation or other kinds of informal strategies. And we also didn't include, users that would send in, say, a lot of reports or notices to Teams. So we just looked at the people who were serving as formal moderators or community managers in those spaces. And this was specifically not at administrative levels. So we didn't talk to Reddit admins and we did talk to Facebook group employees. These were all people who were working specifically with these communities. So, anyways, back to the the examples I wanted to show you. The first one that I think I I love this example because it sort of gets at this, like, big cool idea of what the study captures. It's this idea that moderators view themselves as a gardener. And so this moderator described to us that they see themselves as a gardener who is active because they plant new posts, they remove weeds or the bad posts that they don't wanna see in the community. So poster comments that are very negative and damaging to the community get taken away so it doesn't harm the overall vibe. Now you imagine this gardener role, this almost curatorial, I'm tending to a big box of flowers and plants and vegetables. Compare that to the kinds of values or actions that a filter might take. A filter is focused primarily on keeping bad stuff out of the community, whereas a gardener is tending to and thinking about the way that that content interacts with the community in a different way. And when we pride at these metaphors a little bit more in the interviews, what we noticed is that people who refer to themselves kind of in the gardener style would talk about the the meta the meta values that they saw themselves as like this curatorial work on their communities. And so we saw them in contrast to filters, which I think are the easiest ones, who see themselves as just, like, cruising through the mod queue, just removing stuff because it was really annoying and really, really bad. We also saw other people who would describe their roles more explicitly with labor. So this moderator saw themselves as an employee or as a team member. This person said they wanted to do when they were trying to get new moderators to join their team was hire people with different opinions who are good at different positive things, almost like they were strategizing for a role, like a position that you post for a job. This person described hiring new moderators and that they don't often demote the ones because they're terrible, but because they got really busy with life and those moderators just kind of hang around. Now I wanna be clear that a large volume of these metaphors were given to us by participants willingly. We did in our reinterviews with some of those participants asked if they had, like, kind of a mental model, but most of these are coming from their first round interviews where they're using these metaphors to describe how they envision their communities to work. The reason that I think that this is also cool is because a lot of people kind of naturally gravitate towards these social roles that we already ascribe in our heads to have laborers. So gardeners and employees are laborers that we tend to think about within the larger community or ecosystem, Right? And so if we go back to this list of the metaphors and the moderation, what I think is interesting in kind of reanalyzing this data with a new fresh frame of labor is there's a lot of roles in here that qualify as laborers without actually being classified as employee paid laborers where you use metaphors around employment, jobs, or other kinds of compensation strategies. So things like curators, gardeners, janitors and teachers, police, governors and jury, obviously the managing community's most explicit map, but there's a lot of built in metaphors that deal with the way that we already conceptualize labor. And so in thinking about this for this talk, I actually was, like, really excited that kind of accidentally that there was a lot of this idea of labor popping up in the dataset without us even having to, like, make a category of like moderation as labor. So given this kind of overall trajectory, one of the things that we were that I was really interested in at the end of this study was okay, we've got a bunch of awesome places for mod a bunch of awesome ways that moderator self describe what they're doing in online communities. But what I was really curious about is how those self descriptions actually mapped to the actual practices of moderators within volunteer online content communities. And so when I got to Northwestern during my postdoc, I started this project working on tracking and understanding moderator logs for Reddit communities as a way to understand the actual actions that might go back and support these kind of moderator archetypes or roles. Pause here because I noticed a couple of, questions popped up in chat. This gardener metaphor has strong use of paternalism and supremacy. Yes. I totally agree with you. A gardener who's taken to the extreme can be somebody who almost overwrites the will of the community because they have a strong core vision of what they view the community to be. We also saw a similar kind of metaphor come up as, like, the curator or a custodian of, like, an art gallery, where they have strong incentive and a lot of strong control about the kind of content and community that's developed. So I agree. I think that some of these these metaphors tend to be pretty positive because they're framing the person's work as they do them. Taken to extremes though, metaphors can be obviously damaging to some people. Another good example that we talk about in the paper is that moderators who conceptualize themselves as networkers at the worst are conceptualized as the mod cabal on Reddit, and several mods brought up the cabal of like having too many, subreddits that they engage with and it's like, oh man these people have like twenty, thirty, hundreds of subreddits they're on. It's this cabal of, like, people taking over Reddit when, in fact, a lot of power mods are actually providing, like, back end support to help ma to help communities get through content a lot faster. What kind of tools are used by mods to identify with these metaphors? How might the tools? We actually don't know and that's a really important point of future work, both in figuring out how they use tools to support their roles and their values within communities, as well as envisioning what tools might look like for different kinds or, roles that that moderators occupy. So you could imagine that somebody who's being treated like a quote here, let me go back. Somebody who's being treated like a quote pinata may want a different kind of tool to assist them in not being verbally abused by their community. Whereas a gardener who has a specific vision or a facilitator might want different kinds of ways to deal with community conflicts or other kinds of like rules or curatorial decisions. We didn't ask people specifically about their tool use in part because tools are not developed based on these moderator roles or as far as we estimate and that's one thing we think would be really, really valuable. Could there be a metric like community mood that can be mapped to the efforts of the mods? Yeah. We were also wondering about how the community might view its role to mods and how they believe that peep that the community should like, the mods should behave in terms of what the community wants. And people are obviously going to have differences in what whether they want filter mods or other kinds of mods. I'll talk a little bit more about this in a second when we get into the mod logs research because there does there is some indication that there are different kinds of roles within a given community. We can see that in the mod logs. Gardein, I'm gonna pause here just because a lot of questions are coming up and I wanna make sure I get through all of the content and time. So if I see you and I see your question, I will get back to it later. Anyways, so we are really curious about some of these roles and how they were actually playing out in terms of the actions that people take. Like, do filters actually spend a ton of time just removing content? Are they, like, the removers and they're the person that everybody sees as, like, kind of taking stuff off of Reddit or the person that's banning people on Twitch that then gets harassed? So what we ended up doing is trying to get as many moderator logs for the value or for Reddit communities, as we could. And I'll tell you about how we got some of this data in a second. So I pitched this idea initially to Hamlin, who's a PhD student in TSB at Northwestern. Again, I'm gonna shamelessly plug my collaborators because they're PhD students, and people who don't have full time employment yet. She's wonderful. She's a fourth year and she's looking for internships, so if you want someone to think about data labor or the amount of labor and effort that goes into moderation or ratings, she's your girl and you let me know and I'll connect you with her. She was interested, however, in this idea of data labor and this, concept of taking like, how do people's contributions to communities end up accumulating data that platforms can use to power their recommendation systems, like ratings? And so she's taken this project in a slightly different way than I initially thought about doing it, which was mapping those metaphors to the monologue. So I'm having a there's a little bit of our both of our ideas here, but the majority of this work was done by her. And so I'm gonna pitch this kind of in her framework of what kind of value is generated, in these communities. So what I want what I think is really cool about this dataset is that we're actually able to get private mod logs from subreddits. So most of you might know that there is a bot called public mod log that is available for a subreddit to add to its community. The bot will record the mod log and then publicize it on a website where it's recorded until the end of time. Now this is a commonly used dataset, but it's got a ton of holes in it because it primarily focuses on not safe for work subreddits, people who are pretty conservative, they're anti censorship. They believe that the banning of, like, The Donald and other communities on Reddit is stripping them of free speech, and so they want the monologues to be publicly visible. And a lot of communities started adopting this tool about three years ago when Reddit started implementing more aggressive strategies for quarantining and banning communities, and so many of the communities aren't active. We tried using this dataset before and realized it wasn't actually representative of Reddit. We just had a bunch of not safe for work and conservative subreddits that were dominating the strategies for removal. And so if you imagine that you have a conservative subreddit that has more values towards, say, open speech and anti censorship, probably going to remove less content overall from their communities. So we we actually got IRB approval to get, mod logs from private communities, and we've reached out to about 150 of them. 40 have decided to share their moderator logs with us, which is really, really awesome. And it gives us a really cool insight into the ways that real active communities are moderating their sites. A couple of things that we thought were really cool, one of the power mods from Reddit actually reached out to us when they saw our recruitment message and offered to donate data backwards. So the problem is we we built a little we built a mod bot. We deploy on the community. We can only get data going forward. For the communities that consented to do that, this power mod actually gave us data back that, they had been storing, for a couple years of the mod logs themselves. And that was part of the value that, this power mod in particular had given to this community. They've been storing mod logs for years and years and years. So we're able to get backwards several, communities of data, which really enriched our dataset. And what I thought was really cool is we actually this PowerMod made some introductions on our behalf to some of the larger communities on Reddit that we think would probably not want to, engage in public scholarship or research like this. And so we actually have several communities in the top 50 by subscribers and activity from Reddit in our dataset. Now because a lot of these communities have very distinctive moderation patterns, I can't tell you who's in this dataset because it would automatically de identify probably the communities themselves if I told you who they are. But also the moderators that make decisions because moderator signatures are actually pretty distinctive amongst different mods in a community. But I can promise you that we've got a really really cool and what we think is a comprehensive dataset here about a 100 subs. So if you haven't seen a moderator log before, I found a screenshot online of what the mod logs actually look like. The mod log is just a recording of all of the actions that moderators take on a given community. Now actions are everything that a mod, like, a mod action can allow except for sending mod mail. So this is things like approving and removing comments, creating rules, banning users, all this stuff, which I'm sure, is pretty common to y'all at this point. But this is all filtered in a mod log, and you can actually run behind the scenes pretty cool analyses on what percentage of moderators are contributing to the moderation on the subreddit. This is the one of the ways that I as a head mod would decide when we needed to add new mods. Right? So moderators can see this information already. We built a bot that automatically scrapes this, and gives this information to us with the consent of the community. Now in terms of estimating the amount of time and labor, one of the things we are curious about is how much time are people spending based on these moderator logs? Now you can't actually figure out because of the mod log it only logs when the action happens. It doesn't tell you how much time was spent doing a certain action. A popular strategy that we realized in talking to some mods was that they'd head to the mod queue or they'd log on Reddit and just kinda crank through content and things that they had to do when they were had a couple, like, fifteen to thirty minutes at work and they just needed to get through stuff. And so they'd head to the mod mod, queue and just cruise through things that they needed to take care of. So we use this notion of a streak of actions as a lower bound estimate on the time a specific action would take in the aggregate case. Let me show you what that looks like from an actual mod log. So this is a mod log that I've anonymized. This person got online. This is a single mod, so it's the same person, and got online and approved five posts in about thirty to forty five seconds. So what you can see is the first post posted at eleven, ten, twenty two seconds, and the second post posted at eleven, ten, twenty seven seconds. You can estimate that that second post occurs in five it takes them five seconds to make that decision. Then the second post would be two seconds, ten seconds, and ten seconds. Now we spent a lot of time finagling with the, bracket of how long, spacing between these could be because obviously, like, somebody may take a minute to two minutes to deliberate about a post about whether or not it stays up. But the majority of posts, when they were in these streak boxes, were actually pretty consistent in terms of how fast they would go. Most mods have about the same amount of time that they take in engaging with all kinds of content, whether that's five seconds, ten seconds, all the way up to thirty seconds per post when Ahmad's kind of chewing on and deciding whether or not a post will be approved or not. And so we were able to use this to make a median, number of seconds that an action would take per moderator. And so that top action that we have no information on because it's the start of the streak, we would assume that that was going to take the median amount of time. Now I wanna be clear. Because we have to bound the dimensionality of the streaking, to say, oh, a streak can only there can't be more than, say, a minute to ninety seconds between actions. This lower bounds the kinds of actions and the length of time that it would take to deliberate on something, which we were comfortable with because we don't actually have interviewer diary data that supports the kind of actions and the length of how long it might take, say, for a group of moderators to kick around a post and decide if it breaks the rules or not or how long it might take to edit the Wiki and change rules. Right? That process of deliberation may actually be weeks and not reflected in the mod log when the change rule action takes five seconds. Right? So this is an absolute lower bound on that kind of data. We're comfortable with that since it's sort of the first presentation of these kind of moderation behaviors. But in future where we would hope to kind of supplement this with other stuff that would help understand the length of time that this takes. So given a whole mod log and given these estimates of the amount of time that they take, you can obviously sum up the, amount of time that a moderation action takes and then sum up for a community how long they spend moderating every day or every week. I've pulled five random communities from our dataset. They're not related to anything you've shown so far. And I've shown you the number of posts and the comments that appear on the community per day. The number of subscribers, both of those have been slightly fuzzy, benign, so that you can't reidentify who they are. But the number of minutes per day is actually correct. So subreddit a is one of our, most is our most active community, and they spend, per day, a hundred and fifty five minutes on these moderator actions that appear in the mod log. They get about, 3,000 comments and about 25,000 posts per day. And they're one of the most active communities that we've seen thus far. Means they're spending the the whole mod team, for instance, is, for reference, spending a hundred fifty five minutes moderating. Now what you can see is if you look down this chart, I sorted it by the number of minutes per day that they bought. You'll notice a couple things. The number of subscribers is weekly correlated with the number of minutes per day that moderator teams take. But more importantly, the number of minutes per day is not really correlated to the number of posts or the number of comments. It's incredibly dependent on community factors and the ways that these communities are run about how many minutes they spend per day on these kinds of decisions. Now, one thing I wanna point out here, this doesn't include automod data. This is all of the human labor that goes into this process. Automod definitely impacts the number of minutes that a person would spend on a community if it's automatically removing spam or low effort content based on that, site. That being said, we find that as people use the autobod, these numbers don't actually drop. They just change because moderators go and do something else that they'd rather be doing rather than putting flares on posts, removing things. They just go do other stuff. Now what I wanted to show you too is about this this point about heterogeneity. The post and the comments not really being related to the minutes per day. These are five random subreddits that we found and grouped their moderation behaviors throughout their the dataset. And we grouped them into four categories. Green is content removal, so removing posts and comments. Red is or sorry. Red is removing content, so removing posts. Green is approving content, so approving posts and comments or approving, users to post on the community. Yellow is managing content, so editing flair, distinguishing, or sticking, or locking content. And managing users appears in blue where people are muted or unmuted or banned. Now this isn't all of the actions that are available. For the simplicity's sake, this was kind of the four biggest actions we saw across all of Reddit. And these five communities are a really good example of how differently each community uses its human moderators to do work. So for instance, you can see that subreddit three, which is the second bar, has a much as a comparable portion of posts that are removed to four and nineteen. Two and twenty six, which are the first and fifth bar, however, don't spend a lot of time removing content at all, which we thought was really really fascinating. Now I'm obviously not showing you a 100 of these bars because, God, that would be overwhelming. But the short version is we see this heterogeneity across the strategy of moderation based on a bunch of different subreddits. And moderators by themselves, when you start to look at inter subreddit or within subreddit moderation, moderators exhibit similar heterogeneity patterns where some will be ultimate removers or spam removers. Some go in and are tending to user, flare. So they go in and they flare people very, very quickly, and that's the role that they occupy within these communities. One of the things we're also really interested in exploring is what happens when mob bots or auto mod are adopted. Unfortunately, because we only have a short amount of data for some of those communities and a lot of communities have been using automod for a really long time, we can't go back and like infer who's doing what and when automod got influence. But for those communities where we have that power mod donation, we can actually see the date in which automods started working on their communities and how their labor changed. And the short version is that people still moderate just as much as they did with automod. They just do a bunch of different things. So the last thing I wanna talk about with this is the idea of literal monetary values. Like, okay, you know, Stevie, you know about how much time is this lower bound that we're spending on moderating. Could you actually estimate how much this labor is worth to Reddit? Because at some point, Reddit is taking advantage of and I'm gonna put that advantage in air quotes. It depends on how you view Reddit's relationship with its mods. It's taking advantage of this labor to produce an enormous amount of value. If all of the mods left Reddit, Reddit would fall apart. So what we can do is estimate the hourly wage that a comparable moderator might be charged based on the going rates on Upwork. So Upwork has a bunch of people who will let you be hired to do moderation labor anywhere from between $6.50, $7.50 an hour lowest the minimum wage in The US all the way up to $75 per hour in terms of high skill moderators that you would want to stick around your community for a long time. So we tabulate a median wage based on the going rates for US moderators on Upwork, and we multiply that by the amount of time that they spend on the community to get an overall monetary value that could be a proxy low bound of how much time and how much money it would cost to hire an equivalent person. Now, okay, caveats, caveats, caveats. One, these are highly skilled people who know a lot about the topic and you can't totally compare them to Upwork or Laborers. But there isn't a great specialized comparison because we don't know how much commercial content moderators make. Two, extrapolating this information to from our 100 subreddits to the rest of the rest of Reddit is really, really complicated because of the heterogeneity of the subreddit. And that's where we're actually pretty stuck right now is how do you or could you extrapolate this to the rest of Reddit to give a lower bound cost of how much money does Reddit save, air quotes, by having volunteer moderators on its site rather than paying for content moderators. And you can imagine the kind of provocation that a fighting like that might cause. Right? If Reddit is saving thousands of dollars a day on moderating its content, Things start getting dicey if Reddit starts doing things that might aggravate or irritate its moderators, or takes advantage of their labor by putting advertisements in its face. And so you end up in some really interesting questions about is Reddit responsible for compensating moderators? Probably not. But that starts getting fuzzy when lots of money is being or lots of theoretical money could be translated to replace these people in the We don't have conclusions based on that and whether or not Reddit should compensate its moderators, but I think that these kinds of lower bound estimates let us get started in trying to estimate those kinds of questions. Now because I'm way over time, alright, I'm gonna stop here and just point out that I'm totally open in q and a to talking about future ideas. Ideas you can have about bashing this together with these two pieces of work or with other work that you thought of, poking holes in my argument. I'm really excited to talk to you all about the cool stuff that we've been doing, and I'm gonna stop now and let you all bug me with lots of questions. Thank you for listening.

Cool. I'm gonna stop screen sharing.

can see

Yeah. Okay. So for the second question first, because that's more straightforward. Yes. Within research, we do have techniques of detecting, mental illness within language signatures. This was first done in psychology and psycholinguistics using people's diaries that they would produce during therapy sessions. There's different psycholinguistic signatures within people's language. You can use conversations as a way to figure out whether or not somebody's struggling or might be at risk to harm themselves. There is some work in, counseling kind of conversations between people of saying, you know, I'm struggling right now, I need some help, and somebody's engaging with them, and you can use those conversations to assess risk. Specifically between people and mods, there's not a lot of work in part because those conversations aren't often accessible to researchers. So the work that has shown the kind of two way interaction has come from Crisis Text Line, which is a crisis text hotline that you can text in The US. It's often used by teenagers when they are struggling with their well-being. Now, about the question of responsibility, that's a really, really thorny question that nobody has a good answer to, and I'm sorry that I won't give you a good answer. I have hot takes on responsibility, but the reality is socially. We don't have well defined roles or responsibilities of what you and I should do when somebody is struggling with their well-being. Somebody will tell you that if you believe that someone's suicidal, you should call the police and make them do a wellness check and have them committed. That's one take on what you should do, and some people think that you should leave them alone. Those are two different takes, and they're not the wide variety of opinions that we hold on mental health. It's not clear who's supposed to intervene. If your partner presumably is struggling, you might have more of an obligation than a casual observer or a friend. A doctor has a different legal and social responsibilities for intervention as well. Platforms don't have nearly as much time as those community members and people to describe or to figure out if they should be intervening. Now I think that platforms have some but not all responsibility in trying to do something. I think the somethings that they're doing aren't that effective because they're mostly just sending people those hotline numbers which don't actually change people's behaviors, and community members have told us through interviews that they would much rather have like a mood tracker or like a symptom tracker where they could track the how sleepy they are or how much sleep that they got, or if they could track like if they had a binging episode the day before in their mood, so that they themselves could come to the conclusion like, oh shoot, I've actually been binging a whole lot and I'm going to r slash pro ed or r e d anonymous a lot. Maybe I need to back off of my decision to engage with these communities because my mental health right now isn't good. In terms of removing content that is actively encouraging violence, I think that there's a much clearer, responsibility for social networks to remove those, especially when methods are described. Those are pretty clear in the psychological literature that those can encourage other people to injure themselves because they will directly copy the method that is listed as a way for self injury or suicide. In terms of the opposite question of removing support communities, this gets really, really hazy because many of these support communities contains a range of content from support saying that you really should talk to somebody in person, see if you can talk to your therapist about that, all the way to those methods. And in poorly bound communities like on Instagram where hashtags are the ways that people connect to those, banning the hashtag bans all of that content and bans the good and the bad. I don't know to the extent of how you would disambiguate that kind of content at scale. I have work that tries to disambiguate it and it's not good enough, to be honest, to be deployed in production. The accuracy on it is high but the precision and recall is not appropriate for it to be automatically deployed. So in short, I don't know. It's complicated and responsibility in this space needs to actually be deliberated by platforms themselves similar to the ways that platforms need to have more deliberation and public input about how they are stewards of, but not legally responsible for, speech on their platform.

Oh, that does it right. I think a lot of the Reddit self moderated communities for mental health do it well. So okay. One community I can think of that I really like the way they do it is r slash, oh my gosh. I'm blanking on the suicide crisis, community on Reddit. They do a really nice job at working across a bunch of different, scales of intervention for people who are struggling with suicide, ideation, and crisis. So people can post to the community. They have strict rules about who's allowed to post. You can't post bereavement posts, which is when you're sad or upset that somebody close to you has died by suicide. And you have to be actively needing help right now and be open to taking assistance from others. They also have very aggressive rules for removal, which is you can't encourage methods and you can't encourage specifically other people, you can't egg other people on because there are people who go to that community that fetishize death by suicide. They also have one layer on top of this, and I know this from working with one of the moderators on a research project. If they believe that person is a threat to themselves or others, they do work with Reddit directly to be able to get people's information, to be able to send wellness checks if they believe or perceive there will be an immediate or actualized threat. And so they have set wellness checks, especially when people talk about harming others or going on especially shooting rampages. That being said, the community has a really awesome set of rules and a tightly moderated and really committed community that supports people, that tells people how to engage with people when they're struggling, and the kinds of language and support that you should offer people when they're struggling with suicide, about validating their feelings, not telling them that it's gonna feel better, but talking to them about where they're at and what kind of resources may be available to help them. I think that that's probably one of the best online communities that moderates this content. They have direct access to admins to take action in particularly dangerous situations. And they also do a really, really great job at controlling the content on the community so that creeps can't come in and fetishize people who are struggling with suicide.

No. And that's actually something that I know that my subreddit has struggled with. When people in my community post suicidal thoughts, mods have no idea what to do, and they're just like, oh and then I remember the name of the suicide crisis, it's rsuicidewatch. People will redirect people to Suicide Watch because Suicide Watch does a really great job at having kind of this comprehensive approach to dealing with suicide crisis that a lot of people just like aren't well trained to deal with. And I don't think it's fair to expect all Reddit moderators to take like suicide crisis trading to be able to know how to handle that. Pointing them just to our suicide watch is kind of the best answer. I imagine that you could come up with kind of similar approaches in working with Reddit admins depending on the type or content of the community. Obviously, this has come up a lot with violent speech or speech that incites sedition and rioting. I wonder if Reddit has worked with those communities specifically to come up with ways that they can escalate stuff so that it's removed from the community and actions can be taken to prevent that kind of stuff from spreading. But that also depends on whether or not you think that stuff should be publicly visible, if it should be posted, how long it should remain up, who gets to say what, and how you make those kind of adjudications, which are really, really tricky.

I hesitate to say doing it right because I personally have like a hot take on how I think I should moderate a community, but I also think that there may be more successful metaphors that help people help moderators engage with their community while also allowing moderators to have a steward role in the way that a community should be running what is allowed in the site. And I've been using a lot of Reddit because I spend a lot of time thinking about Reddit. This popped up a lot in Twitch where moderators talk for instance about a lot of the people in the comments implying or acting like that they were 18, and there being this kind of responsibility for them to moderate chat, to model good The target or pinata metaphor I actually thought was one of the more interesting ones where moderators would take the fall for, or bring heat onto them to avoid a situation blowing up. So this happened a lot in Twitch where they would make a unilateral ban decision to, like, ban certain words or certain kinds of motes, and those people seem to have a really good effect on their community because their streamer supported their decision. So one, they did a streamer backing what they were doing, and two, the community blamed them for it rather than sort of the larger ecosystem. Now that doesn't always work, you can't always be the fall guy, especially when a community is, breaking rules or harming a larger kind of community vibe or excluding people. I think different moderators and different metaphors apply in different kinds of spaces too. So another one of the moderators for a Facebook group was a trans inclusive community and they had really aggressive standards for moderating anti trans content, which may not necessarily be appropriate on a politics community, which also appeared in one of our datasets, or as well as, like, the same kind of strategy on a sports community. So while I think that there's some moderator metaphors and roles that are valuable for mods to take over time, they shift based on the needs of the community and based on the kind of community structure topic and appropriateness of behavior within certain kinds of context.

Mhmm.

So short version is I have things I like to do as a mod. They change, and that depends on my community. And so I think that for servicing this list, honestly, gives mods a place to start talking about the ways that they think about their own communities.

Yes. Thanks. Stevie, thanks for, fascinating presentation. I'm always looking to compare where we find ourselves with digital media versus predigital society, if you want to put it out that way. And we don't generally think in terms of moderating and labor in those predigital or nondigital social networks.

Mhmm.

And if we and I'm often thinking of Arendt's definitions of labor, work

And it seems to me that instead of contemplating labor, generally, we think of it as action. And instead of thinking of it in terms of moderating, we used to think, like I say, in nondigital terms, in terms of social norms and ultimately culture. So I wondered if you've come across any architectures, designs, principles that might channel digital social media as we find it in 2021 to something that feels more natural in sloppy terms. You could say it feels more like real life.

Yeah. So what I'm hearing is maybe what might metaphors of pre digital spaces that were moderated look like in the digital realm? Like like physical spaces or places where, like, labor and action intersect with things that look like moderation. Is that what you're asking?

Yeah. It's just we never I mean, when I was a kid, slightly older than you, no one said, well, who's the moderator around here? Right? Right. Communities didn't work like that. We didn't have this role demarcated. It was part of the action, part of the interaction, part of conversation. Yeah. And yet, we find ourselves with these somewhat artificial roles in digital media. And I'm wondering if that's to stay or whether you think that we're this is a phase that we're moving through until it can start to be, for one of a different phrase, more natural.

Yeah. So I think that two, okay, two metaphors of physical space jump out to me that is sort of absent of the digital space. One is the metaphor of academic conferences and Q and A around academic conferences, and the reason this jumps out is this was one of the first metaphors that was we found in the literature going back to, like, the seventies and eighties of online discussion boards related to moderation. And so this is actually one of the we opened the paper talking about fair witnesses and academic moderation as the the metaphor that was applied to moderators. And so one of the places I think that might be useful is the academic conference model where there is a person who's fielding questions. People can be called on from the audience, but there's an expectation of how the audience is supposed to interact. Now that obviously doesn't work in a space where all of those people are supposed to be talking to each other and interacting, which to me reminds me of The Great Good Place, a book I read in my PhD, which talks about the third place as something that in American society would decrease the value of third places, which are not home and work, and the third place would be your church, your bar, your coffee shop, and the place that you would go outside of that. I in my personal moderation experience, I tend to liken my role as kind of tending to a coffee shop. Right? If somebody is loud and obnoxious on a Zoom call in the corner, I'm going to go over and tell them that they're being loud and obnoxious and that they need to turn their volume down when they're on the phone or when they're on the Zoom call. Because there are certain norms and expectations that you have when going into a coffee shop. Now that being said, I don't automatically know those kinds of norms and expectations given a coffee shop. Right? A big coffee shop with two floors and tons of space may allow for those kinds of loud calls and meetings compared to a place that only has 10 seats. And so I don't know a good way of transmitting those social norms in ways that are quick, effective, and lightweight. A lot of those are learned by just spending time, say, in bars or other communities. Thank you for linking the book. I actually really like the metaphor of the coffee shop or the bar, because I think it it hearkens to a a social expectation and norms that we have. There's a communal reason why we're all there. It's because we like coffee, we like to see each other, we like beer, and we also just like enjoy spending time with each other. So those are my, those are my hot takes on possible metaphors for physical spaces, but I think that you could pull more in if you wanted to.

Yeah. I can't promise I'll get back to them all today.

Can we I don't know what your norms are in your Slack about appropriate responses for things, and so I'll have to watch and wait for a little while.

No. I I think it will be fun to chat about this.