I'll share my screen. I am one of these people that has a crazy desktop. Oh, look. We can avoid that. Okay. Good. So let me play the slideshow. Okay. Cool. Can everyone see my screen okay? Good. Awesome. So, yes, thank you, Nathan. And and, as he mentioned, I am. I'm a PhD student at the University of Maryland, where I'm in my fourth year. I've been a part time PhD student for the last, yeah, four years. And over that time, I've been working for congress up until two weeks ago and shout out to Eleanor, my former colleague who also worked with me on the Hill. So I've been sort of studying this issue and topic while actually working in federal policy making for tech policy. So happy to sort of go back and forth on those if you have questions about that. But my PhD is really centered around how do we engage, quote, unquote, nonexperts in technology policy. So what do I mean by that? Right? If you are paying attention to protests and campaigns, you'll see a lot of environmental campaigns. You'll see campaigns around homelessness, around gun control. Very, very rarely do we see protests about people's data or algorithms or injustices in data rights. And so what that means is that policy in this space tends to be really expert driven, whether it be academics, whether it be people in think tanks or people kind of who, you know, study and look at these injustices and then sort of access the policy making realm. So I work on techniques and ways to engage people who are users and impacted by tech in the technology policy process. So today, I'll be presenting specifically on a case study of a citizen panel that used a game and focused on section two thirty reform. So motivation. So why why do I do this work? I kind of already mentioned this, but technology policy is, you know, tends to be expert driven. Participatory action research methods to engage non experts tend to be pretty outdated. So citizen panels, consensus conferences, these go back to the seventies. You know, we're living in a different world now. Relatedly, participation burdens are a huge challenge and we had some of those challenges too so we'll talk about that a little bit. But, you know, the populations that are most likely to be left out or harmed by technology are the hardest to get to come to participatory governance exercises and there's a lot of reasons for that. And then relatedly, the delivery of background information can exacerbate inequalities. So typically, consensus conferences, citizen panels, they require people to do some background readings or information gathering before they show up to whatever the topic may be. And no surprise, some people come with that information and some have have not done the readings. You all are academics, so I think you can relate to that. And so that's really what we were looking at here. So why section two thirty reform? Well, this is an interesting topic that tends to be is much more nuanced than it is often portrayed as many of you know, and it's one of these things that's become sort of divisive politically and simplified to Democrats want stuff down, Republicans want stuff up. But, you know, at the end

a series of values, and that's what we wanted to dive into. So I won't spend too much time on the lit review, but, you know, we mostly looked at sort of work already happening regarding engaging users in content moderation policy. Also looked at gameplay for public engagement, which is a really fun space, and, of course, you know, other case studies of citizen panels and participatory action research. So our research question here was, can a game that builds empathy and understanding of content moderation practices be used to facilitate meaningful discourse, and lead to informed policy suggestions? So so first, I just wanted to start with, like, look at these dates. It is now 2023. So there's a little bit of context. This whole project was happening over the pandemic. And today, I'm focused on the citizen panel piece of it, but I did just wanna show that there was a whole game design process that went into creating the game to teach section two thirty, which is what we used in in the participatory governance democracy process. And I did just wanna point out that we started the project before January 6, and January 6 happened in between two of our trials of the game. And that did end up having some impact here. So you'll hear that come up a little bit. So for those of you who have backgrounds in citizen panels, you know that you often have like a frame where you try to center their participants around so that you don't go all over on a topic, especially a topic as big as content moderation. So we kept our participants focused on these three questions. So what are the values, in order of priority that should be reflected in federal legislation? Given those values, what does effective social media content moderation look like? And what actions, if any, should legislators take?

So this was our agenda for the day. So the first thing you'll notice right away is we did a citizen panel in one day, which if you're familiar with citizen panels, they typically are multiple days or a week, especially if they're kind of governed in Europe it'll be longer than that. Additionally it was on Zoom which is not a common way to do a citizen panel and is definitely something we you know reflected on on afterward. But by and large it followed the same sort of components of a typical citizen panel. So in the morning you had background, which for us instead of reviewing readings we played the game. And then you had some individual reflection, some group reflection, and group preparation for the expert panel. And so then in the afternoon, we did bring in three experts. One was a volunteer content moderator on Reddit, one studied influence operations, and one was a First Amendment scholar. So, it was an interesting group. And that allowed for participants to take what they learned in the morning, what they reflected and discussed in their groups, and then ask really kind of, like, thoughtful questions to these experts. And then the final part, which again is very typical of a citizen panel, was writing the official statement. And we will take a look at that in a moment, but that was essentially the answer to those framing questions that we we got. Typically, you would wanna get a consensus. I'll just tell you right away. We did not get a consensus because it was only a day. So we did, for the final policy recommendations, have to use votes. But, otherwise, you know, a one day session was okay. So okay. So here's the game. I did post it in the Slack. So if you're scared of QR codes, you don't have to use this one. But I do encourage you to check it out. And I have a bunch of backup slides on the design process if if anyone wants to get into that. But by and large, this this was the game. It was a mural board. Now it's in a nice, like, deck of cards, but at the time it was a mural board because it was Zoom. And, essentially, you're playing the role of a startup called Contentter, and you're told that your goal is to balance free expression and community safety to maximize ad revenue. And you're working on a team of three. And as a team, you start by setting your policies, which were inspired by existing policies in other platforms. And you kind of say, this is what we we wanna allow on content or this is what we don't. And then the next round, you're faced with borderline content that is designed to push against some of those policies and you need to be very on the edge. And so you as a team have to attempt to, you know, see if that content applies, to the policies that you wrote. And then the final round are event cards, where basically depending on the choices you made, you lose community safety or free expression points based on those decisions. And there's no way to keep all your points, and your goal is to just sort of do your best and and balance these things. I should mention all of this is based on, like, real examples, and we have, like, a whole reference sheet where all the cards were inspired by real things that many of you, I'm sure, probably have heard of. And we did use Flanagan and Niesenbaum's values at play framework to design the game. Okay. So that was the game. How did we get the participants? So we did a series of just kind of digital outreach, posting it on whatever social media platform we could. We had a very small ad budget to try to target and get some rural population and, you know, some people that will that were harder to reach. And then we had a participant application with a screener. And one of the goals of system panels is to have a demographically diverse group of people, based sort of on whatever population or geographical region you're representing. So for us, that was The United States. So we took US census data and tried to figure out what key demographics we would wanna have on a nine person panel to best represent The US. And I'll I'll show in the appendix in a bit how close or not close we got to that. But we got about 40 applications, and we did we did okay on some diversity metrics. The biggest challenge we had was that our group was overeducated. So we really struggled to get people without college degrees. And that's a challenge I've had doing citizen panels for years, but, it did affect our results, and I'll talk about that more. Cool. So data and analysis. The nice part about doing a citizen panel on Zoom is that you have all session recordings, including the breakouts and the discussion. And so it was really interesting to be able to go through, and we were really tagging sort of just expressed values. So how are people thinking about free expression and community safety as they made these choices and had these discussions? What other values came up? And then we were also tagging for signs of productive discourse. So are people asking questions? Are they changing their mind? Are they, you know, pausing and saying, oh, I haven't thought about it that way versus are they, you know, raising their voice or interrupting people in a in a way that's rude or things like that. So we're kind of looking for those. And then we also had a a post event survey where we are able to ask more specific questions around knowledge generation, how they thought the deliberation went, and satisfaction with data. K. So results. So look, it's it's a case study. It's nine people. So these are very high level of themes. But we overwhelmingly, you know, people said they learned a lot from the day, both from the game, but also from the discussion and the conversation with the experts. I think a lot of them, you know, hadn't really thought about these issues. I should I should have mentioned that from the start. I apologize. No one who participated had any expertise in section two thirty. Like, you're disqualified if you if you did. The whole point was to to not, to show up not knowing that stuff. And so, that was one of the biggest takeaways we saw. Post event, all of the experts contacted me and said I could not believe how smart those questions were from people who, you know, had never thought about these issues. So that was also very interesting. We did in terms of tagging for respectful discourse. I will say of any citizen panel participatory governance event I've ever done, this group of nine was just very, very respectful. And I don't know if that was because of Zoom or just the group we got. I don't know. But, a lot of people really it was very frequent in our data for people to say, I have never thought about it that way. Thank you for sharing that. Or, so that was also very great to see. The biggest limitation I already mentioned was that, you know, it was a bit of an overeducated crowd, and I think that came through. And it also skewed liberal. A couple of people marked themselves as republican, but I think we're more middle of the row, not quite actually Republican. So I think it ended up we ended up having a bit of a skewed perspective there. So here's a few quotes that came from people's feedback. So the first one, you know, just sort of pointing out how nice it was to sit down with a a pretty diverse group of people and peacefully discuss their opinion. So, again, that shows you that it was it was pretty peaceful. And, you know, they said that once you kinda discuss these things, there can be some mutual agreement. Again, I'm not sure if that would have been true with a different group of nine, but that's how this group felt. And then the second quote I really like too because it really speaks to one of the key goals of citizen panels that's not always captured when we do these projects, and that's this idea that the experts should be learning from the citizens as well. So typically, especially in The United States, we have this vision of like put some experts on a panel and have them present knowledge to to a crowd, and there's a bit of a power dynamic. And really the whole point of expert of citizen panels is that citizens get this really strong background, can ask really smart questions from their lived experience, and the experts should be able to to learn from those questions that are being asked from them as well. And that really came through in this expert panel, and I think part of it was because it was on Zoom. So everyone was just in a room together. There wasn't, like, a panelist sitting up in the front sort of dynamic. So, that was interesting for me to watch as a facilitator. Okay. Official statement. So this you can also find us online, and I can post this as well. But like I said, we weren't able to get consensus, which is sometimes the goal of these projects. But what we did is we took all of the ideas of things that the citizen panel participants wanted government to think about or do. And we made, like, a whole list and discussed them, and then people voted. So this is the results that you're seeing here. And I actually, as a facilitator, thought these results were quite fascinating. So the first thing that came up was comprehensive privacy law, which to those of us on this call, like, yes, this country needs a comprehensive privacy law. Right? But I think what was interesting is that the game does not the game is focused on content moderation decisions. Yes. There's a mention of algorithms. Right? There's a general awareness as you're playing the game that contenter has recommendation algorithms, but it's not the main part of the game. And so it was really interesting that this group of people, and I think this speaks to their probably skewing, overeducated, really did put it together that, you know, that data is underlying a lot of this process, and it's not just those content moderation decisions. That was really interesting. The other theme that came through a lot was just wanting better oversight of these platforms and really just thinking that congressional hearings were not cutting it. Again, maybe not a shocking finding to have. And then the next couple ideas I also thought were really interesting, specifically the improving access to voting. So, again, this was the summer right after January 6. And so there was a lot of discussion during the panel the day actually about, you know, the roles of the platform, but also just the importance of having leadership in this country that is using proper rhetoric. Right? That's not doesn't have hateful rhetoric, doesn't spew disinformation. Right? And this this idea that it's not just the platforms. Right? It's people creating the content and people in power and wanting to be able to have more say over that. So I thought that was a really interesting thing that I wouldn't have expected to come out. So, yeah, I encourage you to take a look at the other ideas. I mean, obviously, right, it's a one day case study, but it it is an interesting sort of set of ideas. Okay. Well, that's me. My partners on this project were Sarah Gilbert who's at Cornell now and Katie Shelton who's my adviser. And I have a bunch of appendix slides, but I will open it up to comments.

Thank you so much. We we've got a lot of good time for discussion. Ofer had a question at the very beginning. You you mentioned experts a number of times, but I wonder if, Ofer, you could speak a bit about what you're curious about. Oh, you're muted. You're muted, Ofer. We can't hear you.

Mhmm. Mhmm. All good questions. Let me I actually have some slides on the game design process. Oh, I can show this slide too. So this this is the actual breakdown of our participants based on trying to get to the census numbers. So you'll see, you know, left leaning, overeducated, but happy to come back to that. So experts give me one sec. Okay. Game design. So, first, you know, there was a fair amount of just, like, research that went into the game, right, that I didn't get into in this presentation. So mostly looking at, you know, Gillespie's sort of breakdown of of policies and content categories. So that was, like, the first thing we used to sort of frame. We wanted to make sure we hit all of these controversial areas, so sexually explicit content, illegal activity, cell par, hate speech. And so we kind of used the existing research to come up with those categories. And then this is how we sort of, like you said, sort of conceptualized everything. So we started with sort of the main policy areas. And, again, that that a lot of that came from Galepsy's work because he had already sort of looked at all the content moderation policies and come up with the themes. And then we did, like, a pretty broad search of examples that would push against each of these. And you're absolutely right. I mean, it was subjective. It was like me and two other research assistants, and it was like, what do we think worked the best? What I will say is we had three trials, and our first trial had, I don't know, like, 40 or 50 cards. We had a lot of cards. And the ones that didn't spark interesting discussion, we ended up kinda like throwing out. So what's in the final game are cards that overall throughout all of our trials led to the most interesting discussions. And so then what we did is we took the content. We translated it into text. Now this was partly for ethics reasons, IRB, but also it it also helped with the game because one of the key things we were trying to capture in the game was the experience of content moderation and specifically the challenge with lack of context that con moderators often have. Right? If you're living in The Philippines, you might not have context about US politics. Right? Some of that is missing. And so this helped kind of capture that too besides also being a little bit safer. So we did that. And then same thing. We looked at what kind of pushback do the major companies get when they make these decisions. So what are some real life examples of that? And that's how we made the event cards. Now in terms of, like, the number of points, that was a bit of a calibration over time through the trials. But at the end of the day, we made sure that there were an you can lose just as many free expression points as too many safety points. I think about 500 each. So you so there were as many to lose in the game. So here's, like, this.

Yeah. Well, so what we do is at the end of the game, that's when you can reflect. Right? So so we've had people play the game, and they've kept all their community safety points and haven't left any free expression points. And numerically, like, they don't have that much money because you're supposed to try to balance them, but they think they won. And that's fine. And that's fine. And, so, you know, in terms of the goal of the game, like, you know, the goal of the game was to lead to a policy discussion. And so for us, you know, the points are really just just a a thing to talk about. Right? And so once the game is played, people do sort of say, hey. That one card, we lost 50 community safety points, but I didn't really think we should have. And that's and that's fine for what we're trying to do with the game is to have that pushback of, like, you know, was that was that card really bad for free expression? And that's really the underlying concept we want people to get from the game. That's why they play as a group is so that they realize that, like, they these are a lot of people have different values here. So that's one of the main takeaways.

Yeah. Alright. And like I said, when we were doing the trials, we were testing for empathy and struggles of well, and and our ability to capture what we were calling utilitarianism, but balancing trade offs. And so we had some data on that.

I I think Mark had a question next. Mark, do you wanna speak up?

Mhmm.

No. No. We didn't do a vignette study. No. We actually designed a game that's a card game. Like, you can print out a 120 cards, and you can cut them out, and you can play with them on a tabletop. Eleanor played the first version of the card game, so shout out to Eleanor. So taking a look at this timeline, so this is this is how we did it. So in summer twenty twenty, we made what was a very, very rough draft, and I'll I'll show that in a moment, just to test, like, playability. Like, is this a playable game? Then we did our our major trials where we played with 32 players over a series of six weeks and collected a bunch of data from that experience. And then we made a lot of changes, and made a new version of the game, which we tested again that spring, and we actually coded that that data. And then we only did one citizen panel. I I would like to do more. That's that's something we're thinking about. So in terms of your question about the experts, we've only done this once with those three experts. So but you're absolutely correct. Right? I mean, it's every time you do a citizen panel, it's gonna be different. That's why they're not really, like, generalizable, and and it's a case study. But we did collect a lot of data here. The main changes we made had to do with adding elements around. So one of the biggest complaints we got from the first version was that it was all centered on free expression and community safety points, and there was no consideration of, like, bots or algorithms or fact checkers. So we added a bunch of those cards, and that did help the game quite a bit. And it just led to more discussion, which was helpful. So let me just show that for a sec. Yeah. So we went from this game in phase two, which was just deciding what was allowed in band, to this set of cards, which has both investment funds. So there was more of a money component to the game, which people asked for, and then what we called growth cards. And the growth cards really centered around, do you wanna invest in bot detection? Do you wanna invest in algorithms that have more context, or do you don't want wanna just optimize for time? Do you wanna have fact checkers? Again, this is all very simplified. Remember, this game is designed for people who are brand brand new. It's not designed for you guys. I'm just really have to emphasize that, but at least introduce those concepts for discussion. Did I answer your question?

Mhmm. Yeah. So they

far up, so they get, like, rounds of funding.

Yes. Yeah. Yeah. So at the end of the game, you have, like, the amount of money still in your bank account, and then you have your total free expression and total community safety points.

Mhmm. Mhmm.

And then we do tell you we tell you where you rank on the scoreboard, and then we also tell you sort of, like, what values you captured. But, like, there is no there is no winning. Right? I mean, it's but, like, I would argue there's no real winning in kind of moderation today either. Mhmm. Right? So it's really just discussion.

During the game trials, yes. That's what we are focused on. Like, what what parts of the game worked for you, what parts of the game it it, you know, refinement, and we we tagged and analyzed all that data. Yeah.

Mhmm. Yeah. So we this project is two papers. It's it's the game design paper, which is coming out in simulation and gaming, which talks about our whole game design process, and then the citizen panel case study, which is what I presented here today. So apologies. I didn't wanna take up a whole hour. Yeah. No. Thanks so much. It's really, amazing and and and a lot to go into. And I'm I'm curious about the, community safety that you mentioned and, like, what were the biggest challenge in defining and taking that into account? Mhmm. And then what you mentioned also around, like, data in the findings and, like, yeah, like, if you could expand more on, like, your findings around, like, access to data. Findings around access to data. I'll answer your first question first. Yeah. I mean, this is the thing that's so interesting, right, is that below free expression and community safety are a range of other values. And so, you know, we obviously knew that. We knew that would be discussed. But for the you're making a game, so you have to sort of simplify. So, basically, we just sort of looked like when have platforms gotten pushed back for taking something down. You know? And when have people gotten pushed back for leaving something up Mhmm. That was was causing harm. So it's definitely simplified, but that was one of the super cool things to see was how people really were thinking about the values that underlie both of these things. Like, why like, the discussions around democracy and why certain content shouldn't be taken down, the values around sexual modesty or protecting children or these other things that underlie community safety. So that was one of the reasons it was so important to make it a discussion game so you could dig into those pieces. But, you know, we were really just looking at the real pushback these companies were getting and and using that. And, again, all those references are available in the document. So let me see. In terms of access to data so access to data didn't really come up here. I do a lot of work on that, so I don't know if you're asking me personally about my work on that. But but in this in this conversation, even transparency requirements didn't come up too too much. They were listed as a policy option, but there were a lot of other policy options that were mentioned before that one. As knows, I do a lot of work on tech transparency issues broadly, but that did not come up in this game, weirdly enough. Mhmm. Interesting. Yeah. Thank you. Yeah.

Keep those keep the questions and discussion coming. Mel, are you are you able to ask, or or would you like me to to raise your question? You say you're in a bad audio area? Okay. Mel says, I'd love to know relative time on a player level to understand than play and maybe in game play correction time in the form of rule enforcement. So just some of those those dynamics about Yeah. Onboarding.

Good question. And this was something, you know, through the trials we had to work out. So the game that was used for the citizen panel was facilitated. So myself and two other facilitators facilitated it. We had a facilitation guide, which is available on the website, that walked through kind of the opening game story. And by having a facilitator, what that allowed is, like, those first couple rounds of decisions. The facilitator was there so that when the players were like, wait. I just say a lot of our ban or, like, what am I doing here? Like, the facilitator could remind them. We are now designing and start we will soon be play testing a virtual version of the game that will not have a facilitator. And the version of the game that Eleanor played, the printout card deck was played without a facilitator, and so Eleanor was testing my facilitation guide and gave a lot of feedback on it, which I incorporated. But overall, I mean, the beauty of the game is that it's allow and ban decisions. So you can pick it up pretty quickly, but that is a good question. Facilitators were used for the citizen panel, and now we are working on the unfaciliated, hopefully scalable, version of the game.

Good. And we have a question from, Vasco. Do you wanna, give voice to your question?

Yes. Thanks so much. I think you're on a very, very interesting presentation, and the game sounds fascinating. I just wanted to probe a bit more about the incentives so I can get better feeling for how it works. Were the players trying to kind of minimize, like, you know, risk to society from misinformation or, I guess, maximize some type of protection metric, or were they trying to achieve some, like, positive outcome? What was their main goal kind of?

Yeah. Absolutely. So their main goal was to optimize ad revenue. Oh, okay. And so to do that, we told them, and you all can argue with me if you want, but we told them that to do that, they needed to try to balance free expression and community safety points. And and, again, when we designed the game, we made sure there were the same amount of community safety and free expression points in the game. So, you know, so the game's not, like, biased one way or the other. Like, there's the same number of points you could lose from both sides. In terms of your question about, like, harms to society, so that is sort of captured in community safety, which is a vague term, but that's because we covered six content types. So community safety in the context of sexually explicit content is different than in hate speech, is different than in disinformation, which, epilepsy calls quality contributions. So so you're you're pointing out two interesting things, which is, like, what what was the community safety capturing, which was, like, a range of values that varied by the content, actually.

Super interesting. Thanks. And I had, like, a follow-up question, which was more about, like, the examples you used, were they all very hypothetical, or did you bring, like, real life examples and say, what would you have done, like, in this case, which actually happened? You know?

Yeah. All inspired by real cases. Yeah.

But we knew that people reacted to that or they didn't realize necessarily the links?

The only people who knew were when I did a few rounds of testing with my grad student lab, they knew some of the examples, but no, you guys. Listen. We are in a weird world. We follow this stuff, and other people do not. Mhmm. So

No. That's quite interesting also. Because if they couldn't make the connection, then my other question would have been if attitudes changed over time. But if they didn't even know this was a real issue, maybe not so much.

Yeah. Well and that's something that'll be exciting once the virtual version of the game is ready. I think we'll be able to, like, see more trend trends. Right? This is all, like, qualitative low end stuff. The biggest trends I saw, which probably won't be a shock to anyone here, were just gender trends. I mean K. Absolutely, like, just so striking.

The strongest. Was it women going for more safety kind of the stereotype? Yeah.

100. And And that's

very interesting to document.

Yeah. And that's what we're working on now. Because we weren't looking for that originally in this dataset. We were focused on we're we're just focused on gameplay and and democracy. Fascinating. Thank you very much.

Mhmm. Could you say more about what you're working on now around those gender issues? That sounds fascinating.

Yeah. So once we have the virtual game, we'll be able to sort of put results with demographics, which, again, through the dozens and dozens of rounds of this game I've played, I've seen I've seen the demographics show up in people's decisions. And, also, you know, shout out to, like, Roberts and other scholars who have talked to content moderators and have really looked into the struggle that a lot of content moderators have where they're basically told you know, Facebook gives them a binder, the stick of all the decisions they can make, which we don't do in this game. In this game, you're playing a startup that doesn't have one of those binders. So that is, like, a very different thing. But you're given this binder, and then you basically have to make decisions that you as a person of color or as a Hispanic woman or as a would not make. And so so we do we have some research there, but I think it's interesting to sort of when you take that binder away, what would the decisions be?

Eleanor, do you wanna go ahead?

Absolutely. Yeah. Absolutely. You know, I think a lot of them showed up that morning, like, I'm a social media user. This is interesting. But and then maybe seeing, like, a hearing or a New York Times article about Ted Cruz or yelling about content moderation. Right? Like, they had these very high, high level ideas they were showing up with. But they definitely once the game ended, pretty much everyone universally finishes the game and is like, wow. This is really hard and really complicated, which, you know, we take that as researchers. Like, good. We probably captured the complexity of section two thirty, which is which is good. And from that point, they were able to have discussions and really take what they understood as hard and challenging and complex and put it into questions. And I don't think they would have been able to do that if we had gone straight from the morning to the expert panel. And so I think the experts were expecting sort of basics of, like, what does the first amendment how does the first and, you know, intersect with section two thirty? And what they got were, like, these very nuanced questions, some of which even reference specific cards. Right? So even referencing sort of decisions they'd had to make in the game.

Thanks, Eleanor.

Great. Well, folks, keep the questions coming, but I I I wanna, kinda broaden the conversation a bit and, and just based on your experience in running these games over and over and over, I'd just love to hear some of your insight on, like I guess one way to phrase the question would be like, what would it look like if games won? You know, like, if games become integrated into how we do policy making and governance and in a maximalist sort of way. How maximalist would you go? Where would you see games, like, most useful as Mhmm. As a site of intervention? Where might you see this kind of experience, you know, potentially counterproductive?

Yeah. Yeah. For sure. So I definitely think they're a great tool for discussion. I'm certainly not in the camp of, like, we should have everyone play the game and then whatever you know, if one card was 70% of the time allow and 30% of the time ban, then that should be the actual policy. Like, I'm I'm not in that world, probably for a lot of obvious reasons. Right? Like, majority rule is not also is not always actually the best way to protect against certain harms being like the the major political science response to that. But, but also, you know, games look. Games are simplified and games and what was really interesting about this game is we use the values at play framework, which is underlying that is this theory that game designers design values into the game. Right? So we designed empathy or or this understanding of these roles into the game. We designed this, like, tension of free expression and community safety, but you all could have designed the game differently, and you could have emphasized different values. I know oh my gosh. I'm forgetting his name. He's very popular on Twitter. I apologize. Mike Masnick? Masnick? He has a game like this as well. He's a hardcore two thirty. Like, don't touch it ever. Like, I love it. And he has a game that just, like, gives a bunch of ridiculous examples so that people, like, love section two thirty. We were really trying to not do that. We wanted people to end the game thinking that section two thirty is important and complicated and that content moderation is hard. We didn't want them to necessarily even think that, like, platforms have no responsibility. We wanted to have a game that would lead to an interesting discussion about what should platforms be responsible for or not. But if I was, like, a hardcore lover of section two thirty, we would have made a different game maybe. So it's, like, a long way of saying that, like, no. I you know, because games are biased by their values that are designed into them, they could be a great discussion tool having really interesting democratic conversations, but I don't I don't think you should use the results that come out of a game to shape policy.

Let's say yeah. And it's interesting too just in that, like, policymaking itself is a pretty gamified environment. Right?

Mhmm. Winners and losers and majority rule

and all these sorts of things. And I I I guess, like, one quest like, hearing all this question I have is around, like, why why focus on the subjectivity of the of the start up as opposed to, like, the subjectivity of the regulator or other kinds of

Mhmm.

Other kinds of stakeholders. You know, of course, you got to start somewhere and and do something, and that this might be the most foreign perspective for a lot of people.

But Mhmm.

Yeah. I mean, when when you think of I I wonder what biases to start intervening when when, you know, you're what you're crafting is sympathy for the content moderator as opposed to, you know, the the victim of of attack. All

they're all sorts of

different perspectives one could take. And Absolutely. Becomes this empathy machine for whatever perspective you put the player in.

Absolutely. Absolutely. And, I mean, and this is a thing with all policy engagement games. Right? It's like what state cardinals do you pick and how do you embody them? To answer your question, you know, I think one of the challenges of section two thirty is just that people really, really don't understand it, and they confuse it with the first amendment. And it's it's just become very esoteric, and and I think it's gotten to the point where it's gotten this narrative of, like, if you don't understand like, don't even talk about online safety. You don't understand it. And that's not that's not fair either. It is understandable. It's only 26 words. It is understandable. So we were trying to really think about, like, how do we capture what section two thirty did for the Internet? And and the best way we could put that into one sentence is that it allowed for platforms to make these decisions. I'm not saying if that's good or if that's bad, but that is what it enabled. So let's start from understanding that, and then let's think about these other roles. But I think, your comments, are really interesting as thinking about what kind of questions would you wanna ask after the game is played, Like, you know, pushing people to think about those other other stakeholders. But that that's the answer to your question is we were trying to capture the essence of, like, what did section two thirty do. Yeah.

Amy, go ahead.

The citizen panel should. Sorry. Not the game.

I know. I I I went back and forth with Nathan on this. Like, should I present the game design, or should I present the citizen panel? But, no, I'm a big believer that citizen panels should inform policy.

Yeah.

you a passcode. In just twenty minutes. Yeah.

Yes. Thank you. Thanks. So the game is a game. It's an it's an educational tool. Right? It teaches section two thirty. Where it came into practice with citizen panels is and I don't know how many of you have experience with citizen panels, but citizen panels have its formula going all the way back from the seventies, a three part formula, background info, discussion, and prep and preparation for an expert panel, expert panel, and then and then writing the official statements. So sorry. Four four parts. What we did here is we designed a game to teach section two thirty and to give people an experience of the challenges of content moderation and the complexities around the values that underlying con content moderation. Then what we did is we took the formula of the citizen panel. We took out background reading, which most people don't do and is problematic, and we replaced it with a game. So the game is the background reading. So people showed up to the citizen panel that morning on Zoom having no background on section two. They we I mean, we didn't even tell them what the citizen panel was about. They knew it was about social media, but that that was it. And then they played the game and then did the rest of the citizen panel exercises and the democratic discourse and then wrote an official statement. That official statement, I do I would like policymakers to reference. I don't think that they, like, look at the first thing and be like, oh, that's what we're gonna do. But I think they should look at it the way they would look at any white paper that comes from a think tank, any academic paper. Like, I think it should be a part of their knowledge generation when they're creating policies. Did that answer your question?

Yeah. And that's what Nathan was was asking about a little bit earlier. I I don't like that, mostly because especially when we're talking about data rights and human rights, majority rule is not right. And so in a game like this, where the decisions that are most commonly made are gonna be majority rule decisions, I would have concerns about that. So I would rather games be used as a discussion tool than the outcomes. With that said, our future work is to be able to play this at scale, and, you know, we will obviously see the trends that emerge from that. I don't think I would ever say, okay. Great. These are the trends. Let's make a platform. But I'm just maybe

Ascent, go ahead.

Great. I'm curious if you could talk a little bit maybe about this kind of goes back to Nathan's question about, like, the the starting subjectivity, like, the goal, the premise of the game. And I I see in this this, website, overview with all of the documentation that there's a making and doing one where you can make your own. And that that seems to be one pathway for kind of grounding your own subjectivity or even kind of torquing the game to do the kind of thing that you're maybe find problematic. Like maybe you could sort of construct it in a way where you can actually have non majority decision making and then the kind of results that you get are kind of more less problematic in that sense. But I'm curious to hear like is like the basic premise of startup having to interface with advertising and regulation, like, still at the core of how this making and doing template version works? And then as a sort of addendum to that, I'm risking asking like a two parter, but, you know, you you mentioned that, like, the kind of the advertising is one of the things that really tempers the decision making, like, with the kind of growth events, that you have in the game. And I'm just again, this is more of a game design stuff. But I'm curious, like, how people responded to that parameter. Because in the facilitation guide, advertising actually comes up very, very little. It's only in in there. The the words are only used once for FOSTA. And but so, like, I'm just curious, like, it seems like the thing that was oriented in the discussion and the panel at the at the assembly was about the experience of content moderation and less about the experience of advertising. And Yeah. Just curious to hear, like, how people responded to that kind of parameter or boundary. Sorry.

The Yeah. Very good questions. Okay. For first one first. The making and doing session was actually designed for four s, which was in Mexico this year. And the idea around it was this game was designed for The US context, which I I should have said at the beginning, so apologies for that. So all of the examples are US examples. Section two thirty is obviously The US liability law. And so when we went to four s, which is a very international conference, we decided it would be fun to hear from people from other countries and have them create examples of controversial content that they've seen in their countries and events and pushback that platforms have gotten in their countries. So that's what that making and doing session was, and it was really, really interesting. And I haven't gotten to write up the cards that got created there, but it was really cool. So that that was what the point of that making and doing session was, but you can imagine wanting to make cards for other reasons. So I think that's great, and that's why the templates are there. And then your second question Oh, I had thoughts on it. Can you give me, like, three three sentence three words? Advertising. Advertising. Yes. Thank you. Okay. So that's what I needed. So, you know, it's funny. The first version of the game didn't we didn't talk about advertising at all. And what happened was a bunch of feedback we got during that trial was like, what is winning? Like, it wasn't didn't even even though we told them we told them. We were like, try to keep your free expression community safety points balanced. But there were a couple complications. One, especially in the first round, there are some false assess lawsuits that happened depending on your decisions around child pornography or naked children. Right? It's border lines blurry. And so and there was, like, an ag gag lawsuit at one point too. So there actually were some cards that were more tied to finances than community safety and free expression. So it kind of made sense for us to have the money anyway, and a lot of people just gave us feedback that, like, having a money component would just help with the gameplay because there would be, like, a score. Now how did this impact players? Honestly, not that much. Like, there were a couple people who played the game who were like, yes. I'm gonna try to get the most money at the end. But a lot of people just got really into it and just wanted their values to be expressed on Contentter. Like, I'll never forget. We had this group of three women who had, like, ended up having, like, social service backgrounds, which was I didn't know that was gonna happen. And they were just so proud that they kept every community safety point. Now, listen, they had, like, no money at the end, but, like, they did not care. Like, you know, they would see their balance throughout and then, like, just didn't care. Like, so and Eleanor was nodding her head too. Look, it's like you can play to optimize the money, but you can also just say, this is my one opportunity to create a social media platform, and I'm just gonna create it the way I want. And people did that.

Yeah. That's fascinating. I have a lot of follow-up questions I wanna ask, but, you know, I'm gonna see it because we're out of time.

But we are out of time. Thank you so much for a great discussion, everybody. Thank you for participating. And, as as usual, we will conclude with a live audio applause. So please prepare to unmute in three, two, one.

Thanks, everyone.

I think there is, like, some fireworks along with the applause this time. Thank you so much, Anna, for joining us. That was fascinating, and and I hope you'll stick around in the community going forward.

Yeah. For sure. I'm excited.