You're spelled on. Yep.

Absolutely. Well, as I mentioned, probably ad nauseam, you know, thanks again for having me. So pretty much with this talk, I'm gonna be talking about how our study investigating how to better understand, which moderation systems the public views as the most legitimate. And so as a example, really, to get, you know, our brains going, the Supreme Court has made controversial decisions before but has historically maintained a reservoir of goodwill. So an example of this is that in 1982, the Canadian Supreme Court added the Charter of Rights and Freedoms to the Canadian Constitution, which increased the Supreme Court's power to intervene in policy. While the move was controversial at the time, in the long term, the Canadian Supreme Court was able to maintain strong support from the Canadian public. And so this suggests that these courts benefit from a reservoir of goodwill that is essentially considered to form the foundation of legitimacy as postulated by political science professor James Gibson. When online platforms try to make similarly controversial decisions, we see that they don't benefit from this reservoir of goodwill. So for instance, according to a poll by the Cato Institute and YouGov, 75% of Americans don't trust social media to make fair content moderation decisions. And this same poll actually found that 60% of Americans want more control over the post they see. In addition, another poll done by the Pew Research Center found that 73% of adults believe that social media platforms are likely to censor political viewpoints that they find objectionable, and this belief was common across Americans of all political leanings. So naturally, there's a question of why is it that social media platforms don't get the same reservoir of goodwill compared to courts such as the Canadian Supreme Court? And so has suggested that these issues are symptoms of a lack of perceived legitimacy in content moderation systems in online platforms. And so there are a lot of types of legitimacy. In this talk, we're gonna be focusing on perceived legitimacy, which is defined as the acceptance of authority by those who are subject to it. Perceived legitimacy is important because we have empirical studies showing that institutions with high amounts of perceived legitimacy have greater acceptance of unpopular decisions, more cooperation, and more compliance in the long term. Now how exactly does one calculate perceived legitimacy? In our survey, we calculated it from five aspects. First off, there's outcome satisfaction, then there's trustworthiness of the process involved, fairness and impartiality of the process in question, institutional commitment, and then finally, decisional jurisdiction. Some of these terms may be new, so I wanna explain some of the more novel ones. So first off, outcome satisfaction is the extent to which the poli or the participant or the respondent is satisfied with the way that the moderation process handled the decision. Institutional commitment is defined as the extent to which the participant wants the process to continue making, such decisions. And then finally, we have decisional jurisdiction, which is the extent to which the poli wants the process to maintain its scope of decision making powers. So for instance, low decisional jurisdiction means that the poli wants the process to make decisions with smaller impact. In some ways, you can think of it as demoting the process. On the slide here, essentially below, each of the five aspects are the statements that we ask to assess the aspect for legitimacy within the survey. Alrighty. So before we continue down in this presentation, we'll do a brief snapshot of what we'll be discussing for the rest of it. So first of all, you know, as a high level, we investigated which moderation systems the public views as most legitimate. And so how did we do this? We conducted a within subjects experiment that measured the perceived legitimacy in the context of randomized moderation decisions. Our findings are twofold. First off, expert panels have the highest legitimacy, and second, the agreement with the decision played a larger role than the process itself in determining perceived legitimacy. So taking a step back, let's talk about which moderation processes we investigated. So we analyzed four content moderation processes, and they represent both conventional as well as emerging solutions of content moderation. Pretty much paid contractors and algorithms represent conventional methods, while the expert panel and digital jury are emerging solutions that were designed with the intent of building greater legitimacy. With regards to what we were investigating, we were investigating two research questions. First off, how does the type of moderation process impact the perceived legitimacy of content moderation? And second of all, how does the outcome alignment of a moderation decision with personal beliefs impact perceived legitimacy of the moderation process? Now let's talk about our hypotheses involving the process. We hypothesize that the digital jury would be perceived as the most legitimate moderation process and then that it would be followed by the expert panels, then by the algorithm, and then finally the paid contractor. This comes from the fact that digital juries and expert panels have more independence from platforms, and independence has been correlated with perceived legitimacy. In addition, digital juries strongly embody the concept of democratic representation, which contributes to higher democratic legitimacy. And then finally, prior work shows that overall, algorithms are perceived to be more trustworthy than humans. Now with regards to outcome alignment, we hypothesized that the decisions that align with users' beliefs would be perceived as more legitimate. And so this comes from a whole slew of research that kind of summarized really to show that confirmation biases can motivate reasoning, and then humans have other cognitive biases that are strongly mediated by partisan identification. Alright. So now let's move on to study design. So with our study, we surveyed a 100 US based Facebook users and that we source from Amazon Mechanical Turk. Here, we have a little bit in terms of their demographics. Overall, in terms of political leaning and gender, they were more or less evenly spread with essentially kind of independent participants being a little bit overrepresented as well as gender skewing a little bit more female. In addition, we have some more demographic information. So here we have age as well as education. And so the age is relatively even, although the 18 to 24 year old segment is a little bit underrepresented. And then with regards to education, education skewed a little bit more highly educated than the general US population. Now with regards to this survey design, what we did is that we conducted a within subjects experiment where we showed participants a selection of real Facebook posts that were matched with a randomly selected moderation process, so one of the four that I talked about earlier, as well as a randomly selected moderation outcome, so whether or not the post was left up or taken down. The posts that were shown were real Facebook posts, and they were selected for potentially violating either the hate speech, incitement to violence, or misinformation on sections of Facebook's community standards. However, we don't know whether or not there was a moderation decision made by Facebook, and if so, you know, whether or not the poster to left up or take it down. Now to give a little bit more color to this very broad and abstract stroke that I had talked about in the previous slide. We're gonna show an example from our survey. So over on the left here, we have a post, and then the participant was asked whether or not they believe the post should be left up or taken down. After that, the, participants were then presented with a moderation decision made by a random moderation process, and then they were asked a series of questions about how they felt about that decision. And so this is essentially how they feel about it. And then finally, participants for the individual post were asked to rate the moderation process on fairness, impartiality, and trustworthiness. After showing all of the moderation processes through different posts, at the end of the survey, the participants were asked essentially comparative questions about their perceptions of the four moderation processes. These comparative questions are listed right here, and they are really summarized in terms of, you know, which moderation is the sorry, which moderation process is the most trusted, least trusted, most fair, and least fair. Now let's go into analysis and findings. So overall, we analyzed the survey responses we collected with a linear mixed effects model. This model estimated the perceived legitimacy using process, outcome alignment, gender, and political affiliation as explanatory variables. We also corroborated our quantitative results with qualitative analysis of the free response questions that I had just shown in the earlier slide, where we asked the participants to compare the different types of processes. Overall, our findings indicate that the expert panels benefited from greater perceived legitimacy compared to other processes. And so this is shown with the quantitative model, as you can see highlighted blue, as well as supported by qualitative feedback given by participants through the free responses. With regards to outcome alignment, we saw that outcome alignment, I. E. Whether or not the participant agreed with the moderation decision, was an even stronger factor than the moderation process itself in determining perceived legitimacy. This becomes really evident when we overlay essentially the effect of process outcome alignment on a graph over the process effects. And so this is the blue showing the, outcome preference alignment. And overall, we saw that outcome preference alignment was responsible for 27% of the observed variation in perceived legitimacy. Now for some other findings, we saw that participants strongly preferred group decision making to decisions made by individual moderators. As we saw that either that 42% of respondents expressed that either, you know, a single person makes more biased decisions or multiple people can help mitigate bias, which effectively is essentially two sides of the same coin. With regards to another finding, we also saw that algorithms were not seen as an adequate substitute for human moderators even though they were considered very impartial. You can see this with regards to the percentage of participants that rated the algorithm as the most partial. You know, it beats the rest by a lot with 51%. However, we saw that a third of respondents considered the algorithm to be the most untrustworthy process. And so a really good quote that was shared by a participant was that the least trustworthy would likely be the algorithm due to the complex nature, nuance, and context of the human language. Algorithms cannot navigate the complexity and subtleties of our communication. Now let's move on to discussion. Overall, it seems that mistrust appears appear to override the democratic benefits of digital juries for our participants. At the same time, trust can be created or lost at many points along the way since there are many factors such as jury composition that may lead to greater or lesser trust. And we've also seen this in previous works such as work investigating, you know, effectively digital juries in, situations such as legal legends. In addition, perceived legitimacy is vulnerable to a lack of understanding of the processes involved as well as misconceptions about them. And so here we have two quotes essentially asking about what exactly are we assessing and showing a lack of understanding of what the processes entail. Finally, in a polarized environment, it can be extremely difficult to create broad legitimacy for any moderation process because strongly held beliefs may place a cap on perceived legitimacy. It's also important to note that these beliefs can be shaped by many factors that include community norms, individual experience, in group versus out group dynamics, and finally, prevailing political discourse. So now let's talk about some more novel recommendations. First of all, we saw that perceived legitimacy is vulnerable to misconceptions or lack of understanding. And thus, explaining decisions can help improve feelings of legitimacy in part through outcome alignment. Let's see. And so these ideas have been surfaced before in other research such as research by law professor Tom Tyler. And so examples of explanatory variables can include additional context. For instance, what were the facts that were considered, who made the decision, as well as why the decision was made in a particular manner. Our second recommendation comes from the fact that beliefs are shaped by individual experiences. We saw that survey participants often justify their belief in the trustworthiness of a process based on examples from the survey. Thus, platforms should create or highlight positive experiences with content moderation, and ideally, this should be done at an individual level. So to put some more color into the subject suggestion since I imagine it can be quite abstract, Let's dive into an example. So imagine a user is dissatisfied with the content moderation system, such as, you know, the user had one of their posts taken down or they were interacting with a post of someone, who has had their post taken down. And then in this scenario, platforms could show users examples where the user is likely to agree, such as providing uncontroversial examples of post remove. This could then help the user build trust by seeing the moderation system working well and showing that the moderation system is not necessarily biased against that user and their viewpoint. In addition, another example is that there has been other work that's recently come out, involving end user audits, where, essentially, after the end user went through examples, essentially moderated by a particular algorithm, they came out with, a feeling of actually higher trust in the algorithm. Now let's talk about future work. So there are a lot of areas that can be investigated further, and so I would say they really fall into four categories. The first category is about investigation of other types of processes. As you may recall, we only investigated four types of moderation processes, and so one type that was not included in this investigation was essentially community moderation. In addition to that, in real life, moderation processes are not just we have an expert panel and the expert panel is the one that decides whether or not this post is taken up or down, but they are involve essentially tiered set of processes. For instance, starting out with an algorithm, then it's essentially then followed up by, like, an individual moderator, etcetera. And so we didn't investigate those types of processes, and it is worth investigating whether or not, you know, these changes no effect, essentially, the results. Second, we also saw that in the literature, political affiliation and political debate should have an impact on perceived legitimacy. However, we didn't see essentially anything statistically significant in our study, and so there should be a further investigation to see the impact of these two variables with regards to perceived legitimacy. Then in terms of the third one, we have essentially there should be further work investigating how to further incorporate expertise into content moderation because at the end of the day, expertise is something that does provide greater legitimacy, but there are also practical considerations to take into account. And so this also does tie into a little bit with regards to the first set of areas that could be investigated further, really with regards to the tiered processes. And then finally, there is essentially open an open area to investigate kind of ways to increase outcome alignment that therefore increases perceived legitimacy. And so that's essentially very much tied to the recommendations I talked about a little bit earlier. Alright. So, ultimately, and we our study investigated which moderation systems the public views as the most legitimate, and we found that, first of all, experts have the highest legitimacy. And then in addition to that, agreement with the decision plays a larger role than the process itself in determining legitimacy. And from that, we recommend that platforms incorporate more transparency, create or highlight positive content moderation experiences, and also continue to visibly incorporate expertise into content moderation. Well, thank you for listening. And what questions do you have for me?

I see. So I think maybe just to make sure we're on the same page, are you thinking of let's see. Where's the algorithm one? This slide?

Okay. How about I I'll go forward and so just let me know which slide you were you got that question from.

Oh, I see. Okay. Yeah. So, essentially, the this slide showed our hypotheses.

And so

Yes. And so, essentially, this is a this is a very broad statement. But, really, what we saw in the literature is that depending on the context, algorithms can be perceived as more trustworthy than humans. And so there are other contexts in which algorithms are perceived as less trustworthy than humans. And so one of the things that we were looking to get out of this was essentially to see kind of which of the two areas do algorithms kind of fall into with regards to content moderation. So an example of where an algorithm was perceived to be less trustworthy than a human in the previous literature was there is essentially an a situation where an algorithm essentially had to have a conversation with the human, and then they were being rated on how trustworthy they were. And so the algorithm, for instance, in that case, was perceived to be less trustworthy. Does that make sense?

Mhmm. Okay. Let's see. Also, Amy, let's see. You you had a comment here. So my hunch slash understanding is that if you pull the general user base of social media, they're more inclined to take down content than expert or paid moderators. How does a square with the goal of increasing outcome alignment?

Yeah. So this is a great question. I I think kind of within it, you know, the base of the question is really this understanding that the general user base is more inclined to take down content. And this may this may be the case with regards to the majority. At the same time, there is a pretty vocal minority, and we saw this in the free response, where essentially people would have this opinion of, you know, we can't take this down because I support, like, free speech. And so, I think, quite frankly, this is one of the tensions of, essentially, outcome alignment because, you know, especially with regards to controversial, you know, posts, naturally, you're gonna have two sides that have very different opinions with regards to how they want this post to be treated. And so I think in the case of increasing outcome alignment, you know, those more controversial cases may not necessarily be a good place to do so. And so that's why, with regards to the recommendations, I essentially highlighted that, you know, there are less controversial, you know, decisions that are being made, and so you could potentially essentially, increase the perceived legitimacy by kind of highlighting those cases so that, you know, people don't walk out with the perception of, oh, you know, this moderation process is always against a particular, you know, point of view or is always biased. Does that make sense?

Not quite. I I think really your question at the end of the day very much highlights, like, essentially the let's see. Trying to find the slide here. The point that at the end of the day oh, no. I went too far. Sorry. Trying to multitask here. Essentially, the the strongly held beliefs can place a cap on perceived legitimacy, and so there's like, there's a there's only so much that you can do with regards to outcome alignment. And so, essentially, I was trying to highlight that there are air other areas that are a little bit easier. For instance, like less controversial areas that you could essentially try to increase outcome alignment there to essentially affect the overall perceived legitimacy. Does that make sense?

Yeah. Thanks, Amy. Congrats, Christina. This is just really fascinating, extensive work, conducting this, really extensive experiments. Yeah. Really cool work, and I'm excited to read the whole paper. And I was curious, with regards to exactly on these slides, you have community norms. And I was wondering, in practice, like, accounting for the complexities and potential controversies within community standards, norm, and policies, and how this affects, the work that, the content, moderation contractors, right, the juries and, the algorithmic content moderation and the panel are doing. Right? Because how do we account for the these different kinds of community norms in the process of measuring legitimacy? And for example, I'm reminded of when I was working in quantum moderation, I remember there were, like, maybe more than 40 different kinds of, harm. Right? And you mentioned here, I think, bullying harassment and terrorism and violence and incitement in terms of, like, what you focused on. But I'm wondering about can you overlay the different categories of, kind of policy violation, with this, perception of legitimacy? Right? And in this way, sort of contextualize legitimacy in the context of the particular violation of a policy or kind of harm that people are experiencing.

Okay. Sorry. So I I'm not sure I quite got the question.

Like, I'm wondering how do we account for the, potential controversies in the, I guess, the mechanisms that govern how content is taken down. Right? Mhmm.

Okay. Let's see.

Like, the actual policy is When you account

for the standards. Yeah. I think that this how how do I put it? This in some ways ties to the point I make about incorporating more transparency into the process because, you know, with regards to show social media, content moderation, there have been a lot of controversies. And so, essentially, by being transparent with regards to you know, we've seen historically if you do x, then, you know, we've had this reaction. And therefore, taking it into account, kinda actually be a way to essentially show that, you know, at the end of the day, the there's a lot of work going on to essentially keep to walk that fine line with regards to content moderation. And so I think in some ways, you know, especially if the user base is well aware of particular controversies, it could be a way to actually increase perceived legitimacy. Does that make sense? Like, there's there's an opportunity there, although, you know, it is a very tight line.

And maybe I'm wondering if legitimacy could be higher in some kind of, let's say, misinformation

Mhmm.

Versus building a craftsman. Right? Versus or like in if if it makes sense to kind of look at it in this sort of contextual level and then the particular sort of community standards that how do they impact the perceived legitimacy?

Absolutely. I think that's a that's a great point. I think at this point in time, especially with regards to the types of moderation processes and also investigated Facebook where there's not a strong distinction between the types of harm when you're just, like, kind of going through Facebook on a day to day basis that there is a lot of modeling. And so I I do I I'm inclined to agree with your hypothesis that if you were to try to split things up, like, for instance, with regards to misinformation or, for instance, like, incitement to violence, that, you know, you can see different, essentially, sets of perceived legitimacy kind of factored on those different, cases. Does that make sense?

Yeah. Yeah. Thanks.

Yeah. Thanks. Well, support today.

Absolutely. And, also, would you like to talk about kind of that second piece to the comment about essentially the the traumatized kind of content moderators?

I just shared that investigative journalism article.

think it's, just recently came out. Mhmm. And maybe perhaps that's part of the future work that you were mentioning. Like, I know considering how investigative journalism, right, could factor perceived legitimacy.

Absolutely. Yeah. Mhmm. And also this also highlights other practical aspects that you have to consider when designing content moderation systems because, you know, not only are there factors of, like, you know, how legitimate is the system being considered, but also, like, you know, can you do it at the speed or at the scale that is required as well as, you know, considerations such as, like, you know, how do we protect the people who are involved with these processes so we don't, like, traumatize them? And I I I totally resonate with regards to that because a lot of the posts that we went through when trying to essentially get post for their survey were were quite a bit. So, you know, I would have to, like, kind of walk out and just, like, take a break and, like, go like, okay. You know, that was, like, gnarly stuff And, you know, being exposed to that, you know, on a day to day basis, at such a large scale, naturally would have a major impact.

That's a great question. So okay. Thinking back, it's also been a while since I've seen the free response raw text. Thinking back on the participants' responses I think with regards to domain of expertise, there wasn't a lot really said there. There are more questions about, like, what exactly does expertise mean? And then there were also a lot of calls for diverse viewpoints with regards to kind of all of the content moderation processes, and so experts are included in that. I would say kind of judging from other other sources of information, for instance, with regards to some of the controversies involving the Facebook oversight board, to me, it seems like there's a a lot for a diverse set of experts. And so, like, for instance, there's a lot of criticism with the Facebook Oversight Board with regards to how they come from a very legal background, and so there wasn't as much expertise with regards to how to prevent harm as well. I don't think there is as much with regards to kind of experts with regards to free speech since I think that was covered with regards to the legal expertise. Yeah. That's why that's my 2¢ with regards to it.

Yeah. It's a it's a great question. Let's see. I think the things that this calls to mind with regards to our exploration of the literature is that we've seen that expertise, like, con essentially gets cons solidified over time. And so with regards to, essentially, content moderation, because, you know, we're pretty early on, there's there are more open questions with regards to, you know, what does it mean to be an expert. Because, like, for instance, with, like, law, you know, it's pretty clear what it means to be an expert in the law. Like, for instance, you know, are you a lawyer? Did you go to law school, etcetera? Those are kind of major indicators, but we're we're earlier on. So there's there's a lot more of an open question with regards to that. But I think it's Yeah. Go for it.

Mhmm.

No. You actually got me at the end, so it it worked out. Cool. Let's see. Also, Isaac, thank you for sharing that, essentially case about the Kenyan courts and meta. Okay. Cool. Let's see. Alright. So I'll also just read Seth's comment here. So that's a great response, Amy, because in a way, someone accent. Oh, sorry. If you just talked about it, that's totally cool. Let's see. Any other kind of questions, comments, or things that you would like for us to discuss about?

Maybe a related one around trust and safety. Like, how would you imagine, I don't know, for trust and safety to evolve kind of based on the work? Like, for trust and safety teams to incorporate it.

Sorry. I didn't caught I catch the first part of the question.

Yeah. Kind of building on what Amy was saying about the way thinking about the trust and safety space

to

this work in practice. Like, I wonder yeah. How do you imagine for I mean, of course, you give the recommendations here, right, on, like, how they can adopt it. But I was wondering, yeah, what are trusted safeties teams? I know there was this recent conference, like, from their perspective, like, practically, what do you think that could be the impact within that space?

Yeah. I think that, you know, there is a question of, like, you know, how trust and safety teams can essentially advance this, like, solidification of what exactly does it mean to be an expert within kind of content the content moderation system area. I would also note that, you know, there's there's been a lot of emphasis in terms of designing content moderation systems that, you know, essentially at the end of the day, you know, are as legitimate as possible, kind of barring consideration with regards to how much people agree with the particular, set of decisions. And so I think there is actually room to essentially see and also kind of experiment with how, you know, we can essentially change how people perceive content moderation systems overall. And so that's why I like the recommendations I talk about creating or highlighting, you know, positive content moderation experiences because for instance, not all, moderation process that's not processes. Decisions are controversial. And so being able to highlight that, hey. You know, you're we're talking about the most controversial decisions, but look at all of the stuff that we've done that is not controversial at all and, essentially, highlighting that, maybe to essentially increase the overall legitimacy of these processes involved. Another thing that I recall is that, you know, before recently, the Supreme Court, tended to make decisions. I know, Amy, you talk about this this work quite a bit with regards to the decisions that the Supreme Court used to make tracked pretty well to public opinion. And so that helped essentially build a base of goodwill so that, you know, when more controversial decisions until recently occurred, you know, you had that base to kind of fall back to.

Mhmm. Yeah. Because I I think what like, my my take on kind of this, like, goodwill, this reservoir support is that essentially, if you have something that people are disagree with once in a while, it's very different than, like, having, like, a system where people are constantly saying, I disagree with everything the system makes. And so, you know, highlighting that, you know, there are a lot of decisions that the system has made that people agree with, could essentially change that perception, and be able to build a little bit more of that reservoir of the will.

Oh, great question. So I think a lot ironically, compared to the expert, like, you know, what what should an expert panel have with regards to expertise? There are a lot of comments about the composition of a digital jury. And so a lot of it really stemmed and I'll actually go to the slide since I think the quotes are quite illustrative. There's a a lot of distrust over, you know, what it means to like, over, like, the, quote, unquote, like, average or regular, like, Facebook user. And also a lot of mistrust over essentially the diversity of these panels. And so kind of with regards to the diversity part, the demographics, essentially, that led to a lot of concerns about, like, oh, I'm not like the normal user, so I'm expecting that, you know, the majority kind of user viewpoint is going to kind of override any concerns from a minority perspective. And with regards to the selection of Facebook users, because they're just like, oh, just like random. Like, you know, who knows what they are. There I think there is actually opportunity to highlight essentially that there is quality with regards to, you know, the users that are selected. And so for instance, kind of bringing in ideas of, like, reputation systems, you know, how often is a Facebook user kind of involved, you know, the the size of the group could potentially, you know, help make it essentially help those democratic benefits of digital juries, be more apparent and less weighed down by the mistrust of, like, the peers within kind of online communities overall. Does that make sense?

Yeah. Absolutely. And I would also say that, you know, you had mentioned about essentially kind of more local the jury's made of more local users. And so for instance, I could see that, you know, this may be different if, for instance, you are investigating on a platform such as, like, Reddit where there are subcommunities. And especially on the smaller subcommunities, you know, your jury pool is only so so large, so it may affect essentially kind of the amount of mistrust of peers with with regards to additional juries.

Yeah. I think it would be very interesting to see if these results were different in a community that had just run through an end to end citizen assembly process that that was, like, heavily publicized. So there might be cities or states, or, like, you know, or countries where that happened. And I guess I'm I I'd be curious as a hypothesis of if you had just had a process that had, you know, fairly broad buy in that was, covered heavily within that community that did show random people, able to to be effective thinkers and make good decisions, would you hypothesize, that that would affect these results, or do you think it would not?

Okay. Sorry. There's quite a long buildup, so

so you could

Yeah. Yeah. So so that again. So so there are some communities around the world where citizen assemblies or deliberative polls or other other systems processes that are jury like are used in a very significant way in order to to set policy. And it it it creates an opportunity where people can see that their fellow people are able to think effectively and sort of navigate trade offs and all that stuff. And and and good versions of those processes can help address the group think and and so on. And so in environments where that is part of the general understanding, it seems like there might be an impact on the relative waiting on experts versus their the the other people who might be picked by such a process. Because there's an example, so it's very public examples of people being smart and intelligent and thoughtful as opposed to people being dumb and group thinky and reactive. And so I'm I'm wondering, like, I guess it would be really interesting to to even pick one of those communities and say, oh, you can only only Turkers who are in, like, this city who just had a major process or whatever. Like, what would be the outcome within within that? And and and because that that shows a sort of direction for that that this isn't this is something that's actually based off of the current acculturation and sort of beliefs that might be changed with new information.

Yeah. I think that's very possible. So, with regards to references to kind of off well, we'll call it offline norms. I do remember there was, for instance, one respondent in the pre response who essentially put digital juries as the least trustworthy. And they said, you know, I don't believe in digital juries because I think they're gonna run into the same pitfalls that, you know, I see with regards to juries in real life. And so, essentially, if you change that base kind of expectation with regards to, you know, digit

kind of democratic representation and, like, how people are involved with the with offline communities, I I totally think it would change how they, you know, would perceive, like, digital juries versus kind of expert panels, for instance. And so this is also why, you know, when we did our study, we were very cognizant of essentially only having participants from The US. Because if you, like, throw in kind of additional variables there, it would muddle the results.

Right. Yeah. I think it'd be very interesting just to do The UK and see because apparently, the reputations are very different.

Absolutely.

Thanks. Thanks for the paper, and thanks for the really great tables, like, of of the of the responses and the pros and cons. I really appreciated that.

Aw. Thank you.

Alright. Well, thank you.