Thank you, Jen. Just first of all, I think that URL that you shared is not the one. Let me just share the link of the project page on our lab so that if people are interested, they can also look into that related publication and also blog and applications. Okay. Alright. Let me share my screen and oh, I cannot share my screen.

Alright. Thank you. You already gave a really comprehensive introduction, so I would not give mastery introduction. Just I am a PhD candidate at University of Illinois, and I'm very glad to be here to present this work and also get feedback.

Mhmm. Okay. It's saying you cannot share screen while the other participant is sharing.

Yeah. I cannot see anything that you're showing.

No worries. Okay. Let me see. Can you see my screen?

Yep. Okay. Screen with no notes, screen with slide. Right?

Okay. Cool. Alright. So I wanted to keep this picture here of our team that I have been very grateful to work with for this specific project. And I would start with one sentence here, like, democracy that I'm saying a democratic approach. So democracy is a complex topic. So my goal for this talk is not to show how to kind of forge into a solution for democratizing AI. So what I'm going to do is introducing a possible way to engage end user in AI decision making and emphasizing on underserved group. And the tool that I'm proposing can be divergent to different AI governance context. And thank you all for joining, today and hope, you like what we have to share. So let's dive into it. So I'd like to briefly go over my background as a researcher and do just make sure people understand where I'm coming from. So far I have worked with different underserved group, mainly people with visual impairment and people from global South. So my research involves different interviews, co design, and prototype evaluation with these underserved groups for various technologies, including targeted advertisement in social media, search engine, obfuscation technology design. And this this all of these studies really helped me to understand the need of underserved groups based on their cultural, economical, and local regulation, even developmental differences. And simultaneously, another area that I have been it's my personal interest, I would say. I have been working from the human aspect of web three applications since 2021. And this line of work mainly focuses on the usability, security, and accessibility in context of decentralized application. And, for example, I worked on, like, accessible crypto wallet, usable key management implementation with multifactor key derivation function. And what in really caught my attention in this process was exploring how decentralized app are governed, in particularly, DAO, which is a big part of my academic work as well currently. So, and I also had this interesting experience working as a privacy engineer at Google and Max blank as a research intern. And, like in particularly how organization actually build different kind of metrics and tools around privacy risk assessment. And when it comes to the usage of third party applications in the data pipeline, it actually gave me a kind of real world understanding of industry stakeholders' perspective on governance. So after this very long research background introduction, I'll just start with this statement which known to everyone. Like, we are in the age of rapidly advancing technologies, especially AI. And there is a information gap in technologies between, like, developers, users, and policy makers. So the situation can have a disproportionate impact on especially underserved groups by the design decision of AI. So we are seeing many policy brief then. We're seeing this AI alliances, executive orders that focuses on the vision of trustworthy development and governance. And the overall idea is to mitigate the risk and possible downside of AI. So in my presentation today, what I am actually introducing an approach that engage underserved groups in a deliberative and democratic decision making concerning AI. And notably, this approach actually utilized decentralized governance for scalable user interactions and consensus building on AI design. So, ideally, what in topic that I am interested in exploring how people interact with the system and how organization influence the structure and quality of the technologies. So, just to give another motivation why, I think this is an interaction, to be scalable and consistent building process important. So as a part of my research on, this is a disability for us dataset that I created with blind users. And, so this process, when I developed this novel method of engaging blind users as a contributor for a private visual dataset creation. The ultimate goal for this project was to support the development of AI tools, which is ultimately be used by blind users. So it makes sense to create the dataset with blind users because their photographic style and everything is different. But what happened is actually we did add add value by creating this new data set, but we found that engaging blind users in this process and scaling this process was really challenging. And there are some other concerns as well. For instance, user had this approval sometime very contradictory and conflicting expectations for, technology senate interestingly comes down to the need for consensus either with the service provider or with the fellow community member. So another thing was very interesting at least to me, users would like to see how their input is put into actions, but they found it hard to track once the research is done. And they felt the same for industry and academic research as well because sometimes industry reach out to them to test the technologists and to know their need. But when they use the technology, they don't see those expectation put into action. So as you can see, there's need a regular insight and some sort of consensus. So currently, I I don't know if you are familiar with this recent outcry to show like I would like to just show why regular insight and consensus is important for technology design. This app called be my eye which has been longstanding usage with blind users that actually connect them with sighted volunteers to recognize objects such as, they could read the label of food packets or they could, identify any private content in their, credit card or anything they want to be identified before making any decision. So what happened is, Be My Eye changed the privacy define when they incorporated this GPT based services. So at current state, this tool no longer provide image description to blind users if there is any human shape in the picture. So now the problem is human face and body is one of the private content considered by blind users. And they want this to be identified in their daily life. So failing to identify this kind of private object can be like can prevent them for performing even the daily routine task. So as you can see that there's a clear contradictions of what users need and what companies decide when they are incorporating different AI tools. So this comes down to the broader questions is like how this new design decision has been made and who has the power to make it. It. So from what we have learned so far with this fundamental research, there are a couple of key insight that really stand out from my point of view, what I have done. In terms of underserved populations, first off, they would like to engage in a system in a holistic way to contribute to the technology design and development, which means they expect the system to enable meaningful participation. And second, they want the system to represent their needs. So the first questions comes to the mind, like, how do we even design this kind of system with reasonable or actionable decision making options? So this group of people can meaningfully express their choices. So in particular how to gather this input at scale and when you gather this input, how do we actually incorporate those in a tangible constitute to form consensus? So what do you need a simple, scalable and traceable system or approach for block this underserved groups to participate in the decision making? And I would like to just you to imagine, for example, many of you might be already using different kind of generative AI models. Like, here you can see a text to text model like chat g p t. You provide an open ended prompt like, create a story for an arts, and the AI is generating an response, where it's depicting an artist as a woman. So, it might be possible this, generated description might not align with your expectation based on your background or experience, whatever those different characteristics you have you have as individuals. So what if you had this option to to initiate our main public discussion to first deliver it and then decide whether this AI response was appropriate in the context and followed by a tangible consensus. And in this way, what would happen is anyone could inspect and challenge that value laden decision made by AI. So by value laden, what I am actually trying to say here is, topic related to, let's say here, stereotypical bias or culturally or politically controversial topics such as, I would say, immigration, vaccine, US capital mob, mental health, climate, etcetera. So the summary of this process could look like that they first proposed. They discuss a topic and they reach a consensus of how this AI should be designed. But the major question again, how do we design such system that can deliver process that for large group of people? And at this point, the one approach that you could thought of is active engagement by distributing the power through technologies. And the current technology landscape that was more interesting at this point of that we could empirically use as a test bed, which was decentralized autonomous organizations. So we draw this inspiration from the decision making practices of decentralized autonomous organizations where there is often the lack of centralized paradigms the decision. Though there is still a lot of, I would say challenges or limitation in decentralized autonomous organizations. But at this point of time, if you think about having a empirical test bed to even evaluate users interactions and how even this coordination and consensus process work, then we believe that DAO has a mechanism to even test that with real life users. And specifically, the reason why we think the DAO is a promising AI governance, like, to be used in AI governance issues. Because in the centralized space, to the best of my knowledge, there is a lack of infrastructure to engage for community. There is a, like a way to have a community discussion but not a process for consensus. Again, the traditional legal and policy approach might have some limitations with this fast paced advanced technologies. And that can provide a emerging governance model, at least as as I mentioned, to empirically test the contextual issues at this current point of time. So in specifically, like, we did identify it through some empirical analysis of some metrics of decentralization in the doubt space. And we actually implemented and evaluated various mechanisms like voting schemes that can better engage under subgroups at scale. And let me give you some voting schemes example that we believe could potentially give more voice to underserved groups. So in traditional voting, let's say, I think here, I don't know most of you might have already been familiar with quality voting, but I will still just give pretty much overview why we think that it could be inclusive way to include more advice for underserved groups. So in traditional voting approach, let's say you have four voting power, which means four votes under the one to open one vote policy. So however, this could be challenging for understood groups because they're often, you know, under resourced and might not have a large amount of token. So their voice could be overshadowed by those with a large number of voting power. So under the quantity voting scheme, which is largely used in public good funding, the number of vote is determined by the square root of the token casted. So four four would mean this to four token would mean the two votes. So this could be one way to emphasize the number of voters rather than the size of voting powers. It could be also a way to voice like, give more voice to understand groups. And now about voting powers, another influential metrics in the DAO space for decentralization. So, traditionally, you can imagine if every individual get the same under same number of power, the challenge is, underserved groups are easily minorities. So if everyone gets one vote, then their opinion might be overshadowed by majorities. So to address these issues, what we did is we explored this concept of differential distribution. For example, the application of power to distribution twenty eighty votes where 20% of the community holds 80% of the voting power. So, again, you can think of that case. For example, be my eye that I explained give the example before where the AI feature is used by blind users heavily. So in that type of cases, perhaps they should have more say or more voting power than others. So we use this twenty eighty to mimic a smaller group having more voting powers. So now that I just gave gave this design rationale, so let's go over the system very quick. First of all, as most of you might be already familiar with the DOW space or how the organizational setup has been built upon. So let's say this call a space for the organization where you can set the role for admin moderator or decide on the different voting rules. For instance, who can vote, who can propose, and how many votes are needed for a proposal to be. And when I say proposal, in this context, it's about something related to AI. So for a distributed ledger system to build this system, we are deploying this optimism blockchain. And one of the main reason for choosing optimism because of its current use case in the war coin. And this would allow us to integrate the war coin signing feature for proof of personhood, and which would be very crucial because let's say this system would be deployed in a real life case for AI governance. Let's say companies like Anthropic or OpenAI to engage, individuals, then they also need to make sure that, people who are, voting for different AI, design decisions, they are, they are not bots or spam responses or, one people one person is not voting twice because that could inflate the decision making. So as for a voting power, which is also a very crucial part. So we create the vote token in solidity smart contract and it's the object oriented programming language designed for Ethereum blockchain. So when users signed in, they are given voting tokens so that they can participate in the voting. And the thing is, like, we designed this and developed this system as a, you could say, design group. Because at first, we need to understand, even this kind of system really, be helpful for this kind of, user group because, you know, reinventing the wheel kind of sometimes just working with these underserved groups, one thing is very clear that there are many technologies out there they even don't find helpful at all. And one thing is very important for underserved groups to to really understand their need and communicate with them to have regular insights. And the reason is creating this technology probe at the first place is, first of all, understand if this kind of AI governance system works in their context at at all. So to evaluate this system, we actually designed four varying governance mechanism with treatment condition. So by saying that what I meant is there are four treatment condition we evaluated. For for example, in the treatment condition one, it was quadratic voting with same voting power for everyone. In treatment two, it was quadratic voting with twenty eighty voting power. And in three, there was ranking voting with same voting power. And four, it was ranking voting with 28 units. And the main idea was to assess how a varying voting method or combination of voting power influenced users' perception of the quality of this consensus building process to be democratic in context of AI governance. Now, but what we did is we choose a AI value topic, which is again text to image generation models and concentrated into stereotypical bias, which is a very common bias in different AI models. So we begin by introducing this to users by an AI valid topic. And in this process, what happened is this ai iteratively resolve ambiguities with users and with a multi turn conversation. And the idea was to help user define their norms and expectations when they receive this generated image. And after that, once they are done with the conversation with AI, they can also join a collective dialogue process where they can discuss with other community members so that they can learn the perspective of others of this particular AI issues. So in experiments that are often possible that if you don't give them some topic to discuss, sometimes many people doesn't just come out and discuss. So to facilitate that, we also had some discussion topic beforehand so that if they cannot find a topic to discuss with the group, they can also choose that. And finally, there is a tangible consensus space where governors decision making platforms where people vote for different options to improve the AI models based on their experience and expectations so far. And they could choose and they had this voting powers to vote for different options. And now one might ask that, okay, if they completed the process and provided their different options, how do we measures that people people find this process even fair and they they they they perceive this process as democratic. So to do that, what we did is we adopted this variety of democracy metrics from political science to assess participants' perception of the increasing context of AI within our system. So as you can see, it has nine different scales, and these scales are mainly designed in the political science in context of country level democracy. So what we had to do is for our experiment, we need to adapt it to these measures in context of AI. And when we assess users' feedback, then we found that in the quadratic voting condition, which was treatment condition one and two, was perceived as more fair than the ranking voting conditions with this scale. And we also had this open ended responses from participants, and those actually also support that qualitative voting on the relative skills that allows them to receive more parts and increase the overall transparency as they perceived this process as democratic. Now one thing was also very common in this process that we found, sometimes many of the participant expressed some sort of skepticism that even though they find this process democratic and they think this process could work, but they are unsure if developers of AI system would really incorporate the result of this democratic decision making into the system at all. So to understand this kind of, you know, salient ideas or perceptions, we're currently running interviews with 60 participant randomly selected from different treatment condition to understand their mental models to identify what are some factors that might be also contributing to this perception. So let me just see if there is a questions maybe. Yeah. Okay. Alright. There's no no questions so far, but okay. And okay. So as I mentioned that currently, I am actually interviewing again, the participant who participated in this process and experimenting with this new design form. So many interesting thing that they mentioned mainly that I didn't even think of as a sighted users. So this this feedback are from blind users mostly. So one of the thing that they mentioned is degree of preference. So why they find this voting process with different weighted voting skills in quadrant and even in the ranking weighted voting skills more fair is because that they have a option to choose varying degrees of their preferences for different options. What they don't even find in the traditional survey or different interview study. So they can be more concrete about their satisfaction as well as dissatisfaction. So this could be a potential method even to gather insights from them. And another one is the predevelopment consensus. So as I mentioned before, so they they mentioned that there are different technologies, developers or companies are maybe making advertisement. This is a very, very new or niche technologies to use and have new features. But most of the time, what happened in their context, there's already the technology there to already serve the purpose. So they seem it seems to them like that their the company is disconnected from the community before even making these development decisions. So what currently company does is after developing something, they reach out to the community to do beta testing, but they don't gather this insight beforehand even to build these technologies. And another thing about the transparent community vetting by saying that this participants actually mentioning about every day there is new type of GPDs features coming in and they're interested in using those because, you know, blind users, they they depends on many ways in AI technologies to identify and offer a different kind of like, visual content as well as to learn and use those in an educational context. So what they find very difficult in their context is because every day new features is coming in. They don't know which features is good over the others. So they think that having this voting sports to be incorporated in any features to see why the where the community interest lies. So if there is a visual, a description of these features has been used by, 20 or 50% of the community and they like that, so this kind of information is helpful for them. So this is the different way they think this consensus process as well as the result of consensus might help them in the everyday life when they're using different kind of AI technologies. So I will just stop here because I think I so far presented the initial technology probe and what the responses I received from the users. But I'd be happy to learn what you might think and where I'm where different kind of, I don't know, other ideas can be incorporated.

Just

Mhmm. I think so as you mentioned, where did you see the biggest skepticism, right, from from this underserved group or from the both side of the stakeholders?

Sure. So I can say that both side of the stakeholders. Currently, you know, for AI governance topic broadly, many of the people are working on it, researchers, stakeholders. If I say about a researcher point of view, when I present that on our academic setup, the main skepticism is, you know, DAO is still improving. There is a lot of space in the DAO space standalone to improve, and you are applying the DAO mechanism in AI governance. How that makes does it make sense? Now I I actually I've followed those skeptic business and I know what Huneet is talking about. But if you think about it, it's all about time and any technologies out there, including DAO, they have a space to improve. It's not only about DAO. You think about the different password management technologies we're using, like the Gmail account or the Facebook account. Those password based technologies are also improving. People are now talking about decentralized key management. So every technology has some backdrops, but that doesn't mean that it cannot be applied and evaluated with users. So one of the main, skepticism from researchers point of view, maybe, they're not sure if that would be a possible use case in AI governance. But, the my response to it actually mainly most of the time was that in the centralized space to my knowledge, I didn't find a victim that I can use as a empirical based bit to evaluate the users. What I'm most interested in to first understand the user's interactions from, towards our technologies and even if applicable to, some other context. And, second, the skepticism from user's point of view, what I have found that they find these technologies interesting. They find this voting process and the campuses building by collective decision making, in a community interesting. But, it's also hard if I try to probe them understanding about this blockchain based applications, then that seems like a totally different place for them. And that's another major skepticism is that, let's say, if you if I want to argue that, okay, blockchain based governance can be applicable to AI context, then one can also argue that, you know, there is already some discussion around lack of educational materials to make blockchain understandable to general populations. So if they don't understand your technologies and under the hood security consists of building, then how can you argue that these technologies are well perceived by users? That's something and that's criticism. And third, from stakeholders, company's perspective, this project is mainly funded by OpenAI. And the last summer, I I got this grant for my PhD dissertation proposals from them to work on this democratic input to AI fun. So now they might be interested in the specific area to explore democratic input to AI and they might solve some values. But I don't know about other companies if they are willing to incorporate this kind of ideas into their governance system. Even I don't know if this system would be incorporated in the funding that we got from a specific company. But from research point of view, I see the balance in evaluating technologies with users first. So I think these are some, three different dimension that I look into.

it. Thank you. Yeah. This is such a great presentation. Thank you so much. I have a question around, like so I I do similar work kind of with, marginalized communities, specifically the LGBTQ community and a lot of issues that we've faced, with algorithmic censorship and just online harassment, and how can we, change, you know, algorithms and inform them better about our identities and and expression. So I've I've researched similar things. And I think one one big question I have and always run up against is, like, what are the incentives for marginalized communities to make these tools better? Like, you know, is it as simple as just, like, pay them a little bit? Or can we, like, think about deeper ways of changing, like, the ownership models of these large language models to include marginalized people and remain accountable over time to those people? Yeah. I'm curious what your thoughts are on that.

I think that's a very interesting point you mentioned and that I have also encountered, like, during my last four years working with these blind users and people from global side. I'm really am working with older adults and people of color. So this is the same thing that I have been encountering. And the challenge is they do like to so that group that I worked on because contextually different under subgroups, their characteristic and their expectation might be different. So I cannot speak for all of them. But I could talk most about blind users and people from Global South. So for blind users, one of the incentive they get that, you know, already mentioned in the research point of view, they get some incentive by participating the research. But, but what are the other incentive? You you said the the data ownership or ownership of this, like, AI, models. And that's what actually I was talking about that a continuous process for engaging this underserved group in the decision making process. You know? So if they make a decisions, so then they actually perceive that they have a power to shape the futures of of some AI models. So that that can be considered as incentive. But if you think about tangible incentive, that's another topic to even discuss that I haven't really worked closely with. But as, you know, their, let's say, internal incentive, which is power to be include their voice. I think one of the sorry. One of the book actually I was reading was, like, exit voice and loyalty. So where is, like now how we model this? Do we model the system where consumer perceive this system as, you know, not meeting their needs and they just fix it, or we allow them to include their voice in a tangible manner? So I think that's the questions coming to place. As a researcher, we might need to think a little bit more. And that the design probe that actually I showed today, that might be experimented in this context. Like, in what way we can give the ownership to the underserved group. So I I don't know if that answer your question.

Yeah. Totally. I I will follow-up with you. I would love to talk more about this, though.

Yep. Sure.

I see some questions here. Okay. Let me see. Go out one by one.

Mhmm. Good question. I don't see right now it's a global solution because, again, I I do live one step at at at a time. If I for example, this system, I experiment with 235 individuals from from the National Foundation of White Neighbors and from people from Global South, which is India, Bangladesh, Pakistan. Those are the place I have access, and I actually build a trust the last four years. It it it take a long time to build a trust even being consistent with the participants. So I tested the system with only this 235 people, but I cannot blame that, okay, this will really work, and they this will be well perceived by the other communities, like older adults and people of color and other communities. So that needs work, actually. And, again, I don't wanna overkill. So I'd say maybe in six months. I'll be depending in six months. By then, I'll test with other two underserved groups. I can share more information on that.

Mhmm. Sure. The first thing is the UI UX point of view, I would say, that from research point of view and from the base of our capacity, we tried to make it as accessible as possible according to standard because we tested that with with three so w three a, the guidelines so that it meet the standard of accessibility. And for the second thing about education, right, the new topic, let's say, quadrant voting. Ranking voting choice might be familiar people because they use that in a real infection. But I would say that in in that application, we had designed this, like, videos and in time in time instructions or informations when they need about quality voting, ranking voting, how this work in the real life with, example scenarios. Now, again, we designed this, education, but another thing we actually tested after they they watched the video, we also tested that with the knowledge questions to see if they really understand. So we point in some way, we measure some knowledge level of different, like, voting method they're using in this process. So that's one way to validate their level of understanding. But we also see that people have a different level of background because some of them might be already familiar with some of the blockchain based technologies. So some of them totally unaware of that. So they do have different level of understanding. So now what I'm trying to do in the exit interview to, again, have some rich data, sometimes you ask these questions in the survey, but you don't really get all the information that they have in mind of understanding of this quality reporting process or the governance process. So in the exit interview phase, what I'm trying to pin down is what are the other things they would expect to be explained to have better understanding of this governance process. So hopefully, I'll get some insight on there to better design, the interface as well as the education, for this new components. And third, you mentioned about the wild coin. So the Wildcoin, currently, we had this discussion, meeting with the engineering team at Wildcoin to integrate the Wild app in our inclusive AI infrastructure. The one of the reasons we would like to include that, we haven't already included it, but we would like to include that because in the recruitment process, we found it very challenging to find to validate some of the responses if those are bot responses or real user. You know, now AI being very advanced, so social engineering craft is being more intelligent. So it's tough to even sometimes Qualtrics and even other platform unable to find out which is bot response, which is not. So So that that's another of the reasons we are trying to improve the quality of the sample. So that's another one of the reason that we decided that, okay, while coin could be while app could be a way to verify

is a legit user. If we make the system available in the wild because we did the experiment in the contract environment where we know the people, we know the community, but if we are making the system open source, then that's a problem. That's why we're trying to integrate the wall con wall app. So I think I answered three of the questions that you have, but let me know if any of this is unclear.

Yeah. Clear. That's a good question. So late September, I I went there in San Francisco, their headquarters to present this work, the findings that we received so far. And as a part of the funding process, definitely also share the report, everything to them, as a company executive report, everything. So, the plan from their point of view, this is not just I only will talk about my team, but there are other teams also worked on processing the topic. So they they will share some resources for public communication probably soon. But how much or to what extent this type of process would be integrated? That is not something we had any discussion with. But the work we have done, OpenAI would be the the researcher at OpenAI. They would also be floater when we are actually moving forward and publishing this stuff. So about the industry details that I am not the good person to share more details about.

Thank you. Bye.