Welcome to CDT's Tech Talks, where we dish on tech and Internet policy while also explaining what these policies mean to our daily lives. I'm Jamal Magby, and it's time to talk tech.

CT. Team. As AI systems become more embedded in critical decisions, from health care to hiring, the need for transparency and trust has never been greater. But how do we document these powerful tools in a way that's both meaningful and actionable? In this episode, we'll welcome back Umang Bhat, assistant professor in trustworthy AI at the University of Cambridge, and welcome Amy Weinkauf, senior technologist for CDT as guest host. Together, they'll explore the evolving landscape of AI documentation, its role in responsible deployment, and how emerging standards can help developers, policymakers, and the public understand and govern machine learning models more effectively.

percolates every corner of society now. And I think of documentation as the breadcrumb trail that the AI corporations leave behind for the rest of us to pick up and understand what the corporations are eating. And documentation kind of gives us a purview into exactly what these corporations are working on and helps us put pressure on encouraging these organizations to provide us with transparency into the training data they're using, into the operational workflows that they've tended to consider, the the guardrails that they may or may not have instituted, and whatever, perhaps even regulation, they will subject their systems to during testing. So in some sense, it's our way of understanding how some of these really large multibillion parameter models, actually work in practice. And so I think of AI documentation as our window into their world. Yeah. That's an interesting answer.

Yeah. I think that those two are the primary reasons in the explanation I just gave, but I do think that there are other reasons. So the documentation practices in the software community have been around for a long time, but they're insular in nature. And I think, at least in my head, when I think of the doc types of documentation that my group has been considering off late at Cambridge, these are mostly public facing pieces of documentation. So less so focused on reproducibility, which is a natural natural result of considerations of documentation around code use, around, model development more generally. But I think about that less in the context of our discussion today, but I think that that is another, nice benefit.

So feedback logs actually started as a way for us to consider how to make updates to models as a result of feedback that you might get from some participatory approach. And so from one piece of information or one piece of feedback per se from a stakeholder, there are lots of different changes a developer can make to the workflow or the underlying system. And we actually characterize this in a separate piece, and and took out a lot of that workout. So for example, if you had told someone that there might be fairness concerns with the classifier that they've chosen to deploy, your model developers have a few options. Some might be socio technical in nature and others are, actually concerned the model itself. So you might consider going and collecting more training data. You might consider taking a vast, taking a page from the vast literature around, regularizing for various properties you want. The equalized odds is a is a natural one where the likelihood of having a particular outcome conditioned on a protected attribute like race is equal. That is something you can control for. That might not be something a developer considers until a stakeholder informs them that this is an important consideration in this context. So in the settings where developers are reactive, we were hopeful that we could introduce a form of documentation that keeps track of the information that has been collected and the way it was incorporated. And we just wanted a living record of that. And naturally, there's a separate line of work that has that we were working on for quite some time around, well, how does a developer actually decide how to incorporate that information? Right? Because there's a design decision there. Right? If someone comes to you and says my model is broken, there are lots of ways of fixing it. Right? I might I might go out and procure a new model. I might go out and and encourage use in a in only specific context so you, narrow the use of the actual AI system itself to to prevent misuse. That's a policy effect it is effectively a policy change. It's not really changing the underlying characteristics of the model. It's always saying use it on Saturdays and Sundays and not during the weekdays. So how do you go about creating a breadcrumb trail again of how developers conscientiously incorporated the information that they were given by from stakeholders, whether it's those the internal stakeholders like your in like a CISO of an organization or a regulatory body who's come back and potentially delivered a light slap on the wrist to suggest that you might want to change your system. We never thought that there would be broad adoption of this mostly because there is a not it it is, not exactly clear whose incentives are at play with keeping track of this information. It is an audit trail in some sense, which means that you would need to report everybody you took advice from while building your model. You might not be interested in doing that as a corporation. And it's actually really difficult to justify some of these design decisions we find. So most people cannot really come up with a sensible justification for why they chose to regularize as opposed to collect more data. Right? It turns out to be something super, hard to it turns out to be something that you can't really justify. It's just a a whimsical decision, actually. Just for our audience at home who may not be familiar with these terms, can you explain, what regularization is? Yeah. So regularization would mean that I might care about properties that are not just performance of my model. I might care about properties like ensuring that my model is fair or my model acts similarly for similar people. And I might want that what we call smoothness constraint to exist on top of our models. K. So this means that if you were to change my age by one year, I would hope that my my loan application might not change very much. Okay? Maybe maybe we have a big jump from 17 to 18, but right after that, I I hope it wouldn't change too much. There are other considerations that ought to be at play. K? And so a way you can give a guarantee on this particular system would, would effectively require controlling for that behavior, and regularization lets us get that control. The beautiful thing about the large language model revolution we're in right now is that you might not need to explicitly regularize and control for these types of behaviors because we have access to large amounts of data. K? And when you have access to large amounts of data, these considerations, fall by the wayside.

within a feedback Exactly. Exactly. And that that's exactly what we're after. Right? It's like that design decision needs to be documented somewhere. Right? Like, did you decide to to not regularize because you just assume you're gonna get access to lots of data in the future, or you are choosing to do this because there's some reason why, there you you would like to control for, for for this kind of smoothness constraints or these these fairness constraints or these robustness constraints, whatever you really care about having in your model. Like, right now, there's lots of talk about, thinking machines, but I've had a piece on the stochasticity of the responses from AI system. So how how likely is it for an AI system to respond in a similar way on subsequent on subsequent queries? There are various ways of controlling for this. This. But this is something you might care about. Right? You don't want your, large language model to respond differently in different circumstances to the same query. Right? And so so how would you go about controlling for that? There's a whole line of work there. We just want people to be transparent on their use of that large body of literature or the methodology even at a very high level,

already approached those two kinds of problems? Yep. This is a this is a really good point, and this is why I think the uptick of something like this is better in principle than in practice. The unfortunate part is that a lot of developers get a lot of feedback from a lot of people, and they choose and cherry pick what to incorporate generally. Absolutely. And so Yeah. This would create lit quite literally a list of things that they've learned along the way along with a list of changes that they made as a result. So until the community, which it may never get to this point, decides that it is okay having this backlog of participatory information that is not incorporated and isn't explicitly been, not even denounced, but just ignored, then I think you it's gonna be difficult to get to that point. And so as the adoption on some of these larger scale systems that are coming out of the hyperscalers increases, it's going to be harder and harder to incorporate every voice in the room. And so I can see on one hand what you're saying where we have a lots of really strong efforts around the world or even for within a certain organization specific organizations to collect lots of participatory information and say, here are all the changes that one has prioritizing or here are all the changes we should make. Prioritizing which changes to actually make along with which changes are actually plausible is is a separate question. And and I I kind of see them as sequential. And until there's a culture change, I imagine there'll always be a disconnect between the two. And and I think that makes it very difficult, because there's no one that who's accountable within these organizations for this type of work. And right now, the individual that's accountable for both of these has separate incentives. Right? One is is like an engineering director and the other is is maybe even someone who's involved with a bunch of, like, UX research, for example, is largely where you'll see a lot of this work work take place. Those who are in charge of your internal AB testing platform, for example, would be excited about that type of work. I wonder

offer a way for other people to think about the technology that could be as opposed to the systems that are. We were hoping that this for those UX researchers internally, it becomes a way of speaking the language of developers to kind of provide a mapping exercise for the feedback you're providing to concrete changes that they expect to see. And I hope from that angle, like, this gives them a potential tool or even if they don't use this the library that are that that Matthew and my students wrote. Like, even if they don't end up using that, they're able to, like, at least think in this way where you can kind of take that extra step after you've done participatory research to kind of suggest what concrete changes one would like to see in a language that might help, developers prioritize this. One of the things that I am somewhat bullish on, which fell out of this, and again, is where we had actually started with this, is for developers to look at a piece of feedback and to consider that they're actually so there's a multiplicity in the changes I can make. And this is always never clear. Right? And they never had an ability to justify this. Right? It was very ad hoc ad hoc in a way that we didn't, hold that. We didn't have anybody to hold accountable. Right? And so I was hopeful, and some of the our initial results show for, like, vision classifiers. Like, there are two really drastically different changes you can make. You can change your training data in reasonable ways. You can change your training procedure in reasonable ways. You can change the way your model use and address the exact same piece of stakeholder feedback. And I hope that this alternate reality just encourages people to think about the workflows that they have, right, around incorporating this type of information back into the systems that they're developing, because it is not entirely clear which one is more effective. Some of them might have different costs. So if you have economic incentives, you'll just pick one and move on. But, it is not clear how to actually behave in each of these settings. So this even goes down for the simplest sparsity inducing classifier. So one of the things you might care about is instead of just being smooth, as I mentioned earlier for regularization, you might care about being sparse. So I don't wanna use a lot of different factors about the input to make a decision because that can be really complicated. I only want to consider your age and your body temperature potentially when deciding you have a weight even if I have access to you. It deciding if you have a fever. Right? As opposed to using your entire electronic health record. Right? I just need your body temperature. I want a sparse constraint. I want you to use one thing even if you have access to a million. And so we were able to show and it's quite straightforward to show that, like, even for this, like, old school seminal result in statistics, there are two different ways of accounting for that constraint. And so what that does is it hopefully helps developers think in a way of at least encouraging them to consider all options and justify them in potential ways that are meaningful.

is trying to celebrate the successes of AI use. Okay. The AI ethics community has done an incredible job over the last five years of document the documenting the incidents where AIs have harmed individuals, documenting the risks that are associated with AI systems. But as machine learning models and AI systems are deployed at scale, there is no library for someone who's deploying a system to pull from and see what a success story looks like. And so what we wanted to do is build a repository of successful current deployments of AI systems where we characterize the governance that one puts around the AI system. So I do not care if you're deploying a large language model or predictive even a even an old school digit you we had some a lot of individuals are really excited about just, vanilla logistic regression models that they had been deploying for a long time. But what we're interested in is what governance do you put around it? Where is their human oversight? Where is their institutional oversight? Can you characterize what types of governance you put in place on the individuals who have oversight over these systems? Can you characterize what your governance practices are around as an institution to look at this AI system and say, I have introduced an autonomous AI system or one that is semi autonomous. So it's only autonomous if it's confident. For example, this is a very classic one that you can see. And so one of the things that we're hoping to encourage is a discourse that talks about deployment. And I'm very bullish on deployment of AI systems becoming a site that is an art there for sure, but there's a science that underpins a successful use of an AI system. And there's a lot of excitement about use inspired AI, but that concerns the model itself. It doesn't tell you how to use it. So I might build a diagnostic system that can detect fractured femurs with a 100% accuracy. K? And AI scientists are very excited about getting that metric up. But when push comes to shove and you have to use this tool in practice, how do you do that? Right? Like, what is the workflow? Is there an iPad in the in the hospital, in the ER that someone can pick up and look at? And are there capabilities for overriding? Because if it's a 100% accurate every single time a doctor overrides it, you're by definition wrong. Right? Right. By definition, we've constructed an example. So what are those oversights? How what kind of training does that doctor have? What kind of governance do we put on the model itself? All of these questions need to be documented and answered. And, ideally, you can answer these questions at an abstract enough level where I don't really know need to know the IP of the model itself. A lot of model developers are very excited about keeping that information very close to their heart and not sharing it with the world. And I respect that. But you have, in some sense, a duty to share how you're using this tool and how where are their protections put in place for people? And so this was the motivation for Fabric, where we went out and collected a set of deployed use cases and came up with a design language that lets us think about the governance that we put in place both at the individual level, so for humans who are overseeing these systems, and at the institutional level thinking about how organizations who are adopting these tools put protections in place both for their own personal liabilities as well as for those of their customers. And a rich design language emerges. There are not a lot of interest there are not all too many ways that people deploy systems. It looks something like I might use it sometimes, and the setting the settings where I choose to use it might be some function of confidence, some function of the cost of a claim, for example, an insurance. Right? Like, I I don't care about autonomously and automatically processing claims that are under a $100. Who cares? Right? That's what most insurance companies told us. K? You're gonna you and if the AI system makes a mistake, again, who cares? Right? And so then but but in the settings where you do have potentially more high stakes examples, there are some really interesting patterns that emerge where organizations might have a separate inspection and an escalation pathway that they introduce, and they're all very arbitrary. And most machine learning models aren't trained with this arbitrary escalation pathway in place. So, an example of this is we saw traffic violation use case where you would automatically try to assign a violation, a traffic violation from a video feed. And so if the confidence of that system is high enough, the fine was automatically issued and sent via mail. But if it wasn't confident, which 70% were under this threshold, so there are very few that are being automatically processed in some sense, They had this absolutely obscure inspection pathway where an inspector would first be allowed to look at this feed and decide whether the violation that the AI assigned was right or wrong. They weren't allowed to change it. Okay. They were only allowed to say if it's right or wrong. And then if that person said right or wrong, then it went to another person who then had to say, is it right or wrong? And then, again, a third person had to check, is it right or wrong? And if all three agreed that the the system was right, then the fine would be issued. If any one of them said no, the fine wasn't issued. That's so interesting though, because it's like, you know, there are probably

encouraging anonymity of sharing those use cases. And eventually, we're gonna propose, like, an ecosystem that lets us share this information very publicly, and in a way that it kind of protects the identity of some of these organizations because even your financial organizations, like, maybe they're not using it for trading. They're using it for customer service. And in that setting, it's not even clear if this is a, if this is an airline or a hotelier and hospitality organization because the AI system that we're discussing is sufficiently broad. But I 100% agree that there are a lot of these artifacts that exist. And putting this down and getting a language around it just lets people see how ludicrous some of these AI use cases are. Like, for example, I you'll see a company that is obsessed with the accuracy of their particular system. Like, we're trying to get our accuracy up board level decision. Accuracy must be high. And then if you decide to procure an AI system that has, let's say, 80% accuracy in the next quarter, you're able to get 85, and then eventually you consider fine tuning and you get to nine 87. But at the end of the day, if you let a doctor or a individual look at that AI prediction and then decide on their own volition what to do, you're never gonna get that performance benefit ever, no matter what. There are a lot of concern. And until you see it in front of you, and until you're sitting in front of your board, and you see this saying, I am on we are on an AI adoption pathway. And in practice, this is just effectively pulling a new encyclopedia Britannica source for my employees. That just gives them yet another piece of information which they're allowed to override and still do something on their own.

those realities. And so this is exactly your your this is the exact language that we're thinking about. Right? It's like trying to provide a language that lets people have this conversation without, a, feeling like they're behind in the adoption pipeline, but, b, makes this conversation not about the models themselves, but about the governance that surrounds it, right, and the workflow that you're using. And ideally, in the best case scenario, in a few years, we'll have a literal library of these types of workflows that organizations can use. So you have these artifacts of five inspectors that you've had moderating your content moderation tools for a long time, and you might have new escalation pathways that you can introduce in the face of, ever more capable AI system. Another thing that I like about this is

can start having a frank conversation about the internally internally, if they can come up with a mechanism for sharing their AI use cases outside of a registrar, like a registry, which many are starting to have. Right? It's taking it from a Salesforce DB to something that looks a little bit more like, a little bit more akin to this this idea of incorporating governance into that practice. I think that could be really exciting. And I think it needs to be a little bit stronger than just documentation. Right? Because this is not just commenting on having documentation. This is commenting on what your practices are around using these tools. So I think, like, concretely, it's actually decently free to do. Right? Like, this is it it's just asking your your PMs at your at your companies to basically suggest, you know, here's how we're gonna deploy this tool. And having a frank discussion about that from the get go. And I know that there's a lot of folks who who build technologies and then might be divorced from the way those technologies are used. But in some sense, that is the discussion that we need to have if we wanna have a sensible, grounded discussion about governance. Right? We need to know how people are deploying tools. And the question is, where does the burden fall for that information to be captured and then stored and located? And I'm hopeful that even if internally we can get corporations to start thinking about this, we can make a lot of progress with, with with understanding what our governance practices are for every model that's deployed. Yeah. Absolutely. What else are you excited about? What are you researching now? Like, what are the directions that you're Yeah. This is a lot of what we're we're we have a quite a handful of work in this space of, like, deployment as a science, and this effectively lets us start thinking about the ways in which people collaborate with systems. The natural question that falls out of this is I might choose to only use my model under certain conditions. My model might only be available to me under certain conditions. Yeah. So this leads to really interesting questions around how people incorporate, agents into their existing workflows. So we have quite a lot of work on orchestrating the use of an agent in practice, and going from there to understand what the effects are on individuals who are using those tools. So students who are using these, like, large language models in the classroom or doctors who are using these large language models to educate patients, not for diagnosis, but for patient education, where they themselves might be getting upskilled on a particular disease, for example. Right? Like, semaglutide is is a nice example of this. Doctors are just trying to understand, like, how Ozempic works and then going off and talking to their patients about it. So the use of LMS in that context for doctors is actually pretty high, and and from from some of our colleagues and collaborators. So what does that look like in terms of the collaboration framework that you provide, users with? And so we think about this in in a gamut of different domains. And then we have quite a lot of work on on evals. So what kinds of interactive evaluations can we What's an interactive evaluation? So an interactive evaluation is something ever so slightly different, not ever so slightly, quite literally the opposite of a static evaluation and benchmark that you're seeing in practice today. So, what happens if you watch a user interact with the large language model

The longitudinal effects of AI use are just and it's impossible to capture in a benchmark, actually, I would argue. And so it it's kind of like thinking about and watching. One of the examples of this that we've done so far is what happens if you watch a mathematician use a AI system to do theorem proving, but you let them use it for three days. You let them use it for a week. You let them use it for a few hours. These longitudinal interactions are not captured by static evaluations, but they give us access to a rich history of how we expect people to use these systems in practice. You get notions of over reliance for certain conditions, overdependence, which is very different from overreliance. Some would argue it's a precondition to depend on a system would mean you, you you pick the fentanyl up. Overreliance means you used the you were not take you were taking, insulin injections, not fentanyl injections when you needed to. Right? So are you taking vitamins or painkillers? And and that's the, the the analogy that we like to think of there. And what what does over reliance and overdependence look like on these longitudinal time horizons, which only which are only possible to study over long interactions

sort of analysis to figure out how that could reasonably be done, if at all. Yep. Exactly. And I think this is this is kind of where where we're at right now working with, behavioral con folks and and psychologists and folks who are, who have expertise in crafting these kinds of studies so that we as AI, researchers can kinda make kinda make progress here. Fantastic. Very excited about that. I'm excited about that too.

Yep. Thank you so much for having me. Thank you so much for coming.

Thank you for listening to Tech Talks, presented by the Center for Democracy and Technology. I've been your host, Jamal Magdi. Tech Talks is edited by Jacob Kaufman and produced by Drew Courtney. Check out more of CDT's work by visiting us online at cdt.org and on various social media at syndemtech. That's syndemtech. Thanks for talking tech.