Welcome to Tech Talk. Bye.

Happy New Year, and welcome to CDT's Tech Talk, where we dish on tech and Internet policy while also explaining what these policies mean to our daily lives. I'm Jamal Magby, and it's time to talk tech. Elections have dominated the headlines, and as our listeners likely know, CDT has been busy on that front. Check out our most recent episode for more. But that's just one of our many issues our experts have been working on. Today, we're talking with one of the newest faces at CDT, Lydia X. C. Brown, policy counsel with our privacy and data project and well established disability advocate. In addition to our CEO, Alexandra Gibbons, to discuss CDT's latest report on artificial intelligence and benefits determinations.

Lydia and Alex, thank you both for being here today. I'm excited to be here. Thank you so much for having me.

So so for our listeners who don't know, can you give us a little background about who you are and where you're coming from? My name is Lydia, as you shared.

I always introduce myself by mentioning that my pronouns are they, them. My background is as an advocate and a community organizer in disability rights and disability justice, which are actually two distinct but very closely related movements and ways of thinking about disability in the world. Before joining CDT, I worked with Alex at Georgetown Law's Institute for Tech Law and Policy, where we launched the project on disability rights and algorithmic fairness. And prior to that work, I was a fellow at the Bazelon Center for Mental Health Law where I represented disabled students facing school pushout and criminalization. Outside of all of that work, I've been an advocate on issues of disability, race, and gender for about a decade, and I'm very excited to bring that analysis and commitment to CDT.

Wow. That's an incredible background. I mean, we're super grateful to have you and your expertise at c d at CDT, supporting with our work. So with that, I wanna jump into the issues at hand. If you could, tell us more about how widespread the use of algorithmic decision making is and where specifically you're seeing it being used.

Many people know that our lives are increasingly governed by algorithms everywhere from our social media feeds to high level decisions about policies that may reflect how our decision makers think the population will respond to certain changes. And what many people don't realize is the impact that algorithmic decision making might have on decisions that affect our very lives, our access to housing or to employment, and that's what we're seeing for disabled people. Some of the key areas that our project has been looking at for the last couple years have been ways in which algorithm driven decision making informs which people have been determined eligible to receive benefits, what kind of benefits they're getting, which people might be hired for jobs, and whether you might even be make it past an automatic screening to be considered for a job to even get to a human reviewer, whether you might be surveilled in school while taking tests, whether or not data related to your health is shared with a company that you might know nothing about their intentions, or whether your data might end up in the hands of police, and what that data might even look like. So switching gears here a bit, Alex, you brought this project with you when you became CEO of CDT.

Why was it important to you to have this work happen at CDT? So as Lydia described,

Now, Lydia, you've recently released a report on the use of AI benefits and determinations. What did this report analyze? How was AI being used in these settings, and why is it important for people to know to know about these?

Our report examined the last several years of litigation brought by advocates challenging the harmful use of algorithm driven decision making in benefits. More specifically, we looked at ways in which advocates have challenged algorithms under three main theories, under constitutional and procedural due process violations, under violations of notice and comment rulemaking requirements, how states are required to change policy and solicit feedback from the public before they're allowed to do so, and potential violations of the community integration mandate in disability rights law under the Americans with Disabilities Act and the Olmstead Supreme Court decision. You know, what we looked at in this paper were ways in which different states adopted algorithmic decision making at one or two levels of the process, largely around benefits for people who receive care through Medicaid, home and community based services, or other long term supports and services programs. And the way that most of those programs work are that people might be allocated either a certain number of hours of care or services in a particular week, or they may be allocated a particular dollar amount to budget the cost of their care during a particular time frame, like a week or so. Algorithms can enter the process at one of two places. They can enter the process in the assessment stage where someone's needs, what type of care or services they might need in order to live in their community and to be healthy and to have a fulfilling and meaningful life, what that might look like. And the other way where algorithms can enter the process is in the allocation determination, which is in determining how much care or how many hours the person will actually be approved for. That can get a little bit muddy, especially for folks who aren't necessarily up to their eyeballs in thinking about the intricacies of Medicaid policy. But what it functionally can look like is what happened to two of the plaintiffs in Arkansas, Bradley Ledgerwood and Tammy Dobbs, both of whom have significant disabilities that mean that they need help with everything, with dressing, with eating, taking medication, even assistance to move and reposition their bodies so that they can be comfortable and not get bed sores. And both Tammy and Bradley had previously been approved for fifty seven hours of care each week, and that approval came because a nurse had done an assessment and decided based on what I observed about your level of care needs, like what kind of support you need to live in the community and to be healthy and to feel good about your life, you will need about fifty seven hours of care. And the nurse made a discretionary decision, sent in that paperwork, and that is what the state paid for. But after Arkansas introduced a new algorithm to decide what type of care somebody would need and then automatically determine how many hours that the assessment would then translate into, both of their care hours were slashed nearly in half. And for Bradley, that meant that he and his family even started talking about whether he would have to go into a nursing home just to make sure he could get necessary medication and be moved and repositioned at the right times because they wouldn't be able to afford the cost of care without this help from the state. And all of this happened because a computer decision made a a computer program made a decision based on assumptions about what type of care somebody should need that took the decision completely out of the individual nurse's hands.

So the paper centers on disabled people's stories and cases, and you talk specifically about cases where people have successfully challenged AI systems in court. Can you give us some more examples?

Sure. In the state of Idaho, which similarly adopted a new algorithmic tool for assessment of people's needs k. The state started to use an assessment that would automatically determine what level or range of care somebody would be in based on a series of predetermined categories. And the assessment itself simply forced people to input answers about their needs and what tasks they needed help with and then would spit out a determination automatically to decide what range of care they should be placed in. And once that range of care have been determined or that level, then that would determine later how much hours of care they would be approved to receive. But Idaho devised its algorithm to produce those determinations based on deeply flawed and incomplete data. In the court case brought by advocates at the ACLU of Idaho, the judge found that the data used to create this algorithm had been so riddled with inaccuracies that up to 66% of all the data ultimately had to be discarded as not fully usable or not usable for the discarded as not fully usable or not usable for the purpose of of calibrating the algorithm. And that was because some of the data was incomplete, some of the data was inaccurate, and some of the data was simply unreliable. And it was all based on a relatively small pool of data as well from a few years before it was used for calibration. And that small pool of data was not necessarily genuinely representative of or meaningfully responsive to the needs of the entire population of people in Idaho whose disabilities might mean that they would need support through this program. And so the result was that across the board, large percentages of people who had not faced significant cuts to their approved hours suddenly were facing drastic and noticeable cuts to their hours comparable to the ones faced by the plaintiffs in Arkansas. And the case brought against Idaho centered around Idaho's adoption of an algorithm that was not sufficiently explainable to the people whose lives were affected by it, to an algorithm that did not allow for a sufficient process to be able to challenge the determinations that it made that then affected people's lives and to whether or not it could, in fact, fulfill requirements for due process under the constitution or the enabling statutes for the program for which it was used, in this case, Medicaid, to actually to actually protect people's rights to receive the type of the type of benefits that they had been receiving previously under law. And to which previous court decisions in our history of jurisprudence have established that when you are receiving benefits that you have certain rights associated with them because you become dependent in some ways on your survival and your health requiring access to certain types of government provided benefits.

Now, Alex, why are cases like these important to highlight? What are some of the bigger policy lessons we can take from them?

And, Alex, you mentioned earlier about how important it was for you to bring a disability focused project into a mainstream tech organization like CDT. What does it mean for an organization

And, Lydia, in addition to this report, what are other things you're working on at CDT? What other projects are on the horizon?

We just released another report, which is pretty awesome. Right? And that report came out a couple weeks ago on the use of AI hiring tools. And Alex mentioned this a little bit earlier in our conversation today, but I just find it fascinating and horrifying to think back to even, you know, my own time in high school when I was applying for job after job, mostly in retail, sometimes for office administration type jobs, you know, the kind of things that the average high schooler about to go to college age range might be looking at applying to. And I just remember time after time, it opening on the screen some kind of personality test. And at the time, I just thought, this is weird. I have no idea why I have to fill this out, but okay. Sure. Here it goes, I guess. And I would fill those out. And in four years, I never got a callback for a single job. I have no idea, you know, what happened and what factors went into me not being able to get hired anywhere. And so I can't say was this all because of those tests or not, but it does leave you wondering. To what extent did those automatic personality tests, if any, play a role in ensuring that my resume might not have been seen by any of the hiring managers in, you know, every single store in the mall. Still bitter about it as an adult. But we are building out work to address what does this actually look like at scale. How many employers and in what fields are using not just personality tests, but other automated hiring tools? What kind of information are they collecting on people who are applying for jobs? And how is that used to determine who can even be fairly considered? And in what ways might this harm most disabled people at the intersections of race, of gender, of class, and other axes of marginalization? And beyond that, we're looking at what we might have to say about the way the AI is intruding into our lives in school surveillance, in police surveillance, in collection of health related data, whether specifically of people with disabilities through apps or programs that are designed specifically to respond to disabled people's needs or whether in devices that are made available to everybody that monitor your health. And, you know, I think we all know, just how ubiquitous those kinds of technologies have become. And I don't think that enough people are talking about what the ubiquity of health monitoring and health related data collection will mean for disabled people who have always already been subjected to intense surveillance and to social policies that usually don't have our best interest in mind.

Well, Alex and Lydia, thank you so much for joining us here today. It was great talking to you. Thanks for having us. You can find the disability benefits and AI and hiring reports as well as the plain language versions at cdt.org, and don't forget to check us out on Twitter, Facebook, and LinkedIn at SendDemTech.