Welcome to Tech Talk. Bye.

Vandana and Lydia, welcome to the show. We're so glad to have you both. Thank you so much for having us. It's great to be in conversation with you, Jamal. I'm also really happy to be here. Thank you for inviting us, and I'm looking forward to having this great conversation.

Algorithmic decision making refers to a wide set of tools, both analog and digital, although increasingly digitized and complex, that make an assessment, a prediction, an evaluation, or a decision. And in the context of disability related benefits, those tools can use a variety of data points and compare them or assess them in the context of existing data, whether real world data or manufactured data in order to make assessment about whether an applicant for a public benefits program is eligible for that program, should remain eligible for that program. And if they are eligible for the program, the tool can make a determination as to how much support under the program the person should receive. In addition, government administered public benefits programs may also use algorithmic decision making tools to assess whether a particular person is suspected of fraudulent activity in relation to their continuing receipt of government benefits. On the surface level, this sounds perhaps very descriptive and not necessarily indicative of either a great efficiency or a great injustice. In reality, however, everywhere that we've witnessed government agencies within The United States deploying an algorithmic decision making tool in a benefits program, it's resulted in cuts to disabled people's benefits, removal of disabled people from the program altogether, and very real and specific harm to disabled people when they are not able to receive the amount of support or care, whether in terms of dollars or in terms of hours that they need in order to continue living their lives and staying in the community. So how those tools work might be a person making an assessment might input a wide variety of information about that person's disability, about their particular diagnosis, about their particular experience of impairment, or at least the observations made by the assessor of the what they perceive as the person's level of impairment. And then those data points can result in this assessment, again, based on either actual data reflecting past decision making, past recipients of this program, or manufactured data. But that process, of course, assumes the data being used is reliable, fit for purpose, and will actually result in an equitable and useful outcome. Right? So that's what it means to be fit for its purpose and to result in an outcome as intended by policy and by the ideas behind that policy. It also assumes that the assessment process itself is moving toward the intended outcome and will result in an equitable, accurate, and reliable outcome, and that the people who are subjected to use of an algorithm decision making tool have reasonable and adequate understanding of how it works, what it's doing, and how accurate it might be in their individual case. And those assumptions are not always founded. In fact, oftentimes, they're unfounded. But I'm sure we'll dig very deeply into those questions in the next few minutes. Great. No. I think that's,

explains a lot of, important, key ideas around algorithmic decision making vis a vis disability. Just to sort of add on to that, algorithmic decision making in simple words or in a nutshell is, you know, auto automatizing decisions through use of huge, databases and computational systems with a goal to, quote unquote, I want to say that, like, with a goal to make the process of making decisions efficient, transparent, accurate. And I'm using the language, here within quotes to, indicate that that's kind of where, you know, that's the perspective of the designers of the decision makers, actually, the policy makers who are the decision makers. And they perceive that these systems will make the process of decision making sans the human or humanistic perspective or humanistic intervention will be more accurate, transparent, and, you know, efficient. So if you remove human beings from the picture, then and you feed the system with models, formulas, then it's going to be standardized. The decisions will be standardized across the data, across the population, and, it would be that's the perception, you know, that it'll it'll derive or arrive at the standard decision for all. However, having said that, this process really is really far from efficient and transparent because, first of all, it's based on an assumption that populations are homogeneous because it assumes that the same models will come to similar impacts or will have similar impacts on all, people or here in this case, all people with disabilities. So it tends to flatten those decisions tend to flatten, the category of disability itself, who's disabled, you know, that whole understanding of who's disabled is simplified through a very simplistic, I should say, like a medical biomedical model. So I remember, Lydia, you mentioned that we're going back to the history at the start of this podcast. It it's something, you know, is happening with this, AI and algorithm decision making that, you know, they tend to use biomedical, understandings of disability, to categorize even who is disabled. And even a very reductive biomedical understanding as well. Yes. And, you know, disability community and disability movement has really fought hard to come away from that biomedical model, but we are actually, unfortunately, going back to the medical model, which is very unfortunate because, you know, we had all these gains. We're still not all perfect, but moving to more sociopolitical model of disability, where disability is not just a by an individual situation, you know, or condition. It's an outcome of social, economic, intersectional realities. And a lot of these algorithmic decision making tools do not factor those intersectional social political realities of people with disabilities. So it tends to view all disabled people as seen. And especially, like, the way it gets used in the global South and global North is very interesting, but there are also a lot of parallels here. That in Indian case, we, you know, the systems are used for categorizing who is disabled and the benchmarking or there's a percentage. If you have 40% or above, then you're considered disabled. And these algorithmic tools are used for that measurement tools. So there's very sort of, like, you know, this sort of, kind of quantification, model is really sort of grounded or these algorithms are kind of grounded within that quantification model of determining who's disabled is devoid of social context, social location, geopolitical, also realities of global north and south. And that that weight also tends to exclude, you know, huge groups of disabled people, from benefits receiving welfare benefits. Because if they don't qualify as 40%, if you have 39%, you're not gonna get any benefit. So it it's, it really tends to, exclude a huge group of, disabled people from basic say safety nets like Social Security, which you know, pensions and, like, affirmative action and so on and so forth. Because all of that, especially in the Indian case, is reserved or, is for people who have 40% and above. And if these algorithms are calculating disability from this very purely biomedical,

there's that absurdity. Right? The manufactured absurdity where what if even from that model's perspective, a person is assessed at 39%? Like, is there a meaningful difference between a person assessed at 39% or 40%? And, of course, again, part of that reductive nature of that biomedical modeling is an assumption that a person's disability is static. And in reality, any disabled person can tell you, like, we all know from our Lyft experience, that a person's experience of disability fluctuates constantly. It's not static.

Right. Right. And, also, like, giving back a lot of power to the medical gatekeepers and armed with, my digital gatekeepers. Right? Like, you have, like, double

is in The US context, under federal law, there are more than 80 distinct definitions of disability. And in part, that reflects a fragmented understanding of what disability is even from within a biomedical framework, let alone incorporating a social civil rights or justice oriented understanding of disability. But in part, it also reflects that different federal statutes exist to accomplish different ends. And so the definition of disability as used in the Individuals with Disabilities Education Act is not the same as the definition of disability used in the Social Security and Medicaid Act. And vice versa, the definition of disability used in the Americans with Disabilities Act, the definition of disability that's used in the Workforce Innovation and Opportunity Act, they all end up having some distinguishing components because the policy aims of different statutes and their implementing regulations are not necessarily aligned because they serve different purposes, which, of course, reinforces what activists and scholars of disability have known for a very long time, that the definition of disability itself is often in flux and shaped greatly by political, social, cultural, and economic considerations in determining what a disability is. So algorithmic decision making tools, just like the tools of law and policy, both help to shape and reflect back our understanding of disability. An algorithmic decision making tool that is designed to assess, for instance, whether a person should receive a certain amount of hours of Medicaid dollars for home and community based services is operating to create and to persist in maintaining ideas about what kind of people and, therefore, what kinds of disability experiences might necessitate that number of hours or number of dollars funded by Medicaid for home and community based services or an algorithm that's being used to determine whether a particular person is engaging in fraudulent activity, fraudulent billing from, for instance, electronic visit verification systems. Again, that algorithmic decision making tool does not exist in a vacuum, but it exists in a social policy context where our laws and policies have often presumed that people with disabilities should be treated with an assumption of suspicion, that people with disabilities should be assumed to be takers, should be assumed to be fakers, should be assumed to be liars. And so the algorithm decision making tool isn't operating independently. It is not either protecting disabled people or punishing disabled people. It is operating in the context of an existing social policy and the system that is created and maintained by that social policy that itself seeks to, in some cases, purportedly protect disabled people, but often in reality, to surveil and punish disabled people for needing services, for attempting to earn and save money, for trying to live in the community and maintain some level of independence.

I, you know, I was nodding when you were saying that, about, surveillance and suspicion. I I do agree, and rest. I mean, a lot of my own work resonates that, reality even in the global South context of India. You know, here, the way it gets, deployed or used, as an algorithm decision making is used for determining the percentage of disability. So as I was mentioning earlier that there are these, medical formulas actually, you know, sort of developed by the government of India. This was pre, algorithmic decision making. This was just you know, that's how in Indian context, they use these formulas to determine if if somebody has a disability or not. You have to first go to the hospital, get assessed, and then, those formulas actually go into each body part. It's very mathematically driven formula. Like, if you if your if person is blind in one eye, they get thirty percent disability. If somebody doesn't have one tool, they get, like, 10% disability. So the entire body is computed and calculated on the grounds of productibility, ability, and that that's the yardstick that is used to measure ability or disability. Right? So so there are these formulas that are then fed into the algorithms. So algorithms are trained on these medical formulas and that are used then to determine who's disabled or not. So the whole definition of disability or the whole concept or or who is disabled is determined by these, biomedical algorithms. And stringent more stringent it is, like, you know, as I said, 40% in why that number is very arbitrary how that came up to be. The goal somehow is also to keep the numbers low so that, you know, it also kind of aligns with austerity. So the government funds, you know, there there's also this assumption that people are faking disability. Right? Like, the more and more people want to just get benefits. And, so these algorithms are designed in such stringent ways to rule out, quote unquote, like, those that are faking, disability. And this the the assumption is that the better more than doctors' algorithms can decide more efficiently about the percentage of disability because you're removing the subjective sort of perception of even the doctors. Although we know that, you know, doctors themselves are part of the medical model, but algorithms are seen to sort of go even beyond and, you know, above and beyond that. And, also, what gets deeply ingrained in these formulas or algorithmic decision making is, you know, the idea of productivity. Right? You know, sort of capacity, incapacity, capacity to work, to make sure that more and more people are sort of, you know, disciplined and pushed to work in the labor market even though or despite the fact the labor market is so exclusionary, and labor market is also, you know, contingent upon local realities. Now for example, like, as I also mentioned that these algorithms flatten the cat you know, definition of disability as well. Now I was speaking to a farm worker who got injured while working in the farm, and he lost his two tools. Right? So all he got was maybe 25% disability certificate. And he was telling me that for me, like, this accident has made me totally unable to work in the farms. So for me, my percentage of disability is, like, greater than eighty percent because the way it impacts my livelihood. And the algorithms, you know, basically computed his disability without any consideration of the socioeconomic intersection realities. Now that calculation is also given to exact same person who might be working in city and may have access to technology or in better infrastructure. So there's no difference at all in how these technologies you perceive, shape, categorize disabled people living in different social cultural context. So I think that's very damaging. You know? It it flattens that difference, and disability community too has been fighting back to be represented better because these algorithms don't really represent the lived realities of people with disabilities. And as, Cynthia, you were saying that it's also our state of disability is, unpredictable sometimes. So it's and sometimes and that's how the policy makers don't understand or do not want to factor the contingencies and the nuances to keep it simple to to make sure that, you know, there's they can impose more austerity, spend less. So so these tools aid in those, kind of, like, austerity politics also to implement those austerity policies.

and it would be impossible to cover all of them in the short amount of time that we have together, right now. However, I would name that many of the issues raised by disability advocates and by allies have centered around concerns about due process, whether that's procedural due process or substantive due process. There are many other issues that arise as well that could be particular to a specific benefit program in which an agency is using an algorithmic decision making tool. But in short, for those who are unfamiliar with the language, procedural due process is the set of protections that exist to make sure that somebody is that someone has their rights taken care of, that their rights are actually respected in terms of how the government engages in decision making. So did the government provide a person with the right amount of notice as required under law? Did the government ensure that a person had a meaningful ability to appeal an adverse decision? Did the government make sure that a person was provided with the right information? Did the government make sure that the person was supported in making whatever appeal they might have made and had access to whatever information perhaps pertains to the nature and the methodology used by a particular algorithmic decision making tool. Substantive due process is essentially whether someone's rights are actually being respected in terms of what the purpose of the procedure actually is. Is the person actually being afforded the opportunity that they are supposed to be able to receive? Are they actually being treated in accordance with the intent of the program? And in the general realm that we're discussing, that means are they actually able to receive and have the opportunity to receive the type of benefit that they are supposed to be receiving or eligible for. So in essence, procedural and substantive due process are about how the government goes about interacting with a particular person in regard to their rights or a procedure or a particular program. And substantive due process is whether the aim or the promise of that particular program ends up actually being fulfilled, and that person's rights are respected in the course of administering and deploying that program and the tools used to do so. So the arguments brought by legal advocates have ranged from, you know, the issue of agencies contracting third party vendors that use a black box algorithm where the people who are affected by it might not even know an algorithm is being used to begin with. And then even if they learn that an algorithmic tool has been used, generally don't know how it works, what is actually in the algorithm, and what are the assessments being made by this algorithmic decision making tool? What are the data points being considered? Where did the data come from that was used to train or inform the algorithmic decision making tool? Who is actually using this algorithmic decision making tool? What do the results tend to be? The average person does not know even if they are aware that an algorithmic decision making tool is being used. In other cases, very famously as brought by Kevin de Liban in Arkansas, people who are affected by the sudden introduction on an algorithmic decision making tool might not even receive notice that the government is changing the way that it makes an evaluation or determination of continuing eligibility. In the Arkansas case, disabled people receiving home and community based services funding through Medicaid essentially had the method of assessment switched overnight. And overnight, people saw benefits reductions by as much as about 50%, which put people at risk of catastrophic loss of support and, potentially, the risk of institutionalization even for people who are able to live outside an institution, had been living outside an institution, wished to continue living outside an institution, but faced the potential threat of going to one, losing freedom, autonomy, and dignity because the state cut their benefits pretty much in half overnight because of adoption of a new algorithmic decision making tool

what really, you know, needs to be also under schooled is that India has kind of embarked on this, like, mammoth task of digitalization, digital governance, you know, since 2009, 2010, 2014 more more accurately speaking with with the, you know, with the regime, that was really invested that's really invested in digitalizing India. And not just in the context of disability, but there is this, digital identification system called Aadhaar, which literally translates into, foundation. So this is the largest world's largest digital identification system that operates on very complex algebra computational systems. And what it does is that it every person, every citizen has, like, a 12 digit identification code, and their identity is linked to that number as well as biometrics. You know? So there's fingerprints and then eye, eyes and fingerprints. So biometrics and then there's this number. And when it started, that that was just it was a number. It was one of the identification numbers, but now it has become kind of the end all and be all of one's identification. And every, part of one's identity is linked to that number, like your bank accounts, yours you know, you need it for everything, your income. So everything gets connected to Adha. Your land ownership, your property information, your bank details, your school information. Now even they wanted to make it mandatory even for voting purse purposes. But then, you know, there were, advocacy groups that, sort of pushed against it. But there's huge, huge data that, you know, Aadhaar, collects. And, every aspect of one's identity is tied up with that. So it's a huge big data and survey used for surveillance, and even the disability, you know, why is it important for the disability as well? Because when person goes to get a disability ID or disability certificate in India, we call it disability certificate, they first want your Aadhaar. Right? And so then that disabled person is linked to this big database, and then they, use the big database for means testing to figure out whether this person falls below the poverty line, in order to qualify for disability benefits. So it's very sophisticated, actually, the way it works. Like, the if one has a 40% disability, that's not enough to qualify. You need to be below poverty nine, and your income then gets determined from your Aadhaar information. So Aadhaar is like the root ID, and then everything else stacks up on it. And, so it's it's, as in the case of Indian in in Indian in in the Indian context, I mean, there's this huge network of digital governance that's being used for delivering social welfare benefits, for surveillance, for controlling the population, populations, every aspect of, you know, social behaviors or, you know, like law and order and so on and so forth. So it's, you know, nothing gets really escaped from this huge digital surveillance architecture in a way. But, also, I have to add that this, in order to get, an Aadhaar I mean, Aadhaar system is, also faced like, disabled people also face a lot of challenges in registering for Aadhaar because, you know, there are reports that show that Aadhaar was inaccessible for people with disabilities. Like biometrics, like, a lot of people with disabilities, such as with, you know, people who were on the spectrum, were having difficulty getting enrolled through biometrics or people who had leprosy, you know, with their, fingerprinting. It was, you know, not recognizing their fingers or people with vision impairment could not see, look into the camera very clearly. So they were, like, embodied barriers or there are there are still, embodied barriers in getting digitally included, so to say. Right? So because there's a small assumption that digitalization is gonna include everyone, but it also shows that, you know, people with disabilities have embodied have faced embodied barriers. Secondly, there's a huge digital divide, in the global South. So people in order to get, included in this digital architecture or infrastructure, it it's the process not seamless for all. Like, those who live in rural areas, those, communities that don't have access to technology, they have to rely on government bureaucrats. They may have to bribe them to, you know, get these identification and things like that. So there's another there's a huge new issue of, like, digital divide that's happening because, you know, digitalization has been accelerating, but it's also, like, you know, creating new kinds of exclusions, which are unique to global South, but I should not I should say even in global North, we know it's not just you know, it's not equitable for all communities. I mean, we see a lot of, like, digital divide in The US as well, you know, across, race, ethnicity, gender, disability, and so on. So those kinds of, like, digital divides in in the global South, happen across the lines of, like, you know, morality and caste in the Indian context also and religion and so on and so forth. So there is that. And then there are unique, sort of tech there's unique texture to digital exclusion in the global South because these lot of these technologies were promoted to from, you know, to fight so called corruption. Like, you know, there's an assumption that, you know, every bureaucrat is corrupt. And then if you take the decision out of their hands and just computerize everything and just it's gonna make everything transparent. Right? So but that hasn't been the case because these algorithms too, you know, they they have their own biases. They're they're replicating a lot of, like, social inequalities, you know, and they so those inequalities have not gotten rid of by these technologies. So, I mean, it it's, there are these newer challenges, experienced by people with disabilities who have to now have Aadhaar, who have to go through these digital assessment systems. And just one, interesting comparison I wanted to draw upon, as Lydia was saying that with this new decision make algorithmic decision making system that was introduced, it cut the benefits to half. Right? In my case study, it's actually opposite, but in a very sort of this biopolitical manner. The software the the algorithms I'm studying in in this region of South India, this was introduced, when the government was deciding to triple the welfare benefits. So, this, you know, government the state government decided it's only in one particular state of South India, Telangana, Andhra Pradesh. They decided to expand the welfare benefits by threefold, but they wanted a system that would make it harder for people to qualify. So they put all these algorithmic systems to make the, you know, inclusion criteria very strict. So there was this reverse process that happened, but with the same outcome. While the pensions were increased, it made it very hard. These digital processes made it very hard for people to qualify for these benefits. So there was this kind of, like, you know, biopolitical process of inclusion but through exclusion. So I I just thought it was a good similar but different example there. What do government solutions look like? And are these solutions limited by the fact that quantification

Right. So, government solutions, as in, like, government does perceive these as solutions, first of all. These solutions are creating different problems at the same time. But these, so called solutions were designed to fix some systemic problems. Right? Like, systemic problem of inaccessibility, systemic problem of, say, corruption, systemic problem. But then the corruption like, in the global South, like, the problem of corruption, then these systems are putting the blame on the person rather than the system. The thing that is the I mean, it's the person who's the corrupt person who's trying to fake disability. Right? So it's kind of like these technologies are trying to make the process so stringent, but it's also the state that's corrupt, right, that's trying to impose such such austere excluding systems. But what I was trying to say is that the so called solutions these solutions, of course, in the Indian context were designed with the goal to make the system more transparent, and quantification was seen or is seen as one way to standardize and simplify the decision. Right? Like, you sought population, especially disabled people, into these percentages. And based on the level of their productivity, then you grant them entry into the welfare state or you kind of exclude them from the welfare state. So all those important decision of benchmarking disability are done through, like, quantification idea or ideology of quantification and measurement standardization. And that, again, you know, the process of quantification as we've discussed earlier, first of all, it flattens disability. It does not take into account the unique impairment effects, the social realities that, you know, just, two individuals, for example, if they have a vision impairment, the impact may not be the same because it will determine my the impact of their impairment would be determined by many other factors such as their class, race, caste, gender, and other such realities. And those complex social identities or realities of their life, the lived experiences are excluded from these decision making and excluded from quantification that has now become sort of the foundation for algorithmic decision making. So in simple words, quantification tends to exclude folks who are intersectionally marginalized much more. So that's one aspect of how quantifications become so central and how it, you know, precariously sort of excludes people with disabilities.

For me, the question of solutions, again, to echo Vandana's comments, also raise the deeper question of what is being solved. Right? Because the from the perspective of advocates and the perspective of lawmakers, that answer might not be the same. The problem as a government agency sees it and much less a particular elected official of any party in any location may not be the same as the set of issues as seen by advocates and those who are directly impacted, in other words. And so when I think about solving for harms, I think about risk management and mitigation. We don't currently have comprehensive regulatory or legislative protections against algorithmic bias and algorithmic harm in The United States. And because of that, advocates have to rely upon existing legal tools and frameworks in order to advocate for people's interests against unfair, inequitable, risky, or dangerous decision making enabled by automation and algorithm. Solutions would look like policymakers engaging and collaborating directly with those who are actually impacted from disabled communities, especially those who are multiply marginalized, to devise policies that work for us, that ensure robust privacy protection, that ensure attention to and protection of all due process considerations, and that avoid the unnecessary use of algorithmic decision making tools for purposes which they are not fit for purpose and limiting their use in contexts where it is potentially possible to deploy an algorithmic decision making tool that does not result in wholesale cuts to an exclusion of people with disabilities from the benefits and public entitlements programs that actually sustain life, civic participation,

and continued inclusion in the national economy. Yeah. I would actually like to echo some of that also that solutions. I think I wanted to, you know, reflect to on solutions from the perspective of the communities. I mean, that's really it's it's about really incorporating the voices of the communities into the decision making so that it has to become more and more collaborative. Because I think those the feedback right now or it it is just a feedback. Like, it's not really representation. Right? So I think we need to center the lived experience and the voices of people that are being the algorithmized. Right? I think the for whom the algorithms are designed, like, or targeted towards. I mean, you know, it's there needs to be greater transparency about what the goal of these algorithm is really and these in sort of disrupting the algorithmic the opacity, so to to say, also. And then in Indian context as well, we don't have proper legislation, although there has been a lot of recent, like, conversations around it. And, you know, there's a lot of need for very urgent need, I should say, for legislation because the speed at which these developments are taking place is lightning. You know? So we we have to come up with lightning solutions as well because the tech, the big tech is moving too fast, and it's leaving behind a lot of voices. Lot of, like, important legislations have to come up with, and I think disabled communities or disability justice perspective, and disability communities have to these issues have to be taken up much more by the disability justice. And so I would say that even, like, having education around these issues within the disability justice movement is very important because it's like so much of it is, like, framed within these technical discourses that becomes very, very hard for people to understand what this is all about. Whereas all these decisions are made on their behalf, but people don't even know how to code and decode it and design it. So I think as citizens, as, advocates, we really have to promote digital literacy in some way or literacy around this digitalization within our movement spaces. And I think that's, that's really important. Right? Like and transparency bias all those issues. I think there's already awareness that's growing on those issues, and I think it just needs to be much more.

I I know thank you, Jamal, and wonderful to meet you, Lydia, and have this conversation. I think we should continue this dialogue. A 100%.