Yeah. Thanks so much, Louise. So this talk is based on a report that we just put out, this summer called algorithmic accountability in The UK, how FOIA sheds light on automated welfare. And I I co wrote this with doctor Ali Spring, my colleague at the University of Edinburgh. The report was it was really inspired by, my long standing interest in this topic of, the Department of Work and Pensions use of automation for fraud detection. This is something I wanted to study for a really long time. A couple of years ago, I remember talking to, people who know this topic much better than me saying, I really wanna I really wanna study this. And I'm saying, you can't. There's just no information about it. There's very little information online, or in the public domain, and the DWP is, not going to talk to you, for for security reasons. So I put it aside and then came back to this, just recently in part because I realized that I could use what do they know as a way to get information about the systems that are being used. I'll get back to that. For now, just to say a little bit about the topic. So the DWP has been using data matching and data analytics, we we know, since at least 2010. So this is using their own data on claimants and comparing it to other datasets that have information about claimants. So for instance, who's living in their household to see if they match or to find anomalies. We also know that since 2022, the DWP has moved into adopting predictive machine learning models. So these compare claimants to past claimants who have committed fraud to see if new claimants share similar profiles. If they do, then they'll be assigned a high risk score for potential fraud and error. The DWP has been asked to share more information on these systems. An example of that is in 2023, the information commissioner warned the DWP that it risks being in contempt of court for not releasing more details about this, because very little is known about these systems. So that's why there we can we can say there's a real lack of public scrutiny about automated decision making systems being adopted by government agencies. I would direct you to a really great article by Scott and Edwards that came out this year about this. So they it it's a paper called the inscrutable code, the deficient scrutiny problem of automated government. And they argue that those bodies that are supposed to be overseeing government agencies, so in particular, parliament and the courts, they really don't have the adequate tools right now to address this problem. And that's where civil society, I think, comes in and is playing a a really, really crucial role. I've been noting civil society is is way ahead of us researchers, in in writing reports that are bringing these systems to light. And they do this in several different ways. Public campaigns, you're gonna hear from Big Brother Watch, Jake. They do really fantastic public campaigns to get the word out to the public. Some are are going to court. An example of this is the child poverty action group taking the DWP to court over their automated payment system and the way it affects working claimants unfairly. Online registries of these systems. So the public law project has an excellent database where you can just go to one place and get more information about them. Writing evidence to public reviews, to parliament. And I would say a lot of these are based on freedom of information requests. So these, organizations are getting more information than what's in the public domain using FOIA as a tool. So that gets back to, how, I think we can, as researchers, leverage, archives, like, that have been amassed by what do they know and other websites like this to look across all the different freedom of information requests that have been, put to different agencies about these tools to get a a bigger picture, kind of a gestalt. And we call this the FOIA Mosaic method. So this is where you, combine FOIA requests, that that you can get from websites, like what do they know, with other public documents to get a bigger picture than you could just by looking at what's in the public, public domain. So to do this, we we searched for keywords that had to do with automation and Social Security fraud. What do they know had this great, they you can you can refine the search by asking to only look at requests to certain agencies. So we only wanted to look at those requests going to the DWP. Doing this, we collected 51 requests, and we analyzed these in chronological order. And we also looked at public documents that were being referred to in these requests and news reports that were being referenced. We found that most of these requests are responding to the DWP's, annual report and accounts that come out every year. This is kind of the starring document, because it will say something usually about the fact that they're using automation because it discloses that they are doing something proactively, to try to catch, rampant fraud. So they'll they'll mention it, but usually say very little about what these systems entail. And so these reports come out, and that kind of spurs a flurry of requests by people wanting to know more. Okay. So what did we find out? I'm just gonna give, high level findings. The report has more granular detail on all of this, and I'd be happy to answer questions, as well if you have any if you wanna get more details. Okay. So some of the high level findings. We were able to find out from these, requests. Some of the organizations, the DWP, has been sharing their, they have data sharing agreements with for their data matching and why. We've been able to find out just a little bit more detail about five machine learning models that they've created. These are the predictive models that risk score people. Not much detail, but we know what they're called and in part what data they're drawing from. We've gotten a bit more insight into the various teams that do this work. And we also can see how the DWP is monitoring for risk and for fairness and bias. So this is mostly through, data protection impact analysis that they've, released to some requesters. DPIAs, as they're called, are required by law if you're using personal data in a way that could carry some kind of risk. So they've done a lot of these DPIAs. And so we, so they've, in our corpus, of answers to these requests, we have the primary source DPIAs that were given to people who requested them, both about the data matching and the machine learning models, they're heavily redacted. I mean, in some cases to a point where you really can't get anything beyond, the the names of some of these systems, but very little else. Another really key kind of primary source document that was released to one requester in the dataset is a fairness analysis that was carried out on the first machine learning model. That first machine learning model is called the advances model. It's it risk assesses people applying for universal credit advance for a loan. DPIAs, I should say, are carried out before you start designing these systems or using them. The fairness analysis is post hoc. So you're testing the system once it's been built for bias, and in particular, to understand how it works across different protected characteristics. This fairness analysis showed that there were some issues, with certain, people with certain protected characteristics being flagged more than others. So that that's kind of a high level of what the what the requests told us about what the DWP is doing internally. So shedding some light on these systems. What's interesting about doing this this method of FOIA analysis is you can also learn about how the DWP, responds to FOIA. So you can get, kind of an overarching view of what, to what degree they're actually answering requests versus, refusing them. So out of the 51, only seven were fully successful. 24 were partially successful, so the majority. So that means they would share some information, but reject sharing all of it, all of the requested information. You also have, 16 refused. And then in four cases, they said they didn't have the information. We also see that when they refuse requests, they're mostly relying on on section 31 of the Freedom of Information Act. So they have legal protection not to share because there's arguing that releasing that information could jeopardize law enforcement, so catching fraudsters, essentially. We also found that in all cases except one, the DWP only released DPIA's after the requester made a complaint to the information commissioner's office. What that means is that other people making the same request did not get the DPIAs, did not get those documents. And what we found, therefore, is that there was disparate treatment across requesters. So those who had the time and the resources and they kinda know how to go through the complaints process e the process, the ICO, they would get the information, but people who didn't wouldn't get that information. Okay. One thing I was not expecting to find, and I'm starting to wrap up here, is that you also can learn a lot about kind of civil society actors and the tactics they're using and how they use FOIA. So for instance, often in their request, they'll refer to public documents. So you can trace They saw a public document that kind of, instigated them to make the request, and you can trace kind of their their dialogue as they go back and forth in some cases with the DWP to get this information. You can also link FOIA requests to the news cycle. So in the case of the, of fairness analyses of the first model, that release of that document was given to the Guardian, which, made this issue very public. So you can see how this work makes gives more visibility to these issues around, you know, transparency and the use of automation and these and and the use of these automated tools. You can see how some requests build on past ones. So some, people are actually using you know, they're using, what do they know? So they can refer to past requests and say, well, I wanna know more about this. And what all this shows is is the role, of civil society of making something, you know, align in a public document into a public issue and and giving it, you know, making it something that is publicly discussed and of concern. We have some recommendations at the end of the report. First of all, we think all these documents have been released. They're public on what do they know. We think they should be made kinda more systematically public by the DWP, either through their website or by using the the UK government's, algorithmic transparency recording standard hub. We think they should just disclose, even if they're redacted, all the DPIAs and any fairness analyses that are being carried out for their machine learning models. And this is how you know if these models are biased or not. And we think they need to have a more consistent approach to FOIA requests so that some people aren't getting more information,

Finally, we're gonna go to Jake. So, yeah, Jake from Big BrotherWatch. I run investigations here. So we focus on tech and digital rights kind of, across the board, really. So everything from welfare to facial recognition, sort of, home office encryption stuff that's been in the news today, so all kind of that. But in terms of our work on automated decision making, that has mostly been in the welfare system. Also, it's been around facial recognition as we believe that's an automated decision. And from my experience, I've been working this for five years. There's been a real retreat and transparency over that time in terms of how much the government's willing to tell, how much it's willing to disclose about the ultimate decision spoke to me as civil society, to book Gabriel as journalists, or Morgan's academics, but also to people impacted, which is really important. So a lot of people impacted aren't told about that, aren't told about these decisions. And in the first place, that's that's really a huge problem. If these are making decisions about your life and you don't know about it, how can you then seek redress when it goes wrong, get that accountability? It's really important in the same decision making. I think we've seen a backtrack on transparency kind of in the very recent data use and access bill. Many of you sort of know have worked on that, know about that. But previously in British data protection, we had protections against whole solely automated decisions. These are automated decisions that have big effects that had no human involvement. We've seen that gutted in the data use and access bill. So we are going to see over the next twelve, eighteen, twenty four months a massive in expansion of British state's use of automated decision making. It's going to be less transparent than ever because as the law stood, there was a right. If you were you had one of these decisions, you have to tell you what decisions made about you with a right to be told about it. So many of you, if you applied for a credit card or a loan online, that's been automated, There's been something in the TNCs that you may have seen. You have a right to have human look at this, and that right is going to be watered has been watered down hugely in legislation. So already there, we're seeing a huge drawback and transparency around the use of automations you're making. And the secretary of state has Henry the eight powers. The secretary of state just designate certain categories of decision making as exempt from the few exceptions that do exist. So we might be in a situation where they decide leasing decisions are suddenly exempt from the protections. Some people have to be told that these decisions are made about them automate automated procedures being used. So that really is a problem. In my work over the past five years, I've seen time and time again, even the protections that used to exist, those really been exploited. A lot of government departments on sort of city local government basically use a threshold that was ever so slightly below the legal threshold for telling people that automated decisions were being made. So there's a system called risk based verification. It's mostly been phased out, but was used to effectively risk score triage people applying for housing benefit or council tax support at local authorities. And there were risk scores as high, medium, or low, and this determined how much evidence you had to provide to get access to benefits they're entitled to. And the human involved in this decision wasn't never allowed to downgrade the score. They're gonna upgrade the score in incredibly rare circumstances. But every single local authority in this country argued that that was a human involvement in the decision, so it wasn't solely automated, which meant individuals' right to be informed, that right to for it to be transparent was was removed. In my personal view, and I'm not a lawyer, I think if that was challenging the courts, if that was challenged about whether it was a solely automated decision, somebody would have won. But that's a problem with data protection law when we're kind of looking around this transparency is that individuals have to challenge. It's very hard for organizations such as big brother, what should lead lead? That challenge has to be an affected individual. And how does that affect individual launch a challenge and try and push back that use of these decisions if if they're not aware of it in the first place. So that leaves it to people like myself and Gabriel and Morgan to dig into it and raise their awareness. And we we kinda see that the state is really is really gonna tell tell us either. So that's you end up in this kind of catch 22 where the law is weak. So if its individuals are unable to find out how how they're affected, So people like myself end up having to do these FOI phishing exhibitions to work out what's going on and where can we try and gain transparency. So I think, yeah, this gusseting of 22 and those protections, we're going to see a further decline in transparency around algorithms and automated decisions in the next twelve to eighteen months. It's not quite hitting yet because the law's already kinda just need to force that. I think it's something that's going to be incredibly concerning going forward. But I'm mostly gonna talk about kind of a very small bit of transparency things. Perhaps the most important one, I just want to often go as unaddressed, which is transparency procurement of these automated decision making tools. Because quite often, when smaller bodies of state so local police local authorities don't. They procure them. So we will see procurement notices going up. We've seen them recently for the all the new police forces who have acquired facial recognition funds. They're procuring this this technology. Essex Police procured a different facial recognition technology. The data science firm, Zantora, who are kind of a knockoff British palantir, So they're procured by local local local councils to provide the services they do. And because it's provided by a third party who argues that this is commercially sensitive information that revealing their algorithm would be detrimental to their business, The transparency one would expect from government decision making is almost nil. So a good bit about the risk based verification, which was this, triaging software that was used to risk score people 5% benefits. Biggest biggest part of this was that was that was that Zantora kind of did some palantiric company that quite they're a British company quite big in The UK. After a long FOI, I got hold of actually using the senate. Morgan Cook talks about kind of the mosaic mo mosaic FOI because it was used by a few dozen local authorities. I asked that she doesn't look the authority of documents, and they all gave me slightly different ones. In between that, I was able to really put together quite a detailed look at how they are working. And in these documents, you'd see a list of predictive characteristics that we use to decide how risky somebody was. But they said explicitly in these documents, we are not telling you what full list of, predictive characteristics is because that would be that potentially reveal our sort of, trade secrets, our commercially sensitive information to the local authority, and they knew they were subject to FOIA as well. This stuff might have to get published. So in a situation where people are getting branded high risk by the local authority, being put into huge demands to provide evidence to their benefits claim, A local authority had no idea why. They didn't know how each printed characteristic was, sort of rated, how much weight it was given, but also didn't even know all the ones that have been used. They were just assured by Centura that it was all fine. This is a massive problem. The public sector calls you duty isn't something that's delegable. State state bodies aren't supposed to say, well, our supply tools, it's fine. The duty is is incumbent on the on the state act to to comply with the PSED. And this situation, again, in my nonlawyers view, this was not happening. How can we have transparency in decision making when the state the state actor is fundamentally making decision doesn't know all the things that go into the algorithm? So this is I found this really worrying and kind of as as a practice going forward. How can we have state decisions that then can be challenged and be held to account if the state can't explain in the first place? So that really was a massive problem in the Zamora. And I think there's also, again, smaller kinda smaller organizations as a technical knowledge gap. Zantora often went into organizations and integrate themselves very strongly with kinda local councils. They provide the technical knowledge. So it isn't just that local authorities and these other bodies weren't told what what what was actually being used to make these decisions. It was they didn't really understand it because they these are kinda cash strapped small local bodies. They don't have people who are trained in computer science and machine learning. So it's the reliance on external knowledge and these commercial commercial interest kind of commercial interest secret keeping is creating real problem with transparency. And as we're having more and more kind of AI and algorithms, more companies of various repute selling into The States. We're gonna see we're gonna see this grow because these companies need to defend their their local knowledge because they as a company, which sees their function, they couldn't sell to more and more people, so they're going to defend defend this. I'm going to have peep we're going to have people asking, well, why is this decision made about me? And the state the state won't be able to explain it. And then for people like myself who are trying to get this through FOI, session 43, as much as I often object to how it is used to kind of keep this a secret, is legitimate exemption. So much old to know know these things, making a company go bust, rip yeah. By reading their trade secrets probably isn't isn't about the public interest. So we we need we need to, I think, my view, reform reform the law around this to find a way to make sure that these private companies can't just hide behind motion exemption exemptions and not tell the state how it's going on. Another example is how Essex Police are the only police force in The UK is call site to an Israeli Israeli face recognition firm who are also being used by the IDF. But their pub their equality impact assessment is an absolute mess. It cites assess assessment for a complete different algorithm made by a completely different company as evidence that the course algorithm is fine. It cites it cites evidence from marketing materials from course site from the US Department of Homeland Security. The Department of Homeland Security have gone on public record to say incorrect and really questionable interpretations of some other US studies. Essex Police have never done any any testing themselves. Apparently, it is ongoing now. They've been using use facial recognition for almost a year with an equality impact assessment that is either based on marketing material or based on tests on a completely different algorithm. And that would that'd be like saying, well, Windows does this, so you can do it on a Mac as well. They're not comparable. But, again, because that lack of the this kind of marketing dazzling, this kind of lack of technical expertise within pure embody, it it means that the transparency tests aren't done, that we don't really understand how they don't really understand how it works. So how can they be transparent about that decision making that's going on? So I think for me, kind of, we need to really look at the rules from procuring AI in the first place if you want if you want to be transparent about it going forward. We can't allow commercial interest in commercial that's kind of commercial arguments to shield private companies that have a supply in the state from any any kind any kind of scrutiny. And I don't know whether we need to give the RTO powers to go in go into this and verify. I don't I don't know what that looks like. I'm an investigator or a policymaker. But as it stands, the commercial interest exemptions are incredibly strong, and we're not enforcing laws we even have around the qualities because the people that are procuring it do not understand how these systems work. So that's kind that's that's kind of a huge problem. And I think the black box more more generally again is transparency thing. We could've all heard about it before. But I think as we're becoming more and more AI, we have to think not just transparency for people like ourselves in some society, but actually people who are impacting sales to really matter here. If having people having huge decisions made about their lives and that state can't explain it, that becomes a problem. And I think for me, it's a red line that we shouldn't be using this stuff anymore. So that's something interesting about yeah. But, again, most of the procurement because I think that's it's a huge kind of huge kind of gap. We look at the impact of these systems, but for me, the problem starts when these systems are bought and supplied to this day. Thanks, Jake. Yeah. Hugely interesting to hear about the amount of procurement going on, that that gap in knowledge,

database that they keep on these tools that the the DWP is frequently mentioned, but the one that's mentioned, I I wanna say even more, is the home office. I don't know, Jake, if I have not looked into that myself. My focus has has mostly been on the DWP. Jake, I don't know if you, Big Brother Watch, has looked into this with regards to the home office or if you have other answers as well.

I think the DWP are particularly bad. I I in the tribunal court tribunal hearing at Foyer, I got to say to where they started national security arguments, which were a complete nonsense. The home office think of the worst of the home office because they do have they do have national security duty that often becomes something they hide behind. They're not great. And surprisingly, DCIS as well are an absolute nightmare, I think. So they're kinda three that three three that are worst. Surprisingly, MOD.

Canadians have a decent transparency register they've been trying for a while. I don't know how much detail they've kind of they saw them in, but that was quite good. The Dutch seemed to be reasonably forthcoming as well, especially machines work. And The UK is trying its algorithmic transparency reporting standard, but in my view, it's it's too weak at the moment. The, the various government departments refusing to comply. So on paper, that yeah. The UK system could be good if if government actually kind of followed through and enforced it properly. But at the moment, I mean, it might be they've been they've been too weak letting departments get away without actually complying with what they're supposed to do. I have two examples in The US.

So one is a few years ago, New York state, required that any government agencies that were going to use automated tools to register it and make some details about it transparent. My understanding is that that hasn't been robustly followed through on, but I haven't looked into it lately. But I do know you could at least look up that that law that was passed. Another, I'd say, kinda gold standard is a system, and it's also in The US. It's in Allegheny County. This is a county in Pennsylvania in The US. There is, so this they have a a they use machine learning risk modeling in order to score people for risk of child abuse, and they have made the underlying data available to researchers. So there's actually a lot of research into those systems, kind of similar to what Lighthouse, has done with, using some of the underlying data that they've gotten access to, in The Netherlands. A lot of people have been able to research the system because of the kind of fairly, I'd say, actually relatively radical approach to transparency that they have. I would also point you to understand more about that to Virginia Eubanks' book, automating inequality, which has a whole chapter on that system. So she comes away very critical of it. But in terms of transparency, it's been kind of, like like, a gold standard bearer.

And also add in The UK, Bristol City Council are probably the best in terms of their transparency. They're not flawless, but they are much, much better than basically anybody else. They tend to even they do tend to disclose not the actual dates, but the datasets they use, the algorithm themselves. So as much as they love how they use them, I'd be more complimentary. They at least are much more open about what they're doing. A lot of places are a lot less transparent than that. So if anyone UK based, that's probably good. Let's stop. And two questions from Cara here. With the heightened demand for energy and water,

and I'm I'm not an expert into the, energy, demands of of what the DWP do is doing by any means, but I do think it's important to make distinctions. So a lot of the high energy intensive AI would be generative AI or AI that's being trained on massive colossal datasets. Now this is not what the DWP is doing. What they're using is actually fairly traditional machine learning to data match or machine learning to, predictably risk score claimants. And that, I don't think, would be in the same category of kind of energy intensive, AI that that that a lot of us are starting to kind of adopt in our daily lives, with generative AI. So that would be my answer to to that question. That said, we're seeing just not to go off on a tangent, but we are seeing more and more policy rolling out requiring agencies to start to to adopt generative AI and to and to experiment with it. And so I think that's gonna be an interesting development in this conversation. Not something I know anything about at the moment, it could be because it's so new, but I think this will be kind of, like Jake was hinting at, kind of next steps or next horizons in this discussion.

In terms of the kind of GDP and next, the care the carers allowance there, a lot of the the UDP's tools do in theory seek to find all kinds of kind of fraud and error in the system, but their focus is obviously is on kind of what they they see as criminal fraud. So it may be that this could be that this one small kind of silver lining, this kind of general kind of horrible use of AI could be that some of this stuff gets flagged. But I think it is the approach general approach to the DBP to be fraud and error treated as one which which is a problem which you're going to get. Huge amounts of people who've made a genuine mistake vulnerable situation, making error because people make errors because we're human, and getting treated treated could be a criminal. And I think the carers allowance is a really kind of worrying example of this that people are getting hammered for being able to pay the tiny amount. I think we could see with these kind of large models that are just looking for is a is a claim wrong rather than someone committing criminal fraud, that people are putting through very stressful things for genuine human error, and I think that's it's gonna cause all sorts of problems going forward. Absolutely. Thank you both for great answers.

But, yeah, the reasons behind this trend and what you're seeing. I think the first reason is probably bad press coverage. We've seen we've seen stories like kind of, the stuff Lighthouse has done over in Europe. We've seen this young marriages outcome getting absolutely battering after PLP's work over in The UK. So I think there's that kind of this kind of this, yeah, this kind of bad press coverage is gonna make governance more secretive. And, also, these models get increasingly increasingly complicated to actually put this information out. It's become much more difficult. It's not now just a kind of a basic regression model where it's like, here's some characteristics. Here's the output. It's the black box is inherently opaque, so they become they're coming becoming more more more more opaque in tone. I think they're all just becoming more more more integrated. Governments are as opaque. They've always been. It's just the AI used to be sitting on top, and now it's much more integrated. So as it goes to the half of government, it was gonna get lesser respect transparency as it becomes more integrated into the day to day decision making, I think.

Yeah. I I I agree with Jake. I think so in the dataset I looked at, it was very interesting. The request start in 2018. And at the at the beginning and and and throughout that year, in the next couple of years, they the DWP would respond by, to request for DPIAs by sending very lightly redacted ones. This would be for the data matching. And then there was a point at which I wanna say it was in 2020, where that just stops. And any DPIA that they give after that is heavily redacted. And I I agree. I think it had to do there was a it it was in 2018 that Virginia Eubanks book comes out that Philip Alston writes a scathing report about, the UK government and about the DWP as well with regards to these tools. And I think it's it there was a skittishness that started at a certain time period. And ever since then, it's been harder. But that shows up in the in the dataset, actually.

I think that's a report I put out. The BrotherWatch sort of again sort of the simple solution by design. As I said, you know, on the government's general matching service, the data matching service has been been running for about a decade. And I asked there were some documents already in public domain. I asked for very similar documents this year, and the transparency over over over that time has declined massively. They wouldn't give me documents that are basically already in public domain from a few years ago. They wouldn't give me the new versions. That kind of really, I think, really illustrates the the transparency backsliding