Algorithmic Fairness, Embedded Privacy –– Talking Tech w/ Ali Lange, Gautam Hans, & Michelle De Mooy
Host Brian Wesolowski sits down first with Gautam Hans and Ali Lange to talk about new joint research from CDT and Berkeley that addresses algorithmic fairness, specifically what people think is fair in terms of the automated decisions made about them.<br><br>Then he jumps to Michelle De Mooy to discuss a new joint report from CDT and Fitbit that looks at how health wearable companies can embed privacy and ethics into their research and development process. <br><br>BERKLEY RESEARCH: http://bit.ly/cdtberk.<br>FITBIT REPORT: http://bit.ly/fitbitcdt.<br><br>Attribution: sounds used from Psykophobia, Taira Komori, BenKoning, Zabuhailo, bloomypetal, guitarguy1985, bmusic92, and offthesky of freesound.org.
Top Keywords
- gender 0.010
- personalization 0.010
- data 0.009
- fitbit 0.008
- race 0.008
- research 0.008
- information 0.006
- report 0.006
- company 0.006
- companies 0.006
- results 0.006
- recommendations 0.005
Transcript
Welcome to Tech Talk. Bye.
CT. Tea.
Welcome to CDT's Tech Talk, where we dish on tech and Internet policy, while also explaining what these policies mean to our daily lives. I'm Brian Wasilowski, and it's time to talk tech. In this episode, we'll be talking about fairness, ethics, and privacy. We'll dig into some new research from CDT and Berkeley that addresses algorithmic fairness, specifically what people think is fair in terms of the automated decisions made about them. And then we'll be talking about a new report from CDT and Fitbit that looks at how health wearable companies can embed privacy and ethics into their research and development process. Both of these projects are very real examples of how technology is impacting our daily lives. Most people know that they are being profiled and targeted to some extent online. This personalization of content, whether in terms of the ads we receive, the search results we get, or what prices were offered, might often seem trivial. But sometimes, there are some very real world implications. An ad served up for great deals at local restaurants based on where you live might be welcome. But what about search results that link to predatory lenders based on your income level? Yeah, that seems not quite as welcome. CDT worked with a research team from the from UC Berkeley School of Information to explore this ickiness factor. And more specifically, figure out how individuals feel about specific kinds of personalization. Joining me today to talk about the report are Ali Lang and Gautam Hans, who led the project for CDT. Welcome. Thanks, Brian. Thank you. It seems like I get the two of you on a regular basis together. That's exciting. And you're always such joys to have on. So first, tell me a bit about the research and kind of the questions you were trying to answer here.
Yeah. So we, we're working with this research team in California to figure out how people feel about personalization online. And so it's a little bit different than a lot of the research you see coming out of the tech policy community, and so far as it wasn't focused on what's happening online necessarily or uncovering sort of the technical pieces of how things are personalized or what's personalized or why, but really focusing on how people feel about personalization in a general context in ways that we know that stuff is personalized, in ways that we think it might be, but maybe we aren't sure whether or not it is, and we aren't sure how widespread it is. So we really took a pretty wide swatch of questions and wanted to get, feedback from people on what they thought was fair, what they thought was beyond, you know, the line of fairness, what they thought was really unfair and get a sense of, you know, just how people feel about this, which because consumer expectations are a really important part of privacy advocacy, and it's not something that we have a lot of access to and certainly not in such a concrete and and done with such good methodology as what the researchers, have produced here. So that was really where it started.
Ally did a really good job summarizing the project. And the only thing that I would add because, we were really grateful to the three students we had, one of whom was an intern at CDT, Reyna Cohen. We, had approached them because, we knew that there was some funding available and the Berkeley Center for Technology Society and Policy and the Berkeley Center for Long Term Cybersecurity funded this work. Because it's empirical, we, you know, had to sort of figure out how we were gonna get the survey out to a number of people, and, it was good to have that funding. I think also, in policy, especially in technology policy, we spend a lot of time or I spend a lot of time talking about people feel this way or people feel that way, and it's all very generalized. And lawyers are very bad about having data. And I don't think data is perfect. And, you know, statistics can be interpreted a lot of different ways, but it is useful and helpful to have a sample size of, I think, 750 people to point to and have some numbers. And, you know, we we hope that other people do this kind of research and take it forward, but it's a nice ad. And I think part of the reason the project was so appealing to us at CDT. That's awesome. So you'll be adding all these data points to your talking points moving forward, and your advocacy will be oh so much better. Correct?
Of course. And and, actually, the one of the brilliant things the research team did, and because this isn't the the questions that the survey respondents were asked weren't necessarily tied to something that is happening currently in the moment, you could actually give the same survey next year, the year after, ten years from now, and get sort of benchmarks moving forward of what of people's like, the change in people's feelings. You could also give the survey in different cultures and see if people feel differently about the same Fascinating. Categories. And so it's not tied to, anything in particular. There's not brand names mentioned in it. There's not dates. There's not you know, it's the the context is removed. It's just a general description of something that may happen online. Awesome. So alright. So let's get to
kind of what you found. I know that you're still working on the final paper, so these are kind of preliminary thoughts and analysis on it. What were the categories first of, like, the data personalization that you guys looked into? How did you split it up?
So the the survey respondents were shown vignettes, which are small stories that contained three or three or so, really pertinent piece of information. So it would be something, like, you would see a a story that would say, you know, you are shown an ad online that is personalized to you based on your gender. Your gender was inferred and is accurate. Inferred means that you didn't necessarily indicate it like you might on Facebook, but rather, in fact, they guessed what your gender was. And and then you'd say, how do you feel about this on a scale of one to five? Is it fair or not? And so that was sort of how the format of the question. And the categories that could have filled in in those different places were advertising results, search results, or results, or prices, like, the the thing that was personalized. They studied if it was personalized to you based on your gender, your city or city of or or town of residence, and your race. And then the information was either provided, accurately inferred, or inaccurately inferred. Okay. So that's a huge matrix of information. And the team actually uncovered just tremendous amounts of data, plus the respondents had to fill in a box. They had to fill in a comment box. They they couldn't move to the next page without doing so. So we got a lot qualitative feedback as well. But those so the categories of information that we chose, we did specifically because we wanted to have we thought city and town of residence might provide something like a baseline because city and town of residence is a really non granular location. It's not personal. It's not something that I can sort of reverse engineer who you are. People don't necessarily consider where you live private, I don't think, in in that in that, at that level as opposed to, you know, your neighborhood or your house or something that's much more specific. I mean, we used to have phone books that you would get delivered that was very clear, and no one seemed to think that was icky. But Right. And so so there was that. And then we wanted, to sort of see if compared to that, if, like, race and gender were were perceived differently, and income and household income. And then we also looked at, Advertising is something I think people understand is is a side effect of the the market of the Internet right now. So they're a little bit more, you know, ambivalent about it. They don't feel like it's as threatening as something like search results, which, people feel are are much more important because it, you know, concerns who gets access to information. And then pricing, obviously, is really important to people. So we wanted to create sort of escalating, pieces of information and see if it played out the way we expect
it to. So I glanced through the findings just a bit, and I was definitely a bit surprised to see what people considered sensitive. So for example, like, gender was rated lower than I would have thought on the sensitivity scale. Can you walk me through the the actual findings that you had and kind of what were the surprising ones for you and what findings that you had and kind of what were the surprising ones for you and what what they might mean?
It's interesting you highlight the gender one because I I too, I think, was surprised, somewhat. One thing that lawyers talk a lot about are, like, sort of suspect classes, and this is sort of a constitutional doctrine. And race and gender are two of the, like, things that we've had a lot of history, unfortunately, in this country and frankly throughout the world about discrimination. And so, as a result, I think those two were the ones we thought would be the most sensitive. And it is interesting to think about why maybe race was more of, a third rail than gender would be. The gender thing, you know, and I will preamble all this by saying, you know, this is my sort of personal opinion, and I haven't really figured it out yet. I think we're still trying to sort it out. One theory I had was that, you know, race is obviously very fraught as is gender, but, gender, until recently, we thought of in very binary terms, and I think most people still do think of gender binary terms. Only recently have we started to hear more about people who are trans or genderqueer or don't have an identified gender. But in general, most of society has an a or a b type, attitude, whereas race, you know, race is a very charged thing, and it's not just black or white. We know that, you know, I'm South Asian and, you know, people can be from, Latino descent or Asian or, you know, African descent. There's just such a plethora on a spectrum. And so perhaps to that degree when it's, you know, much more shades of gray with race than with the a or b of gender might explain. But I think, you know, it's certainly a a open question and one that I think we'll certainly be thinking about and debating. Well, definitely one of the strongest,
features of this work is that the research team was really smart about how they designed their methodology. And so, Brian, I think what you may be referring to is there was this sort of initial assessment people the respondents did where they had to rank a bunch of qualities and say, how sensitive is this to you generally? And this was absent context. And so this was absent that little vignette that I gave. It was just, you know, here's I think there were, like, seven or or eight features, and you had to put them in rank order of what's most sensitive. Okay. Or you could check things. I think it wasn't rank order. Maybe it was you could check ones that you thought were sensitive. And gender did what people didn't feel was sensitive when it was when it wasn't given when that sort of characteristic wasn't being explained in context. And then once we put it in the context of the personalization, especially in search and price, it became much more charged and much more sensitive. That makes sense. And so I think that part of what we saw there was really interesting, which was to say that people don't necessarily per se consider gender private or sensitive. But then when you use it against what people see as against them, potentially, it could becomes very sensitive. And it was really interesting to see that play out in a couple of different, you know, pieces of of kind of information. One thing we talked about a lot, us in the research team, was just just how the Internet originally was this place where you you could be anybody. You didn't your your phenotype and your presence was no longer sort of what people use to judge you. And some of that is being lost now without accurate inferences and how reidentify even in a general sense, people online, not necessarily reidentify in the formal sense that that word has a meaning that I'm not trying to imply here. But, you know, when you walk down the street, your gender and your race may not be as private. I mean, the exact details of it, as gotten pointed out clearly, are are nuanced. But, like, as you walk down the street, you present information about yourself. And so it's not that these things are private. Right? What you look like is in private, but it may be sensitive if it's used against you. And I think that that's kind of what we're seeing now as these features are being brought into the online space. We're seeing people respond in the way that they would offline.
So it pretty much maps onto it. It's just that the Internet is kinda catching up to something that used to be pretty casually done in real life. Are there any full on from your research, full on no no's in terms of, like, what sort of profiling or personalization
is okay? Yeah. Don't personalize price on race.
That was extremely unpopular concept. Extremely unpopular. It's funny because I think well, not funny but sort of interesting, that race, you know, as as we're seeing in, you know, culture and society is fraud and even on a sort of depersonalized sort of technology context, it too is very fraud. But the thing that's interesting to me is that, Ali mentioned pricing and advertising. Race based advertising, I think, too, at least in an intrinsic level, you wouldn't maybe assume that people would dislike that as much. And, yeah, I think there are some ways in which race based advertising or advertising that is correlated to race, has some, interesting value. And I think gender too. You know, as a man, I think if I got advertisements for some a product that was really only primarily used by women, I would be, like, confused. Like, I would be like, I don't really need to see this. And conversely, if it was a product that was mostly for male use, then I would be like, okay. That's fine. That makes sense. I mean, they that that's not offensive to me or doesn't bother me. And race too, I think we could pick some pretty easy examples. You know, my sister's getting married later this summer, and I'm sure that for people who are South Asians, like our family, like, if we all got Hannah advertisements, I think we were like, yeah. We need we need a lot of Hannah this summer. Like, that makes sense. And so even though it would be probably correlated to race to some degree, that doesn't necessarily bother other
ways, you're like, this could it's been out a lot of different directions. I think I think Adam's absolutely right to point out the the use cases where your phenotype is or your gender presentation is really relevant to products. And and I think that, even in search results, you can think of examples where, phenotype and and your sort of racial background, if somebody were trying to guess at it and successfully guessed at it, it might improve the quality of the results. And, again, it's gonna come back to sort of for me, a lot of the examples are cosmetics, and hairstyles. Right? Like, if you're an African American woman, you're searching natural hairstyles, you want a specific response. So there are and if you can't get that response, if you can't get that data, that's also offensive. Right? Like, if it's if the search results aren't personalized or aren't aren't providing what you need, whether or not that's through personalization or just sort of a better diversity of of images, which I suppose would be the good first step, You know, because I I think that you see stories periodically where where African American women have trouble finding, hairstyles or they find really offensive search results, when they're searching for hairstyles. And so I think it's really important to keep in mind that the absence of of the availability of of this information is also concerning to people. It's just that the inform the questions that we've presented, the survey respondents weren't anywhere close to this specific. Sure. So it's, you know, our job as the policymaker and the research team's job as academics to sort of take a step back and put these in a larger context like Gautam just did. So,
obviously, you've already touched on this a little bit that this could be used to advance advocacy work and whatnot. How should companies and consumers use this information? Is there any value in it for them?
I think, for the company side, one thing that we thought a lot about was, you know, how do we both, a, make this use as make this research be used as an educational tool. Like, we don't want it to we part of the reason that we didn't really talk about specific companies even though it might have been easier for the, survey respondents is that we didn't wanna be seen as, like, we're trying to say, like, this company is bad and this company is good because I don't think that that's particularly strong position for us. But I do think it really, highlights and the comments that we got that Ali mentioned in the little comment boxes really highlight too that we just wanna put this in front of companies and hopefully policymakers too and be like, this is what what people are thinking about. You are going to take this or leave it, but you should at least be aware of it. And, sort of parking into what I said earlier, there's always a lot of talk, at least for me, about, like, oh, people feel this way without a lot of information. And now we have the information, and we wanted to, get that in front of people so they can, you know, hopefully make more informed decisions from product design to figure out how they can do personalization because it's happening in a way that consumers will be a little bit more comfortable with. You know, the
Emily Pavel and, Reina, who's the the research team, really unearthed just a huge amount of information. What we're gonna present in the paper, that we're hope hoping to have out later this year is probably 15% of the data they collected. And so in in addition to asking about fairness, then one of the another metric they asked people about was trustworthiness. How does this affect your trust of of, you know, the people involved in this and the institutions involved in this personalization. And I think that those results are gonna be particularly relevant for companies when trust is essentially the economic fuel of the Internet right now. People wanna feel like they're being treated fairly, absolutely, but the consequence is that they lose trust if they're not. And so I think that that's something that companies are really gonna wanna pay attention to when the team has time to sort of sit down and sift through the larger scope of the results. But I think even in the short term, there's there's an assumption that people want personalization and that it doesn't really matter, how it's done. And I think that some some of what we've seen here is that, you know, if you even if you're personalizing something really accurately, people still that does not gonna make it so people are just like, oh, this is fantastic. Now I'm okay with it. I think there's an assumption on the side of industry that personalization is good no matter kind of what it looks like or how how it's done. I think these results really call that into question. And I think if you wanna be on the cutting edge of what how to gain trust, which is really how you gain market share, you're gonna need to think about how people feel about these these behaviors and modify your product design to accommodate some of these, responses. So you make sure you don't undermine your user base.
Cool. So you're working on the paper. Currently, 50 pages. I saw a nice draft of that. When should we expect to see it?
I think we're hoping for I saw some some just head shaking here in case you can't see that. I mean, it's a work in progress really much. Part of the reason I'm in town this week is to present this at a paper workshop called privacy law scholars conference. So we're hoping to get some great feedback from that, incorporate it, and my hope is that we'll have it some public draft version of it in the next few months. But I remember how long academic publications take. And so it could be a while till it's finalized, but we're gonna try to figure out different ways, like this podcast, to get some defined into it. Academia is certainly its own wonderful beast at times, isn't it? So this is amazing stuff. I'm definitely looking forward to the paper. For everyone listening, there is a blog post that Ali and Gautam wrote. It gives a nice summary of this, goes a little bit more in-depth,
and share some of the initial findings. So be sure to check that out on CDT's website, cdt.org. Thank you so much for joining in, Gautam. Every time you're on, I know you like to plug your your own blog. So why don't or plug your own blog. Why don't you you go ahead and do it? Well, it's on hiatus, but the ice cream can still be found at Instagram at habeas custard where I've been eating eating my way throughout the country and taking photos of it. We're so proud of you. Alright. Thank you so much for joining. Thank you. Thank you, Brian. How many steps have you taken today? Did you hit your goal? These are becoming an increasingly popular set of questions as more and more people are purchasing health wearable technologies that track a variety of health metrics. So what should health wearable companies be doing to protect the privacy of their users? Probably a lot. And CDT's Michelle Desmoy recently released a report coauthored with Shelton Yuen of Fitbit and looked at how privacy and ethics can be built into the research and development phase at wearable companies to help address these questions Michelle is here with us today to talk about the report welcome Michelle thank you Brian I'm so happy to be here oh it's always a pleasure to have you here even though it's hot you should hear the bloopers that are coming out of this one I don't know what you're talking So you are calling this a first of its kind research project tell me a bit about the research question that you tackled
and the unique partnership that you had with fitbit on this one. Sure. So as far as I know, you know, there really hasn't been a partnership or collaboration between an advocacy group and a company in this way. We were funded through the Robert Wood Johnson Foundation, which was an important part of the collaboration that kinda took the funding element out of the work and that was important to us. And the work was to look at the internal data flows. So how data is used internally in the research and development part of a company. So as you know, tech companies really rely on research and development to figure out if their products are working, what's coming next, what do users want, what features do we have, really, really crucial to their growth. And so what we decided to try to do, a lot of advocates have looked at data that flows to advertising or marketing. And what we decided to do was was kind of shift it that focus and look internally with the idea that if we could figure out the privacy and ethical dilemmas that researchers face when they're interacting with data, this might resonate out. And if we could make recommendations for the industry as a whole, that they might look at Fitbit as the market leader and say, that's if Fitbit's doing this, then we're gonna do it too. Yeah. I think that's an interesting point. A lot of times when you hear all this stuff about big data and privacy,
it is kind of more for that marketing side. But I think the research side, this was one of the biggest things that I took from this report. The research side, a lot of the behind the scenes development and, you know, what companies are trying to do with data to make their products better and better serve customers,
that's kind of forgotten in the debate. So this was a great report in terms of that. Thank you. I I found that really interesting too. I mean, I think it's especially in health with health data and health and wellness, it's really crucial to ask ethical questions too. And so that was kind of another new sort of innovative component of this work. We looked at privacy and security and we created recommendations around that, but we also looked at ethics. And I think, you know, in the report, we make recommendations related to ethics that I think were were really important. So we'll get to some of those thoughts and recommendations first. But first, I mean, you were behind the scenes at the r and d, you know, r and d of a hot company here, Fitbit.
You know, what did you learn from this? You know, were there any surprises in how they conduct their r and d and kind of, you know, how they use data?
Well, they all wear black. No. I'm kidding. There's a they all wear Fitbits. That's for sure. And, what I what I discovered were a couple things. You know, first of all, everybody who works there was really sincerely interested in health and wellness, maybe not a surprise. But what what was a surprise was that most of, if not not not all of it, but maybe the majority of the data that the researchers used was actually Fitbit employee data. And the reason makes sense. If you are working with, you know, a couple researchers and one of them and you wanna test a sensor and you say, hey. Just jump on the treadmill. Let me figure this out. That makes a lot of sense. Okay. But when you have a bigger company dynamic, for example, Facebook went from 10 researchers to 60 in the time we worked with them. They went from a private company to a public company in the time that we worked with them. And by Facebook you mean fit. I mean mean, did they
Keep going.
They, using their employees' data when they were smaller made sense. But when you're growing and becoming a bigger company, it makes less sense and there need to be formal policies and practices in place that make sure employees are comfortable with this kind of thing. So that was maybe one of the first surprises. Another surprise I think was how much Fitbit really wanted this information from us. You know, it it was sort of like sitting in a room with people with, you know, PhDs from MIT and Harvard asking me what should we do. And that was strange but really gratifying because I think what it what it taught me was that there really aren't very good standards out there for this kind of work which is so important. This is where user data goes. This is where the information that flows from devices goes into companies and decides whether the company can make it or not. And Fitbit to their credit was really open to listening to
advocates and and really open to listening to what we had to say. That's great. So in the report you actually apply some, you know, policy and ethics frameworks to it. Can you talk about those kind of how you took a look at the research and their process and apply them and do it without being born? That's your challenge. I'm just waiting for that. Go. Oh, god.
So I will not be boring and say the what we wanted to do was different. A lot of times we'll look at really boring things like the fair information practice prints. These are important frameworks that talk about privacy and security. And we did that. But we also looked at, ethical frameworks. So questions of justice, things like beneficence, how much value are you bringing to the people who are participating in this research? That was interesting. That was definitely new a new way to look at internal r and d. And then finally, what we looked at was what we called business realities. And this was really crucial and especially as I'm thinking about the the process and the project in retrospect, I see how important that was to making this really innovative. The idea is that a company is not gonna adopt a bunch of best practices if, a, it's not suited to what they do, but, b, it's not responsive to things like market pressures. These are the realities. Right? We're not gonna spend time protecting data if it's not in our best interest. And so part of what we tried to do is create recommendations that were real, that were responsive to those things. Awesome. So let's get to those recommendations.
I know in the report you kind of constructed them around three broad buckets or broad groups, the individual, so I assume that's user data, the company, so company practices. Yes. I don't know you're gonna have to correct me. And then community, which is kind of the broader social good, which makes a lot of sense when you're working in a health related industry. That's right. Yeah. Why don't you just kind of walk me through some of those? What were our recommendations in terms of, like, the individual to start? Sure. And I'll try not to be boring in this too. So far, you've done a great job. Good. Thank you. Oh, thank you. So individuals,
the idea here in this this bucket is that dignity should guide how practices are are done, how data is used, what policies are associated with it. And so here's an example. What people expect you to do with their information should guide when you ask them whether or not you can do it. It sounds really intuitive. It sounds like very logical, but it's the kind of thing that nobody has ever really said to a company. Recommendations around those types of things. The second bucket was around corporate stewardship but what we were calling data stewardship. This one has really resonated with companies that I've talked to since we've done the report. And the idea here was that we wanted companies, especially in the health and wellness space, to not see themselves as data silos, to see themselves as a part of this broader conversation that we're having in society right now about what it means to be healthy, what it means to to be well, and and what what is health data. We wanted them to see themselves as stewards of this information. And that led to the third bucket, which is about social good. And this is about really participating in sort of the societal conversation but doing it in an active way. So we said you need to commit resources to doing projects that are related to social good. So a company like Fitbit would be uniquely positioned to talk about things like obesity or heart disease, contribute to that, you know, whether or not that's by putting resources into internal data but it's also a part of sharing it. So we said publish the results of some of these things. And the interesting thing is as privacy advocates, we're not usually, you know, saying you should share more data. But in this context, it made sense. And I think also because we know that they protect it really well. We we did have some data protection recommendations about the escalation of protections based on how sensitive a piece of information is or data. So we knew that they could do it in a privacy protective way. But we said, you know, instead of you holding on to this information that you're doing, you know, you're doing a lot of research that's very interesting and could benefit the public, you need to publish some of that or work with researchers who will. Very cool. So what's your hope for this report? You know, who should be reading it? You know, you know, are people gonna be like applying these guidelines all across the industry? It's hard to say. I have gotten some interesting calls and emails from companies who have said, hey, this was interesting, almost sort of let's let's talk but I don't wanna commit to anything and I don't want my company name next to the word privacy ever. So, but they are really impressed with the fact that Fitbit trusted us to do this. And so I talk a lot about just the structure of how we did this partnership. That that seems to have been really interesting to people in general, companies, but also advocates. And so there's discussions about whether or not we could replicate it, how we could sort of, work with the industry on a more broad way, maybe, you know, having some kind of workshop where we go through these are the different recommendations, here's how you sort of institutionalize them. And so that's that's sort of the the high level. I I hope that for one, I hope that the recommendations are adopted by some companies and I think that they will be from what I've heard. I also hope that other advocacy organizations look at their role. And and what I mean by that is I started realizing that CDT's role is not just about convening and being the bridge, you know, dialogues and and bringing people together. That's important. But I think our role is also to be innovative, to not just accept that this is the way we do it. We we put out best practices and they're sort of dead and that's it. That we try to push the envelope in public policy that we try to, you know, do things that we think will will advance the public good. Well, hopefully that motivates everyone to read the report. And if they need another slight incentive we have with us on the show or doing the recording and producing of this, Tim Hoagland who designed the cover and the report
itself. We have a beaker running on a treadmill. It's awesome. It's amazing. Honestly, I look at this report and it makes me smile every time. So if the content doesn't entice you, maybe the cover will. Thanks so much for joining us, Michelle. Great having you. Thank you for having me. That's it for this episode of Tech Talk. You can find the health wearable report and the blog on fairness and online personalization on CDT's website, cdt.org. Be sure to check them both out. And while you're at it, sign up for our e newsletter at the bottom of the website. I'm Brian Waslowski, and thanks so much for listening.
