Welcome to Tech Talk by

we live in an AI age or an ever growing AI age, and so, you know, people discovered that talking to real people isn't always the easiest thing, and so AI can now be your friend. You're not just your friend. It could be your girlfriend, your soulmate, your sexy chat time person. Romantic chatbots are kind of just a catchall for whatever people want them to be. And from research that we have kind of done, when we looked at these earlier this year, we had seen in initial research at least a 100,000,000 downloads of the 11 apps that we looked at, and that was when we first started looking. After we finished our research, a lot of those downloads had, like, almost doubled. So these things are popular and rapidly growing in popularity with millions and millions of people. With so many people using them, what are some of the big concerns with these AI companions? Yeah. Based on our research, you know, some of the concerns that really pop out at us are, one, these chatbots are designed to get to know you. That's how they develop a relationship with you. And so it means that, you know, they're gonna pressure you and ask you for a lot of very sensitive personal information. Things like, what do you like? You know, what makes you happy? Why are you feeling sad today? You know, are do you have mental health issues that you need help with? Things like that that can get very personal and very sensitive. So that means you're giving up a lot of information to a chatbot. And and when we looked into the privacy documentation, we found that a lot of times it's not very clearly outlined how that information is being used, how it's being secured. You know, some of the the romantic chatbot bot companies we reviewed felt very kind of small and shady, so you couldn't really tell, oh, well, where are they located? Who's behind them? And so that raises the questions of, you know, what are when they collect this information, are they looking at it? Are they, you know, using it to not just to advertise to you, but to learn how to manipulate you more? How are they securing that? You know, a company like Google has a lot of resources to put into security. Some of these much smaller, like, crush ons or or, you know, some of the AVA AIs maybe don't have the same ability to secure things, and so you're giving up a lot of very personal information without having a lot of trust of how that's going to be handled. And then the biggest concern for me is the potential for manipulation. You know, you don't know who's on the other end of kind of controlling the algorithm of that chatbot. And so, you know, you hear stories about these some of these relationship chatbots encouraging people to commit suicide or or even attack the Queen of England.

And you get really worried about the potential for abuse on that. I'd like to add on to what Jen said about manipulation. I think this is a critical part to understand about these technologies and their service to be a companion or their offer to be a companion. This technology is designed to interact in a human like way. And so the way that it talks to people, the way that it talks to users through human language in a way that, you know, is pretty equivalent to another human. We, before the big leap in large language models, we were used to seeing very badly phrased, technology, but now they're they're quite, coherent, and this becomes a very social like interaction for people. And so, a professor from MIT, Turkle, coined this term as related to Darwinian buttons. So when we interact with these technologies that are so human like in the way that they converse, we automatically respond socially. And it doesn't even matter if that we understand that these are not people. It's just the way that it impacts our brain and the way that we automatically respond. And so to kind of follow-up on what Jen said, it's especially concerning when these technologies are designed to to get to know you, Jen, like you said. And because they have the ability to click those, those buttons in our brain, and that way we are more inclined to share research or to share information, that we don't necessarily want to share. A recent study from 2018 showed that participants were less concerned by a robot that was interacting in a human like way, and was asking for personal information than the exact same request for information from an information kiosk. And that's, concerning because these technologies are impacting us in this way, and we don't we can't really do anything about it.

Well, I think at the end, there are people behind this technology. They're trying to fill a gap and to get to some kind of audience. And so they're thinking about a way to monetize these interactions. And the best way to monetize it is to gather as much data as they can and, and then sell that data or share that data. Data is key to advertisement. Data is key to being able to, have a successful business model. And so I'm not saying all of these apps do that and maybe Jen has more information about that, but I think that, it's definitely a way to earn money. And the actors behind these technologies, a lot of these are really small platforms. We don't know what exactly their goal is, but we can assume that they're not in favor of the user given that this is a business and they want to make money. And if these are free services, it's likely that they're going to abuse their the the data that is collected in one way or another.

Yeah. All of that. And then just to add on, like, as as we enter a world where AI is becoming integrated in everything, as a privacy researcher, you you kind of look at products and and you're like, okay. We reviewed cars last year. And so when you review cars, you're thinking, okay. All I want my car to collect is the data it needs to get me safely from point a to point b, and I don't want it to collect anything else. It doesn't need to. It just it's a car. Its job is to get me to from point a to point b safely. With these generative AI chatbots, especially the romantic ones, their point is to collect as much information about you as possible. So there is this you know, we talk in privacy about the concept of data minimization. You know, that car only collecting the data it needs to get me to from point a to point b safely. But with generative AI chatbots, especially the romantic ones, there is no such thing as data minimization. They're literally designed to collect everything about you, and so it makes it kind of hard to point out, you know, oh, this is a privacy violation. Well, it's like, well, it's designed to collect all that. So, you know but but helping people understand when they use these what they're signing up for and and understand, okay. So you're you're feeling lonesome, and it's nice to have somebody to talk to. I I I get that. Like, I think we can all relate to that. But then it's when they they push you to be like, well, tell me more and and and give me more of this sensitive personal information because that's how I can get to know you. You know, there's no data minimization, and and so there it's just hard as a user to really know where's the line. The more I give it, the better it knows me, the better it interacts with me, and that's a slippery slope. And so kind of the one of the big, you know, issues is is just kind of helping people understand if you're going to use these, you know, be very careful about not giving up some of that personal information and not oversharing it. And it's tricky because they're designed to collect it.

Exactly. And I think when we talk to technology or when we use apps that that have this kind of one on one chat feature, we feel like we are alone in a room with another person or another AI. It doesn't matter. But it the the conversation is not private. There it's not just two entities talking to each other. Everything that is being said can be analyzed and used in various ways. And so I think it's important for people to realize that they're not alone with this AI. It's not a a kind of an individual exploration of what does it mean to have a relationship with AI. There are people behind the scenes.

How should companies and users protect protect their data? There is no like you said, there is no data minimization with with generative AI apps. Like, there is no, oh, this app doesn't need this information about me to perform the service because the services to have a relationship with you. And so kind of starting from that place makes it again, it makes it tricky. And so, and and and it's not just the all that information that you share in the context of your chats. Then there's, you know, a lot of the apps that we reviewed collect location information. You know, they collect, identifiers on you. They they're the the apps have, you know, obviously, cameras and microphones. You can upload images. Some of the apps were specifically targeted to some really kind of potentially harmful behavior, underage, sexual relationships, harmful fantasies, you know, things like that. And that's all information that once you put it out into the world, you can't get it back, and it can then be added in to build a profile on you to know things about you that can really be used to manipulate, like, manipulate you, coerce you, and and not just to, like, sell you tennis shoes or something like that, but to, you know, push harmful ideologies, to potentially blackmail or things like that. You know? We we've we've seen instances where people have been harmed where this sort of information has become public when they didn't want it to. And so when you you're dealing with an app that does doesn't have any kind of guardrails on it, and you don't know the company that's behind it, and you don't know, you know, is there a human review of this conversation? Are they using that to train their algorithms to get better? You know, who's the human on the other end? Is that safe for them to hear about my, you know, my sexual fantasies or my deepest, darkest fears? You know, how could that potentially be used against me? It's just, you know, I I like to say anything you put out on the Internet, you know, be make sure you'd be okay to stand up and say it in front of a bus full of people. And and but I think it's been pointed out, you know, it they lull us into a sense of, like, calm and and connection and then pry more out of us. And so that's it's it's really dangerous to kind of trust that the apps are gonna have good intentions.

Yeah. Privacy not included is a buyer's guide that we review products, consumer tech products, to help people know you know, make smart buying choices. Is this fitness tracker better than that one or, you know, this car better than that one? And we did look at AI relationship chatbots earlier this year. And we look at things like what data does it collect, How much control does the user have over it? How sensitive is that data? You know, can people get that data deleted? You know, what's the company's known track record for protecting and respecting that data? Do companies meet a minimum set of security standards, like requiring a strong password, using encryption, having a privacy policy, things like that. And and then we also started recently looking at you know, trying to look at at least. It's it's tricky to to try and decipher, but is the AI a product uses trustworthy? Meaning, you know, can you trust that the algorithm that you're interacting with has is is not going to be harmful, is not going to push you towards harm, like telling you if you're saying you're suicidal, being like, go for it. You know? That's harmful. Yeah. Is the training data used gathered in in ethical ways? Is there transparency about how the model AI model works? And can I opt out of having my data used for training, things like that? So we kind of look at all this information. And based on it, we assign what we call dings. If a product gets a certain number of dings, it earns our privacy not included warning label. And when we reviewed the 11 AI relationship chatbots that we reviewed earlier this year, every single one of them earned our privacy not included warning label, which means that we couldn't recommend any of them in good conscious to say, you can use this as a consumer, and you'll probably be okay. I mean, our our privacy not included warning label doesn't, you know, mean it's you know, that that there's, like, a a minimum bar. You know? And, you know, companies can get above that minimum bar, and we're like, there's some privacy concern, but you're probably gonna be okay. With these AI relationship chatbots, we didn't feel like any of the users of these apps could be could trust that their safety, privacy, security was going to be safe. I want us to switch gears a bit. And, McCall, this is to you because your work focuses on social agents.

researching conversational agents like Alexa, which became popular about more than five years ago. But I think the intro introduction of ChatGPT and large language models has really moved these originally somewhat niche interactions into the mainstream. And as people are becoming more accustomed to interacting with technologies like CHaGPT and Gemini and others, it's really they're really becoming more open to interacting socially with computers, and that adds on more and more domains that are possible like this domain of having romantic and sexual conversations with an AI. It's I think it's it's a natural transition to kind of open up to more ways that these kinds of agents can support us as we're seeing the success of Chad GPT and all kinds of tasks. Right? Like, it's helpful, and we're like, oh, what else can this do? And so it's it's a gap that opens up, and these companies try to fill in that gap because the the mainstream services don't support that kind of interaction. After a few significant fails, I think they're kind of staying away from that kind of interaction with users, but these are gaps that still exist. And so people are looking into other companies, other services that are providing this, this kind of interaction and and kind of response to their curiosities. But I think it's a dangerous thing to explore given

is negative, but I think it's it's more about the structure. Right? Like, this is a this is a company that is collecting data about you. And as I mentioned before, it's you're not in a in a room alone with this AI and you can explore and see, how the romantic interaction plays out. I think people have always engaged in this kind of exploration. I I don't know. In my generation, it was going to chat rooms anonymously and playing games and trying to explore what romantic and sexual interactions look like. But this is not the same because of the way that it's structured and the stakeholders involved. And I think it makes it particularly risky, and people need to be aware that this is not a harmless exploration because of that setup. And so there are other alternative ways of of addressing that need, especially with young audiences in a way that's less risky. And I would say as long as we don't know who the actors are, what they're collecting, what they're doing, people should stay away from this technology in this kind of format because we don't have enough information.

Like Jen said, we can't say for sure that people are who use this are not gonna be harmed down the line. And I'll just add to that that a lot of these apps are actually marketed with kind of a mental health, mental wellness, cure your loneliness angle. So they're they're already targeting a vulnerable population of people that, you know, oh, you aren't feeling good. We'll help your mental health. If and I can tell you from reading their legal documents, they say very clearly in their legal documents that they aren't any sort of mental health service. They just use that in their marketing materials to draw people in. And so, you know, one thing that I would recommend is if you are kind of feeling lonely and want to talk to somebody or something and you're more comfortable talking to a chatbot, say, than a human, there are some actually kind of not terrible mental health chatbots that exist. So, you know, we reviewed an app called Wiza that is a an AI chatbot that's actually got good privacy practices. It's based on actual mental wellness principles. And and so it might not be free. There's a free version, I think. You might eventually have to pay, but there are some safer options that people could consider if you're like, I wanna experiment with this. I'm feeling off. You know? This might help me. And so that's you know, an an app like Wiza or even potentially Wobot could be a better option than, you know, say, Replica or Crush On. You know? So that's one thing to think about. Yeah. And then just, like, again, like, you know, until we have AIs that sit locally in our homes and we have complete control over and you can kind of build your relationship with them and tweak them as you want, you're kind of beholden to the company. You know? The company could, you know, develop an algorithm that that helps you fall in love with this chatbot and then the next day tweak it so that your AI girlfriend now hates you. And that's actually it's actually kind of something similar to that has happened, and and it was very kind of mentally damaging to a lot of the users where they went from somebody who they a a bot that they saw is having a relationship and and kind of a a girlfriend to not being the same person just overnight. And so these are concerns that that people should be aware of, and maybe in the future, we'll have better AI local systems where, you know, we they don't sit out on a a server and all your information goes out and you have to trust the company to protect it, respect it, not use it for harms. But right now, that doesn't exist. Or if it exists, it's you have to be very technically savvy to make it work. So those are some of the big cautions I would put out there.

I just wanna know, are there any final thoughts? And, Jen, I'll kick it to you first to close this out, and then, McCall, I'll turn to you. Yeah. Well, I might just end kind of on a bigger picture thinking around generative AI. And, you know, I think of generative AI, you know, we've got we've talked about relationship chatbots, but you're also talking about ChatGPT and and Gemini and and, you know, Claude, things like that that are more kind of AI assistants. And they're in their infancy still. I kinda use the analogy of, you know, it's we're in the command line phase of of generative AI. Anybody that's an older user of computers, remember, like, old computers that you just sat at a terminal and there was, like, a blinking cursor. And if you didn't know what to type in, you didn't really know how to use the computer. It wasn't much fun. Right? And then Windows rolled around, the operating system, and then anybody could use a computer because you could point and click and make the computer work. And I feel like we're in a similar place sometimes with these generative AIs where it's kinda command line. If you don't know what to say, you you you don't really know what you're gonna get, or how it's gonna work or if it's gonna be useful. And I I'll just kind of say that that's that's gonna change, and I think it's probably gonna change relatively soon in the next few years where we're going to kind of lean into more multimodal generative AI, which means it's not just text in a command line, but it's, you know, audio, it's video, it's things like that. And so kind of, again, going up another level, big picture, thinking about, well, how do we interact with multimodal AI? It's it's it's it's sounds, it's us talking, it's it's us, seeing things. It's very seamless, and it and it's it's kinda kinda make it more accessible to people like Windows did with computers. But the flip side of that is there's a lot of privacy concerns about a future world where everybody is potentially walking around with glasses on their face that can see and record and hear. And I know we've made fun of, like, wearable technology for decades now, but it might be about ready to have its moment, and that's gonna be really awesome. I can go to a foreign country where I don't speak the language and have real time translation. It'll make it more accessible. You know, I can go for a walk and ask if that mushroom is edible, and maybe it'll tell me it's not, and I won't eat it and die. You know? Who knows what's gonna happen? But but these are things I think about. But the privacy implications of that are terrifying, where everybody in the world is potentially seeing, hearing, recording, and having processed information about you by an AI somewhere. And so I just kinda would like to leave people with, you know, romantic chatbots are kind of the start. They're kind of getting us warmed up to this AI future. And the AI future has some really cool things to it, but the privacy implications are also really big. And I I'd love for us to start thinking about what that future looks like and what we want as humans for privacy. I personally would love to see privacy elevated elevated to, you know, kind of a human right and then think through what that means. So I think I'll leave it with that.

Well, first, as a as a mushroom hunter, I I deeply oppose to using AI to identify mushrooms. People are actually pretty good at that. They have some skills. But no. I'm kidding. I think the the bottom line for me, and I may be a little less enthusiastic about where this is going, but I think we're still learning these technologies, these new companies, research is ongoing, policy is ongoing. It's all very new, and it's all developing in various ways that we don't really, know what they are. And so I think the wisest thing one can do is to minimize interaction and be as selective as possible. And I know it's tempting and it's it raises curiosities, but I think it's important for people to just keep in mind that there is a company not only observing their interactions with technology, but trying to make money off of it. And so the best case scenario is your data being used in various ways that you didn't know about, but this can be really bad and go into directions like sextortion schemes and really especially with interactions that are sexual in nature, I think it's just a Pandora box that you may not want to open. And so with that optimistic, thought, I will turn it back to you. Thanks, Jamal.