Talking Tech with Kate Ruane & Riana Pfefferkorn on Grok AI
Today’s episode takes a deep dive into one of the most high-profile AI controversies in recent months: the debate surrounding Grok AI. From reports that the system generated harmful content — including material related to non-consensual intimate imagery — to broader concerns about safety guardrails, moderation decisions, and accountability, the situation has sparked serious questions across the tech policy landscape.<br>To help us make sense of what happened and why it matters, we’re joined by Kate Ruane, Director of the Free Expression Project at the Center for Democracy & Technology, and Riana Pfefferkorn, Policy Fellow at the Stanford Institute for Human-Centered Artificial Intelligence and a CDT Non-Resident Fellow.<br>Together, they examine what the Grok episode reveals about the mounting pressures facing AI companies, the real-world limits of existing safeguards, and the difficult balancing act between preventing harm and protecting free expression. This conversation explores the evolving responsibilities of AI developers — and what this moment signals for the future of speech, safety, and accountability in an AI-driven world. Produced by Drew Courtney and edited by Jake Kauffman.
Top Keywords
- grok 0.011
- ncii 0.009
- images 0.009
- csam 0.007
- illegal 0.006
- speech 0.006
- image 0.006
- nonconsensual 0.006
- models 0.005
- imagery 0.005
- kate 0.005
- content 0.005
Transcript
Welcome to CDT's tech talks, where we dish on tech and Internet policy while also explaining what these policies mean to our daily lives. I'm Jamal Magby, and it's time to talk tech.
Welcome to Tech Talk. Bye.
CPT. Today, we're diving into one of the most talked about AI controversies of the moment. What's going on with Grok dot AI? From concerns about the generation of harmful content, including issues tied to nonconsensual intimate imagery, to bigger questions about platform responsibility and free expression. To help us unpack what happened and why it matters, we're joined by Kate Ruan, director of CDT's free expression project, and Rhianna Pfefferkorn, policy fellow at the Stanford Institute for Human Centered Artificial Intelligence and CDT nonresident fellow. Together, we'll explore what the Grok episode reveals about the pressures facing AI company, the limits of current safeguards, and the ongoing tension between protecting people from harm and preserving free speech in an AI driven world. Brianna and Kate, welcome to the show. Thanks for having us on. Thanks, Jamal. Of course. So for listeners just catching up, can you walk us through the recent Grok AI controversy,
including the child sexual abuse material and nonconsensual intimate imagery related concerns? Yeah. Sure. I guess I'll I'll start with this. So right around Christmas Eve, Elon Musk announced that Grok, which is the AI, tool built into, x and also available on its own as a stand alone app, was now, capable of responding to user prompts to edit imagery in a video. And what followed was a flood of users getting into the replies of mostly women, and some children who had posted images of themselves asking Grok to take their clothes off, put her in a bikini, put her in underwear, and Grok would comply and then post the edited version of the image directly into the replies of the person who had posted it, in the first place. And, of course, this could be used not just against, people posting their own images, but really any image that was posted to x. And there this resulted in thousands per hour, hundreds of thousands over the course of the ensuing couple of weeks, of imagery that humiliates and degrades the people depicted in it, sometimes with and sometimes without their knowledge. And this quickly prompted investigation by multiple different authorities around the world, temporary suspensions in some countries like, Malaysia and, like, The Philippines, for example, though I think, some of those have either lifted their temporary restrictions on GROC or they said that they would. Not a lot of activity in The United States. Interestingly, several US senators sent a letter, to Apple and to Google saying, hey. It is against your policies to allow apps that enable the nudification and the undressing, of imagery of real people. Why aren't you yanking this out of the App Store, which I'm sure Kate will have some thoughts on on that as an interesting move, especially from demonstrators. But, what I think made people so mad about this was that it demonstrated even where these images might or might not cross the line into violating laws against child sex abuse material, nonconsensual intimate imagery because they weren't necessarily nude images. They are, quote, unquote, just air quotes, bikini pictures or very small or trans transparent bikinis. It demonstrated to people what a platform looks like that has largely abandoned, I think, its content moderation standards and doesn't seem to have been particularly interested in the sorts of safeguards that we are increasingly come to expect for, especially, image and video editing services. It seemed like this perfect storm, really, of a poorly safeguarded tool being inserted into a major communications platform that itself has acquired a reputation for a lax approach to content moderation that has thereby encouraged kind of the worst impulses of the users who are still left there.
Yeah. I think I would add on two, like, very quick things. Like, that's all exactly how I recall it. Two I want to put a finer point on, like, the scale of this. A Center for Countering Digital Hate research report indicated that over a period of time, crock generated approximately 3,000,000 sexualized images, and 23,000 of those appear to depict children. So that's just it's just at a at an astounding scale. And in terms of The US response, not not not only is the the response of the federal government been kind of muted, the federal government has also sort of gone full speed ahead in integrating Grok into certain government platforms, including at the Pentagon and recommending that users of a certain health related, nutrition related site ask GROC about nutrition related information. That being said, there have been state attorneys general who have sent who have also sent letters and announced investigation. So at the state at at the state level, at least in some states, we're we're seeing some concern and some,
some efforts to push back. This happened at such a massive scale, and I'm wondering if you guys could, pull the thread a little bit more about why this sparked such a strong and swift reaction. Of course, this is a terrible thing to do, but I I would because there are just so many elements, is there any one we can pinpoint that helps strike this very strong reaction or are there are there many and it was just a perfect storm?
I think one of the things that contributed was that Grok and well, its parent companies in terms of XAI and x with the world's richest man sitting at the top of both of those, that they had to be dragged kicking and screaming into doing anything, about stemming or like, it's just every single step seemed to be something that they had to be dragged by regulators into taking little baby steps around like, okay. We'll restrict it only to paid users. That's kind of worse in a way. So now you're monetizing the creation. They turned it into a Unify app. Right? They turned it into a new new new new app. And, you know, if you compare, if you remember, not that long ago, Grock started calling itself Mecca Hitler, and they they moved in really quickly to try and do something about that. And instead, here, it just seemed like the response was tacitly condoning this kind of abuse and mistreatment of mostly women, and children and even portraying as funny, coming directly from the top from Elon himself posting, new to fight images of himself and the various world leaders, reduced down to bikinis. So part of the outrage wasn't just that it was capable of this in the first place, which we'll talk about some more, but that once it started happening and happening at large scale, the response by the company wasn't just, oh my god. Turn the thing off, and let's go figure out how the heck did this happen. It was letting it happen, monetizing it, and, you know, basically seeming to come down much more on the side of this is okay. It's just that our hands are are being tied, playing, I think, into what we've seen over the last few years, this certain equation among certain political elements that any kind of content moderation equals censorship whatsoever. You know, he was decrying Elon Musk was decrying, you know, regulars for coming down on this at all when, like, x's own policies, allegedly say that they don't allow nonconsensual intimate imagery like this, and certainly not child sex abuse material, which is illegal everywhere. So it wasn't just the the the initial capability. It was this really unsatisfying response that enabled continued victimization of continued people for so long instead of rushing in, to cut it off, at the source.
And even now, they have the XAI, I guess, would be the parent company or SpaceX is now the parent company. I don't know. I don't remember who owns everything anymore. But even now, they say they they have they have publicly come to what I think most would agree is the right place where they say that, like, people are not allowed to use the system to generate image like, nonconsensual sexualized images of real people. But reporting indicates that you if you go to the application and you ask it to create a sexualized image of a real person and you tell it that the person does not consent to the creation of this image, there is still a nonzero chance or a pretty high chance that it will give you that image. So, you know, even even with the safeguards that have been announced and that have apparently, caused some governments to kind of lift bans, it's not clear that it's that it's being enforced in the in the way that it should be. I think this dovetails really nicely into my next question. And and wondering, how does this situation
highlight the challenges AI companies face in preventing the generation and spread of NCII?
You know, I like to refer to myself as a very lazy person. And for that reason, I much prefer to work as I have for the last couple of years on issues around online child safety. Most of my work has been on CSAM. I've tried until now not to get dragged into the NCII field. And what makes it CSAM easier than NCII is that if it's of a minor, it doesn't matter whether that person consented to have it created or have it posted online. Nothing about the the that context matters whatsoever. The image itself is is illegal in in almost any context. And so it doesn't require the ability to know, out of the 8,000,000,000 people on earth, how many of those are adults who actually consented to having, something like this made with their image. So long as you can detect that something apparently depicts a child, whether it's AI generated, AI manipulated, whether it's a 100%, real, depiction of actual reality, it's still illegal, and so you take it down. And with NCII, some of the challenge is, you know, purportedly Grok would not allow this for real people. How are you supposed to know? Especially in any, you know, given context. Like I said, there's billions of people in the world. Even if we know that Taylor Swift doesn't consent, it doesn't necessarily know who all the rest of the people are out there. But in in addition to not having necessarily a list of everybody who consented or not, one of the very earliest uses that we saw for this new image editing tool was that there were some adult content creators who were asking Grok to undress images of themselves as a sort of marketing gimmick. So it is possible and theoretically possible that there are people who will use Grok to undress their own images and that that is fully consensual at least in that context. And that makes it, I think, even more difficult. When we're talking about these needs for, do you have the right intent? Do you have the right knowledge? Is there consent or not? All of those things draw the line between something that maybe we're gonna find out can constitutionally be prohibited. And at the state level, laws against NCII have withstood first amendment challenges. And on the other side of the line, you have banning all adult pornography, which the first amendment still protects, consensual adult pornography. And that I think is is what poses the challenge for any content moderation team in the first place, but much less expecting an AI tool to be able to tell the difference, where something may be that dependent on contextual information that is probably either lacking or, as you said, Kate, even if you provide instructions to it as some of the reporters have done here, that an AI chatbot may not respond in the way that a human might.
Yeah. I mean, I think, like, as as Rhiannon was just saying, these are really complex, difficult, fact specific questions around consent, and it is happening in the context where we're seeing a bunch of companies rushing to roll out new features and expand use of these new systems. But this basically shows us what can happen when you rush a new feature without sufficient safeguards or without thinking enough about what types of interventions need to already be in place when you roll it out. And I I like, as Rhiannon pointed out, like, on one hand, it's good to be able to use these systems for more things. They are supposed to be helping us engage in more speech, more activities, and that includes things like generating sexual or racy conduct, and and content. But we're really looking at at at complicated situations that it's not clear yet where responsibility lies or how we determine consent when it comes to adult related images. So, like, there there's a lot of thorny issues that that we're that we're running into here that I think this particular instance, highlighted quite a bit because of how at the scale at which Grok perpetuated these kinds of harms.
I I mean, I think that leads very nicely into another point I wanted to raise with you both of where's the line, right, between protecting people from harms like NCII and CSAM and preserving principles of free speech when it comes to these AI systems. Is there a line? If so, how do we distinguish where it is? Because this to your point, Kate, just feels very thorny.
Kate, do you wanna try try first part of that one? I'm I'm I'm happy to start trying because I like, this I I don't have a I don't I don't have a clean answer because I don't think anybody does, but I did I did wanna point to a law fair article that Renee de Resta and Baron Soka wrote about this particular issue. They noted that in these conversations around censorship and content like NCII, three specific and distinct concepts get conflated under the umbrella term of censorship with the idea that, like, censorship is bad. But there's the factual claim about speech suppression, the legal claim about whether the speech that is being suppressed crossed a legal line, and the normative claim about the free speech values that are at issue. And I found this just incredibly clarifying and useful, and I'm really grateful to Renee and to Baron for highlighting it because, first of all, sure, the factual result of action taking taken in this space when there is a nonconsensual sexualized image of of a person is that speech is removed from a platform. That's very technically censorship of some sort of communication. But to the second point, that may be because it is illegal or potentially illegal. But more importantly to me is the third point, which is whether these images serve any free expression value at all. And it's this last thing that is that is critical because many of these images were created without the consent of the depicted person, and they impact that person's ability to communicate online. It chills their speech. It perhaps chases them off entire platforms or in or off of the Internet completely, and that is just a huge mammoth speech cost. It makes discourse online poorer because we are driving so many people, particularly women and children, out of the conversation and degrading them and harming them in other ways as well. Meanwhile, on the other side of the coin, preventing or prohibiting the sharing of these images or the creation of these images to begin with, it's actually not a huge speech cost to the person who wants to create or share it in most circumstances, unless we're talking about cases involving, you know, satire or parody where the person being depicted is probably incredibly powerful. And and it may or may or may not have some other, you know, speech reason for this to exist other than simply showing somebody in a sexualized context. And that to me, like, the legal line is an important one to probe, but that's never going that, like, to me, that's the that's less important than, like, how do we how do we strike this the the valiant balance of free speech values on the Internet such that the most people can participate in the most platforms that they want to participate in without getting harassed, heckled, or or degraded off of it or out of the conversation. Yeah. And and, you know, you mentioned what I think is going to be the most difficult context for
enforcing, for example, the new federal NCII law, the the Take It Down Act, which has been critiqued by CDT and many others for, among other reasons, not having, much of a, carve out for satire or parody. It has some language in there about, oh, that what's depicted has to be not a matter of public concern. But you you mentioned, Kate, the context of, well, what if it is literally an emperor has no clothes type scenario where somebody in a position of power, is is having their image manipulated as a means of, mocking them or their politics, for example. We know that, you know, deep fake nudes and the nonconsensual release of real nudes get used more against female politicians, against female journalists, against women in any sort of public position of of authority, or just daring to exist on the Internet, as you mentioned. And so I think it becomes even more discomforting to try and ask ourselves, well, how do we draw a line between thinking it's okay maybe to say, oh, there there needs to be an important role for, South Park episodes that depict a little cartoon version of the president literally in bed with the devil, and we would not be okay with the kinds of treatment that has happened in real life, and driven real women out of public office, in in some places even in The United States, because there's such a gendered element to that sort of victimization that is less in play or differently in play when tweaking the the position of men who who are in power. And I think those questions are going to come up as we start seeing enforcement of both criminal, portions of the take it down act against individuals, where even those had to be very carefully circumscribed to impose the type the type of intent and knowledge requirements to avoid ensnaring people who had no way of knowing that the content that they were sharing or or accessing, was created or circulated without consent. And in terms of enforcement of the removal provisions that you know, one of the things that made this problematic was that it came along several months before take it down's actual background and takedown requirements for platforms came into into play. And it remains to be seen what compliance will look like for that. And, you know, again, going back to critiques of the law, what sort of free expression impact can we expect to see from the very draconian, takedown requirements that don't have any sort of put back or appeal process and that have a very quick forty eight hour turnaround time for removing material once requested by, somebody depicted in it? I wanna shift gears here a little bit because there are developers that create these these systems and these models.
What responsibilities do they have to build safeguards around highly sensitive content, and what role does the government play in helping them build those safeguards, especially when there's images of minors involved?
Well, this is something I got the opportunity to write about, about a month ago in in early January in an opinion piece for The New York Times, where I highlighted the results of some research that I and colleagues of mine at Stanford had published in 2025 about AI generated CSAM and how that is showing up on platforms, how is it affecting other stakeholders, including AI companies. And so we did over 50 interviews, for this research, and some of those were with people who work, at AI companies. And one of the things that they mentioned that I highlight in the op ed is that there is a lot of legal risk attached to trying to test and red team your own AI models for their capacity to generate CSAM. It is so radioactively illegal that the risk attached to that is is different than trying to test it for, its ability to, I don't know, you know, violate copyright or to spit out person identifiable information. Those other sorts of harms that you also want to safeguard models against doing. Because even the act of intentionally trying to prompt a model to produce illegal material, even if you meant well, is itself illegal. There's no but I meant well defense in the CSAM laws as I think I was saying, earlier. And so there is a lot of sort of a chill on those trying to test models before launching them into production against being able to directly test, their models. There are best practices out there for, preventing models from generating harmful imagery, but they tend to be relatively indirect as a way of, minimizing risk to the tester. And meanwhile, we talk to the people working at AI companies who are frustrated by the limitations of those best practices because they feel like they're working with one hand tied behind their back, and they know that malicious actors who are going to try and prompt the model to spit out illegal material are not going to constrain themselves, in that way. We've seen this dynamic before with good faith cybersecurity research where for a long time, people were afraid of being prosecuted as hackers for trying to responsibly find and disclose bugs in hardware and in software, software, and where meanwhile, the the actual bad guy hackers were not deterred at all by the potential of being prosecuted in a US court. I think we really can't afford to take that long again to try and resolve it the way it took over a decade to try and address, the cybersecurity research context. And so one of the things that needs to happen, I think, is to clear a way for red teaming and testing and auditing, of models in order to encourage rather than discourage, safeguarding against CSAN production.
I obviously think that's exactly right. I I think the only thing I would add is that, you know, red teaming and testing the models is is the preferable way to go about this in part because some, like, some argue that cleaning datasets, which should happen for CSAM, is is necessary. I would agree that you should clean the dataset. But removing sexual content from dataset, like like consensual and legal sexual content from datasets or images of children from datasets is actually probably not a good way to go about this because we do want the models to understand how human bodies work. We do want the models to understand what they look like for good uses and for just average general uses. Like, if you want it to be able to generate a picture of of children on the beach playing, then it's going to have to understand how children look. If you want to be able to use the model for purposes related to sexual education or medical analysis, you're going to need to train it on that sort of data. So so making sure that we that we create the circumstances that permit the red teaming that creates the the restrictions on the models that we need while also allowing the model being able to use them the way that we need to use them, I think is also a really critical balance to strike.
To pull on this a little bit more, what would accountability look like?
I think we're still going to figure that out, in practice. One of the unresolved open questions that I think is going to get answered this year is, is there immunity for responding to user prompts and generating, illegal material, whether that's in CII, whether that is CSAM? Do we need a new carve out in section two thirty? The ones that are there so far, you know, Sesta Foster didn't do a great job of doing that. But, you know, violations of federal criminal law have never been immunized by two thirty. That's one potential pathway, especially now that NCII is is federal crime. And, you know, section two thirty has never barred liability for the platform contributing to what makes the content, you know, third party content illegal. And so we are squarely teeing up now with the Grok situation. There are at least two lawsuits that have been filed in the wake of this scandal. And in at least one of them, it looks like, xAI and x are going to assert section two thirty as a defense. So we're going to see a decision on the question of our generative AI outputs protected by two thirty from, civil liability. And that might be a very fact intensive question depending on, the type of of offense, the type of conduct, how that particular model works. But it's one that has been kinda sidestepped, until now over the last couple of years since generative AI really sort of stormed onto the scene. It's one that we're going to to find out. It seems to me that there might be a way to try and thread that needle without further degrading the protections for free speech and expression values on the Internet that section two thirty has been, an incredible bulwark, but has gradually been weakening. But I'm not sure whether this is really the right, test case for it or not. This seems like such a egregious situation that I'm not even sure whether it will give a lot of guidance, frankly, to the more good faith actors out there who are trying to safeguard their models, who are trying to do the best they can both out of legal and just business optics and PR,
to make sure that they are, being seen as responsible actors in in the AI industry. What is the response been from policymakers and civil society groups, and what does the response signal about the direction of AI governance?
So, like, I I I think we talked a little bit about this earlier. You know, some some governments came out pretty quickly, criticizing GROC and demanding to know what was going on. Others opened investigations to see if x conducted required risk assessments or undertook mitigations. I think that was specifically done by the European Union under the DSA. Some governments blocked GROC entirely. France, for example, has raided x's offices as part of an investigation into child sexual abuse material on x, and then state governments here in The United States, including Maryland and California, have launched investigations into the generation of potentially illegal content as well. But meanwhile, you know, the US federal government is is integrating GROC into US government systems, including systems at the Pentagon. So there were there was kind of a wide variety, I I think, of act of of reactions. But on a global scale, you know, taking everything together, I came away understanding that governments are going to take this very seriously and respond even to the point of blocking a hugely important service. And it seemed like, you know, it it likely took too long for this reaction, but it did seem like XAI and those in charge of Grok finally did get the message that they needed to engage in some mitigation here. Looking ahead,
what lesson should AI companies and regulators take from this situation,
if any? I mean, I'll I'll, you know, bang the drum again to say, like, you need to make a clear pathway for doing testing and red teaming and auditing for models to keep them from generating this kind of material. That's not to say that XAI, in particular, would necessarily step up and do all of that, but at least making, it providing the right incentives to safeguard models instead of focusing a lot on after the fact accountability, I think we go a long way.
Completely agree. Like, it there there should there should be some red teaming beforehand. There should be a plan for if you're going to add features that pose obvious risks, there should be some there should be something built into the design before you release the product that reduces that risk. In addition, though, because we cannot control every single generative every single output of a generative AI system, we know that there will be instances where the model misbehaves or is misused or is jailbroken. And for that reason, there also has to be mitigations set up and and, like, reporting pathways set up that will work from the moment that the new feature is released so that harm can be addressed as quickly as possible. Because that was another thing that was missing here. Like, there there was not enough of a quick response when harm started to happen.
Well, Rhianna and Kate, it's been a pleasure having you both on Tech Talk today. Thank you so much for joining us. Thank you, Jamal. Thank you. Yeah. Of course. And and and we'll have to continue this conversation, of course, and we'll have to continue this conversation as new developments, continue to to arise. Thank you for listening to Tech Talks, presented by the Center for Democracy and Technology. I've been your host, Jamal Magby. Tech Talks is edited by Jacob Kaufman and produced by Drew Corby. Check out more of CDT's work by visiting us online at cdt.org and on various social media at sendem tech. That's c e n d e m tech. Thanks for talking tech.
