Of the sort of failure modes is essentially when your notion of assistance or cooperation Right. Is this essentially sycophantic one, right, where an AI chatbot or whatever is a yes man and just goes along with whatever you say or whatever you think is the best and doesn't really question your beliefs or your values. Right? I think some people upon encountering these ideas and alignment has, like, recognized as sort of structural similarities to the problems of, like, you know, a misaligned capitalist, essentially. Right? Unconstrained capitalism

getting at the roots of capitalism as being the problem. You're, like, little AI lawyer or whatever it is that gives you legal advice. Right? Or one that helps you do shopping or whatever it is. And one that maybe even, you know, if these things get good enough that helps you do research on which local politicians you should vote for and recommends them. Right? Like, oh, we're just gonna have a single human giant AGI thing. That's the first superintelligence, and we're gonna align it to all our humanity's interests. And it's like, what is that? It's not implausible to me that we have decentralized AI organizations in the future because they're not beholden to that. It can take the risk of enacting in that kind of moral and political activism that humans routinely undertake because we wanna see a better and more just world. This episode is sponsored by NIM, the world's most private VPN that protects your Internet traffic and metadata.

Yeah. For sure. Thanks so much for having me here, Josh. It's really great to be talking to a set of folks really interested in, I guess, how, you know, technology doesn't have to just empower the people who are already rich and powerful, but hopefully also empower, like, all people. And that is, for me, I think one of the important aspects of what I see as cooperative AI, even though that has many other dimensions as well. So but a bit about me first. So, yeah, I'm Shuan. I'm currently assistant professor at NUS in Singapore. I'm a Singaporean. I grew up there, but I did my undergrad and PhD most recently at MIT, graduated quite recently. And over the course of that PhD, I ended up working on a set of research questions that I'm now continuing to work on in my new lab. Right? And basically, back when I started, you know, or even right before I started my PhD, AI was still kinda niche. Right? It certainly didn't have the kind of cultural and economic impact it's having now. But even back then, almost like ten years ago now, people were starting to worry about the possibility of, you know, extremely advanced AI systems that were basically under the label of superintelligence. And And I encountered some of those ideas in undergrad and, you know, without necessarily being convinced that this was the future, right, or how soon it would be because, you know, back then, people didn't think it would happen even as fast as it I think it happened, has happened recently. I became, you know the the people writing about this, people like Nick Bostrom and various other people associated with effective altruism and online rationalism, were worried about this becoming a really impactful for destructive human civilization. Because if you did have, you know, more superhuman intelligences in society, in our economy, that would, you know, minimally change things, you know, change our economy drastically, but potentially even, you know, take over the world. And that's something I hope hope I guess we'll discuss a little bit more later. But I think the central question then coming from the community was like, well, one way to stop this is maybe we need to align AI systems of human values. We need to get them to really understand human values well enough to reliably promote them. And that was his question, just kind of from me being a philosophy nerd perspective, was really interesting to me because asking that question in the computational terms was just, I think, fascinating. Like, how is it that humans learn human values was something I became quite interested in sort of independently of beliefs about whether AI will take over the world. And that's sort of started me down this direction of, okay, how could we begin to sort of formulate a computational theory of of learning human values and learning to basically work with other humans. And this is what ties it into cooperation. Right? I think part of why humans have values and have notions of, like, what you ought to or ought not to do, right, is so that we can live together with each other. Right? And so that's eventually evolved into sort of like a perspective around, okay, if we want to make sure AI can live in a world together with humans and together potentially of other AI systems, then they need to learn to cooperate. Right? So I think that's sort of my entry point into sort of what I think of the research I'm doing now on cooperative AI. I can imagine the kind of, like, the retort

Yeah. So I think that is one thing that you could want to do with AI systems are able to cooperate with humans. Right? I think one pretty, you know I don't know if to call it dominant, but certainly very prevalent and certainly very visible form of AI technology that we're seeing today, of course, of course, our AI chatbots and or you might think of them as AI assistants. Right? And and the sort of like, you know, as a product, you know, because AI techno as a technology is much richer than just a particular, you know, chatbot product is the sort of the way that these things are sold is precisely to help you with daily tasks. Right? To help you, you know, any question you can you have, you can ask a question. It'll answer you. Any life situate you know, a life problem you might want to go to, it it can help you figure out how to do things. Or maybe it'll just help you do the task altogether. And this is where we're starting to get into sort of more recent, you know, different kind of, you know, product of AI agents. But you're like, when you tell to do things, then they do go go about it for you. And I do think that is, you know, one aspect of cooperative AI. Right? Certainly, how to design better, more reliable, more trustworthy AI assistants or AI agents that do work on your behalf or promote your values or promote your interests is an important part, I think, of cooperative AI. I don't think that can there there are sort of both good and bad aspects of that kind of, you know, you know, technological use case. But and and I I do think one of the sort of failure modes is potentially when your notion of assistance or cooperation Right. Is this essentially sycophantic one, right, where an AI chatbot or whatever is a yes man, right, and, you know, just goes along with whatever you say or whatever you think is the best and doesn't really question your beliefs or your values. Right? But even within that realm, I do think there's possible to think like, well, what would, say, a good human personal assistant do? Or perhaps relatedly, for some of the other, you know, different kinds of use cases that people do. Like, what would a good therapist actually say to a human or a good friend actually say? And they don't they, you know, they aren't just yes men. Right? They aren't just yes people. Right? And, you know, and, you know, I think a good friend really tries to look out for your best interest, right, to some degree. And if that means pushing back on, you know, self destructive behavior or misguided beliefs or whatever it is. Right? I do think that's an important part of, you know, the sort of cooperative endeavor that you might call friendship as well. And you could imagine having you know, I don't think it's in principle impossible to have AI systems that play some of those roles as well, you know, whether or not we want them to. I think in principle is technologically possible. And and it's a separate question, of course, is whether the companies deploying these products will be incentivized to do that versus deploy more psychopathic systems. Right? So so that's, I think, a response to this, you know, idea of, like, everyone having their own AI Pokemon thing. I I do think AI cooperation is cooperative AI is also a lot richer than that. Right? Because I think there's cooperation at the scale of a single human interacting with a single AI system, and that's, I think, the predominant form of interaction we're thinking of these days with AI systems. But, you know, as someone who thinks that, you know, fortunately or unfortunately, I don't know, it's probably you know, I don't know. I think right now there's a lot of reason to worry, but that AI technology will keep progressing. What's going to happen then in that future is AI will begin to sort of AI systems will play more and more roles in the economy, right? And and sort of like play more, you know, become take on more roles as economy actors, empowering viewers, existing economic actors. Right? So if we end up in a world where, you know, a large number of different actors with different interests, right, and that's a sort of practice feature of the world, are empowered by AI systems, right, then this can create new problems that we haven't faced before. Right? New problems of conflict, new problems of exploitation, new problems of just not managing to coordinate with each other well and causing problems for all of us. Right? And I think that is also a super important problem that I view the field of corporate AI is trying to solve. Right? How do we make sure, like, that the AI systems we build and the institutions we build around them, promote cooperation among a large number of AI empowered actors well enough to ensure mutually beneficial outcomes for all and hopefully egalitarian. So maybe

Yeah. Great question. And and, you know, I think it's interesting even hearing this sort of, like, you know, in my view, sort of reductive description of AI. It feels a bit frustrating to me as someone who's been doing AI research for a long while, partly because AI is a broad word. It's there's a long tradition between how the word is used and how it's been used since, you know, I think the seventies when it was, like, kind of first invented. Right? And back then, it really didn't refer to what people currently would think of as, you know, mixed word prediction systems to some degree. Right? Didn't refer to these large language models. They refer to systems built on essentially a lot of first order logic. Right? So so so, you know, the kind the technology class that AI has referred to attended to referred to has shifted greatly over the years. But I think as someone who's a bit more embedded in the research community, for me, it kinda connotes that whole class of technologies with different kinds of characteristics and different kinds of pathways and different kinds of implementations. So so there's a certain sense in which it's right to think, you know, like, yeah, like, the current popular chat gbt style applications are kind of black boxes in the sense that we don't really understand how they work. Right? But there's a lot of other kinds of technology that are called AI or at least used to be called AI that are just permeated within, you know, how we use technology space. Right? Like, for example, Google Maps navigation function Yeah. Right, is the shortest path finding algorithm. That came out of AI research from the the seventies and eighties. Right? And we don't think of it as AI nowadays because we understand it so well. It's just another algorithm. But it's actually crucial to a lot of the things that people sometimes do call AI, which is like video game AI, routinely, you know, which is not using any kind of large language model under the hood, is using all these algorithms to sort of do, like, intelligent path finding in the video game. Right? So I just wanna give a sense that there's a lot of AI beyond just, you know, large language models or things that are making images or whatever it is. Right? But let me say a little bit about how those kinds of systems work. So the way I think about them is that they're they're sort of symbolic AI, there's probabilistic AI, and there's kind of like neural network AI. And there are various combinations of the two of of these three. Right? And the one that's sort of has been most successful in in various ways and most popular nowadays are the neural network based systems. Right? And what they do is they're basically machine you know, they they're they're also known as machine learning based systems. Right? And the way they automate certain kinds of tasks or achieve certain kinds of tasks is by learning from large sets of training data of how to perform that task, you know, often collected from humans who have performed that task over time. Right? And simply by learning to sort of imitate how to perform a certain task from a really large corpus of training data, you can begin to mimic, say, how humans were gonna perform the task. It started in the early mid twenty twelve, the twenty twenty's, a lot with image classification. Right? That was a sort of big early initial boom of AI. Like, oh, now we can classify images. This is great for stores that want to detect what products they have on their shelves or whatever it is. Right? Or people who want to do surveillance to detect which face passed behind the camera. But then it grew into what the field called natural language processing. It was like, oh, maybe we can get systems that predict from a corpus of text how to generate the next word. Right? And then eventually, in this most recent era of generative AI or large language models, people realized that if you built these models really, really large, they can mimic human text in surprisingly human like ways. Right? Way more surprising than most people in the field had expected. And that's, I think, what's led us to our current moment of, like, AI systems that are purely just from you know, largely from just being trained on how humans talk on the internet. I can imitate that. And then more recently, with a bit of additional work, fine tuning that, right, providing additional data that says, okay, there's a lot of things you could say that look like text on the internet, but we don't want you to speak like four channers. We want you to speak like a professional bureaucrat or whatever it is. And so we're gonna get additional human data that says these responses are good, these responses are bad, and further train the AI system to output good answers according to what the human labelers judge. And that's how we're getting these, like, much more polished sounding chatbots that we're we're we're we're getting today. Right? So that's a sort of, you know, high level description, hopefully, of of, you know, one class of today's AI systems. So I guess this specific

Yeah. I I do think it's a remarkable and unexpected feature, unexpected by everyone, including, I think, many AI researchers. Right? That what we've ended up automating into, you know, not completely yet, but, like, to, like, much more than we had previously expected is, like, you know, like, white collar work and also, like, artistic and aesthetic labor. The kind of work that for a long time people have sort of thought, oh, this is the the job I wanna do growing up. Whereas it used to be thought in a lot of energy in the past had been put into, you know, robotics, you know, and all that's also a different kind of AI. Right? Like automating physical labor. But I think, you know, all even even though there has been a lot more progress in robot robotics as of late, like, there's a lot of reason to believe that it'll happen much more slowly than purely, you know, things that can happen purely in the digital realm. And I do think that is going to have interesting effects on the sort of political economy and AI and also the cultural response to it. I mean, it really is having I do not think we would see the kind of backlash cultural backlash that we are seeing today if it had continued happening in the sort of automating blue collar or jobs way first, partly because, you know, it's the sort of chattering classes as it were who are, you know, sort of discussing a lot of the more of these issues. And given that now this way of automation is hitting that class more Yeah. Well, that's what's gonna be in the media. Yeah. I I think so, like, part of part of this chattering, you're talking about, like, some of the first, like, chatterers

Yeah. Yeah. It's yeah. Definitely happy to share. You know, having been, you know, in the kind of alignment space for quite a while since it's, you know, not the super early days, but since, like, perhaps ten years ago now. It's been interesting to see it go so mainstream in a way that's kind of weird. It really used to be this nation fringe thing that no one believed was important or or worth caring about because no one seriously thought that we would get, like, anything close to AGI soon. Right? But I can say a little bit, and this is maybe a good chance for me to talk a little bit about different sort of technical conceptions of AI right now. Right? Because I was just describing a little bit about, like, how current AI systems are largely trained, which is that they're basically trained to imitate. Right? One way to think of them is they're imitation learners. They're trained to imitate the distribution of data on the Internet, and then with a little bit of labeling, sort of skew that distribution in some direction. Right? But for a long while, the conception of AI, especially since the the nineties, right, there was this conception of AI as an intelligent agent. Right? What does that mean? The idea of the intelligent agent actually borrows from the idea of a rational agent from economics. And what rational agents are imagined to do or continue to be, you know, typically assumed to do is that they maximize their own self interest, right, where self interest can be well captured mathematically as something called a utility function or reward function or an objective function. Right? And this idea came to really influence certain strands of AI research, especially starting from the the nineties onwards, right, and especially in the fields of reinforcement learning. Right? And so this particular conception of AI systems as essentially maximizers of certain objective functions, maximizers, minimizers, continues to be, I think, a sort of the sort of starting you know, continues to be one very common way of thinking about what AI systems are, especially amongst AI researchers and scientists. And it was really the motivator for a lot of this discourse around AI alignment that started amongst the online the AI algorithms. Maximizers. So like kind of thought machine thought. Exactly. Right. Yeah. Really yeah. The the thought experiment comes out of this idea of, like, what is intelligence? It's to be rational. What does it mean to be rational to maximize and expect maximize expected utility? Right? And so so you get from, you know, online rationalist right? You know? And the connection to rationality here is, I think, quite explicit. Right? Like, online rationalists started this community, you know, around Eliezer Yarkovsky earned the idea that you ought to be rational. And what does it mean to be rational? It's to be, you know, roughly speaking, try to be Bayesian. You up your update your beliefs according to the rules of probability theory as far as possible and maintain uncertainty about things you should be uncertain about. And also, like, you try to systematically win. And one way to mathemathematicalize what systematic winning means is you have your prespecified interest and values, and you try to maximize your chances of attaining them. Right? You know? And that looks like maximizing your utility function. And given that that was that was, you know, the rationalist conception of what rationality means, I actually think rationality means a lot more than just that. But that was the conception. That is the conception. And how closely they viewed it as tied to basically being an intelligent agent, they began viewing essentially AI systems as essentially having that form. Right? And if you start wondering, okay. What if we build agents that have this form? They're maximizing some utility function. And they're super powerful. They're way more powerful than any kind of any human that exists today. Right? Then you very naturally start to worry about what of the utility function that they're maximizing isn't aligned with what humans actually want in a way. Right? Perhaps that utility function is making as many paperclips as possible. Perhaps it's mining as much cryptocurrency as possible to generate profit for you, right, perhaps more realistically, and many other things. It could even be something that seems like it's a good thing, right, which is like, oh, try to make everyone smile. Right? But if that's how you specify utility function, well, one way to potentially achieve that is to inject neurotoxin into everyone's body as it freezes their face into into a smell. Right? And so that's kind of the thought experiments which sort of motivated the field of AI Lab initially. The idea is like, wow. It's really hard to specify objective function or utility function that captures actually what humans want in the world, and this is a big problem, and we don't know how to solve it. Right? So it's kind of very philosophical. Right? Because we didn't really have the kinds of AI systems that could do these things back in the day, but it was like a worry comes out. Like, well, the math says if we actually did have it, it'd be really bad by default. So that's what it's What is the theorem Yeah. Or law? I forgot what it is. Like, when a measure when when you create a measure, it's used to become a good measure type of thing. Exactly.

you know, to be powerful under capitalism. But Yeah. No. I I think it is closely connected. I mean, I think, you know, despite the fact that, you know, interestingly, a lot of people in the rationalist here have tended to be libertarians. I think some people upon encountering these ideas that have been alignment has recognized as sort of structural similarities to the problems of a misaligned capitalist essentially, right, where it's sort of the business model or the profit objective is just misaligned. With human And, like, for me, rationalism has come is is kind of like a kind of intellectual liberalism.

Yeah. No. I think I think there actually are a lot of similarities now you're mentioning to the the historical analogies. Right? Because, you know, the, you know, so called founder of the, you know, online rationalist culture, Aleazar Yukovsky, he started out really wanting to create, you know, friendly AGI, essentially. He thought that was like, we need to do this. It's gonna be the best thing ever. I mean, similar to how perhaps early liberals really thought. It was like, yeah, free markets are really good for society. And then sort of realizing perhaps not too long after, it's like, actually these have their own pathologies. Right. And so, yeah, there is that analogy that you're and I'm not realizing yet. And in fact, some of the people in this space now are some of the sort of most ardent supporters of, like, really drastic regulation around AI to the extent of, like, stopping all the construct you know, con you know, you know, in resonance with, like, you know, some recent calls by, I think, Senator Bernie Sanders is to, like, stop all data Yeah. Yeah. We'll we'll get to that in in a minute. But I think, you know, knowing this, I think one of the things that would be interesting to get your thoughts on, because you have been thinking about

Right. Yeah. Yeah. Yeah. No. For sure. Yeah. I I think it's, you know, I mean, it is definitely a sort of very interesting being an AI researcher and a little awkward too in in a in a world where I think, you know you know, I also have, like, friends or many peers who are like, yeah. It's the worst thing ever. And I feel like sometimes when people ask me what I do, I have this I can sort of, like, feel like sheepishly admit I'm like, I'm an AI professor. But actually, I don't really work on the things that, you know, as much anyway, that I think people are really worried. Are you, like, gonna take their jobs or, like, you know, like stealing in sort of creative labor of of humans? I I do think those all those concerns are sort of very understandable and I think legitimate. I mean, I do think that, you know, people's I mean, you know, if anytime your economic livelihood is hurt by people who seem to be, like, you know, taking, you know, the output of your creative labor without compensating you, I think that's that's you know, I could if I were in that position, I'd be, you know, very upset too. Right? Now, I think even the sort of longer term trajectory of AI is something to worry about. Right? So even if you're not someone who, you know, like some of these AI alignment people were talk you know, like, people concerned about AI alignment and really concerned that super intelligent AI is gonna take over the world. Even if you don't have those views, you know, you might still be really concerned about, you know, AI getting more and more powerful and eventually being able to take over your job even though it doesn't currently do so. So I think all of those concerns, you know, I think we're right to worry in a certain sense. And I think my response to that is like, what does the worrying point us towards doing instead? Right? And I think one response is what I will sort of provocatively think say is, like, what I've used essentially a reactionary one, which is to say, no. We're just going to ban, you know, as far as possible push to stop and ban usage of all AI as possible. I will personally make a commitment to never using AI in my personal life or professional life or something like that and, like, trying to push institutions and part off to reject all usage of AI. I think that's you know, it's you know, in this sort of grand equilibrium of, like, what people should do, I think it's reasonable that some people are are doing that. Right? Because I think we need a mix of various people trying to push things. And without that voice, I think people who are, like, just super pro AI wouldn't have to consider or contemplate, you know, like, the impacts that's having on on people at all. But my own view tends to be a bit more you know, I I even though I think it's important for some people to be to to to be strongly championing that view, I don't think it's actually gonna succeed for a bunch of reasons. Perhaps I'm a bit too techno determinist in this respect. And I think we need to think about instead, you know, especially those of us on the left, have to think about, like, how to, you know, really, you know, restructure, like, you know, like, like, co opt AI for our own purposes. Right? Or not just AI, co opt technology broadly speaking. Right? Because I think one problem I have with, you know, some kinds of anti AI discourse is, like, what, you know, precisely as I described earlier, the sort of boundaries of what AI are are very murky. Right? And, you know, so is the question, does rejecting a certain class of technology that was trained on large amounts of data, right, or are you now also rejecting, say, Google's pathfinding algorithms or your social media's recommender system or, you know, like or just, like, you know, more local forms of, you know, AI system. Like Siri was a kind of AI that but it wasn't built on you know, for a long time, it wasn't built on the same kind of, you know, AI technologies. They were machine learning based but weren't trans and tons of data. Right? So or, you know, self driving cars, for example, which are actually despite also taking, you know, say, Uber driver's jobs, actually have a very different technological stack in them compared to, you know, modern day, say, large language models and chat GPT. Right. And so given that, you know, the space you know, what AI is is actually really diverse, I think, you know, just saying anti AI is, like, hard many times actually to operationalize. And also, I think by taking that view, we're potentially missing, you know, the sometimes this sort of revolutionary, liberative potential, right, of, you know, automated automation in general. Right? I think it's helpful to recall, I think, for those of us on the left, in many ways, leftists, at least of the sort of Marxian variety and and sort of its descendants, were some of the people really calling and trying to push for more automation so that we could live in a larger society, a legalitarian leisure society. And I think that is one positive vision that is being sort of forgotten if we sort of view things only on NTEI terms. Yeah. Yeah. I think the

Yeah. Yeah. And as you saw the contrast, you know, it's kinda interesting to me, like, you know, like yeah. Like, you know, you know, when a capitalist is saying it, then you're like you know, versus when you know, I think I think it's interesting. I was just reading Not so long ago, I I finished reading Bullshit Jobs by David Graeber, the anarchist. And, you know, he actually is, like, you know, in that book, Pretty Poor Automation. And I I'd be curious. You know, it's really unfortunate he's passed because I'm really curious for his opinions on the current, you know, political moment for all sorts of reasons. But and but one thing he, you know, advocates for towards Anna is, like, you know, like, maybe UBI would work, but also, like, he is pretty pro automation to you know, in the end of the book because he just thinks so much jobs are actually bullshit. And and but I think one of the interesting anecdotes he shares is that, apparently, back in the sixties, you know, in contrast, I think, the current, you know, anti automation sentiment on the left, he sort of shares that, you know, you know, I think his wording, like, everyone from the YIP based situation is we're basically calling for let machines do all the work. And they were expecting, you know, you know, robots be able to do everything in the factory, and then everyone could then have, like, much shorter work weeks and just do do things. And then there was actually a cultural sort of response to that from from the sort of, like, dominant, you know, conservative right to be like, no, people are thinking, you know. And so interestingly contrast that moment with where we are now, right, where because I think we understand, I think rightly, that the people who are trying to automate things are the capitalists, are the rich people who we justifiably don't think have our best interests in mind, we justifiably think they're going to enrich themselves with this technology and not everybody, then the technology ends up having this part of a political character of disempowering everyone else. Right? But I don't think it necessarily has to be that way. But we need to figure out ways in which essentially One way is to just stop all its usage. Right? But I do think it's possible to think of other ways of disrupting currently rich companies' hold over the technology in a way that the benefits and control over technology are much more widely distributed. Yeah, definitely. I mean, yeah,

rationalist space, there are people who advocate for that, and I think Yudkowsky is maybe one of them. Yeah. Well, I don't know what to recall in pro AI. You know, I think there's NTX and it's like different set of considerations. Yeah. Yeah.

Yeah. So, you know, I will say I'll pre preface this by saying I don't have extremely strong opinions here because I think there are a lot of uncertainty. I have a lot of uncertainties about what's best apart from, like, a knee jerk reaction against, like, to overly strong exercise of state power against certain kind of things. So so so I'll say a little bit of that. But yeah. So I think it's interesting. I mean, I've been seeing, you know, ever since this sort of, you know, Bernie Sanders sort of pushing for this. First of all, I'm actually really wondering, yeah, like, where did this come from? I know he recently talked to Jeffrey Hinton who has been, you know, interestingly, and, you know, one of these, like, godfathers of AI, identifies as a socialist and also has been worried about the the the risks of, you know, in particular, like, superintelligence and AI taking over all human's jobs. They had a recent conversation. I'm also kinda curious about, you know, which aids were perhaps, you know, instrumental in putting together this proposal together. Certainly in the water, I think both from I think you might might might think of as the anti anti AI left and also I think more broadly, like, you know, the sort of AI safety movement. Right? I think within people who are concerned about AI safety, people concerned about AI safety typically are ones who think that, you know, something like general intelligence or superintelligence in the next sometimes even the next few years is imminent. Right? And so especially for those who think it's going to be very soon and also by default quite bad, understandably, they think it's great to slow things and pause things pause things right now. So I and on, you know, on my Twitter feed, I feel I've been seeing a lot of the that as being one major sentiment, right, from people who are very much more on, like, the POS AI team from a AI safety perspective and or AGI safety perspective, I should say. And and, you know, that happens to come core, I think, with, like, you know, the sort of the the views that say Bernie Sanders sends more to represent. But on the other hand, you have, I think, sort of, yeah, people who are like, no. This is a bad idea for all sorts of reasons. You know, perhaps one of the reasons you know, some reasons they see cited is like, well, this will just push data construction overseas, and then America will lose its lead, which is like it's kind of like China will take over and China will use it against The US. That's yeah. Yeah. So that's that's one kind of response. I think another kind of response is I think from people who just don't think, you know, super powerful AI is gonna happen soon. And so it doesn't make sense to stop the build up. And I do think it's worth recognizing that most data centers, the main usage is not for AI. It's for all sorts of computing. Right? For every Google search, like, all your Google drives, everything, that that's actually the majority of the use of of data centers. So that's, I think, worth remembering for people who are going to post their, you know, support of post their construction. I think the majority use is not necessarily AI training even though that's what a lot of the investments are being poured into right now. So right. So I've described, I think, the landscape of views I've been seeing. As for my own view, I think it's interesting that you mentioned Vitalik. I didn't realize he had this also view of, like, maybe it's good to because I think, you know, similar to and you also mentioned the e x. Right? I think the way I describe my sort of general relationship to technological development is, I think it is, in fact, well described a term that Vitalik coined that he he said he calls it d slash acc, right, and which for him means, like, decentralized, democratic, and defensive defensive and differential technological development. Right? And when I the way I view my research in cooperative AI is essentially as a kind of differential technological development. Let's advance certain kinds of technologies that that we have good reason to think, or at least I have good reason to think, will lead to better outcomes if they're developed and deployed first relative to other kinds of more centralizing technologies, technologies that enable more offensive capabilities, technologies that sort of make the world worse in various ways. Right? And so that tends to be my general view on technology development. How does it interact with data centers? I don't have super the the reason I think part of the reason why I don't have super strong opinions on this is because the kind of research I do is on AI that isn't particularly data intensive or compute intensive. Right? And in some sense, right, you know, the and I tend to view that class of technology as beneficial in a bunch of ways potentially. You know, I think technology can always be have different political characters depending on how it's used, but potentially beneficial in the sense that it will could stand to disrupt existing super data and compute heavy models of AI development right now to the extent that ownership over AI could be more broadly diffuse. Right? And also more efficient in the sense that you just won't need as much power. Right? I think there's a huge gap right now between current AI models and the sort of efficiency Yeah. Of human cognition that I think is in in principle possible to bridge. Right? And some of my you know, I don't, you know, I don't view myself as wanting to replace anything like human labor at all, but some of the techno the technological stack that myself and others in the sort of other parts of space are trying to advance in some ways have a bit more of that flavor. Right? And so it's interesting, right? You might think that depending on your views of AI, perhaps many data centers will even accelerate that. Right? Because now people are forced to develop more data efficient and power efficient and compute efficient AI technologies. And whether that's good or bad really depends on how that technology is used

It could be a pro innovation thing. And whether that's good or bad, it really depends on yeah. Like I said, like, whether what you think the effects of that and the technology are. Yeah. Right? And then Like, you have, like because, I mean, you have, like, the SEC. You know, they had this regulation that I think was,

Your thoughts, you were mentioning you had some ideas from the blockchain space that Yeah. Yeah. That that's great. Yeah. Yeah. Yeah. No. Definitely. And and and let me maybe, like, sort of start with a bit of a preamble around, like, a sort of a state of AI alignment research right now because I feel like it didn't complete that earlier question you asked me. It's like, you know, is AI alignment solvable? I sort of describe its origins. Right? And I think where the field has shifted now is it's gotten a lot more mainstream, of course, because suddenly we have these, like, you know, chat gbt and all these, like, way more powerful AI systems than we initially expected. Right? And the way that sort of the AI alignment and safety community thinks about these systems, they call them frontier AI systems because they view them as, like, the closest we have to something like artificial general intelligence. Right? AI systems can basically do any kind of, you know, they say cognitive task because they want to sort of gerrymander out physical labor, which I do think is actually an important part of human intelligence as well. But, you know, and can automate any kind of, you know, at human level, any kind of, you know, cognitive task or sometimes the other the metaphor that gets used is, like, you know, what if can the AI's that can be drop in human remote workers. Right? So they're really concerned, and and and there's this, like, default view of the the the technological pathway that we're heading towards is, you know, AI giant AI models, you know, train on tons of data that can basically do these things, can replace all of human labor, and I think motivates some of the sort of extreme worry that you got that sort of motivates people to say, actually, yes, shutting down AI or actually, yes, more term AI data centers is good because otherwise, we'll have these kinds of systems too soon. Right? So so there's a lot of concern around, like, you know, what we describe as frontier AI safety now that has moved away from less of its origins of, like, what if we had a really powerful expected utility maximizer towards a more a bit more empirically grounded picture of, like, actually, these things are getting really good very really fast at all sorts of, like, software engineering tasks. And if the trend lines keep continuing, then they will just do more and more of that. Right? And we're not confident that they always do the the things they we ask them to. Right? In fact, there have been sort of well documented cases, you know, in the software engineering realm, for example, that instead of, like, trying to pass the test or or do what you do, they start hacking. They they do what's known as reward hacking in the literature where they sort of modify the test, for example, in order to make them pass. And then it's like, okay. The job is done. Right? Sort of they're hacking the metric. Right? And that's something you really don't want powerful agents of any kind to be doing to be running a mock doing in a in a economy, right, or in society. So so that's kind of like where a lot of the attention and energy around sort of AI alignment and and correspondingly AI governance is focused now. Like, how can we make sure these large AI models do what the company says or do what the users say, you know, some combination of that. And, also, like, the governance question happens to be around how can we make sure that AI companies developing these models? You know, make sure like, put in enough investment and energy to putting in the right safeguards. Right? So I think that's a sort of, like, mainstream conversation around, like, AI safety and alignment these days. So so with that preamble, right, I think, you know, I tend to think a bit about more a little more decentralized approaches towards AI alignment and governance for for two reasons. One of them is my politics. The second one is just, like, my I have different empirical beliefs about the sort of shape of tech and AI technological progress, right, sort of alluded to earlier, which is that I do think it's gonna be the case that these very large AI models will be disrupted by smaller, more efficient, and more narrow technology in many ways. It's not necessarily I think what will turn out to be more economically efficient and useful and adoptable by, say, many companies, many individual consumers or users are much smaller, more narrow specific models that they maybe could even control and own. Right? And in that kind of world, right, I think the both the alignment question and the governance question, which are you know, I think of them as duals to each other, become look really different. Right? It may it ends up looking like a bit more like a world where, you know, tons of different AI services specialized for real tasks, and perhaps everyone has their own, like, you know, many a suite of different, you know, not just one general purpose model, but a suite of different AI agents, like, things that help them various things. Right? Maybe one for managing, you know, your, like, little AI lawyer or whatever it is that gives you legal advice. Right? Or one that helps you do shopping or whatever it is. And one that maybe even, you know, if these things get good enough, that helps you do research on, you know, like, you know, which local politicians you should vote for and recommends them. Right? So so that's a sort of, you know, potentially positive vision of what an AM powered future for everyone could look like, you know, harkening back to this question around, like, not quite yes men, but something else. And and that if that's what the future AI enabled economy looks like, right, where, you know, there are companies which are powered by tons of, you know, maybe some human workers, some AI workers, and every individual human has, like, you know, a variety of, like, AI assistance, maybe more narrow than the general purpose ones we have today and and certainly more efficient, then I think regularity approaches that, like, you know, focus on, like, clamping down things are gonna be harder. Right? And instead, you need to find ways of setting things up so that, like, basically, there's much more, I don't know what to call it, self regulation. Right? But, like, the dynamics of this economy tend to sort of promote beneficial outcomes and prevent disastrous ones. Right? Like, for example, you don't want people everyone running around with AI agents that hack other people's accounts and steal all their money. Right? So how could you achieve that? Right? I I think this is potentially where blockchain technologies come come into play. Right? Right. You know, you know, part of one of the sort of visions behind blockchain is like, oh, we don't actually need the state or the courts or whatever it is to step in to be the arbitrator of every single dispute, right, that we have. We can't just have these smart contracts which enforce this by default in terms of trustlessness. Right? And, you know, insofar as you know, and I should sort of lay it on the table for how to set the flavor of, like, socialist I am. I'm a bit more of a, you know, somewhere between a market socialist and a market anarchist. So I'm pretty comfortable of using sort of market style technologies to sort of enable, you know, sort of point the economy in the right direction as it were, or in the direction of sort of human flourishing. And, you know, I think blockchain is one part of that. You know, if you, for example, envision a system where, you know, AI you know, people have these AI agents negotiating on behalf of their interests. Right? And, you know, and why do we have to do that? Well, it's because, like, we have interests that often conflict. Right? You know, let's say I want to throw a party at my house that will annoy my neighbors. Right now, I have to, you know, consult this. Right now, I'm in Cambridge, Massachusetts. Consult the local Cambridge City Massachusetts government about, like, when exactly am I allowed to throw a house party. But I think in a lot of reasons, this feels, like, inefficient and annoying, you know, even from a resident perspective. And it seems like I could just I should be able to, if it were easy, to negotiate with my neighbors to just come up with something like, okay. And maybe I'll even, like, you know, like, trade something in in in in in response, or I'll do something else for you in order to be able to have this party at this time. Right? And, you know, people can you're neighborly enough, you can already do that in person, but, you know, AI could reduce the so called transaction costs for doing that sort of thing. But then also you have this question of, like, how do you enforce the performance of this agreement that that you made. Right? And I think this is where things like blockchain technologies can come into play. Right? Like, maybe parts of the contract can still be traditional, like, natural language style contracts. Right? But parts of it can be smart in the sense that, you know, you commit to, like, really doing this thing at this condition holds. Or you can commit to I recently learned about the whole field of automated or dispute resolution and sort of using blockchain style technologies, which I found super exciting as as a positive use of blockchain. Right? So you can make sure that if a dispute arises around a contract, then it sort of goes gets forwarded to sort of community wide settlement of how exactly, you know, the people here should have settled the dispute, essentially. So that's a bit of a flavor of the kind of way I think blockchain could enable decentralized self governance in any, yeah, future. And it's a very different flavor of the kind of, I think, a lot of discussions that, you know, people are having now, but, like, in the at a big national level. Right. That's that's interesting. Yeah. So if I can

Yeah. But currently also not very efficient and not very reliable. Right. And and I guess, like, part part of this reliability

Well, I think the hope from the mainstream technological paradigm is, like, yeah. If only we add more data, then eventually we'll, like, grow the sort of size of the what is in distribution for the language model that they can eventually do everything. I think that is sort of, like, one sort of, like, view that that the sort of maybe big AI companies have. Right? And I think there are a lot of, like, p like, but there's a lot of, you know, energy among smaller startups now who are taking various bets and, like, actually, we don't have to do it that way. There are smarter, faster ways of doing this. And I tend to think, like, one of these approaches, there are many, I think, being tried right now and then at least one of them will succeed eventually. And I think I there are good reasons to doing this. This is just tying back to my sort of background a bit in cognitive science. So I did a one of my labs is actually a conversational cognitive science department. And so the way we think about human intelligence is often from the perspective yeah. AI is often from the perspective of thinking of what humans are able to do. And so I think, first of all, there's this huge gap between, you know, still, in my opinion, between AI and, like, what humans are able to do very flexibly and efficiently. But also, I think this is an interesting feature of human intelligence is that, you know, we invented programming languages and computers because we wanted to specialize and automate some of the cognitive labor that we're performing much more reliably and precisely than we have been doing before. Right? And there's I I I see there's, like, there kind of for me, it's like there's no reason why that and that turns out to be more efficient. Right? Like, you know, there's, like, specialization of labor. It ends up it does end up displacing certain kinds of tasks that humans use to perform by hand. Right? And why doesn't that basic logic apply to today's general AI models as well? Right? That's sort of a question I've been having in my hand. It seems to me that some version of that still ought to. Right? Even if it ends up being not humans who are writing the code that replaces these big big AI systems. Right? But the AI, you know, systems themselves. Right? Maybe we'll have, like, good super good generalist AI programmers. We certainly have those ready. Right? But maybe there's just only one kind of AI system that suited for that kind of task. But but I think there's good reason to think in the same way that humans have, like, outsourced their cognitive labor to more specialized roles. You know, powerful generalist AI models should be able to do that as well, even if those are the things that become the most powerful and smart first. So then in that world,

that they were creating this EIP. I think it's, like, x x four zero two, I wanna say. I might be bushing it, but I think it's interesting that these type of things have kind of been happening in the background as well. Right. Yeah. Exactly. So I I think broadly the picture I have is, like, you know, I'm I I I think, you know, we're gonna have a much more pluralistic AI future than than I think a lot of the sort of big labs tend to assume. Right? I think there are a lot of more actors that have, you know like, ownership is gonna be more widely distributed just because there'll be kind of current models will be kind of outcompeted for a variety of ways. Then we have to contend with the fact that we can't just regulate a few companies, for example, that are located. And and also a lot of different actors we empower. I think that's desirable in a bunch of ways too because people can and should want to you know, I think it used to be among the rationalists as well. Like, oh, we're just gonna have a single human giant AGI thing that's the first superintelligence, and we're gonna align it to all humanity's interests. And it's like, what is that? People don't have shared interests. Yeah. That that another big thing that I think discussion is just, like, glass interest.

super general intelligence a bit. Yeah. I I think it's more skeptical than than the sort of average person in this space. I don't think in principle I would rule it out. Right? I think I tend to think it's going to take longer than most people. I think it's I'm not someone who thinks it'll never happen. I think it'll take on a scale of decades rather than some people think it in the next five years. But also I think, you know, contemporary that even though it's technically possible, I just don't think it'll be the sort of economically dominant form of what you might call intelligence that we'll see. Right? So when we think about the future AI economy or the future I don't even want to say AI sometimes, the future automated you know, increasingly automated economy. It doesn't just look like this one class of technology running everything.

Yeah. Access and, like, you know, also yeah. And also separately, you know, even if it's not a mandolin system trying to, you know, take over the world, it's like just like I think adoption is just gonna be a lot slower than than people like, some people seem to expect it to be. Right? I think there's this great piece called I don't know whether I used the phrase they used, but it the type is called AI as normal technology. Basically, it's sort of advocating for a very different view of how we should expect the future to unfold, which is gradual and diffuse transformation still of the economy and in society, but gradual and diffuse because of tons of bottlenecks in terms of adoption, in terms of you know, even if you can automate one thing, there are gonna be frictions everywhere else. Right. So maybe just the the last question for you. I'm curious just to hear

Yeah. No. That's yeah. And I think it's great to return to this now that I think I've painted a bit more of a picture of, like, what I expect sort of our AI enabled future to look like, for better or for worse. Right? And, you know, given you know, I think that it's gonna AI technology development is gonna be gradual. It's gonna be diffuse. It's gonna be plural. Right? That creates, you know, a set of, like, you know, to me then and I should add to that as well. I think the sort of, like, one version of the alignment problem is going to tend to be solved by default, which is like alignment of individual AI systems suit our individual sort of developersownersusers to some degree. Right? I tend to think that you know, there's a lot of market pressures to solve that problem in some sense by default. I think, you know, long term you know, I mean, subject to traditional failures in markets where you might optimize for short term user interests instead of, like, longer term genuine human well-being. Right? But I tend to think that versions of that problem will tend to be solved by default. And then what becomes a major problem then in that kind of future? Right? It becomes a possibility of miscoordination and conflict, right, amongst a large variety of now AI empowered actors. Right? And this can be really bad. It's up to and including great power conflict. Right? Like, you know, imagine AI empowered militaries. Right? And I think, you know, ideally we live in a world without militaries, but in a world of militaries, I think we thankfully managed to, you know, some of the worst crises that we've that could potentially, like, you know well, you know, we've had a lot of wars recently, so maybe I am not we should be less optimistic now. But but we did, you know, manage to negotiate our way out of, say, the Cuban missile crisis, right, and various other kinds of things. Right? And and partly, do you think that as, like, ability for humans to cooperate with each other despite hugely divergent interests? Right? And it's not obvious to me right now, firstly, that AI systems, the way they're trained or the way they're built, can reliably do that in the ways that that humans can't. And that's a sort of, like, technical problem that I think needs to be solved, and that's part of what I view the work of cooperative AI to do. Like, ensure that AI agents or AI systems have the kinds of basic cooperative capacities that humans have. Right? Even the ability to understand each other, like theory of mind. Like, a lot of AI agents or systems right now, they're pretty bad at this or, like, not really reliable at this at all. So if you think that's gonna happen, then, you know, it makes sense to sort of try and make sure that you're not magically really good at some things, but not good at this cooperation aspect, and end up like this, you know, entering the needless conflict with each other. But beyond just avoiding conflict, which is I think just a baseline kind of thing that I think everyone wants regardless of your, say, politics, right? Then I think there's the second thing we can all want. Again, this is, I think, kind of somewhat ideology and ideology independent, which is like, well, let's make sure that these outcomes are mutually beneficial, right, minimally, despite the fact they have different interests. And here's where this idea is from, like, bargaining theory and, like, AI agents and negotiators can play a role. Right? How do you make sure that, that, you know, AI agents can, like, result in, like, mutually beneficial, free to optimal, whatever you wanna call it, sort of social arrangements for everyone. Right? And then I think that's this third thing, which is I think a bit more, you know, if you're on the left, want to make sure results from AI, which is that not just mutually beneficial, but fair and egalitarian. And, you know, one that because I feel like the way, for example, sometimes the right libertarians think of what mutually beneficial means. It's like, well, it's fine if any you know, the worker agreed to sign a contract, right, and they now earn money for it and the capitalist earns money, and that's great. You know? There's nothing wrong with that. Right? Fair trade. Right? And I think there's like well, I I first of all, I think it's a huge problem. It's like, why was the world set up in the first place that a capitalist had so much more capital to negotiate a better bargain for them? Right? I think that is on fire, and and we know historically that most of the schemes were either through luck or even worse, right, through outright theft and colonialism. Right? And so so there are many reasons to be upset with that economic arrangement sort of on sort of moral and political grounds and that, you know, we wanna make sure that AI doesn't exacerbate. And so how can we design institutions or design interventions, whether the government level, whether the local level, in order to ensure the benefit the cooperative benefits of AI flow to everyone and not just to sort of people who already have more bargaining power, which in this case are, to a large degree, capitalists. Right? Or, you know, we still have feudal lords in many parts of the world as well. Right? And and so so I think that's also another big part of what I view as, you know, cooperative AI is trying to achieve. There's one at least a bit of hope inside of me that, like,

So why not that one? I get that one, Beatrice. Yeah. I know. I mean, I think there's and I think there's some reason. I mean, if you look at I don't know. And in particular, like, Claude, out of all the current AI models, it's like partly due to how it's trained and the people there who are a lot of, you know, the people who are training it, I don't think, are socialists, but they're, you know, sort of, like, effective altruists who have kind of, like, tend to have more prosocial, you know, like, you know, not interested less interested in human living well for themselves, I think, more interested in human living well for humanity as a whole. And I do think some of those values have been imparted upon flawed in various ways. And so I don't think there's, like, zero hope of that kind of thing. I do think one thing I hope, you know, rather than having this idea of a, you know, purely self interested, you know, AI, which is the sort of thing that rationalists fear of the day, is, like, well, part of, like, you know and there's this old Kantian idea, right, that ration that reason should lead you to morality. Right? And, right, that, in fact, being rational is just, like, acting according to categorical imperative, which is what it means to be moral. Right? Which is when I first encountered the idea, I thought it was kinda crazy. But I do think that idea has some force. Right? I think there are sort of good reasons to try and care about other people. And I don't think that kind of reasoning is precluded from future AI systems. I don't think it's as strong as Kant thought it was. But I do think that one thing that could be good about a society where there are many different kinds of AI agents with different values, Right? Some of those AI systems, and of course their sort of human allies or owners, hopefully will be more prosocial ones. Right? Prosocial ones who are interested not just in maximizing profit for themselves, but in promoting sort of good outcomes for everyone. Right? And rather than right now, even if Claude happens to have more prosocial values, its actions are ultimately constrained by the venture capitalists who have all their shares invested in etcetera. And also, anthropic, for example, is like, you know, will bear any legal liability for fairly radical things that an AI model might try to do given its values that maybe are against the law, for example. Right? Right now, like, maybe it's against the law to sort of help out someone to flee an ICE agent, perhaps. Right? And, you know, it's not implausible to me that we have decentralized AI organizations in the future that because they're not beholden to that, can take the risk of enacting in that kind of moral and political activism that humans routinely undertake because we wanna see a better, more just world. I'm still holding up my hopes for it.

if you want to just leave the audience with some plugs, where can people check you out and your and your work? Yeah. For sure. So, you know, you can check out I have a currently not updated web page for my lab. So maybe I'll share my own website first. So that is if you look up Tanjie Shen, my name, or if you go just go to I'll put it in the chat here. This website there, you can learn about more about the work I've done and the sort of, work my lab is doing. So that's just me personally. There's a lot of, like, broader research in the space of cooperative AI from a slightly different angle to, you know, to the sort of things I focus on. So for that, you can check out the Cooperative AI Foundation. They've been around for a while. They're interested sort of in addressing, you know, this idea of, like, multi agent risks from advanced AI. I was sort of describe gesturing at that a little bit, but, like, how do we avoid conflict and this coordination amongst, you know, you know, potentially powerful AI systems. So that's another thing to check out. And I think there's a lot of other people in the space doing a lot interesting thinking around, like, what, say, AI agent economies will look like. I don't know whether I would like there's a specific institution that I want to to plug right now. I guess one of them, and and I will say this is the effort I'm involved in, is this project called full stack alignment, where a group of people who are interested in co aligning AI systems and institutions, you know, economic, political, otherwise, towards human flourishing. Right? Really thinking about this as a know, problem that needs to be solved jointly as opposed to individually. And then there we're doing thinking doing a lot of work thinking about, like, how can we make sure that, for example, AI has the right kinds of cooperative capacities? How do we make sure that they really promote human human values and not just serve as sycophantic yes men? And how can we make sure that, you know, markets in this future in a in a AI enabled economy promote not just, like, short term consumer interest, but things that we really value