Hammond Metagov 2023011
Speaker 1: So welcome, everyone. It's my sincere pleasure today to welcome Lewis Hammond to the medical governance seminar. Lewis is a I can say this the right way, not a PhD student, but a DPhil student, Institute of Science, the University of Oxford. He's also the is it still acting or just executive director of
Top Keywords
- cooperative 0.015
- cooperation 0.010
- might 0.007
- work 0.007
- agent 0.006
- human 0.006
- humans 0.005
- agents 0.005
- welfare 0.005
- preferences 0.005
- systems 0.005
- well 0.005
Transcript
So welcome, everyone. It's my sincere pleasure today to welcome Lewis Hammond to the medical governance seminar. Lewis is a I can say this the right way, not a PhD student, but a DPhil student, Institute of Science, the University of Oxford. He's also the is it still acting or just executive director of
Still acting, technically. Okay.
Still technically acting executive director of the Cooperative AI Foundation. And he will be speaking to us today and giving us a little bit about cooperative AI and what that means. So I'll let you take it away, Louis.
Cool. Thank you very much, Josh, and thanks again as well for the invitation. It's really nice to to be able to talk to you all. I kind of stumbled across MetaGov only kind of relatively recently. But, yeah, from what I've seen, it seems like this amazing community and I'm kind of very interested in kind of slowly moving myself more into thinking about the sorts of things that you all think about. So, yeah, very much looking forward to the discussion later as well. I'm gonna attempt to share some slides, which I hope will work. I probably won't be able to see you after I share the slides, but this is fine. Yeah. If you want to shout at me during the talk, you absolutely can. Otherwise, I guess we can probably shelve most of the discussion points to the end of the talk because it's only gonna be a short one. Cool. So hopefully you can all see those slides, I think. Cool. Alright. So okay, this is this is what I'm gonna be talking about, this idea of of cooperative AI, and then at the end, as Josh kind of hinted, I'm gonna briefly give this little plug about the Cooperative AI Foundation, which is the place where I am doing a lot of my work at the moment. So very brief outline of this very brief talk, a little bit of motivation, which is probably, I don't know, probably not needed for this audience, but but I'll do it anyway. Brief intro to cooperative AI and what it is. Just a quick little plug for this foundation and then some discussion stuff at the end. So as I said, kind of maybe this is not necessary or all that necessary, but I'm gonna go through it anyway. So broad kind of like motivating starting point is just that many of the most in problem many of the most important problems facing us, by which I mean kind of maybe, you know, civilization, humanity, kind of life on Earth, whatever kind of however you want to draw, that kind of ring around those things, are problems of cooperation, So this might be climate change, war, pandemics, etcetera. And yet also many of our kind of greatest successes in the past have been due to our ability to cooperate. So trade, treaties, various other forms of kind of collaboration and so on. And one of the points here is that as societies, economies, militaries, etcetera, have become more powerful and more connected, then the need for, but also the opportunity for cooperation becomes greater. So while we're all kind of hunter gatherers living in little kind of islands of kind of not really kind of that interconnected, then cooperation, we could kind of get away with a lot less, and now it's a lot harder, and the stakes are a lot higher. So second kind of point to follow-up point to this is that I'm gonna claim at least that AI will become increasingly more relevant to problems of cooperation. In particular, by increasing both the power and the interconnectedness of actors, those two features that I mentioned before, and hence the need for and hopefully the feasibility of cooperation. So on the one hand, AI systems might end up exacerbating some of our existing cooperation problems. So this picture on the right is a little thing from the flash crash, which I guess many of you will have heard about. It happened back in the kind of earlier in the 2000s when a set of automated trading agents kind of ended up kind of getting locked into this kind of weird death spiral kind of dynamics and wiping like trillions of dollars off the stock market, after which it like miraculously bounced back, but still kind of yeah. There's the jury's still out on some of the features of exactly what happened there, even though it happened ages ago. So that's one way that things might kind of get worse potentially in the context of AI systems. Just one way. And then the other, we on the other side of things, we might ask if we can also kind of reverse this question as well. So this is what this very kind of, like, hokey and generic kind of stock stock image photo of the bottom right here, and is trying to symbolize is basically how might advances in AI systems help us, to solve cooperation problems. So before continuing, I also just wanna take the take a moment to mention that there you can also think about the possibility of downsides to cooperation. So the idea here is that cooperative capabilities can be thought of as dual use in some sense. So for example, forming credible commitments can be used to kind of make offers and binding agreements and so on, but can also be used to make threats Reaching mutually beneficial bargaining solutions could help agents lead to collusion when we don't want them to, so there could be some negative externalities there. And forming alliances could be used to create larger factions and thus greater risks of conflict. And if if we can kind of team up and kind of raise the collective stakes for for winning, then this might kind of tip the balance in favor of conflict potentially. And so ideally what we want is advances that lead to something called differential progress on cooperation in the context of AI systems. So what do I mean by differential progress? I'm gonna point at this picture here, which is a cartoon, is very much just a sketch. But the idea is essentially that if we're building or attempting to build intelligent machines, and we want to build them with certain capabilities, then we would do well to try and differentially kind of advance work on more, say, cooperative capabilities than other kinds of capabilities. And so the point here is that here is more just that, you know, there are many ways that we can kind of build intelligent machines and we should kind of push more in the directions of of intelligent cooperation. Okay. So that was a kind of like motivational kind of spiel type thing. And now actually just talk about, like, what more concretely do I mean by this phrase cooperative AI? Because it's kind of I think it's it's open to multiple interpretations, but there is a specific thing I mean by it. So before the what, I'm gonna briefly talk about the where. And so in this broad project of this kind of making AI systems go well, which you might, one way you can try and carve that up is technical work and non technical work, although again this is just a cartoon, and under technical work you might have things like alignment and interpretability research and robustness and cooperative AI and a bunch of other things. And on nontechnical work, you might have things like policy, norms, institutions, all that sort of kind of governance y stuff, a whole bunch of other things too. And obviously, there's a lot of overlap between these two things. It's not as black and white as the picture here would indicate. But the main point is that cooperative AI is more or less on this side of things. It's looking at kind of technical advances in AI systems to solve cooperation problems. So, yes, this is just basically what I said. It's a subfield of AI that aims to improve the ability of AI systems to engender cooperation between humans, machines, and organizations by which I'm kind of just loosely gesturing at the idea of groups of humans and machines and the institutions that they build and so on. And so the aim here is to kind of help reduce AI risks resulting from corporation failures as well as maybe to see some of those opportunities that I mentioned before. And in doing so, it kind of forms this natural complement to technical work on the problem of alignment, which you might have heard of, as well as nontechnical work in AI governance. So another way we can frame this is the task of improving the cooperative intelligence of AI systems, where one kind of maybe working definition of this might be something like, the cooperation the cooperative intelligence of an agent is its ability to achieve high joint welfare in a variety of environments with a variety of agents. So if you're familiar with this, the leg cutter definition of universal intelligence, which has been quite influential in in AI, then then this might be familiar, but it's not it's not necessary. I also wanna note here that I'm focusing specifically on cooperative capabilities, one's ability to cooperate rather than one's disposition to cooperate. I might for instance be an incredibly kind of skilled and fantastic negotiator and able to reach this really beneficial joint outcome for everyone, but if I don't want to then I'm not going to. And so I think this is an important dis an important distinction which is kind of, yeah, often often missed. And so this latter thing, the disposition thing is also really important, but it's slightly more a question of alignment, and at least when we are considering kind of the interests of, AI systems which are deployed on behalf of humans or groups of humans. Okay. So just to kind of make that point kind of pictorially, one way of framing this is that this kind of alignment problem is classic principal agent problem. I'm a human h, and I wanna make sure that my robot r is gonna do what I want it to do. And similarly for h prime and r prime can be seen as a kind of vertical problem where the kind of y axis here, the verticality represents something like normative kind of priority or precedence. So we care about the human's interests more than the robot's interests here. And then cooperation can be seen as something of a horizontal problem. And of course the world is actually much more complicated than this, and obviously still much more complicated than this other toy cartoon on the right side. But that's maybe one other kind of thing that can help get this intuition across of what I'm talking about. So kind of relatedly to this and I'm kind of this is originally part of this talk was originally framed towards a slightly more kind of AI safety type audience, people who are working on kind of doing that sort of thing. So some of the stuff I'm gonna say is slightly couched in in in terms of that, but it might just be interesting context anyway. So lots of work in AI, implicitly or
otherwise focuses on the single agent setting. And multi
agent work is often on the single agent setting and multi agent work is often adversarial. So we've seen lots of progress in recent years in adversarial game playing. So here's AlphaGo. And similarly on the kind of more safety and alignment setting, a lot of work is is focused on kind of this single agent setting, though there is often as well the a human kind of involved. So here again, this is you there's a good chance you might have seen this. This is a GIF from some experiments that OpenAI ran back quite a few years ago now, where they teach this little kind of sausage leg thing to do a backflip just from human preferences, from human feedback. So humans kind of watch videos side by side of it trying to do the back flip, and they say, oh, this one's more like a back flip. This one's less like a back flip. So this is all about getting an AI system to act according to a human's preferences. But again, it's kind of just this kind of one human, one agent setting. Cool. So I think finally, just to conclude this little section, I'll talk a little bit about some kind of open problems in in kind of cooperative AI. And in the kind of longer version of this talk, I have a bunch of slides about examples and so on which I won't go into here, but if there's time and interest, I can always try and be a bit more concrete in the discussion. So first thing here is kind of why are cooperation problems in the context of AI different? Why is this, like, even a thing? Why is cooperative AI a thing? Why is it not just cooperation? So one point here is that cooperation can be harder because AI systems don't possess some of the features that allow biological creatures to cooperate, such as in the case of humans, things like theory of mind. But cooperation can also be easier because AI agents have or can be constructed to have non biological features that might allow for better cooperation. So for instance maybe the possibility of very kind of you know particular information structures or abilities to communicate and so on which might be kind of hard for humans to replicate. And these features introduce a range of open problems and in fact there's a paper just called Open Problems in Cooperative AI which I encourage you to read if this sounds interesting to you. So the first is around the question of understanding and in particular understanding the world, the behavior, and the preferences of other agents and dealing with recursive beliefs by which I mean things like I believe that you believe that I believe that you believe and so on which comes up in in game theory all of the time. There are things such as forming commitments via devices such as delegation, contracts, hardware, etcetera and maybe some of that is slightly more relevant to this audience in places. There are also things about communicating effectively over common ground, dealing with problems of bandwidth and latency, teaching, overcoming games with mixed motives, and so on. And then finally, and again, slightly more relevant to this audience maybe, is the creation of institutions such as norms, reputation systems, and so on. Cool. So that's kind of most of the content of the talk out of the way, and I just wanna take the last few moments to just give a quick plug to this foundation that I mentioned before, so Cooperative AI Foundation. So that's me in the bottom of the middle and surrounded by the five trustees. So this is Alan Dafoe, who's at the Center for the Governance of AI and DeepMinds, Eric Horvitz from Microsoft, Julian Hadfield from Toronto, who's also based at OpenAI and doing bunch of other things. Bottom left is Ru Donnelly, who's president of Polaris Ventures, who are our backer, and on the bottom right is Dario Amadei from Anthropic, previously OpenAI. So we're a new charitable entity, and our mission is is here. It's to support research that will improve the cooperative intelligence of advanced AI for the benefits of all. So backed by an initial philanthropic commitment from Polaris of $15,000,000. And if you want any more information on kind of what it is that we're about, then there are a couple of papers. There's a nature commentary, and there's this open problems paper that I mentioned that came out one or two years ago now, which are a good place to start. And then finally, if you're even more interested in this, you can check out a number of online seminars that we have that will be kind of returning that seminar series later this year. We have a mailing list that everything goes out via. There's a website which is still growing, but it's up. And in future, they'll we'll be looking for kind of a range of kind of working groups and looking for interdisciplinary collaborators and grant making opportunities and so on. So for example, we have one thing coming up soon with some people from the Collective Intelligence Project, who are great and who you might have already come across in some form, working on running a workshop to do with AI for institutional design and how it might we kind of leverage AI for institutional design. So, yeah, if you wanna chat about any of that then I encourage you to to contact me, and I'll drop my, contact details in the Zoom chat afterwards. And that's it. Thanks very much.
Awesome. Thanks, Louis. So I'll get us kicked off. But for moderation purposes, if you have a question, please do either raise your hand here or just post it in the chat, and I'll kinda, like, get people in order. Otherwise, if you have a discussion, a comment a response immediately relevant to, like, an existing discussion, just, like, communicate that directly. Like, I I would just, like, maybe, like, raise two fingers. That's an often, like, good practice. Okay. This is so I definitely I think there would be a lots of questions about, like, the institutional design part because, obviously, that's, like, something that's quite relevant to this group of people. I love that alignment versus cooperation diagram because it it kinda yeah. It's very clear that this is how alignment works. This is how people typically think about alignment and AI safety as, like, humans dictating things. And then but, obviously, the world is more complicated, and cooperation can mean lots of different kind of, like, multipolar kinds of, like, arrows going many different ways. That's really very interesting to even think about, like, how to how to think about that in a technical way. So in those diagrams, it seemed like cooperative is posed as a technical discipline. So do you have, like, a technical definition of what constitutes a cooperative capability?
This is something we're still kind of really trying to nail down, because I think people have different intuitions about what this should be. And I guess I'm the first thing to say is maybe that I'm kind of using cooperative intelligence and cooperative capabilities somewhat kind of synonymously. But there's already kind of things that you kind of differing intuitions that people might have here. So one example that I gave is this kind of ability to achieve high welfare, in a variety of situations with a variety of different players. And you can kind of get somewhere in kind of formalizing this, when we might think about kind of, okay, well, what environments? And how do we kind of put a prior over environments and so on, a distribution of environments? What agents are you gonna be playing against? And there are some kind of interesting and principled ways to try and do this, such as using things like a kind of Solomonoff prior over environments or this thing called the Darwin Wallace distribution over different players that you might face face. So there are things you can say there. There's also this idea of, well, maybe that's not really what it means to be co opted in intelligent, maybe what I really want to be doing is achieving my own goals, but conditional on achieving my own goals, then I want to make sure that I'm doing it in the most cooperative way possible or something. So there are other things you can say there. Yeah. This is a bit of a fuzzy fuzzy notion still, and I think some of the work that we're kind of most interested in supporting and pushing forward at the moment is very much on kind of doing this kind of conceptual kind of clarifying and so on. Yeah. There are a few other things I can say there, but I think I'm gonna pause for a moment at least.
I I have a follow-up question, but let me pass it first to Seth and then Sarah.
Thanks so much. I might come in through.
Yep.
Okay. Great.
Great. Yeah. Basically, I just wanna know, you know, what an outcome with an AI looks like depends a lot on the human, and even, like, low level perceptual stuff. You you you put the AI in a, like, smiley robot or you put it in, like, a a a blank computer turn hole and, like, really cute kinda nineties human robot interaction work shows dramatic differences. And and and and and they stick. You know? But, basically, what you know, it's ambiguous. Just like interacting with all kinds of, like, ambiguous agents, it's ambiguous. It's not clear whether I should use game theory or decision theory to model a human being's interaction with an AI. And if tiny little switches are so dramatically changing how we would, like, capture the human side of those interactions, how can we get good models with that of AI, without good models of people, and, you know, or empirical work? Is there an empirical side to to this work? And or is it all modeling?
No. There definitely is an empirical side to this work. I think you're you're totally right to pick up on this though, because most of the work I spoke about just there or the kind of things that I was describing were very much more at this kind of high level abstract conceptual game theoretic kind of modeling type level. And as you rightly point out, there's been a ton of also kind of interesting work much more on the empirical side of things, human computer interaction, kind of psychology and sociology and all of these kind of other kind of social sciences, stuff coming into. So there is work in that area and that is work that we are excited by as well. One instance of this for that we can talk about kind of quite concretely already is work on the context of large language models. So there, you're kind of already seeing lots of interactions between single humans and kind of and single kind of models. But there's plenty of, I think, work to be done in the kind of broader context of thinking about kind of, you know, multi in a multi agent setting basically when we might have multiple humans, multiple models, and so on, all kind of mixed up. And a lot of this is naturally much more kind of empirical work. And in fact, this might even be the direction many of the kind of core insights are coming from now that we have these kind of bigger models and kind of more complicated systems that are harder to analyze using this kind of standard kind of, kind of almost in some sense, it's kind of naive kind of game theoretic approach. Yeah.
Sarah?
Oh, can't hear you. Sorry.
Oh, you're Adi.
Thank you so much for, sharing your work and pushing it forward every day. I'm curious about when you talked about kind of I think in your definition was a a definition of welfare and thinking about how you might have an accurate identification of what the welfare is. Right? Because I think we don't have, as humans, a shared agreement about what welfare is always. Like, who's well like, what's best for us all is and is a thing we struggle with all the time. And so kind of how how do you think about how that gets defined or how one the the accurate knowledge of that is defined, right, and in the feedback loops, etcetera. And then kind of relatedly, there's also this question of at what scale, what group, what group's welfare are we most interested in, and at what let like, where does the cooperative container exist? And I'm curious how you what you all are talking about in regards to that.
Yeah. Definitely. So there's a few things that I can say there. I think the first is, like, in terms of, like, yeah, how big are you drawing the circle, kind of whose welfare are you really interested in? I'm interested in, like, everyone's welfare, like, you know, and I don't know, like, it depends how you're going to draw the line for like moral patienthood or something. But I think plausibly that kind of circle is a very big circle and might include things like future people and kind of, you know, non human animals and all of this stuff as well. So I'm very concerned with kind of welfare in general, and that's kind of implicitly what this benefit of all thing is getting at in the mission statement. But that's also my kind of more of a personal take. And to kind of drill down a little bit more on the things that you were saying before, I think there's there's at least two things to say. The first is that, yeah, there's no kind of one definition of welfare that that makes sense, and this is kind of a well understood point in the literature and kind of computational social choice and so on. So that's that. Though there are still things you can kind of say. So for instance, if we can get if we're not at the Pareto frontier, and we can get closer to the Pareto frontier, then this seems like a like a kind of at least a bar. This is this is at least something that we can kind of aim for, even if it doesn't tell us where the best point is up on the Pareto frontier or anything else like that. It seems like there's still a lot to be gained, that we can do kind of collectively. But of course this kind of implicitly kind of assumes that we have some way of kind of capturing and learning about preferences of various humans and so on, and I think this also goes back to Seth's point that this is actually just like really difficult and messy and can change depending on, you know, very many kind of factors. And so one kind of small part of the of the kind of cooperative AI landscape and also linking to this work on kind of alignment is on kind of how do we how can we build machines that can better understand and work towards human preferences. And so there's loads of work on kind of preference elicitation, reward learning, all about all of, you know, all a lot of stuff in that vein as well, which I think will be quite important in in helping us solve some of these problems. It's a great question. Thanks.
I wanna do a quick follow-up to Seth's question, which is that as we both of the previous questions sort of like pointed out, like, okay, defining this well for functions is an extremely difficult it's like it's a very messy sort of setting. Once you add in all of these additional actors and you can accommodate, like, what are their preferences? So it does feel like there's some sort of, like, combination of, like, game theoretic ideas, decision theory, being sort of inserted into, like, what is traditionally an AI setting. Right? And I have a model of, like, AI empirics. And I have a model of, like, the kind of is it fair to call empirics? Like okay. Empirics that you see in economics. Right? You know, whether it be simulations or whether it be, like, sort of, like, know, observational studies. Because I mean, there are not really any laboratory study studies. And, you know, does cooperative AI because it's trying to, like, in some sense, synthesize these fields, does it have a good idea of how to tackle those empirics? Because it feels like it would be you wanna sort of take an AI approach because I I I perceive that as more successful. But, you know, you're obviously hitting all the barriers that economists typically start to hit when you enter into these settings.
Yeah. I totally agree. And I think there's, yeah, there's some stuff you can talk about kind of in theory and even maybe in the lab in kind of toy settings and so on. But then there's this other thing about like, okay, no, what we actually care about at the end of the day is kind of how this kind of can help people and things that we care about kind of improve their joint welfare and so on and reach more cooperative solutions. And and kind of thinking about that in this kind of broader and socioeconomic context is is just much harder and bridging that gap is is difficult. I think you can you also just don't see that very much work in that intersection more generally. I think AI researchers are kind of not very good at, like, often, like, running kind of these bigger kind of more real world social experiments and kind of with with kind of humans in there as well and a bunch of stuff. And maybe kind of people who are better at doing that. I don't necessarily kind of aren't necessarily like hooked up to all the kind of technical people as well and thinking about the kind of fundamental kind of theoretical and kind of algorithmic things that kind of lie behind those technical advances. So I think the short answer is that I don't have a a good a good kind of, yeah, proposal for how to try and bridge this gap, but I do think it's a really big gap to be bridged.
I'll I'll just say that for the economic side, Philip Zahn, who's kind of, like, in this space and collaborator, and I are working on a kind of research proposal slash grant related really coming from the economic side, running like a simulation engine modeled off of, like, Dellinger, which is like a tool for producing experiments really quickly within psychology. But actually, it would be really interesting to see, like, to what degree, like, the classes of tools could be useful for the needs of cooperative AI.
Yeah. Definitely. There's been a little bit of work on things like kind of bridging agent based modeling type stuff and AI in a bit more kind of depth. So stuff like there is a few things like that kind of in that gap, but kind of relatively few. So that does sound interesting. I'm familiar vaguely with Philip Zahn's name, but, yeah, I'll ask you about that at some other time. Thanks.
Of course. Nick, you're next.
Hey. Thanks so much, for the talk. This is this is super, super cool. My question is a little bit loosely formed, but I'll try to formulate it better in words than in my out loud words than I did in the the text chat. So when we talk about human cooperation, it seems like it's it's pretty easy to kind of know it when we see it, define an active if I have some data set describing people playing a cooperation game like I can see like oh that's a cooperation action that's not and it's normally because you know I gave money to fund something that's not a private good or I gave effort in a way that's not a you know not resulting in like increases only to my private welfare. And normally, it's like it's money or or acts is is, I guess, how we would define that. And on the AI side, it seems like if you're imagining some environment of human agents and AI agents, like, the main two things I could think of that would be a human to AI collaboration would be giving the an AI agent more information or data or, you know, helping make sure that their computer on which their server in which they're running doesn't, you know, get the cord plug the get the plug pulled. Are there other things other than that that would be, like, these human AI links that that you think about in your work?
Yeah. I'm just gonna ask. My Internet connection was really annoyingly dropped out just during the second thing that you said. So you said that I heard the first thing, about kind of sharing information and so on, and then I missed the second example. So I just wanted to ask about that before I answer. Oh, so
I guess I'm thinking specifically of, you know, improving an AI agent. I I'm cooperating with an AI agent because I decided to share my unique knowledge about the world with them that that, you know, lets that agent be more accurate or have, you know, better calibrated predictions. Or I I agree to stand guard outside the server room and make sure that no one unplugs the the server. Right? Like that like that seems like a I'm using that as, like, a broad category of, like, things that, like, infrastructurally make the agent continue to exist. Are there is there like other categories where we might think about what it means to have like a human AI cooperation act?
Yeah. Sure. So I think both of the examples you give there are kind of they're implicitly about enabling the AI system to better achieve its goals, either by providing it some more information that you able you that you have, or by kind of, you know, preventing it from being disabled or something if you're standing out and stopping stopping it being switched off. So implicitly there, there's this notion that this AI system actually has some kind of objective or some goals. And then if we can use that, then we can presumably also talk about the extent to which it is able to achieve its goals. And then in the same way, we can apply that to humans and we can think there you have your kind of concept of welfare when we kind of might be able to kind of aggregate those. So I think that is definitely those those are two kind of nice kind of concrete instances of that. There I think there's like an interesting question about, like, do we actually care about the do do we just care about do we actually care about the goals of AI systems kind of non instrumentally, or do we just care about the goals of AI systems because they represent the goals or hopefully they represent the goals of the the people or the kind of organizations and so on who are deploying them? My personal take is that maybe we're kind of, like, not quite at the level where we have kind of digital sentience or something or kind of, yeah, concept AI systems that we should care about for objectives in and of themselves. But I do think it raises interesting questions there too. Sorry. That was a tiny bit of a tangent. Feel free to push back or follow-up on that if you want to quickly.
No. That's helpful. And I'm I'm just realizing from this answer that I think that actually this is kind of, like, maybe very related to Seth's question. So, you know, if we if in answering and discussing that, we might get at what I was really trying to ask.
Sure. Yeah. Happy to circle back to that and and try and give a better answer as well if if that if that opportunity arises.
Next is Janice.
Yes. I wanted to echo a little bit Sarah's question about how do you define the, let's say, the local, like, multiple agents, they locally maximize, and then you can aggregate on the whole kind of like a a a phantom phantom might be the utilitarian calculus. And my question when you involve AI is how would you establish like, how would you train your neural networks? Because I I I surmise that the the the local utility functions would be kind of like dynamic. So even if you train them, let's say, you have some data, you say, okay. I kept a static approximation at the current time, and you train your models and they make you know, they they they found you. They do the maximization and all that. That, you may need to constantly keep retraining, and it seems to me that it's gonna be like a constant, like, the dog chasing, you know, it's tail, you know, that they're in wiping the dog. You know? So is that something tractable? I mean,
at at this
at this, like, research. Yeah.
Sure. So two points here, I suppose. Well, the first is that I agree. I think I think you're right. And so, yeah, there's at least two things to say here. The first is that multi agent learning and the theory behind behind multi agent learning and so on. It is still, especially in the context of kind of large deep neural networks and kind of modern AI architectures, The theory definitely does lag behind the practice, and we don't have kind of necessarily always a good understanding of the dynamics that emerge there. There's all sorts of kind of interesting kind of pathological cases where you can get weird kind of chaotic dynamics or kind of orbits or kind of things happening which don't seem desirable. So that's the first thing to say. The second thing to say is that I think, and maybe this is again kind of getting back to some of the earlier points is that having and studying situations where there is some kind of human in the loop, so to speak, or multiple humans in the loop, adapting to them, kind of really somehow kind of learning to take account of their preferences and so on and and kind of working in this kind of more kind of collaborative setting is, I think, a a a large part of of kind of work on cooperative AI is at least at the moment. And it it needs to be there. You know, if nothing else, because, you know, you train some AI system and you just have it as a static black box versus you can train some AI system to be constantly kind of updating based on what you're saying and learning and kind of making sure that it's acting according to your preferences, you know, that that second system is just gonna be way more useful and and better for to help you kind of, you know, achieve your goals as well. So there are kind of strong incentives there for people to try and solve those problems just more generally in AI research, but it does come up within this banner of under this banner of cooperative AI too.
It is are there any studies on the computational complexity of such models where you want to factor in the adversarial nature of the of the of the agents.
Yeah. Definitely. A bunch of bunch of computational complexity results in that sort of area. Yeah.
I Do you think it's financial?
So it it can be. Yeah. So so for instance, like, one one example of this is there's this there's this framework called cooperative inverse reinforcement learning where you have one human, and they have some kind of hidden preferences that the agent doesn't know about, and then you have an agent and the agent and the human, the ASs and the human, they're both interacting in the world taking actions, and the one way you can view this is that the human's hidden preferences are some kind of like a hidden state in the environment. So there's a kind of level of, like, partial observability here where the robot gets to see what's happening in the world, it gets to see the human's actions, but these kind of preferences are kind of, or at least we model them as kind of latent variables that are not observable to the agent. And so in this setting you essentially end up with a kind of what's called sometimes a POMDP, so like a partially observable Markov decision process and the complexity of solving that is and I'm gonna be embarrassed if I get this wrong, but I'm pretty sure it's extra time. And so, yeah, so there's like the the this possibility of hidden information and different preferences and so on definitely adds to the kind of computational challenges that are present here.
Seth? Seth?
No. I'm I'm I'm inclined to pass, so I just sort of re I realized I just kinda reasked my question. Except Nick saw something in it, so maybe it's worth revisiting. So artificial intelligence is a very loaded term. Right? You know, there are computer programs, and they create an active environment for, you know, conscious humans. And calling artificial intelligence sort of accepts the the conclusion in a way that we should use an agency model to understand their behavior, to understand their own world, understand the thing that they do. And so are you, I don't know, vulnerable to any blind spots by accepting the premise as the conclusion and and using, for example, game theory to to as the basic tool of modeling AI's role in cooperative interactions.
This is such a yeah. This is a great really great question. I really like this. I think I mean, the short answer is I think, yes. And my kind of usual kind of thinking about this is to like, at least personally is to to kind of take this kind of almost like, like, Dennett like position, where it's we kind of model these things as agents because that has kind of predictive power and that enables us to to say things. But it's not it's very much not always the right level of abstraction. And I think as you say, this the the the whole word the phrase of artificial intelligence is this kind of, very kind of nebulous thing where the goalposts are always changing, you know. Once upon a time, the chess engine that you can, like, use on your phone would have been, like, the, like, pinnacle of artificial intelligence, and now it's just like a boring tool and where it's like a calculator and we don't care about it. And then, you know, and maybe this dynamic has continued. In terms of like more more pointedly, I think, and I think you're I think maybe the kind of yeah. The the follow-up point to this that you're making is like is is something more like, well, if you're going to adopt this stance, then what what might you be missing out on, and and what might you what problems kind of might you be not considering or overlooking and so on by doing this. I think that's something I have much less, I don't have a particularly good answer for at the moment. I think this also I think it raises I mean, so there there are some things there are some things you can say. So for example, a paper that I'm kind of helping write at the moment with someone called Wolfram Barfus, who was previously based at Princeton, now is at Tubingen, is about the idea of collective cooperative intelligence. So how does kind of how do cooperative kind of capabilities and so on emerge from small numbers of kind of more basic fundamental simple kind of agents. So very much taking a kind of a complex systems type collective intelligence perspective on problems within a cooperative AI setting. And so there's in some sense there, there's a kind of lens that you can take and those agents themselves aren't kind of, you know, traditional agents in the kind of homo economicus kind of traditional kind of paradigm of of what we think about as agents often, at least in in canonical AI settings. But so there's, you know, there's something you can gain there, but that was only a small kind of step away from this the kind of existing paradigm that I've been describing, which is more of a kind of game theoretic and decision theoretic one. And I think there could be a lot more to be gained from bringing in other perspectives, from other people who have different ways of thinking about and talking about AI systems. And yeah. But I I don't I I wanna talk to those people and I wanna learn from them. But, yeah, you don't find them in the computer science department very often, annoyingly.
That's fun.
One of
the most fun conversations I've ever had about AI was I was randomly placed at I think it was I don't know if it was High Table or just formal table. But I was placed next to a religion scholar, and we had a very long conversation about humanist implications of AI automation, and it was deeply informative, actually. So
Sounds great.
Yeah. Look. Go to go to the this actually even have a divinity school, it must. I mean, there's a room called divinity schools. Yeah. Accent.
Yeah. The Domenity scholars are always really interesting to talk to, really about any topic. I I I think you brought this up a little bit when you were talking about cooperative and first reinforcement learning. I'm curious to kind of learn a little more about the different job workers in in this context. My kind of generalized understanding is that a lot of the work they do is very itemized and individuated. So we're not this type of, like, multiplayer environment that you've been describing. And so I have, I guess, I have two questions that I think pair in some ways. I've been reading through a little bit of this paper called constitutional AI, by, where they're sort of talking about the, in some ways, the scalability issues of supervision and the kind of problems that come with large preference label sets. And one of the solutions is to kind of hypothesis hypothesis is to set up a constitutional AI that allows the preference modeling to be much more scalable. And they also kind of allude to the fact that in Miami Dade anyways, that the kind of work that crowd workers are doing to some extent can eventually be done by these systems themselves rather than by individuals. So if we arrive in a position where the crowd workers themselves and their positions have become obsolete, I wonder if we'll start to find ourselves in situations where different models advertise or point towards the way in which the preferences were set, and if there might be kinds of ways of embedding cooperative AI principles to influence the behavior of crowd workers that continue to work, but because they're working, like, under the influence of some sort of cooperative AI principle, or if it's more applicable to kind of embed these things in the kind of constitutional model, and then have the kind of scaling effects, of cooperativism happen happen there. I I hope that makes sense. It kind of makes sense. Yeah.
It's a no. It's a great question. Somewhat embarrassingly, I actually still have not read this paper, got around to reading this paper. It's one of those many's that I have just saved and and not read yet. But I am loosely I'm I'm vaguely familiar with the kind of overall paradigm and the problem of what's sometimes referred to as scalable oversight. So this idea of yeah. What if potentially you might be able to leverage advances in AI to help kind of give oversight to or kind of train and make sure that kind of subsequent AI systems are doing what you want them to do, which is kind of broadly the thing adopted here. In response to this specific question, I think this is nice because I think it's some it somewhat returns to this idea of cooperative kind of dispositions versus kind of cooperative capabilities. And if we're kind of thinking about kind of imbuing the kind of crowd workers with certain kind of presume in your in your kind of case, it sounded like you're it was more about the kind of, you know, if they're abiding by some principles, then maybe that's a sort of kind of dispositional type kind of attitude towards in favour of kind of cooperation and so on. So that's, I think, that's one thing. And then when we're thinking more about kind of, oh, how does this actual framework work and kind of technical, can we thread some of the principles of cooperative AI more kind of directly into that, then that might be something more kind of capabilities y kind of thinking about how these systems kind of I mean, there's you're still kind of making potentially, like, value laden kind of design decisions and so on. I don't wanna kind of try and wiggle out of that. But but maybe it's it's slightly more kind of leaning towards that. And so in that sense, I think that's the probably the area that most naturally falls under the kind of cooperative AI banner. But, you know, only the cooperative AI banner insofar as, like, I have, like, just defined it and described it. And I do think that, you know, that's just one way of carving up a space and and and these things sit very close to each other within that space.
Yeah. That's really interesting. I guess maybe the services for me that one of my assumptions is that by providing cooperative capacities, you're opening it up to more potential for cooperative disposition. You know, I'm wondering how that applies to the kind of work of crowd workers who are actually setting preferences instead of levels of doing that work. Thanks.
Yeah. No. I think that sounds really interesting. I don't have anything intelligent or useful to say about it or just crowd working stuff in general, which is the whole kind of interesting thing and phenomenon unto itself, which I have little kind of experience with. But, yeah, it's great points. Thank you.
Eugene?
Yeah. And following up to, the various points that have kind of been coming up on, the capabilities versus disposition side. Because, obviously, the or not, obviously, but I think there's obviously a lot of tremendous amount of challenges just on the technical side and with the AI and thinking through those two balances. But even if we, quote, unquote, solve the that side of the equation with capabilities and and disposition and the right balances between the two and capturing the right complexity, it still feels as though there's a huge gap between humans' capability to coordinate, which seems tremendous, and our disposition to actually do so. And so I was wondering if you've seen any kind of experimentation with actually using playing with AI as being a mechanism of creating in humans more of a disposition towards being open towards coordinating and collaborating. Because even if you give the perfect collaboration tool to someone who is not in the mindset of collaborating, would would it even function then, or is then it it kinda going to be fundamentally misused? So that that question was not fully baked out, but hopefully that make made sense.
It did make sense. I think it's a very interesting question. I actually I don't think I I would be surprised if there wasn't some work out there on this. I actually don't know of any off the top of my head. I I don't know. It's I'm always when when people talk about, like, oh, using AI systems to kind of influence human values or kind of tendencies, like, there's obviously at some level we think like some amount of kind of like moral or ethical education is okay and so on. And also we might think that, you know, being able to use, AI tools and systems and so on in in the service of education is oh, sorry. That's, my side. And it is also okay. But, some kind of combination of those is, like, potentially dangerous and and kind of the kind of yeah. There might be kind of bad effects there that we that we might get. Yeah. This is sorry. This is not this is not very saying much in your in your in response to your actual question other than, like, I don't know. I think it's interesting. I think it also isn't, like, potentially a tiny bit of a Pandora's box, and we need to be very careful when we think about how AI systems might shape or manipulate humans' preferences and values, which is not a particularly controversial point, but anyway.
Absolutely. And I just wonder, just exposing people to if you bait have a well baked out model and just expose to what variable changes that are saying more pro or less pro coordination, how that actually affects the system model, and seeing if that makes people think of, like, oh, maybe I should act a little differently on a day to day basis. Are there any just loops on that kind of not even the pure technical experimentation, but more you know, if a human is presenting models to other humans, can we use the the structure of such a model to inspire certain thinking on personal growth and change? Thank you for for that.
Yeah. And again, unfortunately, I don't necessarily have anything that comes to mind, but I do think that would be very interesting and certainly you know we could you can yeah as an educational tool you can kind of look look how great kind of cooperation can be and so on, and you know all of these sorts of things, then then, you know, maybe maybe they can serve as kind of inspiration in in that sense, which I do think would be potentially beneficial.
I was gonna I just wanna say that on that thread that Eugene introduced, I was thinking, oh, this is like Facebook. You know? Like, actively manipulating people in order to run some experiments. And then I thought, oh, you know? So, yeah, imagine if they ran an AI that actively and adaptively intervene on people and would promote, let's say, cooperation. And I, like, wrote this out thinking, oh, this would be terrible. And, actually, isn't that terrible? Like, would it be that bad? Wouldn't it be that kinda nice? I don't know.
I I have, like, some intuitions pointing in the direction that this would be nice in the same way that we kind of, you know, rely on our institutions and norms and so on to kind of shape people towards hopefully doing nicer and more cooperative things. And I think that if we are able to kind of effectively and safely use AI systems and so on to do that, this also seems like probably a win to me. But yeah, I've just seen the phrase Pandora's box and I yeah, I agree.
We need to stop you from the account for this. Yes. But this is our next project. We're gonna we're gonna have an a TikTok voice.
Someone secure this handle on Instagram right now so that we can we can run with this.
Let's do it. Nick, you've been posting a lot. Is there anything I think we have time for one last comment, and then we do a short round of, like, quick announcements. Do you have anything you wanna share?
Because I
Nothing to highlight. I was
just I was
just jotting some thoughts from my earlier question. I mentioned there was some connections, and I wanted to flesh out a little bit more, but we can we can go straight to the updates. I think that makes sense.
In that case, why don't we take a second? As is tradition for everybody in the next three seconds, please unmute yourselves, and and then we're gonna give Lewis a round of applause, and also hope, you know, wish him well as he tries to survive this blackout in his room.
Has has this I couldn't tell if this was on just on my screen or on everyone's screen. Hang on.
It's on everyone's screen.
Alright. Okay.
I'm gonna
try I'm glad you're okay.
I'm still
here. But maybe in three seconds. 321.
Thank you. Thank
you very much.
And I turned the lights on as well. It was magical.
Hey. Thank you.
Yeah. I also just wanna say thanks thanks so much for the great discussion as well. I've still been in so many seminars where you give the talk and then there's just kind of dead silence for like the remaining ten minutes and you're like, come on, no one's just given a fifty minute talk and no one's even got ten minutes of questions. And this is like the absolute opposite. I gave a tiny talk and everyone's got loads of questions and loads of amazing things to say. So this is really incredibly refreshing and, yeah, very much appreciated. So thanks. Thanks again for having me.
Thank you for joining us. Oh, and if everybody has announcements, now would be the time to share them. I'll have a I have a quick announcement that, thanks, Nathan, for pointing this out. I actually just saw the email as well. I'll be doing this innovative in residence thing for the Unfinished Network, which is a group of nonprofits doing kind of web free governance or not web free, but just Internet governance. And, actually, as part of this, I I'm supposed to be hosting office hours, which you guys are in Wellington, but we're gonna be running I'm repurposing that time to do some work on DAOs for nonprofits. So if you're interested in nonprofits and you're interested in running them in, you know, new decentralized ways, who knows what that means? There's a variety of experiments that I've kinda, like, dreamed up, and it'd be kind of fun to have people participate if you're interested. If you're interested, just ping me, and I'll I'll send you the invite to the calendar thing, which is gonna be February 19, I believe. We have the first meeting.
Me and Josh, we have an awesome awesome student, Lucia, have a a paper. It's a big survey of, people in crypto on their political, economic, and governance beliefs. Chasing down the crypto lefty.
We have Jesse Kate Schinkler, posted recently in our slide about a reading group that that's gonna be happening with the the Harvard Berkeley Institute Institutional Theory Reading Group. I think it's gonna be with her and Primoera. Biweekly on Thursdays at 7PM CET starting January 19. So that should be a really good meeting. Given just in here a talk about two or three months ago on institution theory. So if you go back with the archive, that's also a good primer. Next week, we have a series of what we're calling short talks. We have Val Affante and Blaine Hansen, who are gonna be talking about Resate's future preference toolkit project and, the concept of persistent democracy. So that'd be really great. If you are part of the community and want to participate in in one of these talks, you can go to that sheet and it's, permissionally sign up. Feel free to reach out to me if you have any questions. And well, we're gonna have our first membership meeting, for Medigov members next week. If you're not a Medigot member and you are interested in learning more about that, I will share a link to the policy so that you can read it, and it also links out to the Open Collective where you can sign up to become a member, and they will do. We're gonna be, establishing the purpose of the program, and going through a couple of other formalities. It'll be fun, hopefully.
So have a
look at that and become a member and and, you know, get involved in governance at the community level with MediGov. I think that's everything for me.
Awesome. Alright. In that case, I'll just say, in the interest of time, I'll close it out. One final thanks to Luis for joining us. It's been an incredible discussion, and look forward to seeing at y'all at the next meta governance seminar. Bye, everyone.
Sounds great. Thanks very much, everyone. Bye.