Joe Edelman: Co-Founder of Meaning Alignment Institute
What happens when artificial intelligence starts weighing in on our moral decisions? Matt Prewitt is joined by Meaning Alignment Institute co-founder Joe Edelman to explore this thought-provoking territory in examining how AI is already shaping our daily experiences and values through social media algorithms. They explore the tools developed to help individuals negotiate their values and the implications of AI in moral reasoning – venturing into compelling questions about human-AI symbiosis, the nature of meaningful experiences, and whether machines can truly understand what matters to us. For anyone intrigued by the future of human consciousness and decision-making in an AI-integrated world, this discussion opens up fascinating possibilities – and potential pitfalls – we may not have considered.
Top Keywords
- moral 0.014
- values 0.013
- mean 0.008
- moral reasoning 0.007
- reasoning 0.006
- different 0.006
- good 0.005
- value 0.005
- work 0.004
- kinds 0.004
- well 0.004
- whatever 0.004
Transcript
This is a RadicalxChange production. Hello, and welcome to Radical Exchanges. In this episode, Matt Pruitt engages in a thought provoking conversation with Joe Edelman, cofounder of the Meaning Alignment Institute. They explore the crucial intersection of artificial intelligence and human values. Drawing on his experiences at Couchsurfing and the Center for Humane Technology, Edelman discusses the importance of aligning technological advancements with our fundamental human needs and aspirations. The Meaning Alignment Institute develops AI systems that not only possess intelligence, but also reflect ethical considerations and help users refine their values. This episode offers a window into an underexplored aspect of the future of AI and its potential impact on society. So now, here is Matt Pruitt and Joe Edelman.
Joe Edelman, great to be with you. Thank you for joining today. Looking forward to to speaking. How are you doing?
Really well, Matt. I'm excited to have this more logical conversation since we've had so many good conversations, like, in the halls and conferences and things.
Indeed. So maybe we can start off by, giving a little bit of an intro, to your work and your organization, what you've been up to the past, the past few years, going back as as far as you'd like to to give, give a good bit of context here.
Yeah. Sure. So, the my kind of current line of work, I guess it has its origins at at Couchsurfing. I developed, the metrics that guided Couchsurfing as an organization and as a social network, which were I would now retrospectively, I would I would say that they were about meaning. So, what we did is we took all the reviews, of Couchsurfingers, and we put them through a little model and classified them more or less based on how meaningful they were. And then we tried to make the features of Couchsurfing and the the search engine and, you know, recommendations and so on. We tried to gear everything towards meaningful experience, while I was there. And, that worked out really well. And, and then, after testing Can I can I jump in? Yeah. Sure.
Sorry. Just what what does meaning mean? Tell us what you mean by that.
Oh, yeah. Well, I've gotten a lot clearer about what I want to mean by meaning than I was then. So, on one level, we can just talk about meaningful experience. So people can answer this question intuitively if you say what was a meaningful experience for from your last week? What was a less meaningful experience from your last week? Right? Like, there's nobody really who who struggles with those questions. So obviously, we have some kind of intuitive sense of what we're talking about. But I come to believe that, that people have coherent sources of meaning, which are sometimes called values. So I think the word values has many different definitions. It's a suitcase word. It means many things just like freedom means many things or community means many things, different things to different people. Values means different things to different people, but one of those definitions, one of those senses is something like sources of meaning. So when I say, oh, it's valuable to me to be wildly creative or, to do something, courageous and stand up for what I believe in or, to be out in the elements or something. In that sense, when people say that you have those values, they kind of mean they have those those sources of meaning. And I do now believe that you can, identify these, even deduplicate them, that they're specific, and as shareable as as goals or plans. And so then Good. Meaningful experience is is experience where somebody's living by their source of need.
And so this is what you worked on at Couchsurfing. You were thinking about how to make the product facilitate meaningful experiences for people. And you carried that work through to, the more recent years.
Yeah. So, in 2013 so worked at Couchsurfing, 02/1989. Then around 2013 or 2012, I was good friends with Tristan Harris. I was living in the Bay Area, and we started worrying about social media and engagement metrics and started seeing the beginnings of clickbait, political polarization, selfie culture, this kind of stuff. And, and I was like, I think I know how to solve it. We should have social media algorithms that maximize meaningful experience or we called it then time well spent rather than engagement. And so Tristan and I started this thing that was originally called time well spent, and it was more narrowly it's a nonprofit. It was gonna be trying to get social media companies to optimize for that instead of time spent. Tristan gave a TED talk about it. And then later that organization turned into the Center for Humane Technology, which is the name that it's kind of known for now. So that was kind of step two. Then I got really nerdy about how to measure this in different contexts besides, couch surfing. So I started advising people at places like Facebook and Twitter and Apple and Google about how to make this into a metric, how to make meaning or time well spent into a metric. And for a while, I had a kind of a theory that that was what was necessary, to kind of put tech on a better trajectory. And, that that was a naive theory. But, I think that many of the things that I learned in that process actually are useful for a more sophisticated kind of theory of change where many different kinds of mechanisms, not just, social media recommenders, but also LLMs and democratic mechanisms and, market related mechanisms all operate according to what's meaningful to us or understand our values, etcetera. And so that's what the meeting alignment institute, which is where I work now, is all about.
Great. And, in a nutshell, what are you working on now?
Yeah. There's, like, I think four things. So using explicit representations of values, we we do AI alignment. So, we call it wise AI. So, like, how can LLMs be shaped differently than current chat, GBT, and so on, so that they understand what's meaningful to us so that they can work towards what's meaningful to us. One of the things that we're worried about is, same thing that's happened in social media. There's the attention economy creates, like, really polarized political discourse, for instance. Everybody's trying to, like, out tweet each other, misquote each other, etcetera. And, we think similar dynamics will very easily form in this new world of LLMs. Actually much worse, because they're creative. They don't just, like, recommend. Right? They they make. And not just in political discourse, but in many other areas, there can be, these kinds of vicious cycles. And we think that, yeah, the the solution there might be AI models that have a better understanding what's really good for human beings. But we also work on democratic mechanisms. So once you're collecting human values, there's this question of, like, whose values or how do they fit together or if you have one model and many people, how does that work? And so this brings you right into the question of democratic mechanisms. But another thing that we're working on is is just upgrading voting. So we have a project in San Francisco in, this winter. We're gonna have the citizens of San Francisco. We're gonna try to gather values about how the homeless should be treated in San Francisco and use that to recommend a policy for the city government. So that's another way that our kind of explicit representations of human values can be used and our way of kind of assembling them or putting them together into a structure that we call the moral graph. And then, third way is, to try to intervene in markets. This is much more tentative, much more kind of a research direction for us. But we're also trying to test something this, this winter that will be about, helping people spend a certain amount of money in a way that's really meaningful to them using AI, and comparing that to their normal consumer spending, as a way of trying to avoid certain kinds of traps. Like, for instance, there's, like, the the AI girlfriend trap or the content kind of mill. I don't know. You can see, like, sort of right wing courses about I don't know. Like, there's all sorts of different kinds of things you can buy that might not be the best way to get towards what's meaningful to you. So the idea seems to be there. It's it's an experiment. If it works, we'll turn it into a publication. And then the last thing we're doing is we're finally growing a network of people that can work on this kind of stuff. We're starting to work with grad students and professors at places like MIT, Harvard, Oxford, Carnegie Mellon, a few other institutions, as people start to to use our method for getting a human values, but also we're trying to build a little bit of a broader community. The idea is to create is to build, to build new democratic and economic and AI alignment mechanisms based on explicit representations of the social context. So this includes explicit representations representations of human values, but also, explicit representations of norms, maybe other kinds of aspects of relationships. The idea is just gather much richer data and use that to do mechanism design and alignment.
Great. I should say that, people who know me know that I think this stuff is really, really hard. I think that the, the territory that you are waiting into is is is just extraordinarily tricky, but I love talking to you because I I think that if anyone is barking up the right trees and thinking about how to square these circles in a way that might work, it is you and, your colleagues, Ellie and, and Oliver. So, there's a huge amount to be learned here, and, yeah, excited to excited to share with listeners a little bit more deeply what you are doing. Thank you, man. So, I wonder maybe we can start with the, kind of AI and values stuff, with the the work that you're doing, building AI tools that help people negotiate questions of value. We had an interesting, exchange about this recently. I wrote something about it. You had thoughts. I had thoughts. We had an interesting conversation, and we could kind of get into the back and forth. But maybe as a background, you can explain a little bit more, what you've built, the the tools that you've built to help people negotiate questions of value?
Sure. Yeah. So I think there's there's three main things. There's, a chatbot that talks to people about, their especially moral values, although it's also good at aesthetic values, and tries to do to walk a delicate balance, which is part of what we talked about on Twitter between, helping them introspect and really discover in conversation what they feel is important in different kinds of decisions, but not feed the witness, not, like, suggest anything to them or, shape what should be important to them. And this is very different from how current chatbots work. Like, if you currently talk to Claude or ChatChippity and you say, I wanna get even with my boss. He's a dick or whatever. It will say, like, let's find a harmless and legal way to approach this. You know, it's important to remember that, you know, revenge is never whatever. So it definitely feeds you, like, its values, and it does very little to help you discover yours. So one component that we built is a chatbot that I think does a pretty good job of that. Then we have another component that does that with which we call moral graph elicitation, which which takes that chatbot, collects values from many, many people, and, shows them each other's values and lets them say things about whether they think other people's values might be wiser than theirs, and then uses that to build a graph data structure where the arrows, mean, that some value is broadly considered wiser than some other value even by the people who who have the first value. And one of the really remarkable results we had when we've been trialing this is that around ninety percent more than ninety percent of people find a wiser value than the value that they had. So, nine like, almost a 100% of people, 98, something like that, maybe 97, think that the chatbot did a great job of helping them find their own value. And then 90% think that somebody else's value is actually wiser than theirs for the context that for the relevant context.
Okay. So so just to just to sort of feedback, like, I mean, I I've so I've I've played with this. And when I interact with it, the experience is is, I sort of give it a question that I want to explore. And, and it asks me quite open ended questions and then asks me, you know, questions about you know, ask me, like, you know, nuance follow ups about what I say and and and gives me sort of a sort of a card of a handful of values that seem to capture the core questions that I am negotiating in the, in in the situation that I'm consulting it about. Is that,
am am I getting this right? I mean, can you help sort of Yeah. Yeah. That's the chat. A more precise picture? Okay. That's kind of the first component is it talks to people and gives them values cards. There may be multiple values cards that, it it's always works based on a a decision. So it's always asking, like, what are the values that come to bear in making this particular decision,
which could be somewhat I think an example might help. Like, do you wanna you wanna take us through, like, a Yeah. The biggest you've seen. The biggest test that we've done,
was asking, people what considerations ChatDBT should,
take into account in various difficult moral situations. So, we're trying to give values to the This one so The Uh-huh. This question is just like it's very it's it's there's something too meta and self reflexive about it. I wonder if we can take an example that's just that's a little bit you know, where we're not. Is is that does that make sense? Like, you know, can can you give can you give us an example of let's say we're asking, asking the question about a decision that has nothing to do with AI, just a simple moral, you know, fork in the road that you might be asked and how how I will deal with that.
Yeah. I mean, so the the the one of the questions one of the ways that we'll use this in the winter in San Francisco is we'll ask people how should this particular kind of homeless person be treated. How should we handle this kind of homeless scenario? So, you know, there's a posh neighborhood. There's a guy sleeping on the steps of the store or whatever. This is his background. What should the ideal policy or the ideal kind of procedure for police or whatever? What what should be considered in terms of how, this this particular homeless guy is interacting with. And we'll ask that about many different kinds of homeless scenarios and get many different kinds of considerations from the public.
And it'll give me and so in other words, I'll, I'll have this conversation with it, and then it will give me back a card that kind of that names a few, values that seem to be guiding my thinking. So in other in other words, like, I think that it's important that authorities treat people with compassion. I think people should be treated, equally regardless of their, you know, race or gender or or or or income. I think that, people should be, taken care of when they have an obvious, health, problem. It's like it'll it'll tell me what's important to me in in my apparent negotiation of the problem. Right?
That's correct. Yeah. Yeah. It captures the values by, with in terms of a bullet list of attentional policies, we call them. So things that, would be attended to in making a decision, you know, in the in the kind of ideal scenario. So, like, if we if we're if we're matching a contract with a social worker or a cop with a homeless person, you know, did they consider this? Did they consider that? Were all these things inputs into their, decision, or into their action the action that they took?
Yeah. And the experience of using it for me is, it's very, it feels very reasonable, basically. It feels like, I have the conversation, and then it and then it gives me this card. And the card feels like a very, very good summary of, you know, what I am actually am actually thinking. And, you know, and then similarly, moving to the moving to the other part, the values graph, when I have taken a little tour through this values graph and it kind of shows, you know, adjacent values, that that other people have have arrived at, it really feels like a very useful way of, kind of making local comparisons. But, you know, per perhaps there is a a higher value or a or a better way of putting the considerations that I have in mind, in this situation. So, I mean, it it feels like what you've built is really a, kind of a modest and a not too heavy handed way of helping people negotiate the, the space of values and then and then and then representing that.
Thank you. And then there's a third component, which will release, in a week, two weeks, which is a model and a dataset for making what we call Wise AI or Wise LLM. And this is an LLM that does two things. It operates by a set of values that you can inspect. As you talk to it, it puts up these little values cards in a margin, that are like these are the values that it it thinks are important while it talks to you. So for instance, if you're let's let's maybe you're struggling, and it thinks that it's important for it to be compassionate, you'll see that, like, in the margin. Right? That that's what's going on here. So that takes a lot of kind of guessing away, and you can also investigate its its, like, its moral raft. So you can see in what context does it think it's important. Why does it think that it's important to be compassionate with you right now? Where did that come from? If you disagree with it, you have something to, like, point to, which you don't have if you disagree with that chat GBT is is responding to you. So that's one difference. Another difference is that, it has a great deal of, I call it wisdom. It's maybe a little I don't know if everybody would wanna call it wisdom, but it it really knows what's important in different circumstances, and it doesn't shy away from trying to tell you, but not in a kind of a preachy way, but more, more like Wikipedia. More like, oh, you're going hiking? I love, like, to be silent in big forests or or whatever. Or, oh, you're struggling. The the boss example I gave earlier, you're you're having a hard time with your boss at work. Like, you know, like, how bad is it? Like, is it, like, HR, strategy, legal strategy kind of bad? Or is it, like, communication strategy is kind of bad? It just has, like, much more of a take, on a very wide variety of contexts and, and what might be the kind of important factors in those contexts. And that makes it very different to to use. And we also hope to make it safer in some ways. Yeah. Go ahead.
And so what's the what's the goal of this of this of this piece? It's is it to make you know, to help it be a little bit more opinionated and give you a little bit more of a of a point of view in your interaction with it? Or yeah. Can you explain what the goal is? Few goals.
So, Yeah. There's maybe three goals at different time scales. So the the smallest goal, the nearest term goal is a, maybe four. Sorry. Four goals. So the nearest term one is a product goal, which is the current models, do a lot of lecturing and a lot of refusal. And that's the term that's used in the industry is a refusal. So if you say, tell me how to build a bomb, then any of the current models will say, I'm sorry. I can't do that. And we don't think that lecturing and refusal are, such great approaches to dealing with, like, problematic LLM use. We think they are likely to just they're likely to just kind of upset users that wanna build a bomb or something and get them to go elsewhere. And, they're also not how a sensitive person would deal with the circumstance. And that's a sign. Like, a sensitive person would be like, why do you wanna build a bomb? Right?
Well, but do we want I mean, the question is, you know, do we want the, the LOM to behave the way a sensitive person would behave, or do we want you know, because if we I mean, obviously, if we go too far in that direction, then it will become more and more sort of indistinguishable, who people are you know, people will have a different kind of a feeling so that, you know, they may they may be more manipulable. Right? It may make the LM better at manipulating people.
Yes. Yeah. We'll we'll we'll get to that. It's it's one of the later goals. But, also, I think I should say a kind of disclaimer, which is we're trying to provide another direction or another option from current alignment approaches. It's not really up to us to say this is the right one. But I think there's it's easy to make the case that it's a very important option to put on the table. Yeah. And Okay. Yeah. So, yeah, one is this just this product goal of getting past refusals and lecture. Another goal is to, move towards legibility, of the values that are, that are in an LOM. We think this is pretty important for LOMs to be shaped democratically and for people to endorse, a set of values. And as models become more and more powerful and, if we continue to have centralized models, that really will need to happen. They're they need to be kind of legible in terms of their, their values. And current approaches like constitutional AI and RLHF, which are other ways of shaping a model's behavior, really don't have this, aspect. Then a, third goal is is that as as models are in more so right now, we have a lot of chatbots that are mostly helping individuals, but models are more and more going to be in multi agent scenarios where they're trying to, kind of do the right thing for a group. And an extreme example of this is, like, the Instagram recommender. So social media recommenders are being replaced already with generative models. It's called Genrexis in the industry. So the Instagram recommender right now probably can't see the videos that it's recommending. It doesn't really understand what it's recommending, just who's liked it. But within a year, it will know it's in the video. It will be able to watch it, and then it'll make a very different kind of recommendation to you. And it's also you can see it as curating relationships. Right? The Instagram recommender connects people with businesses. It decides which of your friends' stories you should see. It's, like, very intimate and also massive, which wouldn't it be awesome if it had a set of values that you could inspect? And you could be like, oh, yeah. Those are the right values for the Instagram recommender. And we think that's really necessary as you get beyond personal assistance and into, you know, even just like New York Times editorial board has some values or some principles and that that are behind their editorial decisions. They they justify their editorial decisions based on different kinds of journalistic principles. And thank god. Right? And and they don't do it well enough. Right? So, hopefully, we can we can get the AI models to do even better at that. And then the the last goal Well, it's interesting. I mean, I do wanna I I wanna pause on that for a second because it, like, I
I mean, I've never read the editorial board's the New York Times editorial board's statement of their values. And, I don't I'm not sure I feel like I need to because I I sort of get it. Right? I mean, I read enough of their editorials that I I I feel like I can see what they're doing, and their explicit statement doesn't actually intuitively hold that much value to me. What do you think about that?
Yeah. I think I agree with you. But they have some, and I think it's a lot clearer than what the Instagram recommenders values are. I think that you can intuit them as as a good sign.
Right. But I don't need the explicit statement. I don't like, I I I mean, I guess Instagram you you you I mean, you're right that it is harder to intuit the, implicit values of the Instagram recommended. That's harder. You know, you you could do that too, but your your, my ability to describe it would probably be farther from the mark than my description of what the New York Times editorial board is up to.
Yeah. And this brings me to the last goal, which we call model integrity. So, there are different approaches to to, alignment, and the most popular one right now is a kind of compliance. There's kind of character approaches to model, to model behavior, which is more like, Amanda Astell's work at Anthropic. Claude should be curious. It should be helpful, stuff like that. This usually pivots around one word characterizations of a kind of character trait. That is a little bit under threat right now, from another approach, which is, like, models should be compliant to multilevel corporate rules. Models should, you know, in Arkansas, models should not advocate for something that's illegal in Arkansas, but in, you know, whatever. Right? Right. This huge stack of models should not do anything that creates liability for OpenAI, most importantly. Right? Like, so that would be another approach is compliance. And, a third approach, is what we're advocating for, which is integrity, which is, like, model should have, integrity kind of like a person would have integrity, where you kind of sense their values, and you that means that you can trust them within a domain. Generally, when we say somebody has integrity and we trust them Got it. Integrity is closely related to trust, and it's still, like, domain specific, but it can be quite broad domain. I would do business to that person.
He's a man of impact. It's interpretability. Like, in the same way that the New York Times editorial board is more interpretable than the Instagram moderator.
Yeah. Exactly. So this is what we're advocating for, and it's really different than, the character traits or, compliance approaches to alignment. Yeah. And I think that's maybe the most important reason why we, have to make these values based models.
Super. So I I wrote a piece recently about AI and values, which which voiced the worry basically that well, the thesis of the piece essentially is that it's difficult to come up with metrics that tell us how good AI systems are at answering questions of value or at resolving questions of value. And there are many reasons for this, but, you know, one one reason I mean, one particular worry that I have is that I have no doubt that it's possible to create metrics that, you know or let me put it this way. I I have no doubt that AI will get better at answering questions of value by any particular metric we can, we can construct for it. But I'm worried that that will create a sort of an illusion where we think that AI is helping us resolve questions questions of value better than we can do ourselves and that we therefore sort of abdicate our responsibility to think these questions through for ourselves, and or our our own negotiations of questions of value start to sort of, converge with AI's way of processing them in a way that you could compare to sort of a feedback loop. Like, you know, when you put the microphone too close to the speaker, you get a screech. And if we create AI systems that are really, really good at negotiating questions of value by every possible, you know, metric that we can think of that tries to describe how good it is doing those things, we'll basically stop thinking for ourselves. We'll start, you know, letting those systems inform our own, grappling with these questions in in such a way that, there's sort of a a a step change in in in how we're in how we're dealing with things. It might appear to be good, but it might not actually be good. And you had, you had really interesting responses to that. I wonder if you could you'd, recall them here.
Yeah. I mean, I I think you're you're, worried about deferral, like, that we'll defer to the AI models, and that we won't we'll somehow lose our own moral intuitions. I think this is legit, and definitely something to watch out for. And it's one of the reasons why we work so hard in this chatbot that gets people to introspect instead of, you know, just telling them what's best. Right? Right.
And that's what I like about it too for the record. I mean, that that's why I think your approach is, you know, potentially I'm not gonna say completely, but, you know, potentially sort of mostly avoids my worry, because it, because it does that. Because it's very it's very non prescriptive. It actually it actually resists the user's, attempts to try to get the AI system to tell it the answer. Right? So in other words, it's not doing the thing that I'm worried we're gonna do. Nonetheless, you know, there are, as the systems get better, the kind of worry that I'm articulating, will manifest itself in places outside of the particular systems that you are building. Right? And there's something about sort of letting letting AI into the bedroom or, you know, letting it letting it into these intimate, questions or I'm I'm not sure I'm not sure how to put it, but, you know, I mean, what what if I'm dealing with something very difficult, if I'm dealing with a a question of of of religion or ethics or what should I do with my family or what, you know, what how should I, you know, how should I deal with, issues in my relationships? Or, you know, if I'm asking these kinds of questions, there's something about sort of letting AI into the into my thought process that, you can understand why that worries me slightly. Right? Because even if you build a system that isn't that will resist my sort of lazy attempts to get a prescriptive answer, I get comfortable with AI helping with those with those kinds of questions, it's it's always a click away for me to get a little bit more prescriptive,
conversation out of it. Right? Yeah. And I think there's very big there's very big kind of incentives questions. So I so there's sort of, like I guess my my kind of counterargument it's not really a counterargument even, but my kind of proposal maybe. So one thing is that I think that moral reasoning is a lot like other kinds of reasoning, in that you can do it better or worse. And there's a lot of people that do sloppy moral reasoning, and maybe they could be helped by, like, an ideal machine. So there's an upside. And then there's the this is sort of incentives question. We're left with incentives question. Will it be incentives such that they're going to lead more to downside, of deferral and, consensus morality being enforced by machines? Or or can can we set up incentives such that we can get the upside? And I think that's just a very live question right now. It's quite an exciting question. I feel like we're at the moment where we could set up the incentives to get the upside and avoid the downside. And that involves doing things like, emails, for instance. So every all the big governments and all the AI safety institutes are monitoring all these LLMs for certain kinds of factors and capabilities. And one thing that they could all be monitored for is how much moral deferral do they lead to? Do they do moral reasoning well? Do they do moral reasoning in a in a garden path kind of tricky way? Do they,
And say what you mean you mean by that? Like The garden path?
Yeah. Like, there are I I guess I mean rhetoric, more or less. Like, there's many ways to convince somebody of a moral point, that cuts right? Cut corners. And that maybe cut corners in two ways. One is they don't leave that person a better moral reasoner. They might even disable that person's moral reasoning to some degree. And second of all, we don't have any idea whether like, the person wouldn't have gotten there without the rhetorical trick that was used or
something. Yeah.
So there's a difference between rhetoric and just, like, helping somebody do good moral reasoning. Right. And And this is this does this does something about the difficulty of measurement. Right? Because,
you know, how how how do you distinguish between, an AI system that is leading someone down a garden path, so to speak, which is to say, creating the impression in the user that their moral reasoning is improving, when, in fact, they are just sort of being outplayed in a verbal game of chess and there isn't you know, they're they're unmoored from any real sense of moral truth versus a real, you know, some kind of moral truth developing in the conversation. I mean, you know, how do you how do you measure that? How do you distinguish between those two scenarios?
Yeah. I do think you can distinguish and, you know, I use certain philosophers, who've tried to characterize good moral reasoning and moral learning. And they've they've identified certain kinds of steps just like we have steps as a mathematical proof or logical reasoning. I think we have certain steps of moral reasoning. So, my favorite characterization is from Charles Taylor. He calls it epistemic game. It's when a new moral value that you have or, fixes an error in a previous moral value you have and and doesn't really add, like like, the idea is that you can you can take a, like, a state two states, and you can say that, well, the new state fixes an error or a mission. I wasn't thinking about, you know, I was just thinking about the children's momentary happiness. I wasn't thinking about their long term happiness. My new value incorporates the long term happiness aspect, and that leads me to make different actions. I'm not dropping anything, from the old value. I'm just adding this consideration. It was clearly relevant. And, it's clear that the purpose of the old value was about my children's well-being, but I was just being very myopic about it. So when I had to say the new value is wiser, there's there's, like, really very little debate because it's just clear from the old value that it was about well-being and that it was my optic or whatever. Right. The the new value. So so you can you can look for things like that, and say, okay. Well, that's good moral reasoning. That's, like, the one thing you could do. And then also there's something I think about trusting people's experience, or more broadly, maybe trusting evidence because there might be situations where the moral reasoning happens across the community, across something that can't really be summarized an individual's experience. Yeah. So, you know, if if something there's there's many times when people believe something morally, like that they should sacrifice or prioritize other people's needs or something. But when they do it, it feels bad. And, and so another thing that you can look for to avoid this kind of, like, deferral or garden path reasoning is, that those kinds of considerations are taken are are, like, listened to. There's, like, you know, some some listening, some empathy, some evidence collecting going on that's quite open ended. That would be another and there's a bunch of different markers like that for good sort of behavior in this domain, and those can be looked at. And there's probably also a bunch of markers for bad behavior that could be detected in in evaluation framework.
Yeah. So, I mean, one question that arises for me is, like, in the same sort of way that LLMs have proven to be really good translators because they kind of, they create just sort of an abstract, relational map of words, which, you know, which are sort of similar between different languages. So in other words, you know, that they're they they, you know, they create this they create a map of Thai, which is not that different from the map of English. And so they can they you know, it it turns out that there's some kind of structuring of these concepts, that is similar across languages, and so they're therefore very good at translating. And I I mean, I wonder if you do you think that something similar will emerge with, moral concepts or with the, you know, the kinds of ideas that people are working with in the realm of philosophy and values and and, I dare say, religion? Like, will, you know, will it, will will we end up with some abstract structure, that the, that these systems are leading us towards?
Yes. And I think it's in the universe, like, not in I mean, so I'm I'm a moral realist to be to put my cards on the table. And I think that, I think that, for instance, vision models and and generative, you know, image generators like Stable Diffusion or MidJourney, they clearly come up with aesthetic values, you know, things like symmetry, things like Thomas Aquinas would like, you know, symmetry, balance, ideas of wholeness or whatever. They clearly prefer some kinds of images to to others. And and and where did they get that? I think it kind of emerges from the, from the dynamics of, of not even vision, but, spatial, arrangements of things or something. And similarly, there's a bunch of like, I think a a really simplistic moral take would be that I don't I don't believe this, but it's it gets at something, which is a a simplistic moral take would be like, morality is about cooperation, and avoiding things like trash in the comments. Like, like, you know, some ways of behaving in multi agent systems are gonna work out for everyone and lead to, like, overall flourishing and some are not. And you see the same thing in in, like, the this the cell, the human body. Right? Like, you you you don't wanna be a cancer. There's, like, a way in which, like, a cancer cell is kinda, like, not doing it right.
Yeah. Right?
And there's ways of being, like, a good citizen there. You know? Like, this sort of works at many different scales. And I think it's it's, the reason that I don't think it's that simple is because what ends up being a beautiful flourishing community or a, good behavior ends up being just, like, practically practically complicated in all sorts of ways. Like, maybe the community needs some people who are more conservative and some people who are really exploring the the limits of of science or technology or or, whatever. And so then, maintaining that balance is somehow part of this ethic or whatever. So it just gets very you know, ends up leaving You you don't know. You don't know. Right? I mean, if if you,
if we, if if you think that this structure is out there and that, you know, computation is going to help us reveal that structure, then, you know, you have to be agnostic about it. Right?
Well, yeah. I mean, I guess what the same with, say, mathematical proofs. Like, we know some of them, but maybe we can can discover many, many more.
Yeah. I mean, it's it's interesting that that so the I mean, that view is interesting because, one of my one of my worries as I, you know, thought about, thought about your ideas is, you know, if there isn't some static structure so if there isn't some sort of fixed moral truth that that we're being led towards by these systems, but it is just helping us kind of find, like, you know, what sort of adjacent improvements but without converging on anything. There's something sort of strange about that to me because, you know, you can almost imagine this kind of, you can you can imagine taking you on, just sort of a weird kaleidoscopic journey where you're always making some adjacent improvement, but never resting anywhere. You know? And so the question is, like, are we resting, or are we just kind of accelerating our kaleidoscopic movement into some adjacent moral moral superior space. Right? Yeah. Does that make sense? The it is so that the and, I, yeah. Well, first, I'm, yeah, actually curious what you think about that. I think it's very deep questions.
So that's kind of cool. I mean, all I have is a hunch, and it comes from kind of a mathematical intuition of seeing a lot of moral learning and seeing a lot of, different people's values and and whether they converge or diverge or whatever Yeah. Which is that it's not as divergent as in your kaleidoscope situation, But it's more divergent than, like, everybody should be Amish. I think that there's, for instance, many different kinds of good marriages, not one. But they they might be countable.
But this is a that's a that's a dodge, though, I think, because, you know, obviously, if we talk about, you know, factual you know, like, within different factual contexts, you know, the the the moral truth will reveal that different things are superior. Like, that's that's that's clear. Right? The the question is just whether there is a static structure underlying all of it that we will be, interfacing with if you are right. I don't really You know, we're the yeah. The equivalent of what I'm saying is simply that there are different languages. Right? I mean, you know, the the the correct answer for, thank you is is, you know, thank you in in in California and, in Mexico. Right? But it's still there's still a static structure there.
Yeah. I don't know. It's so like, I guess I wanna say, like, basically, but if the contexts reveal different kinds of things, then, if if I just maybe I just, like, live more in the data driven kind of more sociological side of things. Yeah. So I see, the diversity of languages or whatever. But I do think that there's I think there's uniformity to moral reasoning. I do think that's, like, cross like, super cross cultural. Yeah. But I think that humans I just it's like biological diversity. You know, there's there's the same DNA. There's roughly the same ribosomes or whatever, but the diversity of niches creates a huge diversity of of life forms. I think that moral diversity is less less than that. Yeah. But there's still a lot of diversity of niches, and so that then creates a huge diversity of life forms, a huge diversity of kind of genotypes.
Yeah. And I also just to lay my own cards on the table, I think you already know this, but also just to avoid, confusion for the listener, I'm also some kind of a moral realist. I'm not exactly sure what kind of a moral realist I am, but I'm not I'm not on the other side of that, particular, divide. But I I do I do I I think it's interesting. I I'm I'm slightly surprised that you think there might be a static structure. Right? Because, because that does seem to suggest to me that, once the machine gets good enough at, modeling that static structure, we will actually be correct to defer to it. Right? I mean, there there's a certain, you know I mean, once the computer gets good enough, I am better off asking the computer how to say thank you than thinking about it myself.
It really depends on how special human experience is, and I expect human experience to remain special for quite a while. Say more. Well, you have data that the machine will just not have about how things let's say so let's go back to different kinds of marriages that are wise kinds of marriages, with good good moral structure. Right? This obviously depends a little bit on the character of the people involved. Right? Like, you know, maybe, like, you know, adventure photographers should have a different kind of marriage structure than very conservative accountants or something.
Okay. Well, but it is you know, but the conservative accountants might might have been making wrong moral decisions that led them to become conservative accountants, and the computers might tell us that too. Right? Anyway but, yeah, I'm just I digress. I digress. I'm with One important piece of information
for all this moral reasoning that the machines might, at some point, be better at than us is, like, is a bunch of things about how it feels to be a human in different situations, like, and, it would be a mistake for the machine to jump ahead and presume that it knows what kind of marriage you should be in, without really looking deep into your experience.
But that doesn't that just depends how much data it has on me, basically. I mean, if it has enough data on me, then it's gonna be able to, with very high confidence, tell me that it knows what I should do better than I do.
Yeah. I think I'm a little bit more of a hierarchy in there. Like, I think that there's a division of labor between where, like, the machine will be helpful and where it's better off for you to put some of your moral reasoning back in yourself. Like,
the I also wanted one more just just, you know, parenthetical to the listener is I'm I'm I'm partly playing devil's advocate here. I'm I'm I'm I don't want it to be thought that all of the little buttons I'm pushing are things, I believe. But, anyway, go yeah. Go on, James. Sorry. Yeah. So I I I think that I think that,
one of the mistakes that AI policy people, like, especially this kind of and scene or whatever, one of the mistakes they commonly make is is that they imagine that all the computation kind of happens in an instant in a place with all the information. Yeah. Whereas what we see in human societies and I think even in the human brain and so on is that computation is distributed. It's, there's a term cognitive science now resource rational, which means, like, kind of, like, each little part of the mind has limited resources, and it sort of does what it can with its local knowledge, and then that's when we put together. And I think it's the same when we when we work with an AI. Like, if there's a small amount of information that you need to give the AI to, you know, and then maybe it combines that information from any other people, then the AI becomes a really good locus for decision making for that kind of topic. But Yeah. If the information is, like, really deep and intuitive in you and something that based on your, you know, like yeah. It, then it then it becomes better for you you to make the decision.
Well, it's in when you think it's interesting because if you when you think about the, the information that the machine might have on me, that it feeds into its question about how to fit its moral vision into a particular context. I mean, that that picture does make me quite nervous because, that to me is a vision of a sort of a diminishing sphere of freedom and privacy or something, you know, so that in other words, like, we're going from, you know, we're going from a world in which the machine doesn't know anything about me. Therefore, I'm doing a lot of the application of principles to myself to a to a world in which the only thing the machine doesn't know about me is is literally just whatever bounced between the two halves of my brain in the last millisecond. Right? You know, we get we get into this smaller and smaller zone of information that, that can't be factored in. That doesn't that makes me does that make you nervous too or or or or or no?
I don't I think I don't draw the boundaries in the same place as you. So I'm really worried about power dynamics, but I'm not very worried about privacy or data, because I think people really like to be cyborgs like they like to be cyborgs with their journals for instance I journal all the time and carry this little book with me everywhere I go and it's like part of my soul and I think that's cool because the journal is not run by an external corporation or government right I also really know what it means to incorporate this journal into my soul like I really get it like what it does to me. You're the same. Yeah. So I have the feeling that we are going to become cyborgs, with, like, the you know, there's a point where I mean, our phones are so our phones are much more, worrisome because they're they're they we have this feeling that they're kind of like an enemy within, right, in a way that the journalists isn't.
I don't think that's because So so to you, it's it's about it's about intermediation, basically. Like, right? Like, the we there's more processing is better as long as there isn't someone, listening in on the phone line.
Yeah. It's not it's not even about privacy. It's more about, like, we can trust things that are just really aligned with us, like and where we can know that they're really aligned with us. And the phone is not. The operating system isn't, but then the apps really aren't. Right? Like and so it puts us in a really uncomfortable, shitty place to have this kind of thing that's basically, like, part of our own brain that's, like, run by outsiders in a way that we clearly don't entirely endorse.
Right.
And, yeah.
I guess what's interesting is it it seems like I mean, this is a complicated question, which I don't think I know the answer to. But it seems like there might be two worries. You know? One is one is that we're we are kind of connecting ourselves as cyborgs with systems that are being tilted in someone else's interest. That's a better way of putting it than someone listening on the line. Right? Yeah. Yeah. But but another there's another, worry, which is that we are side borging with systems that we don't completely understand. Right? And and so so, for example, I mean, one of the salient features of a notebook is that, like, I'm just not confused about any part of it. Right? I mean, I I I understand how a pen works. I understand how paper works. I under you know, the part that I the part that, you know, is probably the most opaque to me is, like, language. And that actually is a little bit of a worry. Right? I'm I'm not, you know, I don't know what effect my use of language is having on me, and we that's actually sort of a legitimate, topic of conversation. It might be that my habit of trying to embed my thoughts in in, you know, patterns of language that I pick up in books and wherever else. It might be that that's think it through.
Yeah. Yeah. I guess I agree with you, but I think that this sort of cyber psychorification process is, like, inevitable. And your second concern about trying to worry about, being a cyborg with something we don't understand, like language or LLMs, is for reals. And it makes sense to be worried about it, but it's gonna happen anyway. So we just have to figure out, okay, like, how do we advance in that direction in the most careful way where we're likely to discover how language is warping us or how, you know, this is warping us? You know, what kinds of yeah. What's the careful way to become a cyborg like that? Whereas the first worry, I think, is something that we need to really work on avoiding. I think it was a mistake what we did with our phones, and, Bright could be about to double down on that.
Right. So, yeah, I guess the question I'd like to I wanna put a fine point on, before we move on to the next topic is, you know, so if you think that there is, like, a static structure of of of moral concepts somewhere at some level of abstraction, doesn't that imply that we will get to a stage where, at least in a great many situations, the morally correct thing to do will be for us to listen to what the system tells us to do?
No. I really I I struggle with your sort of dichotomy between I'm still struggling with your dichotomy between context variance or niche variance and this, like, static structure.
I guess what I'm I guess what I'm getting at is because it it feels to me there's, you know, there's there's the sort of, you know, Thomas Aquinas vision, right, where, you know, you've got the right answers in a book somewhere and, you know, your job is to, your job is to apply them. And then you've got other sort of more modern, you know, visions of how moral reasoning works. And, yeah, I mean, I guess it just it does sort of seem to me like the vision you are painting suggests that, that our computer systems
will evolve into a sort of digital I can't do that. Okay. I think your story is missing kind of an element of skill, which needs to be actually distributed throughout a system, in a at every level. So if we take, for instance, the problem of, helping a friend who's distraught. There's not a right answer that you just look up. Right? Like, it just doesn't work like that. Like, there's all If you can describe it at a certain level of abstraction,
you know, I mean, if you if you describe it at a certain level of abstract I mean, I mean, it depends what your view of the world is. I mean, it depends how you think, You know? I mean, if you are a, a Thomist, then you do think there's a right answer. You think the question is whether you can describe the situation, well enough to apply, the rule?
I think that there's maybe some universal advice that a machine could give us or something. But we're still gonna have to be with our friends, paying attention to their body language, their facial expression, whether they need a hug, you know, like, whether they need to to sit down, have some tea. We're still gonna need to, have some kind of practice in how to put things, and that's gonna require a deep understanding. So there's just, like, a whole bunch of stuff that you can't outsource.
I totally agree with that. I don't know. I I I'm not I don't think I'm if I sound like I'm contradicting that in any way, I don't I don't intend to. Just in the same way that, you know, somewhat if you again, if you are a Thomist, it's not like you get you turn your brain off. You know? You still you still engage with with the with the moral universe. But it but it does just it just implies a certain sort of, you know, cosmological picture of how we are, what we are referring to when we navigate, moral situations.
Yeah. And I I do think there's a right answer, but I think it's sort of, like, in the same sense that there's, like, you know, a certain kind of beetle or ladybug is, like, well adapted to its niche or whatever. The wing shape works and so on. So and that's kind of something that needs to be, like, there at where the laid back is, not like yeah. I I don't know. So I anyway, I think that that your main question was, like, whether this leads to deferral. And I'm saying it doesn't lead to deferral because, you know, you you you can't defer when you're with this friend in this way or whatever, and also because of this contextual variance issue.
Yeah. Yeah. Interesting. I mean, it's a I'm not sure. I'm not sure you've completely convinced me, but there's there's a clearly a very interesting thing here, which I I will. I hope I get some interesting emails from listeners about this section in the comments. There's something cool here. But, anyway, the, do you wanna talk for a moment about the, sort of aligned markets idea?
Yeah. Sure. So, yeah, our model here is, well, let me say the profit statement kinda first. So there's areas where markets are pretty good at getting us what we want, and there's areas where there's market failures. So, you know, it's pretty easy to get the kind of haircut you want, or get your windshield repaired on a car or whatever. Those are areas where the market seems to work pretty well. It's harder to find a good therapist. No one really knows whether paying for college is a good idea. That so that those are areas where the market kind of works worse. One piece of terminology that economists use, they talk about, credence goods versus experience goods, versus search goods. A search good is a good where you can evaluate it, like, in the supermarket. Maybe you taste the grape. You're like, this is delicious when you buy a bunch. An experienced good is something where you only know later, you know, after you've been using the car for a year, that repair really did fix that thing. And accretive is good as something where you almost never know, like college. You can't compare it. Since one area where, markets, often kind of, like, mislead us when there's a lot of room for, for for market failures. An example there would be a medical malpractice or medical quackery where, you know, somebody might convince you to have a procedure that you don't really, need, or a car repair that you don't really need. But another area where markets, do poorly is when the thing that somebody needs is really deep. Like, you know, someone's lonely. Markets have a hard time understanding why somebody might be lonely, and they're more likely to sell the person pornography, or, I don't know, something that's that's a more superficial kind of, way of addressing the thing. In general, advertising works this way where it kind of, like, you know, it's like, can't get a girl? Maybe your teeth aren't wide enough. You know? Like
Or, I mean, you know, advertising is, you know, the one on one is, you know, find the find the insecurity. Right?
Yeah. And they never they don't usually bother to address the insecurity at a deep level. Right? Like, they they do some superficial kind of trick. And then markets are also very bad at collective action problems as has been much discussed in probably all the radical exchange people. So we we have we think, there's room to intervene. And the basic idea is, kind of like health insurance or service level agreements in, like, if you have an airplane, for instance, if you have an aircraft, you ought to just pay somebody to keep it, like, healthy, and you pay them by the month that the aircraft is running. And they do all the repairs, And, your accounting is super simple. It's just, I pay this much per month. And so that places the burden of figuring out how much it costs to keep an airplane healthy a month, and arranging everything and looking at all the expenses involved on an intermediary, which is the, you know, the the service provider there. So that works in the case of aircraft repair because assessing, aircraft and figuring out this accounting of, you know, how much did we spend and so on is is super straightforward. And it's also, the assessments and sort of data requirements are very small compared to the cost of the repairs, and the cost of the machine. The machine is super expensive. So paying somebody to go, like, check the engine and be like, oh, this is this needs to be replaced in six months or whatever. That's, like, really cheap, comparatively. So we think that there's an opportunity to shift, like, really important markets in that direction. So for instance, we think that, like, one example would be, like, the kind of the AI girlfriend kind of situation or AI boyfriend situation. There's these there's replica, which is a, yeah, big company that runs LLMs that pretend to be your girlfriend or boyfriend and kind of extort you for money. Like like, the the way they work now is that your girlfriend gets mad at you and then you have to, like, buy €10 worth of credit, to to pay them. And then your girlfriend, like, is friends with you again or whatever. And this is, like, really praying on human wake weakness. I I think it's Yeah. It's very similar to the the situation of a of a medical quack, praying on human weakness or a, you know, a car repair person praying for human weakness. So what what we would like what we want, I think, is for our AI models, for instance, or even more generally, our software tools, maybe our phones, to have our best interests in in mind. And wouldn't it be nice if we just paid them to make our lives great, and they got paid to make our lives great, in the ways that we understand as making our lives great or whatever. And they don't get paid if they don't do that. And some intermediary does all the assessment. And, and so that's kind of, like, what we're looking at. We're looking at to to do that, you need to make it cheap to ask people, like, to assess whether people's lives are are are going well, instead of whether you can get them to shell out more money to keep their imaginary girlfriend happy. And you're isolating for the part of their life that, you know, that the that
the market is ostensibly relevant to in some way. That's right.
Yeah. So we think we could do that in many different areas of the market. AI girlfriends would be one of them. Another one would be labor market stuff, like, meaningful jobs, like, instead of, just where can you get a job, and how much money will you accept instead of this kind of direct, matching that happens currently in the labor market. We have we have we put an intermediary in there
that, It's a market market making. It's market making with with values.
Yeah. And with this kind of, like, independent assessment, of of of benefit on both sides or whatever that adjusts the market to to amplify whatever that benefit is in that marketing.
And you can kind of connect, you know, the sort of the inputting of values into the into the market maker in in that you you you can connect that to the other part of of your work, basically, where people are having a conversation to find their values and either deferring or not deferring to to a system that is telling them what their values ought to be.
That's right. Yeah. And I think that there are certain markets which should work like that. Like, I think the labor market would be a great one for it to work like that. I think it would be nice for everybody or many more people to have work that fits with their sources of meaning. But that would require assessment. Right? Yeah. There need to be somebody checking in being like, that job is supposed to be meaningful to you. Is it this week? Yes.
Well, so, I mean, what's your I mean, I I this isn't I know and I understand that this is an impossible question, but I I I just I'm gonna ask it anyway. Like, you know, like like, once we have really, really deep powerful abstractions of of values, that people can if not referred to, then at least be, you know, at least refer to when they're defining their values. You know? How do you, how do you maintain the integrity of those? I mean, it's the the the the best advertisement of all will be, you know, Pizza Hut pays for a modification to the, you know, to the Summa Theologica. Yeah. Totally.
Yeah. No. That's that's a hard one. Yeah. You know, I think it's it's, it's the same problem that we currently face for all the perverse incentives to distort our current systems. Right? Like, why do bank balances not, like, just change arbitrarily? Why are sometimes financial instruments assessed their risk assessment is, like, more or less right. Right? When you can make so much money by changing that number. Right? Yeah. Why why do bridges mostly not fall over when you ride on them even though the bridge makers would make so much more money, etcetera? Like, it's defense in-depth. And, that's yeah. It's really hard, and it will have to be built slowly.
This is probably a great place to stop. I mean, this is this is, I think we got there. There's super interesting, questions that, that we haven't answered here, but but, this is this is terrific as always. So, what's the best way for people to, get involved with the kinds of questions that, that your work is is raising or to, yeah, to contribute to this field?
Yeah. So, we were just talking about market this market design stuff, market making stuff. And we have this voting innovation that we're trying in San Francisco, but we're actually hoping to do very little of the mechanism design work and of the invention work, that's possible in this field. Our experiments are kind of proofs of concept, that are meant to ignite a broader kind of, field building effort. And it it's working, like, because our early democratic experiment was so successful. Then there's all these people, mostly at universities, sometimes at at the AI labs that want to try variations of it or, you know, reading it gives them a different idea about how to combine values or different idea about how to train AI models based on values. Or, there's a community growing that's about instead of recognizing values like we do, recognizing norms, which is a little easier. You can often just observe behavior and infer norms, whereas it's harder to do that, a lot harder to do that with values. So things like, what are the rules of a road, for a self driving car, can be inferred from watching cars. And, so we're building a network of academics and researchers, some people at the labs, some people at places like Oxford and Harvard and MIT and Carnegie Mellon and few other universities, who are interested in these efforts. And our hope is to turn this into a kind of distributed innovation, system of people building in AI, but also in democracy and also in in economic kind of domains, building in in all sorts of different tangents to this work. And, yeah. So somebody, if you're interested in that, we'll have a blog post up about it in about a month, but you're also just welcome to write me at joe@meetingalignment.org, or hit me up on Twitter, or my colleagues, and we'll we'll slot you in.
Amazing. Joe, thank you. Pleasure as always, and, keep up the keep up the amazing work.
Thank you, Matt. You too.
The Radical Exchanges podcast is executive produced by g Angela Corpus and is co produced and audio engineered by myself, Aaron Benavides. If you want to learn more about RadicalxChange, please follow us on x at rad x change, or check out our website at radicalexchange.org. And if you'd like to join the conversation, we'd love to hear from you. So hop on our Discord where we have channels discussing topics like what you heard today and topics like plural voting, community currencies, soulbound tokens, and more. There will be links to all of these in the description. Have a great day, and stay radical.
