It is incredibly valuable to have a publicly accessible, decentralized, append only ledger that has, like, strong economic security properties. The only problem is it's far more useful for speculation and fraud than a lot of other things. I think that just being open to, like, do you think that humanity might not make it? And if the answer is, yeah, I'm willing to entertain that humanity might not make it, and then you look at the probabilities, the types of scenarios, civilization starts looking really fragile. Yeah. I'm sitting on a bunch of zero days. Some of the lyrics curl because I haven't vetted them. And, like, that seems asymmetrically offensive if that capability shows up in the wild. I do think that unemployment is going to spike at some point because I think that in the next few years, it feels pretty likely that we'll get something that's like a drop in knowledge worker. I think that as intelligence gets cheap, that agreement or consensus gets very expensive.

Yeah. Absolutely. Thanks for having me, Josh. Into a bit of who I am, where I came from, any blockchain connection because Atlas doesn't really have much of a blockchain connection. I shortly after finishing experimental physics PhD, I went looking for hardware startup to join and a college buddy of mine convinced me to join his peer to peer file sharing startup in 2017 for an unconventional fundraise. And that was the Filecoin token presale. Imagine the people here might be familiar with Filecoin, IPFS, LIP2P. They may or may not know that those are all from the same company, Protocol Labs. I've had I've had the delight of surprising people. Yes. Those are all, in fact, one company. Protocol Labs did a lot of various things, and I basically built out the research and meta science infrastructure. So this is everything from, like, I think I interviewed basically every researcher who joined the company for a period of six years. I, like, set up many of those interview flows, set up the grants program the research grants, set up our advisorships, sponsorships, figured out how we, like, leveled and promoted researchers, also ran a metascience team, also ran a special projects team for a good long while, also ran a for the last two years there, a venture studio focused on mechanisms for improving public goods funding. The idea was that Protocol Labs was going to spin out all of its various teams and products into different companies. Some of those would be creating public goods. How do you how do you properly incentivize the creation of valuable open source software? There's a lot of really interesting questions around governance and, like, social choice theory related to that because what is the value of this public good is something that is entirely subjective. You would quantify it. In fact, most things that are quantified are in fact subjective, but a lot of interesting questions there. I, like, helped out on the Filecoin white paper and the the economics of Filecoin white paper. A lot of, like I like to joke that I did some experimental macroeconomics, for a bit. Yeah. About three years ago, I left to start, Atlas computing mostly because one of the researchers on my independent research team, David Dalrymple or Davadad, was working on a mechanism to or approach to quantifiable safety properties for AI systems. And this seemed very promising. It seemed like there was a lot of work to be done. We decided he should go to ARIA and run this as a program there, and I should go start a nonprofit to do all the things that would be useful for that for that approach that couldn't be done effectively as a government. And over the course of the last three ish years, I've landed on Atlas being kind of something that fits between the think tank and the incubators where the think tank will have a really good proposal for how like, understanding of this problem and a proposal for how to solve it. And we'll have a lot of relationships with people who can speak to the problem, people who would you be users or stakeholders on the solution. And the incubator assumes that a founder has shown up with the right solution. And so I'm taking basically all of these gaps that experts have said, like, yeah. It's this is this could be a problem, and no one is working on it. And I have been doing some amount of work to make it easier for someone to take that problem. I like to quote that between the Venture Studio Protocol Labs and Atlas Computing, I think there are about seven founders, many of whom were not, repeat founders, where I introduced them to the problem and the potential solution and their first funder. And I think that really lowers the barrier for what it means to be a founder type. I think that people who would disagree with me on that would say that all of those people are founder types because they did end up founding a thing and that are that's going well. But I have I think that maybe those people would agree that there are a lot more founder types out there than are currently empowered or clearly see the affordances to go and do the thing. So I I will sometimes jocularly refer to myself as either a research fixer or a professional career derailer, where I help people find the thing they want to do. I I approach people when it seems like there is a much better way to achieve goals that a a person might have with their skills. And I say, you have the right skills. Would you like to be doing what you want to do, but more effectively in a new organization or a different organization or something like that?

read that type of job description before, which is, like, really, really interesting. But I can't Yeah. If I can't get excited about a job by reading the job listing, that's, like, in many instances, the most excited someone will ever be about a job. And so, like, I I think job descriptions must be held to a very high bar. This is actually something I inherited for Protocol Labs where, like, I think consistently, I would write job descriptions and be like, cool. Do I want do I want this more than I want my job? And if the answer isn't a little bit, then I, like, would go back and rewrite it.

I think the people that listen to this podcast probably understand why I might shutter to be associated with the majority of them even though a lot of your listeners have probably done the same thing as I have for the same reasons, which is technology just moves faster than a lot of the other aspects of society. And if you want to try to fix important problems in society, it is important to kind of, like, steer the fastest moving thing. And Mhmm. It was delightful. Like, I think Protocol Labs, very well known in the, like, the subset of crypto who were very ideologically motivated. And I still, amongst AI people, will say, it is incredibly valuable to have a publicly accessible, decentralized, append only ledger that has, like, strong economic security properties. The only problem is it's far more useful for speculation and fraud than a lot of other things. And so it's really hard to actually get adoption. There was at least one, like, really impressive thing that like, deal that Protocol Labs made. And part one of the riders in the clause was you can't use this in your advertising because we don't wanna get you accused of misusing taxpayer money. Like, that is not a good sign for your economic structure.

It will be related to Atlas Shrugged. That did come up when I was, like, floating it to people. And I was like, I don't think I think this is just a function of who I'm talking to. I don't think this will come up that much, and it hasn't come up that much. Right. Mostly, I couldn't come up with a better term for a set of maps or something that was related to making a lot of maps. The plan for Atlas was try to do, do a lot of the strategy work to make it easier to chart courses to better futures. I feel like technological non determinism probably has a meaning, but that I don't mean to invoke. But this idea that, like, if you move, I guess, differential development is or differential Okay. Development is, like, maybe the more appropriate term where, like, if you develop technologies in different orders and you could do this somewhat intentionally, you can lead to different outcomes. And I think this is an important thing. It's fairly technocratic and, like, fairly rationalist coded as ideas go. It's interesting how much I feel, like, having spent so much time at Protocol Labs, there was, like, a very consistent vibe and ethos and a lot of interesting, like, philosophical questions that were, like, I find it very helpful to reach for, like, the decentralized solution to this thing first because it turns out that in the event that there could be a, like, singleton superintelligence, you want something that is much more robust to, like, power concentration or things like that. And I think having focused on finding decentralized, democratized, accessible solutions is good, but so is having the security mindset of, like, okay. But it's really easy to think of something that sounds secure. But, like, did you kick the tires? Did you really try to see if someone can use this as an attack?

So I thought that was that was funny. The physics analog of the map is not the territory is all models are wrong, some models are useful,

Yeah. I think that it's interesting how many of those view like, those those downside scenarios get caricatured and sometimes dismissed. It is really interesting to see that, like I think that there's a much higher correlation between does someone acknowledge that these catastrophic risks are possible, and did they seriously at some point consider whether or not they might be possible rather than, like, how much evidence have they been presented with or something like that. I think that just being open to, like, do you think that humanity might not make it? And if the answer is, yeah, I'm willing to entertain that humanity might not make it, and then you look at the probabilities, the types of scenarios, civilization starts looking really fragile. I'm specifically looking at neglected problems where, like, because AI capabilities are growing, you could imagine new capabilities leading to asymmetrically offensive or asymmetrically oppressive technologies or technological equilibria. And the question becomes, what else could you put into that world to make it asymmetrically defensive? That, like, now we have this, and as a result, it is net safer. And so an easy example would be AI based red teaming seems like it is getting scary, good, very fast slash already. There was a great talk from Nicholas Carlini at unprompted, like, last week Yeah. I think, where he talk says, like, yeah. I'm sitting on a bunch of zero days. Some of the Linux curl because I haven't vetted them. And, like, that seems asymmetrically offensive if that capability shows up in the wild. That said, there are a bunch of things that you could imagine having. Like, first off, I would be very excited to have a bunch of funders decide to get a bunch of high profile cybersecurity firms on retainer to just take these and vet them themselves because they shouldn't get blocked on Nicholas Carlini and a bunch of other people at Frontier Labs who are finding these in order to vet them because they're doing lots of things and their job description isn't doing this. And paying someone to do a thing is a really good way to make someone pay attention to a thing. Another one would be if we had public key infrastructure so that if I am trying to red team my server, my server like, the AI that I'm using for that red teaming knows that this server is mine. I can sign I I can, yeah, I can generate a signature that validates to the same public key as what the server is sending out. And so you don't have to worry about if is Evan red teaming or, like, actually attacking a different team rather server rather than red teaming this one. You could imagine a lot of progress in hot patch updating where you don't need to reset the system to update the code. I feel like that there are lots of like, as AI gets more capable, the window between, like, was this discovered and was this patched needs to get much shorter because the ability to attack in that window might be much greater. And so being able to update the system trivially as soon as you know without having to reboot anything or having to interrupt anything seems like something that would be very useful. So these are the kinds of interventions that would lead to something that's more asymmetrically defensive. I think that I when I think about risks, I ask, like, would this prevent like, I want a world where AI listens to people, like, follows instructions, and where, like, people like, where there isn't something that looks like catastrophic harm. If it's not catastrophic, I'm not really that worried about it. Not because I don't care, but because I think that there are enough catastrophic harms to keep me busy, and you kinda got a scope. If it's bad already and AI makes it much, much worse, I care. If it's bad already and AI makes it a little bit worse, I consider that a little bit out of scope for me. I wouldn't necessarily, like, rule it out if a funder came to me and said, like, hey. I really want field strategist on this, and they brought a field strategist candidate or something like that. That would be fine. At this point, I'm, like, trying to scale as quickly as reasonable. I think of in terms of, like, following instructions, this is like, includes, like, safety, alignment, trustworthiness, control, a lot of, like, model model robustness, lots of terms for, like, very niche things in that in that category. But I also care about, like, economic impacts or epistemic impacts. Do you trust the information that you're getting and your ability to make decisions based on that information? Do you like, is are is the cyber physical infrastructure robust? If AI is totally aligned, but someone manages to convince it that, like, the right the moral thing for it to do or the best thing for it to do at this moment is to completely destroy the electrical grid. That seems like a bad thing. There are two points at least where you can intervene here. One would be to secure the grid. One would be to make sure the model doesn't go and do things like that. Lots of other interventions. I strongly believe in defense in-depth, which is this notion that you should try to defend at multiple instances. The Swiss cheese effect is another thing that gets flagged here that everyone got uncomfortably familiar with during COVID. But in terms of, like, economic risks, I think that there are probably a lot of things that people haven't started really thinking about. I do think that unemployment is going to spike at some point because I think that in the next few years, it feels pretty likely that that we'll get something that's like a drop in knowledge worker. I mean, the prices of the, like, AI subscriptions has been going up roughly an order of magnitude a year. And, like, what would you pay, like, $2,000 a year for or $20,000 a year for or sorry, per month for? $2,000 a month is, like, something that someone might pay an intern.

And, like, you should expect it to have at least that level of capability if they're going to charge for it because I think that market is at least pretty efficient. I like to flag that another possible category of concern is, like, I think that if AI automates legal work, then lawyers benefit. But if it automates driving, the truck drivers don't benefit. It's the probably the consumer that benefits on that one or, like, the firm that owns the the shipping companies. And so there are a lot of questions like this that I have around, like, what as AI gets better, what breaks? And Right. If something breaks, then, like, how can you either prevent it from breaking preemptively or, like, shift the equilibrium into into something better? Like, if if unemployment spikes by, like, 5% in a month, then I assume that in The US, there would be a UBI bill that gets written and probably passed. And I don't think it would be a very well thought out bill because it would be passed that month like, written that month. So I'm, like, very interested in trying to get people to start drafting versions of that bill and talking about it. It would not solve all the problems. One of the biggest problems that I think of is, like, one of people often say, like, dignity of labor and things like that. I worry less about that than something called the intelligence curse, which was a phrase coined by a friend of mine and his co author. And, like, the the common notion like, there's a concept in economics, the resource curse. And this is that if you have if your country has a lot of natural resources, you don't invest a lot in the population because that's not where you get your tax revenue from. You invest a lot in the, like, extraction of these natural resources. And the you don't really have a financial incentive. The, like, power structures of the country itself focus on, like, you need to secure the supply chains. And so you get a lot of, like, dictators from mineral rich countries. You might see the same thing with intelligence. If intelligence like, if you have a knowledge economy or a service economy and you replace all of that with AI, then, like, you probably as a government don't have a lot of incentive to support the people because the people have no connection to your tax basis or things like that. I also worry about, like, possible like, I I worry a little bit about monetary risks from inflation from UBI, but I honestly think that if we end up in that scenario that the growth of the, like, service outputs probably are so big that, like, you can really start insanely pumping the monetary supply without losing buying power. I'm a little bit I I'm MMT curious, so I, like, I don't worry as much about that. It may come out at some point also that I have strong Jourdist inclinations, which has impact on other things that I I think about these things. For those who haven't seen it, I would definitely recommend the book Blood in the Machine. Have you come across this one? Yeah. Yeah. I actually bought it very recently. I went at at CCC, actually. I bought the book. Yes. Yes. I I enjoyed it. I think that some of the conclusions drawn at the end about limitations on AI systems are incorrect, and I think that it is probably scarier and probably would have been made a better call to action if the author had shared more of my concerns about what AI will be capable of. But in terms of the like, I was I was shocked by how everything I thought I knew about the Luddites was actually the, like, post rebellion propaganda that everyone was getting. Yeah. Yeah. And also the fact that, like, a lot of the things that people are talking about now, like, attacks on automation or, like, absolutely discussed back then by these, like, specialist guild members who were prohibited from competing in many ways.

Yeah. Yeah. I I I think that it was, like, a very interesting awakening when I realized that, like, law is to human actions what software engineering is to information. It's just the best that we've come up with. And economics is actually the science of decision making, not the science of money. Like you're saying Theory, like, oh, shit moments in my in my improvement of the my of modeling the world.

I don't follow it. I'm too skeptical for for bad reasons. My one line take on quantum computing is that people who are very, very excited about quantum computers and aren't deeply familiar with complexity theory should probably look a little bit at, like, what is in BQP? What is in BPP? What might be in, like, other complexity classes that are, like, interesting and possibly relevant? That feels unintuitive to me is that, like, there are problems that AI systems are really good at now, like protein folding. Hamiltonian minimization is the name of the general problem that solves, and that is an NP complete problem, which means that if you have an oracle that can do that, you can map basically any problem that is in NP to that, and you have you now have an oracle for that other NP problem. I bet I'm, like, 90% sure that everything I've said is correct, but Claude would do a better job. But yeah. Like, there are reasons why that why you can't use AlphaFold as, like, a universal NP oracle, but, like, that doesn't mean that we couldn't build oracles for a lot of other things. Another thing to look at is the fact that SAT solvers and this gets into a little bit of formal verification stuff, which we can talk about later. On the formal verification side, it's, like, proving SAT is also NP. Like, set there's there's a satisfiability problem, which is, like, I have a bunch of ands and ors and variables together. Is there a set an assignment of their of, like, true and false to these that leads to the overarching expression being true? And solving this is also NP complete. And yet we have solvers that solve most of these in polynomial time. Not all of them, obviously, but, like, it it gets weird fast where you like, the problems that we care about are often easier. The halting problem is another one of these where it's, like, it does this program stop. Every programmer listening to this has solved the halting problem for, like, a finite number like, a finite nonzero number of problems. So it is solvable in in the cases that we care about, just not in the, like, arbitrary general case in polynomial time or no. In finite in finite time. Yeah.

I feel like it's kind of I also wanted to mention, if you like Pantheon and haven't read QNTM's, Lina or Emma Macedo, this is a very short story that's, like, the other possible end of a spectrum of I love science fiction. Pantheon is amazing. Would also recommend the Jean Laflambert trilogy by Hanu Raimini for, like, similar, like imagine there are uploads. I've heard great things about the age of m. I have not read read it yet. I would also strongly recommend oh, man. There was another thing. Oh, we are Bob, we are legion is another one of these fun, like, upload flavored ones. I think that a lot of a lot of fiction seems to unsurprisingly handle the notion of superintelligence really badly, and even, like, the TV show upload, like, doesn't acknowledge that, like, yes. If we can slow people down, we could also speed them up. Or at least I I haven't watched much past the, like, first season or so, but they like, I think that it is I think there's some, like, concerning back of the calculations of, like, yes. If you did in fact upload a person, like, someone tried to convince me that you could run like, depending on your model of the brain, run a human in a pretty good MacBook Pro. And, like, the weird thing is, like, if that is not true now, it might be true in four years if you're originally off by, like, a factor of 10. So, like, the the the brains are getting more complicated. At some point, a lot of these really weird things become answerable and a lot of questions of, like, what is identity? Who is me? Like, the teleporter paradox gets gets weird in, like sorry. It gets real in weird ways. And but this goes back to what you're saying about how, like, a lot of the philosophy questions seem to be getting more applied, Less than real.

I love the adage that well, I don't know where this came from. This might have come from me, which might be why I'm particularly fond of it. But it is easier to get it probably came from DavaDot. I feel like my my most clever things kinda came from DavaDot. It is easier to get a hold of a GPU than it is your anymore, but it is harder to build TSMC than it is a centrifuge. And so, like, there is a there is a natural choke point. If, like, if The US and China decided, like, hey. We think that a pause would be really useful. You could actually like, you can't do this with a 100% guarantee, but I put people to flexhegg.com, I think, is the URL. This is a proposal for a robust tamper responsive modification you could make to GPUs where you could have something like you embed some cryptographic keys, you formally verify the entire tech stack to prove isolation, and the fact that no one from the outside can, in software, modify these properties of, like, there will always be a there must always be a monitor that is signed by some authority that gets run, and then you have a blockchain out in the world that the various world like, national governments are, like, federated consensus participants in. And the like, every time there is an update to these classifiers, it gets signed by the multisig and broadcast to everyone, and these devices have to be able to connect to that blockchain every certain amount of time. Otherwise, they shut off. They have to be able to, like, run the classifier on every output from the AI. Otherwise, they shut off. They have to be able to, like, see and report and, like, reliably share metadata about any training run they're a part of. Otherwise, they shut off. And if you try and open the thing, the chips get shattered. Like, there are things like this that you could design and build and implement. It would take a lot of international cooperation. I'm not sure we're gonna have that. Right. It would take like, upstream of that, you need agreement that, like, this is a catastrophic risk. We are on the verge of losing control. Maybe there's what is what they call in the biz a warning shot. This would be a, like, recoverable catastrophe that leads to people saying, like, oh, we should do something about this. I think that it is I think that I vary on a lot of the, like, what is the most likely outcome from all of this. I think that doomer has become an interesting pejorative to just dismiss concerns generally, but I think that, like, it's reasonable to have a term for people who are, like, there is nothing we can do. We're all doomed. And so I so I, like, push Doomer over there, and I do not ascribe that label to myself. I think there's a lot of things we can do. I would point to Dean Ball and Zvi Mofowitz as, like, probably the two clearest voices in the space who are, like, very vocal and public about a lot of these things. They are not, for the most part I think neither of them is, like, is dumb in any in any takes that I have seen of theirs, which is a very high bar.

No. Dean well, Dean is a, like I guess he's a researcher, but he's a policy guy Okay. But also very, very good with with the models. He wrote he was the main author on the AI action plan. And Zvi is a very well known EA rationalist writer who does a lot of regular updates about the course of AI progress. And, like, I would they are obviously not in agreement about a lot of things, but they are very reasonable. And, like, it I feel like if we were in the version of America where I think it was, like, 20% of the population when it came out read common sense is the typical estimate by Thomas Paine. Like, I think that, you know, if we if America were like that, then, like, Dean and Zvi would be, like, the main two voices that any anyone is listening to, and there wouldn't be a lot of anxiety about the water usage from data centers or things like that because the the stakes are much, much higher than water.

I think that there's one of the biggest risks I see is, like, this mimetic attractor of you don't care about my problem, therefore you are wrong, or you're not you aren't putting my problem first, therefore you're wrong. And I think that, like, I worry about water usage and, like, access to clean water, generally. I'm not saying that that is zero concern. It's just not the highest concern. And I think that people who say it is, if they had the context that I have, would probably agree. And Right. Like, I think that they particularly, I think the political left, because there isn't a notion of returning to some idyllic past where all the problems were solved and, like, we don't need to, like, we don't need to go forward. We just need to go back to something that never existed. Would strongly recommend Ada Palmer's Inventing the Renaissance for those who haven't come across it. One of my favorite sci fi authors, even though it's a a history piece, which is actually her specialty. But there's the conservative, like, we're gonna go back to the golden age, and, like, let's reinvent the golden age for that. But the like, if you want to improve the world, there's a question of in what way do you want to improve it? And it's really hard for people on the left to build a coalition because there are a lot of different visions for what the future could or should look like. One of Sure. I I think that there are a lot of it's another Ada Palmer quote is that science fiction helps us or it enables us to fight the moral skirmishes that we will face as a society before we actually have to face them. Right. And I think that it is very good to engage in worlds where super like, to in the idea that a superintelligence could exist. I think that there are some good science fiction efforts around this, the ones I mentioned. There's also things like the culture series gets invoked a lot. Greg Egan's diaspora is another one where I think this is actually, like, a version of things going quite well, actually. Maybe maybe that one's the best. Like, one of my one of my favorite, like, things going well versions. It's not his most popular. People usually reference permutation city. I think that they're wrong. Diaspora is better. But also, like, if you if you read that and you don't like it, there's a glossary in the back. And also, like, I may have liked it for math y physics y reasons that other people might not like. Anyway, the I think there's a lot of ways this could go, a lot of thick concerns that people could have. I don't know what it looks like for humans to have influence over superintelligences. Maybe this looks like AI systems generate fiction for like, fictional universes that it could build us too. And, like, if you really enjoy Harry Potter magic or if you really enjoy, like, exploring space, then the fraction of people who engage in those universes changes the allocation of resources. But, like, there will only be a finite amount of compute for all time. Finite amount of energy being captured and allocated towards it. How does that get decided? Is it a small number of people? Is it the intelligence itself? Is it a set of superintelligences? Does do those last two matter in terms of the human experience? Why would why would the AI systems listen to people? And even if they did, how would you know? Why would you trust them? Right. Like, a superintelligence should be better persuading you that it's listening to you just like it's better at solving all of the other problems.

one of the greatest disservices that the media has done the world at this point is convincing people that intelligence looks like characters from the Big Bang Theory and the ability to, like, stand up at a black blackboard and recite facts. Yes. I hate the Big Bang Theory.

Yeah. It's a dumb person's idea of a smart person. Yeah. Exactly. I think that one of the more controversial takes I have is that like, I think that there probably are really influential genes for intelligence. It just seems weird that there wouldn't be. Like, your eye is sensitive to the polarization of light, and it's not because it's useful evolutionary. It's because it's really, really hard to make a polarization and sensitive photon detector. And, like, I think it would be really, really, really hard for evolution to make all humans exactly equally capable of grasping abstraction, which is kind of how I think of intelligence. But also, like, this is not something that should be I think that people shouldn't be branded as, like, being on one end of the political spectrum for holding this belief because the like, there are polar there there are two different polarized takes you could have about this. One would be, like, therefore, some people are better than others, and therefore, they should have more or something like that because they'll distribute the resources more effectively or something. You could tell how I probably infer how I feel about that take. Or, like, this is very important to know, therefore, we can help people who are disadvantaged, and we're actively disadvantaging them more by pretending, like, they are just as capable as everyone else. I also don't believe that, like, it's a 100% deterministic. I I think it only attribute accounts for part of it, but, like, this shouldn't like, it could be an info hazard. It could be that civilization right now isn't well equipped to know which of these genes it is. I worry about, like, monoculturing, like, brain phenotypes as a result of this in the short term. I worry about a lot of weird things. Right. Right. Right. But, like, I think that it's, like, I and when I say monoculturing in this way, it's, like, I think that I'm clever enough that, like, if I'd grown up in a family of lawyers, I probably could have been a decent lawyer. If I'd grown up in a family of doctors, I probably could have been a decent doctor. Like, that's transferable, but, like, I don't think you can take Mozart and put him into, like, Einstein's house as a baby and get a get Einstein again or get Mozart again. I think that that just doesn't work. Yeah. And so, like, there you need some amount of diversity in that to get the level of flourishing that we have now, and I don't want people to suppress that because they're, like, intelligence maxing or something. But, like, there's yeah. There lots of concerns in this. Lots of Yeah.

Yeah. I remember the the point that I was trying to make it to is I think that a lot of people think, like, going back to the they think intelligence is the big bang theory. They think that intelligence does not actually help solve problems, at least the real problems that they they deal with day to day. And I think that this is a mischaracterization. I think the people who are like, oh, like, I've I've done so much better than so many intelligent people. I can like, I'm not worried at all about a superintelligence. I think that the people who have that are actually just smarter than they think they are, and they have that raw intelligence, and they don't realize it. And they think that they are not intelligent, and therefore intelligence is not this useful. This thing's been useful to them. And, actually, it is a core thing that has made them much more effective. And Right. The fact that society has convinced them that this is separate has led them to fail to extrapolate what that looks like much, much better. Like, Robert Moses was, like, a a general like, an he was a natural general intelligence with a slightly higher clock speed and a slightly longer context window than everyone else. And he just wrecked all of New York governance controls in ways that, like, people should be aware of. And, like, I would point politicians to that as opposed to, like, 2,001 or The Matrix or Terminator when I say, like, this is what superintelligence can do. It's like, imagine if Robert if you had 10 Robert Moses's who could all coordinate with each other and we're all deciding to back one particular person. What if it's a 100? What if it's a thousand? Like, that Yeah. Yeah. Should hopefully feel more visceral.

anytime. Also, I mean, maybe I was maybe I was a bad example. Mistake. But yeah. No. I I think the point's in. There's there's, like, this notion of non non scalar intelligence also is very relevant to AI. There's a great, take from Helen Toner about jaggedness of AI capabilities and, like Right. Yeah. AIs are really, really good at some things. In fact, like, most of the things that I think about most of the capabilities I think about are, I think, things that basically no one uses, like cyber red teaming or, like, like, assisted bio design and things like this where, like, it takes some amount of skill with AI to be able to evoke these and knowledge of the field to be able to use them well. Like, my understanding is that at the Frontier Labs, humans don't write any code anymore. And, like, this is something that seems like it should be having economic implications soon, but I think it'll take a procurement cycle or two. I yeah. I like, people don't if if the procurement people don't understand that this isn't just a tool, this is actually, like, an employment like, an employee real replacement, then, like, they're going to slow this down. So I think that there are bottlenecks, but, like, there are also definitely things that AI systems can't do. And some of those things are things that people are much more likely to ask AI systems to do, and so they can be much, like, justifiably skeptical. But I think that the, like, the the radio chart of, like, where is it better than humans? Where is it much, much, much better than humans? And where is it not as good as humans? Like, it is good to understand intuitively more than, like, trying to say, like, oh, yes. It we're not at an AGI yet, or we are at an AGI now. Like, the those terms just stop holding meaning.

AI and in ways that, like, seem deeply irrational to me. I do think that there's a connection to, like well, I ended up thinking about formal verification because I think that it is a very useful construction for a lot of different purposes. Formal verification is this notion that you have code and you have a specification that is a mathematical description of how the code should, like, mathematical or logical description of how the code should behave. And then the there's a mathematical proof that takes the definition of the programming language as the, like, starting axioms and then proves that the code has these properties. This is what's used for the most, like, critical of software infrastructure for a lot of things. Like, you can prove parts of the stability of aircraft systems, and that's why you never hear of, like, oh, yeah, there was a bug in the code. You hear, like, oh, there was a problem with the sensor. Because, like, obviously, if the sensor's broken then the code is gonna misfunk malfunction, but, like, you formally verify a lot of the critical systems so you know that, like, if, like, two trains in France are not going to crash. This is another, like, prime example of where it gets deployed. I think that you could, hypothetically, generate a system where if you have, like, widely deployed flex hex, as I mentioned earlier, you could imagine having something that looks like a bill of rights encoded into AI systems so that before they take any action, they have to prove that their action is, to the best of their understanding, consistent with that bill of rights. This seems like a delightfully, like, like, western liberal direction for how one might govern AI. It requires an a separable auditable world model in which the AI can test, like, does this, in fact, do this thing? But, like, it was I had I came to this with some delightful conversations with Mark Miller of Agoric about who, like, takes some very strong libertarian views on basically everything, And we landed on like, yes, this would actually be really cool because you could embed you could imagine embedding in AI systems a something that is as fundamental as the laws of physics are for us, but because they all exist in compute, if you can put this into all compute, you could put this into all of the compute that the AI systems generate in a, like, reflection on trusting trust ish kind of way for those who get that reference. Like, you could just embed this and it would would propagate in a useful way, and you could have constraints on, like, you should value, like, honesty. You should, like, do not take actions where the transmission of withheld information would lead your interlocutor to, like, have certain responses, including changing their mind about collaborating with you kind of thing in in ways you could start defining. This is this starts getting very far from formal verification in that, like, specifying those things objectively in properties that you can describe in the simulation sounds completely intractable today. But you could start thinking about what does it look like to like, what do things in the interim start looking like? And between here and there, we could have mathematical like, logical descriptions of, like, the building code, tax policies, legal contracts. A lot of these things are like, the tax policy, as a great example, is software. Like, the canonical implementation of the tax code is running on a server somewhere. For The US, I understand that is not running on a particularly well updated operating system or language because it still works. But, like, that's actually what the tax code is, plus all of the infrastructure and, like, the institution around it that lead to modifications thereof. But it's, like, it's not actually the law, and it would be really useful to be able to formalize all of those things into something that looks like software or at least property like, pre and post conditions of, like, if the input satisfies these properties, the output will satisfy these properties. That would be an example of a spec. You could have specs for all of these things and you could have, like, one click check. Does this meet the building code? Does like, is this the correct tax? And we could have all of these I'm actually working on fundraising for a focus research organization at Convergent Research that will build tools to help people validate. Is this the correct formalization of this math theorem or this software property? Or it could apply to, like, this hardware property or this piece of the tax code or this piece of the contract. Like, help me understand the difference between this and a variant I might generate. Why would it matter? When would it matter? And I think that we could imagine a world where we have AI systems that we govern not by, like, saying this is the correct set of virtues, embody those virtues, but rather here's a set of laws that are we consider fundamental as, like, ground rules. Do not break these. Whatever else is fine. And I think that there are versions of this that would feel much, much better to most people who have grown up in, like, western liberal democracies.

Within your science, actually. But, like, you you end up with, like, some formal logicians from the the math side. And, yeah, it it's, like, trying to trying to make things very explicit. It gets used in a lot of places. I think it should be used in more places. It's been mostly limited by the fact that you need people who have deep expertise in formal logic to learn these languages and then to generate these proofs, and all these things are very hard. They require time and expertise. And, thankfully, time and expertise are two things that get much cheaper as AI gets more capable. So I think that we're seeing a lot of really cool things happening. There's a a different focus research organization. The and, like, FRO focus research organizations or FROs, are these, like, 10 to 20 I'm sorry. 10 to $50,000,000, three to five year research efforts to create public goods that are too big for university lab, too small for, like, a big government center, not profitable enough for, like, private industry, very startup flavored, but it's like, the the goal for each one is to build a product that seeds an ecosystem that drives a scalable revolution. And Convergent Research is, like, I I wear a hat there as a research fellow, and I do so delight delightedly. And they've created 10 of these FROs, I think, and are maybe maybe more at this point. And there's a lot more interest in those. And each of them, like, you check the conversion research website, each of them seem poised to revolutionize, like, single cell proteomics or connectome map like, brain connectome mapping or, like, intergalactic astronomy and things like this. And, like, they're very cool. I am trying to help them build things that are useful for AI resilience, AI security, etcetera. And there's another one that it built Lean, the proof checker, which has been getting a lot of attention in a lot of, like, software security a little bit. Right. But definitely the math world a lot. And I'm, like, very excited about some of the progress they're making where they had Claude generate, like, an instance of the compression library in lean entirely and prove that unzip of zip of file equals file for all file, which which would be an example of a a simple spec that you would want for your compression library. So, anyway, there lots of cool things. Specifying things is very hard. And, like, you never know you specified the right thing. And if your model of the world is wrong, you've also not, like, there will be gaps. But I think it's it's much closer to how we much closer to the, like, legal form of how we deal with society. We don't point to, like, that's the most moral person, be like them, which is what alignment is. It's more like follow the ground

Yeah. I I love that, like, in the original context, that was not that, like, like, software as physical law, but rather Yeah. Yeah. It was intended as, like, software as, like, legal legal code. Like, software code is legal code. And, like, I think that that one makes a lot more sense than, like, it makes sense to try and optimize the checking of these things. If you can put more effort into saying what you want, then you don't have to check to work as hard to know if what you got was what you wanted.

apologize that I have to rush it a little bit. I think that as intelligence gets cheap, that agreement or consensus gets very expensive. And I think as software becomes cheap that, like, the moat around a lot of software infrastructure and information processing generally disappears. And I think that having consensus, having agreement, making that agreement quantifiable and objective and verifiable is very, very useful. A lot of the like, I look at lots of risks, and I try to map out what could be done. And, like, blockchains do show up maybe surprising amount to a lot of the people as, like, possibly the best way to have this because you can start reasoning robustly about security models. What at least one of the ideas like, one of them I mentioned. Another one, I am hesitant to say because I'm not sure it falls nicely on the line. I I think it becomes very good in some scenarios and very bad in other scenarios. And so I, like, don't wanna put it too much into the water if I don't know which scenario we're in. I think that they're like, a lot of the questions around labor and capital become very, very interesting. And I I expect that people who have thought very deeply about implications as we move into making more labor synthetic will be very interesting. And there are a lot of lot of interesting discussions around, like, a permanent economic underclass being a possible outcome. And, like, it's very unclear what it's very unclear what good even looks like, but also very very unclear how we would go and get there separately.

Yeah. Very much appreciate it. I would look forward to another conversation very much. I would say that our efforts are currently to try to scale the field strategy work that I've been doing of, like, making it very, very easy for founders to pursue new efforts. That looks like a combination of, like, take going from a problem to a plan that has been somewhat derisked with buy in from experts and the potential, like, users or stakeholders. And I'd like, the rough skill set is, like, can you go into a meeting with, like, a Fortune five hundred director and convince them that this new solution is viable and that they should agree to spend time possibly engaging in collaboration to address some market failure that would lead to a better outcome for society or civilization more generally. People who can point at a thing and say, this would be very useful. No one has done this, and I can tell you with confidence because if they had, one of these seven people would know. And I've talked to all of them, and they could all they all agree that no one is doing this thing. I it's a like a weird skill set that shows up in, like, startup product market fit or, like, being an entrepreneur or being a, like, head of strategy at a big company, or like sometimes in consulting, often in research. I assume it shows up in, like, the intelligence community. A bunch of weird places for different fields. If you as a listener are very, very interested in doing that and worried about how AI will break society in the very near future, particularly for cybersecurity risks or biosecurity risks or maybe epistemic risks, I would love to to hear. You can find more information at atlascomputing.org. And then, also, we're somewhat funder funding limited. So if people are sitting on a, like, large capacity to donate to a US Five Zero One C Three, we'd love to be we'd love to outreach in that for that too.