What the Internet Archive founder predicted 50 years ago | Brewster Kahle (DWeb Camp)
The Blockchain Socialist | 2026-10-05 | 33:51
Brewster Kahle started planning the Internet Archive (creators of the Wayback Machine) in 1980 at the MIT AI lab, when he figured that machines worth talking to would need a library to learn from. Forty-six years later, he says the biggest obstacle to that future is a handful of companies that want to control information instead of selling it. We recorded this conversation live at DWeb Camp 2026 outside Berlin. We talk about where the early web went wrong and what Europe's copyright carve-ou...
Top Keywords
- internet archive 0.011
- archive 0.009
- publishers 0.008
- make 0.008
- libraries 0.008
- internet 0.007
- library 0.007
- nonprofit 0.006
- build 0.006
- infrastructure 0.006
- work 0.006
- publisher 0.006
Transcript
Speaker 0
0:00 – 1:07
But it had a couple of basic flaws and we're really seeing those to this day. One is we never built in a money mechanism to try to get people paid and so we ended up with the ad model which tends towards monopoly. Big problem. And never trust a publisher to do a librarian's job. They're just bad at it. They're going to manipulate it. They're gonna change and delete it, and they do. And they do it at scale. And the biggest holdup we have now in the AI front is actually these monopolies saying, no. We're gonna control everything. We don't want libraries anymore. What libraries do is they collect, preserve, lend, and interoperate with other libraries. Those guys are all about control of information, and they work extremely well with authoritarian governments because they sort of think in many ways the same way, and they're structured in many ways the same way. How can we mail a decentralized system that actually works to get some people paid and some independence and control over what it is they're putting out. That's the only way to make a free and open society work. We now have actually a live podcast
Speaker 1
1:07 – 1:50
recording, which is pretty cool. This is the All Models Are Wrong podcast, hosted by Joshua Davila, in conversation with a person who in many ways started all of this, Brewster Kahle, founder of the Internet Archive. As a young computer scientist, he dreamed of a library of Alexandria for the digital age, and forty years later, that dream is core Internet infrastructure. The show's name comes from the idea that no model is ever fully right, only useful. A fitting frame for a talk about holding fast to a vision. Please welcome Joshua Davila and Brewster Kahle.
Speaker 2
1:55 – 3:41
Awesome. Thank you for the introduction. That means I don't have to do the introduction. We can just get right into it. So we're here today at the Web Camp 2026, just outside Berlin. This is the first time that they've done it, I believe, outside of The United States. I think it's really interesting that this is happening during a time, during the rise of artificial intelligence. The technology has improved quite a bit since the past year. And also Brewster was one of the, employees of one of the first AI companies, I guess, called Thinking Machines, not to be confused with the new company called Thinking Machines started by one of the OpenAI cofounders. As part of this, I kind of wanted to contextualize what's happening right now within the long arc of history since, you know, we could say from 1996 when Internet Archive started or maybe even back in the eighties when you were working at Thinking Machines of where we are today and how we've gotten here and kinda where we're going. And so kind of one of the things that I noticed about your work when I do my research is that, like, when I think about what's happening right now and then I think back thirty years ago, because now I think thirtieth anniversary of Internet Archive, the thing that's really striking to me is, wow, you must have, like, you must have thought about the future with AI and what that would look like and the kind of types of civilizational institutions and infrastructure that we would need in preparation for that future. And so what I wanted to do is start off maybe bringing us back thirty years ago or however many years ago you want to go back to the young... Younger Brewster, founding Internet Archive and kind of what was going on in your mind during that time,
Speaker 0
3:41 – 3:49
and, like, the history that got us here. Let's dial the way back machine another another couple steps. Sure. So 1980.
Speaker 2
3:49 – 4:01
Sorry. I don't understand. The AIs want to join the conversation. They're they're omnipresent now. Make sure you don't say anything that angers them.
Speaker 0
4:05 – 11:45
So back at the AI lab at MIT in, in 1980, I was looking for a project of going and being a geek that would be useful. And the ideas... There was... We had been promising the Library of Congress on your desk for decades by then. And it's like, how hard could this be? Right? We just go and and and either just burrow through it and digitize it. But we thought, okay. What is gonna be needed to be able to make that come true? We knew even back then that we're going to be making machines that were gonna be... It had to be interesting to talk to, to have them learn enough to to be, useful to be around. So we thought, okay. And they would have to know a lot. And so I thought, okay. That sounds like a great idea. Let's go and build that AI future. And, well, first, we're going to need to have computers that don't suck, and so we built a connection machine supercomputer. It was the first real search engine, but it was be... While we're still in the ARPANET era. Then we needed the ARPANET to transition to the Internet, and we needed to go and get, a free and open source software because we'd screwed up the copyright system. And that sort of... That was in the early eighties. And so I was not the leader. That Stallman was, but I was there with him to try to go and try to get this stuff to work. Then we needed to have open protocols, to make this all go. We had to get, these new personal computers that were coming along that had operating systems that didn't work. There were single thing that you could do on a Macintosh that, you know, blah. So we had to try to get all of these pieces, going. Then we had to get people publishing on the net because the idea of of going and just, say, buying a library and cutting off all the bindings and just digitizing it seemed, kind of shortsighted. We wanted to try to get the new publishing system going. So I put the New York Times, Wall Street Journal, the the first publishers on this Internet thing, which was kind of a military, research project to try to get it so that people could make money by publishing on the net and try to make everyone into a publisher. So I did a system before the web, called Waze, to go and try to make it so that everybody would be a a publisher. But Tim's system worked a lot better. It was a lot simpler, Tim Berners Lee's World Wide Web. And so we kind of all piled in behind that because it was an open protocol, open system. But it had a couple of basic flaws, and we're really seeing those to this day. One is we never built in a money mechanism to try to get people paid, and so we ended up with the ad model, which tends towards monopoly. Big problem. I'm a make... Much bigger fan of royalty systems. It's how the book publishing industry solidified in Europe in the early sixteen hundreds. But we didn't get there or we haven't gotten there yet. Crypto was supposed to help and then, that's been having real trouble, based on regulator problems. So, we... But we got a bunch of the pieces. We got a bunch of the pieces. The World Wide Web worked, but one of the problems of the World Wide Web is unlike publishing in the past where you'd... A publisher would go and make copies and and sell them or give them away, and they would be in people's collections or in library collections, The, in the World Wide Web, there's only one copy. It's got one URL. It's in one place, and it's changed or deleted within a 100 days. Things would... It's kind of a problem there, because that's putting way too much control in the power of that one publisher and it tries to force them into being an archive or a library. And never trust a publisher to do a librarian's job. They just bad at it. They're going to manipulate it. They're gonna change and delete it, and they do, and they do it at scale. The other thing I got wrong was the, monopolization of things, the drift towards concentration that is now allowed and encouraged by governments that, don't have, it... So we now have monopolies, basically being built in every field. And so we have media consolidation that is making it so that we're having to refight the same fights over and over again, where we had to fight AOL to go and build a decentralized, distributed interesting system. We won that one. But kind of they're back. Right? Do we have Facebook? We got we have, Twitter. We have, YouTube that goes and sort of dominates whole media types. And this is not the way you can make things diverse and interesting and, working. So, where the Internet Archive is the latest phase towards building the datasets for AI, we have this counter trend towards monopoly. And the decentralized web work of... Sort of started really for us ten years ago to try to get the decentralized web to happen, was a hail Mary to try to counter the monopolization forces. And, that... It's, you know, got 500 people here in Germany in its first year, indicates that there's real interest in trying to build an alternative to the to the, future that we're being sold or forced fed, of this monopolized structure, which doesn't work very well. It's it's not a very innovative, interesting system. So long arc here, I started out trying to build a library that's smart, that would be a global brain, that would be a thing that would help people make better decisions, where the technologies have been unfolding. It's not like AI happened in, you know, 2023 with, the the the OpenAI thing, though that's awesomely cool. You know, Google has been basically AI for a long time. How can you type a couple of words and end up with good documents out of billions? And that is freaking amazing. And they've been basically tuning all of that technology for a long time. They're not widely deployed, but now we're having some of these new chat systems, which will... Is that kind of, can you talk to the library? And the biggest holdup we have now in the AI front is actually these monopolies saying, no. We're gonna control everything. There... We don't want libraries anymore. What libraries do is they collect, preserve, lend, and interoperate with other libraries. That's what libraries do. Four things. And the publishers say you're not allowed to collect, you're not allowed to preserve, you're not allowed to lend, and you're not allowed to work with other libraries. You all have to deal with me. I'm gonna surveil everything. Everything is going to be, paper drink or I'm gonna control everything. And as a top lawyer at one of these media conglomerates said, it's not about compensation, Brewster. It's about control. So that's the world that we're, living in, and DWeb is essential towards building a different view towards how do we go and make it so that everyone's a publisher. Some of us, will be, paid for that work, and we don't have to work for media conglomerates to be able to be a good reporter. That those are the systems that we need to have, a a work long winded question... Answer to a very short question.
Speaker 2
11:45 – 12:05
That was, that was great. I think the the thing that comes to mind for me at least is that it seems like Internet Archive plays in a space perhaps in the middle between publishers and big tech. So I think there's, like, been a lot of, you know, over the years, battles between publishers and big tech where, at least my impression is that they're not,
Speaker 0
12:06 – 13:48
a... No. No. No. No. Come on. Come on. They're the same. Right? So so so the publishers and big tech are the same type of people. They think the same way. They try to carve out and own particular things. Totally. And even if you're a new distributor like cable, then you end up owning the studios. Now we're having the the, Reddit is indistinguishable from a publisher. Amazon bought MGM. So the... All of... Those those guys are all about control of information, and they work extremely well with authoritarian governments, because they sort of think in many ways the same way, and they're structured in many ways the same way, and having a fewer, so in The United States, we're starting to see the merge, right, of of our big media companies with, alignment with a political party. So we're, so I'd say the... One of the brilliant, things of the Internet was the definition of the ontology of organizations into .com, .edu, .gov, .mil, and .org. We think differently. We are different. We we wake up in the morning with a different success metric. If you're in the military, you think success is very different than if you're in a, educational institution or in a .com, or or organization. So I would... No. Don't don't just... This this false dichotomy of publishers. No. They are the same people, and they are, they basically think the same way. They're all about control of information.
Speaker 2
13:48 – 15:02
Sure. Yeah. I guess the the way that I was thinking of it was more about, like, how they kind of, like, usurped into each other. There were, like, early battles between them, and they've kind of... They found a compromise between each other, to some degree or bought each other out or, you know, something happened. But I guess the... So the the the direction I kinda wanted to go with this question a little bit was now... You know, right now with the expansion of artificial intelligence, when you say, like, I I think the the the motto for Internet Archive is open and universal knowledge for all. Like, is... Was the thinking thirty years ago, like, the fear around the, the inability to share knowledge freely with the growth of these types of companies, and that there needed to be... Like, the the point of Internet Archive was to provide, like, a a more, truthful, like, database of knowledge than what maybe you could get, either because of authoritarian governments or because of, like, a paywall behind, like, corporate databases of of knowledge. That makes sense.
Speaker 0
15:03 – 20:27
I I actually... It shows my naivete. I I didn't expect the monopolization. So the the library system, funds about 20% of all of the book publishing done. Right? So it's it is the fun... It's a socialized support of local production of knowledge. It it it keeps... Traditionally, it kept local authors, local publishers, local newspapers all going. It was how we socialized and funded information production back in the old days. Right? You know, early. Right? I've got gray hair here. So, and so it... It's... It bought one of everything. It kept things around. It it it was an infrastructure that was... The publishers would go and make the new thing, and then they'd sell them to libraries or individuals. And then, they would go off and work on the next one. That's not the way that, these companies think about themselves and they're threatening libraries all the time to make sure that they don't collect things, that they can't get things. They don't sell anything anymore. So it used to be that you actually would sell a book and a library would own it. But an ebook, no... They... None of the big publishers will sell an ebook. They're they're... They don't even let the libraries... They don't trust the libraries to have the bits. They basically only allow their, patrons for money to go into their surveillance oriented ebook lending systems. And those... You better believe those are surveilled. Those those companies are now starting to call themselves data analytics companies and not publishers. So it... It's really changed, right, how this whole, ecosystem has worked. But the library system, I think, is a good one. And I... We... If we didn't have this monopolization, we would have local news. We would have local authors. We would have authors actually paid somewhat reasonable amounts of money, but it all stops with these middlemen layers that are getting very good at standing in the way. But we still have to continue on. Right? We still have to learn what our world is about. And wouldn't it be great if you could pay somebody, for their song and know that 80% or more of the money that you go pay would go to the freaking musician? Wouldn't that be awesome? It... It's just not happening. It's all sifted off. And so, we we... Oh, well, it's an honor to go and be on Spotify and not get paid. And it's like, no. I... It's like, what what what... How can we mail a decentralized system that actually works to get some people paid and some independence and control over what it is they're putting out? That's the only way to make a free and open society work. Otherwise, we end up with a kind of Soviet centralized control, company system, and and control economy. Anyway... Right. So, but where are we? We're at the point where we have some of the AI technologies that are kinda great. Right? We can actually talk to these things, if we can be allowed to build them. Europe has done something awesome. They made a carve out in their copyright, law called article three that if you're a collect... If you're a cultural heritage institution library or a research organization that you can go and let the robots read. And you can let the robots read to do text and data mining as long as it's scientific research. Awesome. Let's build public AI in Europe. This was the... Let the robots read is what The United States said that allowed search engines, to happen and what we ended up with, a .com boom in The United States that didn't really happen in Europe because the publishers didn't allow it. But there's a carve out now in, in Europe to be able to build public AI. And I think this, opportunity to build public good AI. Let's build climate GPT. Let's build health, use these new tools to help with health care. Big questions, big problems that we have. And not just selling shoes or whatever it is the ecommerce based search engines or AI companies are gonna wanna, push on us. But but the big hard problems of education, how do we make it so that Europe... If you're a a parent in Europe, and you know that your kid is gonna be brought up by these large language models. Right? That's gonna be a major part of how the education system going... Goes forward. People 40 largely got brought up in the search engines, and we ended up with, well, one, really, and it wasn't European. So are we gonna do that again in Europe? Is it... Is the choice parents are gonna have to have their kids brought up on a large language model that has an American accent or a Chinese accent? Is that the is that the, the real opportunity? I would say not. Let's go and build things with European values in Europe, and have it reflect Europe. Let's have Africa reflect Africa. Let's have a more decentralized world that allows us to go and carve our ecos... Our information ecosystem to suit more than just the commercial interests of a few monopoly players that are probably thousands of miles away and that don't have your interest in mind.
Speaker 2
20:30 – 21:10
Well said. Yeah. I think this kind of... I think you kind of, brought into my next question is about, civilizational infrastructure. I guess the the way that I would frame Internet I... Archive, I mean, we've been kind of blessed with its creation. I think it's something that, yeah, I mean, clearly, states or governments didn't create something like the Internet Archive, and it required, you know, someone like you to actually make it come alive. What are some other types of, I'm using my term, civilizational infrastructure, that you think we ought to be building right now?
Speaker 0
21:12 – 25:11
Oh, cool. Civilizational infrastructure. Well, let's take some of the preconditions that allow something like the Internet Archive to have been founded, or Wikipedia, is a freedom to go and build nonprofit infrastructure, that does not just advocacy, but but real infrastructure. So, in The United States, we've got nonprofit, universities. We have nonprofit hospitals. We have nonprofit... A lot of operational infrastructure. So it wasn't that odd in The United States to build a nonprofit library that was independent from the, from the government. In Europe, there's not as much of that. A lot of it's really within the government, infrastructure. So I would say we need room to go and experiment with new infrastructure without the top down nature of of of government institutions. So I think the nonprofit structure, I think, is really important one for in... For cultural, moving, for civilizational moving forward. And I'm gonna surprise you with one. I'm gonna suggest another piece of civilizational infrastructure that we need is capitalism. Very spicy. Very spicy. I bet you did not expect that. I I... Okay. So let me define that as, buying and selling things. Like, having large numbers of small competitive players. Like, I still like the mom and pop, restaurants. Right? Those are really awesome places. Right? Unlike, say, the franchises. Right? I'm against monopolies and these large score... Large scale corporate structures. They make for homogenization and blandizing and horrible lives, and you don't wanna work there and blah blah blah blah blah. But having large numbers of of small players, sort of the Adam Smith for a version of of capitalism, where you actually buy and sell things. We're being told now that you're not allowed to buy and sell anything. Everything comes with a license, a terms of service. You don't own your phone. You don't own your car. That's a connected device. Every time you tap on the brake of a Tesla, it's reported back home. This is awful. You... You're not allowed to repair your tractor. You're you're... You know, all of this, tanglements, of of sort of you're just a serf. You're just allowed to go and use the land for something. We know what that was like. It was feudalism. We had, we basically had the, enlightenment was kind of some breaking the shackles of that. When I try to think of the freedom that Europeans thought they were getting by moving to the new world, it was trying to get out of all of these obligations. That was these these tangled obligations that basically made it suffocating. And that was the sort of like a brand new start. You can get your own land that you own. Okay. It came from Indians. It was a problem. So, but the idea that we can go and have commerce happen, small scale competitive commerce, I think is a great, a great thing to have happen. So I would say the nonprofit infrastructure is is a way that we can build international organizations that function well and outside of all of the constraints, within the government structure, but are still in the public good side. And then let's build lots of small organizations that buy and sell things. On the Internet, let's try buying and selling files. Let's not just do this license garbage. It's the only way I think we're gonna make a decentralized work, world prosper.
Speaker 2
25:13 – 26:31
Interesting. Yeah. I think, I think it's funny that you mentioned Adam Smith. I think, you know, if you take many of his quotes from his original text, many people... You would you would think he's a socialist based on that. So the things he he said... I mean, the neoliberal era, I think, was one where it sold ideologically commerce, but then in the end, what it actually did in practice was, concentrate capital in very few players in practice, because we lost the public institutions that helped facilitate, like, more small scale competitive markets, so to say. One thing... This is one of the last things I wanted to talk to you about because I think this is a a part of Internet Archive history that maybe not so many people know about, but I think is really fascinating, is that at one point, you tried to actually make a credit union Yes. Through the Internet archive, or with the Internet archive. Could you explain a bit what happened during that that journey, and what were some of the the learnings from it? Because I think part of the question that we have to answer for ourselves, is not just the the knowledge part or the sharing of information, but also, like, the financial sustainability of this entire infrastructure and, how we buffer ourselves from the more extractive forms of of finance capital.
Speaker 0
26:33 – 27:07
Yeah. Well, there's a... This... The great financial crisis, 2007, 2008, and things just, went went really wrong. And it was interesting to see how it went wrong from my perspective. Yes. This... The, there was a monopoly structures that were unleashed in the banking industry that screwed up the housing market, and there, there was a there was a crash. The government went and bailed out the banks, didn't bail bail out the the, the homeowners. Shame on them.
Speaker 2
27:08 – 27:12
And so I... You could say the the model that they had was wrong.
Speaker 0
27:13 – 31:54
Oh, yeah. I mean, I just... You know, you're looking back on it. There's... You know, we bailed out the wrong people, and we basically ended up taking away houses from an awful lot of people. And they ended up in Berkshire Hathaway going and buying up a lot of single, family houses. And, you know, we have just the rise of private equity. We have the rise of billionaires. There shouldn't be billionaires. So we have this, situation that was really sort of taking off with this financial crisis. So we pulled, our our employees. There was a, a a well... A shovel ready sort of welfare to work program in San Francisco that they would basically cover the direct expenses, if we cover the indirect expenses of hiring welfare people that had been basically just put on. So people with kids, poor people, and so we started scanning books, for the blind and dyslexic. With that help from the government, we just said, okay. Let's go and and and do part of our mission of of supporting the blind and dyslexic at a big scale. So we hired a 140 people. If they can guarantee that it would stay around for two years, I would have hired 2,000 people. But they couldn't guarantee it. And in in fact, one of the parties went and filibustered to go and make sure that all the money was clawed back, and, they made... They they took the money away. Awesome people. Anyway, so we pulled our our workers to go and find out how were they doing, and the answer was badly. And, the the reason why was rent. And, so trying to figure out 40% of their income was going to rent. This is stupid. And so, it's, how do we... Can we go and get around this? We're smart people. Is there some way to go and and make this, not work? Because... Look. I I raise money for for an organization, and about 80% of it goes to people. And then if 40 or 50% of that is going, to rent, then, basically, I'm paying... I'm raising money to pay rent. And where did that rent go? It turns out it's not taxes like you hear people. Taxes. It's like, it's not. It it, where it's going is mortgage, on that apartment building. And about two thirds of it, half to two thirds of the money goes to paying mortgage. So I'm raising, money to go and put into Wells Fargo and Bank of America. It's like, oh, I don't wanna be doing this. Okay. So, you know, reading David Graeber right there, you know, the Bitcoin thing came out at the same time. So all these sort of like, okay. What do we do about this? We said, okay. Let's make a new banking system. And, let's go and start credit unions at a velocity. Let's see if we can start 3,000 of them, because they're member owned nonprofit ones. And so I thought, okay. Let's make a model that can be replicated, over and over and over again. And, to basically help people raise, make money to go and buy houses and get rid of the debt and keep the debt off. Longer conversation. But, anyway, so we made a credit union, with a banker friend of mine who is sick of those people. And so we said, let's go and do this. And, what I didn't realize is how against credit unions the banks are. The banks have, have basically put together... In 1972, they put together the National Credit, Union Administration as part of the United States government to slaughter, credit unions. They they were started at three three hundred to 500 a year since the New Deal, which is, the world that I sort of grew up with, up to 1972. 300 to 500 new credit unions every year until 1972, and it's been going down at three to 500 a year ever since. They can only shut down a credit union a day. That's sort... That's all... So... But that's what they are all about. And, so I was going into a diseased area that was controlled too much, and so they really didn't allow us to do anything. And so we we we we shut down the credit union. I think there's still real reason to go and try to build alternative financing structures for people to go and take control and de financialize the housing system. This... Just The United States, we don't have social housing like the the wonderful examples we have in The Netherlands, and other places. United States, we don't have that. But can we build it in a nonprofit structure that's out... Outside of the hands of one political party which would destroy it every time they're in power? So can we go and build definancialized housing was the idea, and I failed. In terms of, you know, big sort of face plant failure in my career, my credit union was was definitely one.
Speaker 2
31:56 – 32:00
But this is something that I think, probably still worth pursuing. And if you're interested in that,
Speaker 0
32:01 – 33:08
Absolutely. I think there are ways that we can pursue this, but just don't actually go through the strict banking system. Start to do lending outside of that system. And we have enough money these days to be able to do a lot of this. We are... We did make foundation housing. Foundation housing is to go and make nonprofit housing for nonprofit workers. There's 12% of American workers work in nonprofits, and they're completely screwed. So they're, like, if you work at EFF, you're commuting in from Oakland. If you're if you're a teacher or a policeman or whatever it is in San Francisco, you can't afford to live in San Francisco. Can we make nonprofit housing with... For nonprofit workers? We bought an apartment building. It's working. It's a model. We can go and do socialized style housing in The United States outside of the government infrastructure. It's doable. We've, we have at least a demonstration. It's only one data point, but it's working and it's worth, sharing some of this idea. You guys in Europe have got, more functional governments, towards... In some of these aspects, so you may not need it as much as we desperately needed in The United States and other other countries in the world.
Speaker 2
33:09 – 33:37
Cool. So I think if you're interested in, decentralized cooperative banking, check out bread.coop. QuickShill is my project. Okay. So this is the first episode of the of the podcast, All Models Are Wrong. If you enjoyed this conversation and you want to listen to more of these types of conversations, allmodelsarewrong.online is a website I set up yesterday where you can subscribe or scan the QR code if it's easier for you. Yeah. Thank you guys for joining us.