S.1 Ep. 13 Impact, Evidence, and the Future of Web3 Grant Governance with Mike Cooper
In this episode of Governance Futures, hosts Jamilya and Eugene speak with Mike Cooper. Mike Cooper is an experienced social scientist with 17 years of leadership in impact strategy and measurement for numerous multilateral, bilateral, and other social impact groups, including the World Bank, various UN agencies, USAID, FCDO, MCC, and others. He specializes in the curation and use of evidence for decision-making in decentralized systems. He is currently working with Metagov on creating standards for impact planning and measurement. Mike brings his background in international development to the challenges of Web3, exploring how impact should be defined, measured, and planned for in decentralized ecosystems. The conversation covers the culture of Web3 grant programs, the pitfalls of vanity metrics, and why problem definition must come before funding solutions. Mike shares insights on how decentralization does (and doesn’t) correlate with impact, the importance of creating an “evidence commons” for governance experiments, and lessons Web3 can learn from commons management and collective action models. The episode closes with Mike’s one-word vision for governance: variety.Timestamps: 00:00 – Introduction and hosts’ reflections on Web3 grants 04:52 – Mike’s background in international development and impact framing 07:13 – Defining impact: problems, strategies, and measurement 10:24 – Grants as marketing vs. solving real problems 12:36 – Web3’s potential as a transformational tool for social impact 14:14 – Lessons from decentralization in international development 16:22 – Culture of Web3 grants and gaps in transparency 18:05 – Comparing Web3 grants with traditional gold standards 20:23 – Emerging standards and the role of Metagov’s Grant Impact Handbook 22:04 – Why decentralization doesn’t guarantee impact 23:41 – Governance paralysis, inefficiencies, and planning gaps 25:36 – Performative decentralization vs. honest centralization 27:32 – Experimentation, evidence, and governance design 29:58 – Outputs vs. outcomes vs. impact 34:08 – Network growth vanity metrics and flawed assumptions 36:23 – Problem definition as the foundation for impact 38:04 – Measuring long-term impact and sustainability of projects40:00 – Developing the Grant Impact Handbook 42:25 – AI, mechanisms, and knowledge translation in grant governance 44:01 – Mechanism libraries and evidence standards 46:24 – Building an evidence commons for Web3 50:12 – Cultural and organizational hurdles to adopting evidence use 52:11 – Incentives for grantees and grant programs 54:29 – Funding pressures and bull/bear market dynamics 56:26 – Leadership, hierarchy, and who drives impact culture 58:44 – Capital allocation’s role in ecosystem success 01:00:12 – Learning from mistakes and failure festivals 01:02:12 – The case for an evidence commons in Web3 01:05:50 – Champions, culture, and incentives for evidence use 01:08:14 – Toward performance standards and shared learnings 01:10:11 – Quiz: commons, principles, decentralization, variety 01:11:12 – Closing thanks and outro
Top Keywords
- impact 0.015
- three 0.013
- grant 0.012
- evidence 0.011
- ecosystems 0.009
- problem 0.009
- governance 0.008
- capital allocation 0.007
- different 0.007
- mean 0.006
- allocation 0.006
- standards 0.006
Transcript
You're running your funding program or your grant program as a marketing campaign and not as a thing to actually solve a problem. If you're not solving a problem, you're probably
enabling problems or you're creating other problems. Well, 99%
of projects that go through hackathons, they actually just die. It was Einstein. If I had an hour to solve a problem, I'd spend fifty five minutes thinking about the problem and five minutes thinking about the solution. It almost seems like the decentralized
nature of the governance has actually led to inefficiencies. Right? So from the state of Web three reports, you start to actually see this. There's actually a centralization happening, and there's a lack of transparency.
I personally find let's bang them bang very problematic.
I don't think there's a correlation with type of decentralization. I think that decentralization brings possible benefits in achieving impact.
Hello, and welcome to the Governance Futures podcast. I'm Eugene.
And I'm Jamila.
And this week, we had the pleasure of talking to Mike Cooper, who's a senior researcher at MediGov within the Grant Innovation Lab and leading the Grant Impact Handbook project. So, Jamila, how did you think the conversation went?
I really enjoyed this conversation, just because I personally don't really know much about web free, grant allocation, impact. So to me, it was insightful because I was able to ask all of those questions, and, his answers were very, very insightful. I'm sorry for repeating myself, but that's just the only word I can really use. And I also think there's so much overlap to what I was doing and I'm doing as part of my PhD research, as part of my work as governance facilitator. So it's very interesting because sometimes you're saying, oh, no. Definitely, there should be no overlaps. Even if we are in web three, it feels like it's a very distant to what I do. But I encourage everyone who might not know much about web free grant allocation and might not be as interested in that, potentially entertain that idea of listening to it because I found it, great. It's just so refreshing to hear people who know what they're doing, who are so knowledgeable, and they're so eager to pass that knowledge too. What about you, Eugene? What do you enjoy from this conversation considering that you actually have a lot of experience working with web three grant programs and everything?
Yeah. I mean, I I what had me most excited to suggest, Mike, and have this conversation, was really also stressing the importance, and I'm glad we got to touch on it, there's frame of impact beyond capital allocation. So I think it's easy to say, like, oh, I wanna spend money. What do I wanna accomplish? And then kind of backing into the steps you have to take. But, you know, Mike said something to the effect of, you know, like, governance and impact are inherently intertwined. And that's not a framing that's frequently brought up about governance that I think no matter what your view is, in terms of the specifics of the type of governance system that you wanna see, framing it through, what is the ultimate impact? Why are we doing this in the first place? How is the world supposed to look different because we do this? I just think that that's language that, like, we don't get to use enough that I haven't heard brought up enough in other DAOs. It's just not the place where you normally start from, and I I think it can, on a longer time horizon, really help us reorient to why we're all doing this in the first place, which just gets me really excited. Thanks to conversations like we had with Mike,
I personally take away that there's so much potential, and there's so many reasons why we should continue what we're doing. And regardless of the environment and regulatory frameworks and just overall general volatility that everyone experiences in the space from time to time, surely, I think it's very important to just continue working on things and not give up in all senses of the word. So, yeah, with that in mind,
here's to our conversation with Mike. Thank you for joining us today, Mike. Excited to have this conversation about the intersection of Web three and Impact today. And it feels like a very useful place to start with. What do you mean when you say Impact?
Thank you for having me here today. I'm excited. As we've talked about, I've, listened to the podcast and, you know, I I appreciate you giving me the chance to be here today. So I think when we talk about impact in relation to governance. Right? I mean, I think that, you know, for myself, you know, I started my career working with different bilaterals, multilaterals, and foundations in international development space. Right? So, like, the World Bank, different, you know, UN agencies, national governments, foundations, and things like that. And so, you know, when we talk about, you know so that's kind of my background. So when I talk about impact and and governance, just kinda keep that in mind because I'm taking a lot of these lessons from the non Web two world and trying to integrate them into the Web three, world. So when we talk about impact and governance, you know, governance has all kinds of different functions. Right? And the ultimate aim of those functions is what we I consider to be impact. Right? So a lot of what we're gonna be talking about today is, you know, specifically capital allocation. Right? Because one of the first things that we're working on, right, is improving the impact grant programs within web three ecosystems. And so if we're just gonna talk about, you know, a function of governance being capital allocation, you know, it's the function of that capital allocation, the ultimate aim. And that is largely a product of planning. There always has to be some kind of planning. Right? A lot of Web three ecosystems, have these visionary statements that, you know, the the the purpose of the ecosystem. Right? Whether it's, you know, to improve, access to different financial services and products, whether it's to improve or empower different marginalized communities, whatever. Right? The impact of that Web3 ecosystem is the, literal plan to achieve that ultimate, vision for the ecosystem. You know, and I emphasize this planning function, right, because we we see a lot of visions. We see a lot of kind of, you know, mandates the ecosystem is gonna do this, but there's no kind of plan to say, like, this is or strategy. This is how we're gonna do that, And you can't have impact without that strategy, and measurement that Yeah. Says, this is how we accomplish the strategy. Here are the milestones we use to to measure. Here are the actual empirical measurements. Although and so when I talk about impact, I mean, that ultimate aim, the strategy to achieve the aim, and the measurement of that strategy, and the achievement of the aim as a change in state. Right? An actual change from a to b as a result of the implementation of the strategy and the resulting behaviors involved, with that strategy.
And so if if thinking of it and let me know if this is not a reasonable oversimplification of that. But if oversimplifying impact as the change you wanna see in the world and the plan of how to get there and measure it. I guess, first off, is that a reasonable oversimplification,
or what is that missing? No. That's a I mean, you probably said it better than I did, to be honest.
So I guess it if that's the case, where is usually the starting point? Do you start with the plan, or do you start with defining what change you actually wanna see in the world regardless of what kind of specific program we're talking about?
We start with the problem. Right? If you're not solving a problem, you're probably, you know, either enabling problems or you're creating other problems. Right? And so the an impact if you achieve your impact, the only thing that should mean is that you alleviated a problem or a set of problems. Right? Because that should be the primary purpose of any type of governance system is to mitigate different problems. And so, you know, you define your impact by identifying the problem you want to solve, unpacking that problem through different types of methodologies. Right? There's all kinds of toolkits and methods out there to do it, you know, root cause analysis and all kinds of, you know and but essentially, like, you know, if you're gonna do a root cause analysis on a problem, you get down to the root causes of that problem, and there's different kind of layers between that root cause and your original problem statement. Right? And, you know, the this kind of manifestation, how this problem rears its ugly head empirically. And so your impact should just be like, well, you know, we alleviated the root cause of the problem. Right? And then your strategy should just essentially kind of reflect your planning framework should kind of reflect all those kind of, you know, layers between your root cause, your problem, and and the manifestation of the problem empirically. Right? And, you know, the layers between that root cause and the empirical manifestation of your problem are going to be, you know, your intermediate objectives within your planning framework to achieve your impact. So if you're not starting with problem identification and analysis, then you don't you're not really
going after an impact. You're just kind of doing a thing. Yeah. Which I I guess to put in the reality to jump ahead to the reality of web three funding programs, that's you're running your funding program or your grant program as a marketing campaign and not as a thing to actually solve a problem. But I also yeah. I I don't wanna get ahead of ourselves because I know we're gonna talk about the impact handbook and more about web three. But to to keep the flow going, I would also wanna kinda double check. You know, you have this background in aid development. You know, you've worked with groups like USAID, RIP, and, other international orgs. When did web three first come on your radar? What what was it about this space that made it appeal to you given that's the realm where you were really focusing all, you know, your career before that? I mean, I think there's been an interest in web three in international development for a really long time because,
you know, initially, when Bitcoin first came out, the idea of using blockchain for more what I call incremental tasks within international development. Right? Whether it's using the blockchain to manage different supply chain logistics within international development is a huge issue. Right? Whether it's getting food aid overseas, vaccines, different medicines, or whatever assets we're transporting. Right? Remittances and things like that was one of the first kind of things within international development we thought of for using a blockchain. But when we started to wade into the space of, you know, decentralizing social impact, you know, using different governance frameworks like DAOs and things like that where because a big part of international development has been how do we better tap into the needs of local communities? How do we empower local communities to define and solve their own social impact problems? And how can we use international aid to facilitate that? That's where I think blockchain really started to gain more interest within the national development space because it's you know, there's a a a long history within international development of top down mandated, impact objectives. Right? And going back to what how we define impact, we this top down approach that we were using, you know, kind of mandated what the impact would be on local communities, whether that impact actually addressed their problems or not. Right? But what if we could integrate, you know, decentralized infrastructure to manage that entire process and empower local communities to do this themselves? That's where I think it really started to kind of the rubber hit the road for me, where we weren't just using blockchain as an incremental, tool. Right? We could use it as a transformational tool and actually how we we literally achieved, impact.
And I guess as a last question for this context setting before passing it off to Jamila, you know, I was wondering with that recognition that so many, say, development programs or etcetera have this kind of combination of on the one hand, you wanna understand the problem from the bottom up, what is actually being experienced by people. But on the other hand, you have the mandate coming from top down around what's actually gonna happen. You know, how would you characterize the development world? You know, how much decentralization was there in defining the problem, in setting the strategy, in coming up with the approach?
I mean, I think it varies. Right? I mean, I think that international development is not, this kind of homogeneous space much like web three. Right? What we're gonna talk about when we like, later, we're gonna talk about the use of evidence within web three, and we definitely see variance between ecosystems on that. I mean, I think it's the same for national development before USAID was dissolved. You know, there was always this emphasis there was this big kind of localization agenda, right, where it was this big emphasis on, you know, more grassroots empowerment and things like that. You know, the World Bank and different bilaterals have had similar things, but, you know, generally speaking, they were not very successful just because I think that when you are looking at the foundational NGO I mean, I think that the NGOs, have been much better at tapping into this, you know, empowering grassroots and communities to define and solve their own social impact problems. I think just because they themselves are less centralized. Right? It's very difficult for big bilateral, multilateral donors that have so much bureaucracy and are very inherent to to change and are very risk adverse to adopt that kind of approach. Right? And in a way, it's very threatening for those organizations. Right? Because their mandate is to do exactly what they're they're doing. So, I mean, I think that there are some NGO actors like the, like the the greenhouse, grants initiative and and groups like that have decent oh, I'm sorry. I'm getting that name wrong. The Global Green Grants Fund. I'm sorry. You know, they have a decentralized model at their core for their capital allocation in their grants programs. And so they're they're what I would consider to be more of the gold standard in the development space.
Interesting. Yeah. Yeah. I really liked how you dissected root cause problem, manifestation of the problem. I wanted to go back to a general culture of grant programs in web three. I think for people who are familiar a little bit with the culture and landscape of web three, they would know that there are a lot of projects, a lot of grant programs that fall somewhere between promises of access to financial products, empowering community. You would often hear things like let's bang them, banned, which I personally find very problematic that statement alone. Just to, from the perspective of are we promising what we can actually deliver. And, I my question is perhaps I'm starting with rather pessimistic note, but I'm curious to hear from you. What is that that distinguishes Web three grant allocation? What is that apart from technological solution that can facilitate that positive change, what do you see radically different to, let's say, Web two or just, like, ordinary charity, nonprofit work?
I think that there's been a lot. I mean, I think there's already been good observations and evidence from things like the state of Web three grant reports and things like that that Medigap puts out, which has documented. And I think, you know, you go to, like, conferences and you hear this all the time that, you know, generally speaking, it seems that web three grant making, despite these kind of statements about, transparency and accountability, these benefits that should come from decentralized governance aren't actually there in a lot of ways. Right? It seems like the decision making about what Web three grant programs focus on, how they are awarded, how they are monitored, and the final, you know, impact of the performance of those grants is largely centralized, not very transparent, and doesn't use very good, frameworks for, you know or principles, practices, and tools for, you know, identifying the need and measure and awarding solutions, which are your grants, and then monitoring the results of those solutions, which is, you know, the performance of the grants. And so I think that that I think that's pretty well known. Right? Now that's not saying that every ecosystem does this. Right? I think there are some ecosystems that perform better and and others less so. But I think generally speaking, this is the case. Now we contrast that with kind of like gold standards and non web three spaces for grant alloc grant programs or capital allocation. Right? So I think that there's all kinds of examples to to take from this. You know, some of the examples, you know, Eugene has mentioned that, you know, we're working on this grant impact handbook with Medigov. And some of the examples we're looking at, for that handbook are examples from, you know, commons management, collective impact management, and, you know, different decentralized groups, consortiums like that that manage different types of common assets or have different types of capital allocation for a collective benefit or a collective impact. And they all have kind of standards, right, that they use for that they adhere to for for capital allocation, for grant making. And, you know, those standards are something that we're really trying to bring more into the web three space. Right? So there's been really great efforts by Medigov and DaoStar on creating data standards. You know, Medigov and Block Science or others are now leading efforts around KoiPond, which is creating standards for different types of knowledge products. And I think that, you know, what that work enables is, you know, improving the use of standards within the web three space, allowing for more comparability between ecosystems for performance and increases that accountability and that transparency. But I think the main thing that we really need is performance standards. Right? Some of the, you know, like, how do we actually define an impact? What is required for planning for that impact? What kind of measures should we be using? What are the trade offs in the different data? You know, how should we be collecting data for these metrics and things like that? Right now, it seems within the web three space, there is you know, we are getting some standards used around, you know, metrics and things like that that are starting to emerge. But, you know, they're pretty they're they're they're still emerging, and they need a lot of work. Right? We're not really seeing any type of measurement of performance on Grant's past outputs. Right? So we're able to track things like workshops are held, you know,
different Let me ask a follow-up on this because I'm very interested on this connection. From what I'm getting, the success of particular web three, grant program is very much tied to the impact. Correct?
I would say that yeah. I mean, the success of a grant program should be a measurable impact. There should be a measure that the entire community can see and assess themselves.
I'm curious where decentralization, how does it can like, do you have any evidence maybe from your research with Medigolf that would support that the more a particular program is decentralized, decision making is decentralized, the more impact there is. Is there any evidence that would support that? I don't think there's a correlation
with type of decentralization. Right? Okay. I think that decentralization brings, possible benefits in achieving impact. Right? So kinda going back I mean, say what? Go ahead. Such as Yeah. Yeah. Which one? So, I mean, I think that just the sheer fact that we're using a blockchain. Right? Using a decentralized lay ledger, it creates all this opportunity for on chain may or measures being more on chain and visible to entire communities. Right? That's the first obvious one. Transparency, basically. Yeah. Transparency will be worth. Mhmm. Transparency and improvement and count and accountability. Now, again, like, we are seeing that to this date up to the point of of kind of, like, what we call output measures. Right? Mhmm. You know, we we spent these resources. These grants were funded. These grants produced this many patches or this many workshops and things like that. But what we're not seeing is any type of measurement or very sparse measurement beyond that. Right? That actually leads, you know, kind of more outcome and impact measures. Now, like, you know, and to your point, there are benefits that come with decentralized governance structures to achieving impact. Right? But we're not really seeing that in the web three space, and there's no correlation between decentralization and the achievement of impact. I mean, I now this is purely observational on my point. The point I'm about to make is purely observational. But I would just say that given the work we've done up until now, it almost seems like the decentralized nature of the governance has actually led to inefficiencies. Right? So from the state of Web three reports, you start to actually see this. There's actually a centralization happening, and there's a d and there's a lack of transparency. And a lot of these benefits that we're talking about aren't actually materializing because you're actually have paralysis in the governance, right, because of decentralization in part. Right? I would say that, you know, the paralysis isn't, just happening due to decentralization. It's it's also due to a lack of actual planning and and structure to that to that governance and capital allocation mechanisms.
No. I'm, curious and I'm very thankful for you sharing those observations as a person who works with web three grant programs. I'm just curious whether you see that as a problem or perhaps it's just a matter of what you manifest. If you come to the space claiming that the decision making of grant allocation is actually decentralized and there's no, like, just a centralized body that would decide who gets the funding, then it's one thing. And another thing is perhaps it's okay for Web three programs and grants to just come out as, hey, we are a group of experienced people, we have money. I mean, it's a gross oversimplification oversimplification. I apologize. But we, will use our expertise to grant whoever wanted to grant, but at least it comes out as honest. Everything is still on ching, but we just decide to do this thing in a very centralized way because we see it as more efficient. So guys, we we don't want to play in this performative decentralization. So personally to me the problem is, like I don't see a problem when project grants, DAOs come out as well, DAOs, I don't think it can be centralized in a way, but like projects let's say. Let's say, well, you know what? This type of governance works for us. So, you know, you either come join us or not. Like we're not forcing you. But I do see a problem when the project or grant problem trying to act as if it's a community owned decentralized decision making and then it turns out to be actually, it was all theater.
Yeah. I mean, I think that, you know, I think especially and you still see this, right, in the early days of blockchain, and you still these debates happening around, like, you know, decentralization isn't a spectrum. It it's nominal. It's either yes or no. And there's only one way to be decentralized. And, you know, there are are so many different design variations of web three governance systems that can have all kinds of different types of decentralization, and we're barely scratching the surface. Right? But, you know, there's, you know, governance design requires evidence. Right? Because your govern your your governance design should be built around what problems you're solving. Right? And I think there's, you know so I mean, you could be centralized. That's cool. Right? You could have a centralized governance structure that still uses a blockchain to solve your problem, and you can be very effective in that. And there's nothing wrong with that. Right? Mhmm. The Web three landscape Web three is just gonna have a plethora of all these different types of ecosystems out there or, like, we hope that there will be. Right? But the thing is is that there needs to be going to what you're you're saying. Right? There needs to be some transparency in what those governance structures actually are. And I'm not really sure that we are there yet. There's not they're not very defined. Right? White papers and charters have a lot of open endedness in them. And I think that even that in itself is fine because to a degree at some in some way, shape, and form because, you know, you should have some open endedness to some things because we still need to experiment and find out. Right? But Mhmm. Experimentation requires evidence both to design the actual experiment in terms of what governance structure you're actually testing, why you decided to test that specific structure or or or whatever thing that you're testing, and what a successful test looks like, how you're gonna monitor that testing. This all goes towards these principles and standards around measure I, measuring performance, creating evidence, and using evidence that we don't really have yet in the web three space. So going back to kind of like, you know, commons management or, you know, collective action models and frameworks, Again, you're seeing the same thing there. Right? Some of those frameworks and governance structures and and and those groups are more centralized than others. There's different cultures and things like that, and they evolve over time. But I think the thing that is always the death knell for those ecosystems is when there is too much variance between what they're actually doing and what they say they're doing. Right? Whether it's Yeah. The Right. Yeah. Whether it's, you know, the actual level of transparency or centralization in their governance structure or the actual impact they're making. Right? They report a bunch of vanity metrics, but their marketing is talking all about impact, and they don't address the the delta between the two.
So to ask a follow-up on that, I wanted to kinda get at something you alluded to, but I I I didn't wanna interrupt the flow before. But this element of, you know, output or is it outcome output, impact, right, in terms of understanding those kind of differences and with what you're getting out of what problem are we actually solving? Like, because it seems so often in web three grant programs that the problem to be solved is defined in that kind of level one of result of, like, we want more TBL. We want more of a certain type of vanity metric like you were saying. But the question I always try to bring up whenever I'm talking to grant operators and that comes up is, like, but why does the how does the world change because your ecosystem has more TVL? Like, what what is supposed to happen after you get more TVL? And it seems like there's a disconnect in that thinking of, like, oh, no. But we just need to grow the network and to be more than our competitors. Isn't that already the impact that we want? When in reality, it's like, well, no. That itself should trigger other changes happening, which then leads to a bigger change in the world. So I guess the the question here is, like, I know with Jamila's question of there's the centralization, decentralization element. And even there, that's on, like, the design, the execution, the who's reviewing it, the which projects get funded. But there also is this fundamental element of what is a good frame of mind to have when defining the problem to be solved, when defining the what change you wanna see in the world. Because whether you're doing that as one person or as, you know, a multi thousand person community, if you're not doing that thing, it doesn't matter if you're centralized or decentralized. You're just never gonna solve the thing because you didn't define what you wanna solve.
A few things. I mean, I think going back to your first step your first kind of statement on there's things in between outputs and impact that we're currently not planning for, creating strategy for, or measuring performance of. Right? We stop with kind of vanity metrics of of outputs. But we our marketing is all about impact, but we just don't have evidence. So yeah. I mean, I do think that and it's not just within web three. Right? I mean, we do not I do not wanna phrase or or paint this as just a web three problem because, you know, that gap between outputs, vanity, and vanity metrics, and ultimate impact exists everywhere. Right? Private businesses struggle with this. Everybody struggles with this. It's just that within Web three, we have all kind what web three really does is it creates all kinds of different, what we call, causal pathways to achieve impact. So we have a lot of different ways in which we can achieve impact. And so the only question is, you know, how do we best generate the evidence for for how to get to that that that impact for more collective learning? And like you talked about, Eugene. Right? Like, you know, net network growth is is the big one, is probably the number one vanity metric within web three. Right? Because it's easy to measure, and there's just there seems to be this natural assumption that if the network grows, then all these other benefits will will accrue. Right? Will manifest themselves as a result of a bigger, you know, more, you know, bigger network. Right? More people on the network. But there's not a lot of thought of okay. So for the problem that we're really addressing, if we are addressing a problem with a lot of Web three ecosystems haven't really identified the problem that or problems that they're meant to address. It's there's not a lot of discussion of, well, who should be engaging with the network? What types of specific actors should be engaging with the network? What kind of actions do we expect from these actors? You know, do these actors, and their resulting behaviors help to diminish the problem? And, you know, so there's a lot of assumptions being made. Right? And those assumptions, whether they are clearly articulated or not, are usually where most ecosystems fail. Right? They grow network, but they don't achieve impact because a lot of the assumptions that they made failed to to materialize. Now in the non web three space, there's all kinds of, you know, like, again, going back to kind of more decentralized governance, ecosystems with, like, you know, commons management and collective action forums. They have methods where they actually articulate and state assumptions that they are making, and they monitor those assumptions actively. Right? So they're not just monitoring the performance of actors in their ecosystems. They're also monitoring those factors that can influence their ecosystem was kind of like, yeah, the mindset around problem identification. And to be honest, I mean, I think that if we literally got and problem, there's all kinds of toolkits and methods out there within for how decentralized, governance models approach problem identification and problem analysis. Right? But, you know, there's that old Einstein saying around I forget it exactly, like and I I hope I'm attributing it to the right person. You know, like, 80% of the solution should be, you know, thinking about the problem. And that holds true for this as well. Right? Most of your planning and strategy actually comes from unpacking and analyzing your problem. Yep. Right? That's that's 90% of it. So if you're not doing that,
then there's you really are setting yourself up to fail. Yeah. Real quick, the exact quote was, and it was Einstein. If I had an hour to solve a problem, I'd spend fifty five minutes thinking about the problem and five minutes thinking about the solution.
Yeah. I would I would say that rings true. Right?
I remember our conversation, that we had in our podcast earlier with Jen from Lido, And we were talking about, AI tools and, we're just curious about what do you think about AI tooling that we see in the space. And I don't remember the direct quote, but she also said something within the lines that we have too many solutions. And she said we don't see that much demand for it. Although we do see, many projects, receiving grants or receiving funding for development of certain tools. And there are a lot of tools. It's just like not enough demand of people who would Yeah. Use it. And, from my work experience at Scro, I also can say that, there are a lot of service providers that I came across. And in my mind, I was like, okay. This sounds like a great idea, but I actually don't understand how that would incorporate to what we're dealing with. And it's also very interesting because some service providers, that I've encountered within, like, just five past years in crypto, they don't actually care about the problem so much. They're like, we have a product. It's cool because it's this, this, and that. They're like, have you actually looked at what we do? So that is a very interesting point. But I wanted to, maybe hear more about how you measure impact. And I know that, there's no, perhaps just like, you know, checklist. And you've mentioned before that, obviously, it depends on goals. It depends on what is that objectives, and there's also some easy, metrics that you can just see on chain. Obviously, if you do have certain goals to onboard x amount of users or repost created, I don't know, transactions, comments, any of that that could be, like, easily, counted. I'm also thinking perhaps, survey, interviews, activity, but also more so like, the long term sustainability of a particular project. Did that initiative for the project survive after x amount of time after the grant? Did it attract, follow on funding? Because there are some, some evidence that, for example, 99% of projects that go for hackathons, they actually just die. And I'm curious to hear from your side, what is that, like, from your work, Amerigo, from your professional experience? What are those metrics that we should be looking at in terms of how do we evaluate and measure impact?
So I think one of the things that we're doing with the grant impact handbook is, you know, we've created a manual that helps ecosystems plan out their their strategic objectives and their ultimate impact, identify requirements for their grant programs, you know, gives criteria and tools for how to correlate, you know, proposed solutions with identified problems, coming up with metrics that are being used, metrics that can be used, etcetera. Right? So hopefully, you know, the grant impact handbook helps somewhat in in in helping ecosystems plan for and measure impact in that way. Right? But there's a lot if you're do I understand it correctly and full transparency, I haven't seen that handbook, and I would Mhmm. Appreciate we will link it to you. We're still crafting it, so no worries. Yeah. Oh, it's not out yet? No. It's not out yet. It's not gonna be out for a few months, but we're developing it now.
Okay. So, I'm part of Blockchain GOVE research effort, and, we, coauthored and collaborated with Project Liberty on this, like, governance toolkit. Mhmm. And very short is basically because I I think every often, every so often researchers get approached like, oh, but how do I design governance? And, like there's no single recipe. Right? It just really depends what you put into that recipe and, what the outcome would very much depend on what you put in there. And our toolkit, cookbook, tries to address that. There's no, like, golden recipe. It really depends on your objectives, really depends on your, like, value system and what you want to see, what are the outcomes. And more so, like, don't be surprised that if those are the ingredients that you're putting in your pie, it's probably gonna come out as this and not that. And you have to just be honest with yourself. So is it something similar to what you're creating in Medigold? They're trying to just basically create those, like, interpretations and basically provide that education to people that, hey. Those things leads to that outcome. Those things leads to that outcome. So So or by the planning of anything else.
No. I mean, I would say I mean, it's it's it's somewhat that. Right? So, essentially, what the grant impact handbook is attempting to do is saying, you know, we have all of these different types of grants that we can fund. Right? Mhmm. Here are the types of problems that these grant types are best suited for, and here's some measures you should be thinking about that others have used. And here are some measures that, you know, but at the same time, we've talked about there there just hasn't been a lot of impact measurement within Web three capital allocation. And so we're also proposing measures and manners in which they can use to measure impact. But I think that, you know, it doesn't it's not prescriptive. I think it says, here's things that you should be thinking about. Right? And here's some of the trade offs involved. And so I you you mentioned AI, and, you know, there's this explosion of, you know, how we're gonna use AI in this space moving forward. You know, I think that AI, generally speaking, one of the things that we need most in web three is better knowledge translation, data translation as a part of overall decision support. Right? So as so let's assume that we start getting better at planning for and measuring impact, and we start to use AI in diff for for different functions within, you know, that that life cycle. Right? There's going to need to be the capacity within web three ecosystems to say, okay. You know, here are the trade offs involved with all these different kind of decisions you're making in regards to, you know, this is the type of data you're gonna use for this decision or this is the you know what I mean? And so I think that that's a function that's really messy or missing within web three. You know? And I think going back to what we were talking about earlier in terms of how web three ecosystems have all kinds of different capacities they can use to achieve impact that non web three ecosystems, don't have. The big one is, you know, mechanisms.
But what do you mean by mechanisms such as
So, for example, a quadratic funding, right, is a new type of funding mechanisms being used. The mechanism library or the mechanism institute, I'm sorry, is creating their mechanism library where it's essentially, you know, listing out all the different types of modular mechanisms that can be used within a web three ecosystem. You know, what they do, what their function is, etcetera. You know, kind of a profile for that mechanism that gives kind of initial guidance on when and how that mechanism should be used, how that mechanism's been used in the past with, like, links to examples. You know, that, in my opinion, this is just my opinion, is one of the biggest value ads of web three ecosystems is the ability to use novel, mechanisms to achieve objectives, and to experiment with those mechanisms. Now, again, there's that word experiment. Right? You you you alluded to the fact that, you know, web three culture is really driven by innovation and experimentation. Right? And there but there's just not a lot of, evidence that is informing that experimentation, or there's not a lot of evidence being gathered that captures the results of that experimentation and how and there's not a lot of decision support on how the results of that experimentation should be integrated into overall governance decision making on what governance should look like, what capital should be allocated towards, etcetera. So when we talk about things like AI and, you know, mechanism libraries being built, You know, that mechanism library really we should be able we should get to the point where we're generating enough evidence that for each mechanism, right, there's just like this nice, you know, big piece of guidance that comes with it that gets very specific around, you know, these are the when this mechanism's been used. These are the results. These are the specific types of problems. You know, this is how you should test the mechanism, you know, in terms of protocols, you know, for like, I used to work with a group called, you know, three IE, and we used to create these things called evidence gap maps that essentially kind of plotted out, like, you know, all these different types of interventions and how successful they were to achieve different types of outcomes and the quality of evidence that is reflected in the performance of those interventions and achieving those outcomes. If we could get something similar to that, and, you know, correlate it with actionable guidance in something like a mechanism library, that's what I'm talking about in terms of web three performance standards that Mhmm. You know, are roughly the equivalent of the types of performance standards we see in in other ecosystems.
Thank you. And you alluded to the knowledge problem, and we had professor Ella Reni join us in the past episodes, and, she talked about koi point experiment. So perhaps you could maybe share with us any learnings or insights how projects such as DAO star and koi point address barriers that you currently see, maybe projects or people who want to enter Web three grant program or they want to make sure that it's as impactful as it could be,
that would be great to hear from you. I mean, I think that a lot of the kind of building blocks are are coming into place, like, what you know, what you're just talking about, whether it's the data standards being, you know, worked on by DAOSTAR or, you know, the knowledge modular knowledge objects with Koipond or even the mechanism library by the mechanism institute. Right? These are all things that, you know, these, you know, I'm gonna call them constructs to oversimplify. Right? But these are all constructs that we're gonna need to build our solutions for our problem solving. Right? But what we and, you know, it's very easy to create these constructs because there's just low barriers to their creation. So there's really nothing to stop people from just keep generating, you know, the the mechanism library can get to be, you know, whatever. It can be huge. Right? And so the only the only the primary limitation on that library is the evidence to guide its use, right, in problem solving. And that's the thing that we don't have because we don't have standards and protocols for testing all of these different constructs, whether it's a mechanism, whether it's a knowledge object, or whether it's, you know, a data standard. We don't have the protocols in place to
to test them and to see how successful they actually are. Sorry if I jump in with a quick question here. So I guess to like, let's say we wake up tomorrow. Right? The impact handbook is out. DowSTAR standards get, you know, all these things progress to a point where there's enough legibility, which that's already a big leap ahead. But let's just say that magically happens overnight. I mean, what is your experience been on the culture and willingness of groups to acknowledge the work that goes into this. I think I I remember from a previous conversation we've had when you were telling me about the norms of how much the evaluation portion is of the overall spending budget. And, you know, when you actually tell that number to most grant operators, they're like, oh, I can't do that. Right? Like, I I can't spend 10% of my money on evaluation. And so I guess what because there's the technical part of what knowledge needs to be available to make it possible, but then there's the what actually makes people change or groups change. Mhmm. What do you see as the main hurdles outside of the standards, the protocols? You know, what else is gonna be the biggest thing that needs to actually change for this kind of stuff to get adopted?
So I think when you look at the evidence for what are the enabling factors that drive good impact design measurement and adaptation within any ecosystem. Right? There's usually a type of champion that actually has the ability to leverage change within the ecosystem to focus more on evidence use, evidence curation. Right? But even a champion, right, like, you know, this the seed thrown on stony ground. Right? There has to be some kind of culture within that ecosystem that, you know, we talk about within web three. There's this culture of innovation. Right? Are we sure? I mean, like, I'm not really sure. Right? Because, like, you know, innovation isn't just building stuff. Right? Innovation is building solutions. And if you're gonna call the solution, it means that you have empirical measurement that that thing solved a problem. And I think that we just don't have that yet. So I do think we're talking about, you know, shifting the culture where innovation just isn't seen as build fast, build faster, but test. Right? And what all does test what all is does testing actually involve? And so I would say it's, you know, shifting the culture to promote evidence, building an evidence use, really advocating and enabling champions within ecosystems that are pushing for it. You know? And I think that when we talk about incentives,
right, that's when it's to ask them to, you know, push that. I'm like, let's talk about incentives. How do we create incentives that actually foster this desire to champion certain initiatives forward and, you know, don't lead to this apathy or just this burnout or sensation that whatever I did actually doesn't bring as much impact as I thought it would.
Yeah. So, I mean, I think that okay. So I guess if we're talking about, like, capital allocation or grants programs, right, you look at, like, what the Global Green Grants Fund does. You look at what Rockefeller does or Gates or any of the bilaterals or multilaterals. Right? The funding comes with, requirements attached for, you know, collaborative evidence building. Right? Whether it's, you know, hey. You have to report out on these metrics. Or, hey. You have to take part in workshops to talk about, you know, they call them, like, failure festivals. Right? Like, what didn't work? And so I think that it's a lot easier to create that incentive when, let's say, you have a champion that has achieved, you know, a stable foothold and can actually mandate this on grantees that they actually are required to do that. Right?
So, yeah, that's that's easiest.
But you still have to deal with the problem of, like, well, what's the incentive for the grant program to even adopt that standard to begin with? You know, I think when from my personal observation in the past, when there's just been an absolute lack of performance and and the the grant ecosystem or or organization faces this decision of, you know, we are not we're not achieving any type of return on our investment, and we need to make a a shift. Right? I think part of the problem with Web three ecosystems is that there hasn't been this kind of financial pressure to improve capital allocation up until now. Right? Maybe there I mean, again, this is purely observational. Right? I do not have good kind of empirical data. This is purely observational, but I would venture to say that maybe those pressures increase during, you know, bear markets. Right? But there's just not a lot of incentive at this stage in in web three maturity during bull markets to really focus on evidence curation and evidence use. Right? Now the question is gonna be once Web three really matures and there's lots of competition between ecosystems
and things like that, there could be more pressure. But now it's it's it's tough, to be honest. And a quick question on you know, you mentioned the importance of having the internal champions who are interested in in kind of pushing for this. I think another, you know, unfortunate reality and, again, this is more observational anecdotal as opposed to empirical data, but it seems as though those who are tasked with building and running a grant program are just not part of leadership. And so, like, in in the in the equivalent of the hierarchical orgs outside of Web three, right, at best, they're mid level managers. Like, they are definitely not the people at the at the top of the pyramid of the power hierarchy, so to say. So when it comes to your experience outside of Web three, how much does this kind of change start happening until you get someone who's in, you know, the equivalent of a c suite for these orgs being bought in versus are there have you seen actual successful campaigns where it's like, hey. We as an org might not care about impact evaluation, but it starts from kind of the lower or middle parts of the org to truly propagate that culture. Because, obviously, we're smaller orgs. We have flatter hierarchies, but I have yet to meet someone who's the head of a grant program who's, you know, like, has the seat at the table equivalent to the ED of the foundation or the c suite of the labs.
I mean, I think that it depends. Right? I mean, I've seen some instances where I'm thinking of one in specific, you know, there's this government agency. We had this massive capital infrastructure program in a variety of countries, and we were not seeing the results from those capital investments in infrastructure or different types of institutional building that we wanted. And we realized because there were a lot of capacity issues that we weren't addressing with the capital, investments, and so we needed a grants program to, you know, address a lot of these things that we didn't really plan for when we made the initial capital investment. Right? And so then all of a sudden, the grant program was very important to over the overall investment. And so in that instance, you know, the grant program leads, they were at the table. Right? And so I do think that it's kind of this question of what is the role of how important is capital allocation to ecosystem success? Right? I think is the primary question. If and I guess my question would be is, like, well, if it's not, then why have it? But I would also really question the prominence or, you know, whatever the role that is assigned. If they if they don't feel like capital allocation is important enough to track impact on, then why have any type of capital allocation? Or you're just wrong in assessing its overall importance to your ecosystem. And I would say it's probably more the latter than the former. Right? Now and so then you would make the argument, like, well, you should be, you know, assessing its impact because it plays this this role in your ecosystem, and that role is really important. Now I but I also think when we're talking about, you know, governance and what type of what should be the design of the decentralized governance within the Web three ecosystem, that's also an impact question. Right? Because there's a lot of different thing. You know? You should be assessing governance performance. You should be testing different types of design features within your governance frameworks. Right? So I would yeah. That would be my argument for for that as well.
I wanted to go back to your point. I'm not sure did I remember it correctly,
failure fest or Failure fests. Yeah. Failure festivals. Yeah. Right.
Which I actually think is so important to learn from our mistakes, to pass that knowledge, and to hopefully not repeat those same mistakes. And unlike you both, I'm not very close to the world of grants, within web three or outside. So I do have a question to both of you. I have a sensation that within web three, well, we all know that that everything moves on in web three as just like much higher speed than it does in the normal world. And I have a sensation that very often, we also forget about some of the mistakes and we just repeat them again and again. And sometimes we're like, why are we again in the same situation? Have we not learned? So I'm curious, what do you think about, this passing of the knowledge of the mistakes and learning from the mistakes? What are your observation when it comes to Web three grant allocation?
I mean, I think Web three needs an evidence commons. We're building all of this infrastructure to to, you know, create functional web three ecosystems, but we don't have any type of reliable, consistent knowledge about how they, you know, make those ecosystems work the way we we vision. Right? And I think the dangerous thing is that we don't have standards and structures in place to kind of mitigate failures before they become catastrophic. Right? And so you get a Web three ecosystem. It it's it's chumming along. It's it's, you know, completing grants. Maybe 80% of its grants are completed successfully. Right? Quotation marks in terms of those vanity metrics we were talking about earlier. They all report positive vanity metrics, but the ecosystem just all of a sudden fails. Right? We've actually seen this a lot. The point with impact planning, impact measurement is to derisk experimentation. Right? To where you're plotting out a causal pathway towards success. You can measure your incremental progress along that pathway, and you can start to measure when you start to kind of lean one way or the right, and you start to go off the rails. And it allows you to use evidence to take corrective measures, before you just have your entire ecosystem fail. Right? This is and so it it's there's a big risk management piece to evidence building that we haven't really discussed. But the long or I should say the the short response is we need an evidence commons. Right? We need standards to create an evidence commons so that people actually know how to use all these things that we're building. Thank you, Mike. And, Eugene,
we know that you have a huge experience when it comes to grounds, within MediGolf, Web three, and even before Web three. So curious, what is your take on this?
Yeah. I mean, I I love the the call to action around focusing on an evidence commons. That would be a beautiful thing for us to actually see have developed, and put into the world. I think, you know, coming back to what I was alluding to with the or or getting at rather with the previous question on the hierarchy and the power sort of there, I mean, I I've also talked to a lot of grant operators and managers and, you know, even the people who are kind of, who they report into. And there's this recognition of, like, oh, we know this isn't working great. But it's kind of somewhat like, at least we got a little bit out of it. And so Yeah. That's enough for us to not justify spending more on justify spending more on personnel to then actually work on solving the problem. Because whether it's, a, how do you justify funding the Evidence Commons? Right? Like, that's its own problem that is not unique to web three. Right? How many core, like, security primitives or tools just in the digital public infrastructure outside of Web three are, like, one dude in a basement because they feel like it, and, like, what happens if that dude or gal disappears? And then it's like, oh, this whole core part of the Internet is just going down. You know, so that part does not feel unique to Web three and nor does the other side of it, which again comes back to the, like, the people tasked with running and solving these problems. Are like, I have so many things to deal with, and I'm getting yelled at for the top of the list. So Mhmm. This kind of stuff, which is nice to have on the bottom of the list Yeah. Yeah. Never makes it to the top because, like, I'm not gonna get fired because I didn't solve this part. I'm gonna get fired because I didn't grow TVL or I didn't have x amount of projects go through. But it does seem this disconnect again, and, like, Mike, let me know if this seems more endemic in web three than not, but it does just feel as though it's a common startup and capitalization problem of, like, the thing that the VCs and the people with the most pressure on them on, like, growth and performance just aren't worried about impact evaluation. That seems like a nice to have when we're in a world of abundance, and we can do all the nice things that we haven't had time for, but it's just not like a top one, top three, top 10 business priority.
I mean, I think that in non web three ecosystems, even decentralized ones, right, you've seen consortiums and collaborations around shared standards, principles, and even, you know, repositories for evidence, right, within, like, the collective action forum and, you know, similar groups. Right? They represent decentralized entities that were able to collaborate across ecosystems, create the standards, you know, create kind of a culture around implementing those standards, guidance for implementing those standards, and actual and their versions of what we're referring to as evidence commons. Right? Learnings around how those standards are are applied. Now so I think we have examples, non Web three examples, even within decentralized spaces. But I think, yeah, Web three is different. Right? We know it's different. I don't have a perfect answer for what that Evidence Commons actually looks like or how it's created, but I think that we're approaching a space where, especially with the integration of of AI tools and functionalities within all these different life cycles where, you know, now that we've got, you know, things like the mechan all these modular tools coming online, whether they're knowledge objects or data standards or, you know, whatever mechanisms, the ability to create Web three ecosystems has never been easier. Right? But we just don't have the knowledge to do so. So I would say that anything we can do and I think that meta gov is in and I think that's actually underlying the importance of organizations like Medigov. Because when you do see evidence, comments, and standards and principles being developed in non Web three spaces, it is usually the action of somebody like a Medigov, right, which is cross ecosystem, this kind of collaborative champion. Now when you talk about funding, right, that's that's, you know, where it starts to get stickier, but, you know, there's no reason why Web three can't do it. And I would actually make a strong argument that as Web three matures, there's an interest. There's an incentive to do this. Right? I, you know, I think we're also talking about timelines here, right, where you're gonna see a lot of ecosystems that, yeah, they're in this, like, startup mode and and things like that. But even in that race to an MVP, right, if you're in startup mode, that in and of itself is still just a different type of evidence comments. Right? There's gonna be evidence in that that's gonna be useful in that race to an MVP. Right? Even if it's just principles. Right? There's all kinds of things that can be in that evidence comments, whether it's, you know, principles or, you know, standards or, you know, more specific case studies around the use of different mechanisms. And it can be oriented to different types of ecosystems, whether they're DeFi or or some or focusing on something else, and maturity of ecosystem. Right? Start up to sustainable. But I think that we would all agree that, you know, decision making of any type that doesn't use evidence is at risk. Right? And it's just a question of what type of risk are ecosystems comfortable with with taking.
And, unfortunately, I know, time is, flowing on by, and we gotta start transitioning to our final portion of the quiz. Before doing so, I did kinda wanna offer you the opportunity to bring up anything we didn't have a chance to touch on in this discussion or any kind of final words you wanna leave before we transition over to the quiz part. I mean, I think that maybe just on the last point about, you know,
what should this Evidence Commons look like? Right? Because, you know, the Evidence Commons should in and of itself solve problems. And so I would just say that, you know, in the process of creating the the the grant impact handbook and beyond, we're very eager to talk to different ecosystems about, you know, what decisions do you need to inform, what evidence do you need to inform them. And I think that that's the starting point for the creation of a of an evidence commons or whatever that
looks like. Right? What decisions do we need to make, and what evidence do we need to inform them? And that could be a a fun workshop somewhere around DevConnect as a lot of those folks will be together. But we'll I'll follow-up with that with you offline. But to to start bringing us to, the next part, which is our wonderful quiz, where as a reminder, I'll be asking you a series of questions looking for one word answers. Hyphens are okay, but one word answers. So the first one I wanna ask you is, where do you think Web three folks should be gaining more inspiration from or learning from?
Man, I'm gonna say commons. Right? Because I just think that, like and I and I know that this is, like, Austram's ever been longer than one word already. I'm just Oh, yeah. Yeah. I was gonna give an equation of the one word. The one word is commons. Uh-huh. Okay. Think about think about, read about exactly. Exactly. Right? I mean, these are not new problems we're dealing with here. It's just that we add new tools.
For sure. Yeah. So then looking at web three governance, the best idea you've seen in governance is?
Principles. Okay.
What gets you most hopeful about web three?
I mean, I know this is gonna sound lame, but decentralization.
I mean, given who you're talking to, not lame at all. Yeah. And last but certainly not least, what do you see as the future of governance in one word? Variety. Oh, yeah. Always interesting to see where people take it with that one. But, yeah, Mike, thank you so much for taking the time to chat with us and to explore this intersection much, governor. Thanks a lot. I appreciate it. Thanks for tuning in. The Governance Futures podcast is sponsored by the Scroll Foundation and produced by the governance team at the foundation, Jamila Kamalova and Eugene Leventhal. Any music and photos are attested in the episode description. Feel free to subscribe, leave a review, or share with a friend. Until next time.
