Thank you, Eugene. I'm very excited to be presenting to all of you. I met Eugene at the DSi Denver conference in March. We had talked about discourse graphs as a way to synthesize knowledge, as pertaining to collective governance. And so I'll tell you today about how we use, this tool called discourse graphs to synthesize knowledge and push forward new research in my cell biology and biophysics lab. K. So let me share my screen, which you can see now. Okay. So in my day job, I'm an assistant professor of biology at the University of Washington, so I run a research lab. And now about half of my time and attention is is spent in applied metascience, developing software and approaches to to improve how we do distributed, collective, scientific research. And that takes the form of building, software, that helps us run our lab, as a collective system to do collective sense making and internal nano publishing. Okay. So the the main motivation for this project were that there are a number of systemic issues in science, that I encountered over the course of my of my training, that I thought we might be able to address with with, software that has a different, organizational approach. Okay. So the first barrier, that I, have encountered and noticed is that there's a high barrier to entry to scientific research. When you're entering a new field, when you're a trainee, when you're trying to make an original contribution to any research field, it takes a long time to acquire sufficient context and to maintain orientation to say, is the question that I'm asking one that's already been answered? Like, what is the state of knowledge about this question? You know, what studies have people done? What are the what what's the the hard evidence that they've collected that informs my question, am I at the edge of knowledge or not? This is also informed by some qualitative research suggesting that one of the factors that contributes to when a researcher starts to consider themselves a scientist is when they make an original research contribution that can be used by other members of the field. And traditionally, this happens when you publish a paper, but in our field, that takes five, six, seven years from when you start your your graduate training. And it creates this huge gap in when you are not sure what contributions you're making, and it's one of the factors that is sort of lowering the mental health of of research trainees and also contributing to an inequitable attrition from from the research ecosystem. Second, from an organizational perspective, I'm really interested in helping to develop a research environment that's fueled by intrinsic motivation. And I found that the hierarchical structure intrinsic to most academic labs is not conducive to that, which is part of the reason why I'm so interested in the decentralized science, movement. So some of the factors that, contribute to intrinsic motivation, based on other qualitative research are autonomy, sense of purpose, and expertise. And so to maintain the type of autonomy and and agency, it's and having all of the ideas and sort of action plan consolidated with the PI acting as a manager isn't really, conducive to, to the students or the trainees maintaining that type of, of agency. And so the alternative seems to be a a decentralized network of researchers, and they're excited about, DAOs and other ways to, organize research, in a in a decentralized manner. And to realize this this vision, it seems to me like we need a coordination layer, so that a given researcher sort of knows where they're at, how they can contribute, who to collaborate with, etcetera. Finally, the researcher roles tend to be uniform in academia. The the analogy that I like to make, which is initially created by Greg Nelson, who's a human computer interaction researcher at the University of Maine, is likening the current research ecosystem to unsustainable agriculture, where PIs sort of all have the same needed skill set, and it's basically the only way you can contribute to research after a certain point, which is really missing on a huge base of diverse intellectual contributions that we think from people who wanna be contributing to the research ecosystem. So our hypothesis is that we can address some of these systemic issues with new collaboration tools as a means to facilitate cultural and, organizational change. We do this by breaking research into modular parts that can be shared and cited, and those modular parts have a semantic element, which I'll describe in a second. Okay. So we aim to facilitate the following things, on onboarding into a new research group, bookmarking gaps in knowledge to motivate new experiments, the coordination and collective sense making between researchers interested in a shared question, and finally, the regular exchange of discretized results between researchers in a way that's much more flexible and reusable and sustainable than traditional research articles. Okay. So the the main approach we use is to decompose scientific arguments into questions, claims, and evidence. This comes, from a model of argumentation created by Stephen Toulmin, who's a science philosopher. And the main premise is that science, rather than being a series of monolithic truths, is instead a series of claims that are backed by more or less evidence. So the textbook has the best claims at the moment, but it evolves as more evidence arises. And this is part of a collaboration I've had with Joel Chan, a human computer interaction researcher. We started in in 2020 because we were interested in the same sort of distributed graph based note taking tools, and we finally met for the first time earlier this year at at DSIGHT Denver. Actually, the very first meeting was was through Eugene when we were when Joelle and and Eugene were creating this doing this discourse graph, workshop. Okay. And so the idea is to take the atomic units of a scientific argument or any research argument, questions, claims, and evidence, and connect them into a graph. So this is the schema for a discourse graph. The question is the motivation for your work. It's the thing that doesn't yet have an answer that you'd like to to answer. The claim is your current proposed answer to the question, and the evidence is a specific measurement or observation or outcome that supports or opposes the claim. And each piece of evidence is grounded in some source, in some study or some experiment, etcetera. And so every time that you make a claim, you support it with this discrete observation that has been made by most by a researcher of of some kind or another. And, of course, it's tied into a graph so that, the edges between these nodes can can be labeled as supports or opposes. So let me give a concrete example. Let's say your question was, what is the fastest land animal? And I claim cheetahs are the fastest land animal. And And the Internet and ChatGPT give me a bunch of a proper full evidence for informing that that question, so we need something more concrete to inform this this question. And luckily, a researcher made a rigorous measurement of the speed of a of a cheetah. This person published a paper where they tacked a piece of raw meat onto the back of a jeep, It had a stopwatch and a track of a defined length, and he measured the speed of this particular cheetah three times, to get the speed of 64 or so miles per hour. And the methods include information like, at the time, the researcher was an athletics coach and so was well trained in stopwatches. This is the stopwatch that I used. The wind was blah, blah, blah, blah. So it gives you all of the contextual information you need to evaluate for yourself the strength of this evidence towards this claim. And if you wanna make an alternate claim, this approach really encourages you to bring your own evidence and add it to the graph so that everybody can weigh the strength of one piece of evidence or another so that we can sort of increasingly arrive at a stronger and stronger claim. Let me show you an example in our lab. So let's say we wanna do we a journal club rather than about a paper, we do it about a question. We call that synthesis journal club. And for the sake of brevity, I will just show you what such a discourse graph looks like in a this sort of open canvas that we've created. So let's say you have this, research question about, like, how cells attach to their environment, which is something that we study. So you start with a bunch of sort of premise questions. We divide up the relevant papers between people in the lab that, might inform this question. Everybody pulls out a specific piece of evidence, which is, like, one observation from the paper, and you can see over here, like, the key information is, like, there's, like, some figure that has a a, like, single measurement that that gives you the outcome, and there's some contextual information about, like, you know, what were they measuring and what was it on so that this individual piece of information is sufficient to understand what the outcome was. And here, people in the journal club are sort of, like, arguing with each other about and bringing evidence for why they said that this protein interacts with this protein. And then you end up with some summary. So you're like, here's our diagram of this molecule interacting with this part of this cell, and we use it to create this claim, this claim, and this claim. And rather than just hoping that those are true, we have we have pieces of evidence that that support that claim that we can refer back to. So we consider this to be sort of getting to the edge of knowledge, the state of knowledge about this particular research question, and we can return to change the claims as we acquire new evidence. And I'll show you some more examples shortly to make that more concrete. So that approach has been great for getting up to speed on published literature and doing sort of a systematic review of the state of knowledge, but I realized that most of my time was actually spent trying to, be in the lab or run a simulation doing sort of original research, and I wondered how this schema could affect my ongoing research. And so the moment was when I realized that, like, what else am I doing in the lab except for trying to collect new pieces of evidence that are supporting a new claim, which I usually call a conclusion, and write that up in a paper. And I thought, well, if I organize our lab's research through the schema, this is gonna let me crowdsource a particular project where each person can be, testing a related how strongly a related claim is supported by running their own experiments to contribute a single observation or result that helps to support this claim. And when we think about decentralized science, I think that the schema could be a appropriate level of abstraction for people to make contributions. So everybody who's interested in the same research question could be make different claims based on the evidence or point us to what new experiments need to be done. So we think of each of these as the minimum shareable unit for collective sense making. How does this work in practice? So here's a plot of structure as a function of effort, where as you go up the the plot, there's increasing, structure that's afforded to your sense making. So in the normal course of our research, you're sort of going on and doing some sort of study or experiment. You get a new observation, which you say, okay. This is the, like, the take home, result from from my experiment. And now that informs my question. I can make a new claim. Oh, but I can't really make a stronger claim as I really wanted to, so I'm gonna call that a hypothesis because it's sort of an untested claim and go back, and I wanna make a new go carry out a new experiment as a result. And in practice, when you don't have time to run that experiment, you can save it for later, and you call it an issue. And you say, well, I or somebody else should really go do that experiment because it's gonna help inform our hypothesis. And so I'll show you some examples of how we how we do that. First, let's show you who's working on this. We have a great integrated team of developers who have been supported by Protocol Labs and Schmidt Futures and experiment.com who are building plugins currently in, Roam Research, this graph based note taking, software. And then the users are my cell biology and biophysics lab, and so we have a, let's say, weekly feedback loop where we're filtering for features that are specifically useful in this, research lab environment. Okay. So what does our lab graph look like, and what do we use it for? Okay. So let me just show a sort of inventory of the types of discourse nodes that people have created in the lab, recently. So some of these are results, new results from their experiments. Some of them is evidence from the literature. Some of them are new claims. Some of them are are questions. And the important thing is that each of these elements is tied to the source material. So in the case of our ongoing research, let's see what happened yesterday. So Valerie has a result that's tied to a ongoing experiment. This is a tissue culture experiment that lets you tie like, drill down into the nitty gritty details of on the lab notebook level, what exactly did they do? This is usually too much detail for somebody to make sense of, so you usually stay up at the level of this of this, result instead. But if you really want the the detail, it's accessible, in in the graph. And so this is the first 12 of, we of what are now 1,200 of these of these, discourse nodes that we've generated, as a research group over the course of the last, let's say, two years. So here's a graphical, this, sort of schema, showing, some of these discourse nodes where, like, each dot is one node that's like a question or a saved experiment for later, an issue, or a claim, or a piece of evidence, and gives you a sense of how interconnected this our research graph is. So it has on the scale of, let's say, 60 experiments, 80 hypotheses, 250 pieces of evidence from that are all tied to to source material. And so this is a wonderfully densely interconnected graph that essentially serves as the operating system for our for our lab's work. So how do you make sense of such a densely interconnected graph? There are a couple ways to do so. You can synthesize the your your research, on a canvas similar to the one that I showed previously. So here's another example. So that's this would have been created organically by by one such researcher. You would have the experiments, the results, the conclusions, and the follow-up questions. Current work now is this question, this experiment, and this result. And once again, the result has, like, some key, sort of plot summarizing what the outcome is, as well as the sort of the, like, lab notebook level, note taking that that researcher needs to do just for their own personal sense making. Let's see. Okay. And I'll show a couple more examples. One thing that I think would is we found to be especially useful for collective research is bookmarking and handing off ideas between researchers. And we do this with what we call an issue sport. So every time we're trying to make sense of our project and we say, well, there's one more analysis that needs to be done. If there were five of me, I would just do all of them. But but we've done the intellectual work. Let's just save this for later and and hope somebody else does that. So we collect all of those elements in this in this issues board, which you can filter by is it based on analysis? Is it based on the literature search? Is it based on the simulation? So that new researchers can come in and pick one of these and use it as their entry point into the lab. So we ran this experiment over the summer with a new undergraduate who joined in March or so, And we essentially gave her a query of a couple of these issues that sort of had a type issue type of an analysis, and we said, hey. Like, does this, like, piece of this this single page plus the sort of contextual information that you can click into give you enough background information to to make a substantial research contribution. So it contains the description. It contains the the, you know, graph connected hypotheses that the issue was trying to to address. And and so the question is, can you claim this issue and then make that your ongoing experiment? And she was able to do that. So we gave a little some background information. Of course, she was collocated in the lab so she could get more hands on information from us. But over the course of just several weeks, she figured out what analysis needed to be done, ran it, carried out, found some some results, and then put them together in this in this graph. And this so this is great as a question. Let's see what the literature says, results, etcetera. Okay. For time, which I think we are about out of, why don't I skip ahead and just advertise that large language models can synthesize this information too. We are sharing individual results both sort of in a peer to peer fashion through a search of my discourse graph and somebody else's discourse graph. This is ongoing work. And to individually which we're experimenting with sharing results individually with with the world and and having the and and how having the sense making happen outside of our of our lab through LLMs, etcetera. I wanna advertise that we're putting together a user experience pilot where we're going to have a number of other groups interested in a shared research question, build one of these graphs, either in room research or some related note taking software and give feedback in exchange for getting their their labs set up. So let me know if you're interested in participating. And we think that this lab notebook software can serve as the coordination layer for decentralized, research by creating scaffolding for trainees, creating a more decentralized organization network for researchers, and to create a more sustainable culture of research contributions by researchers of different types where the fruits in this schematic are the discoveries and the roots connected with each other are these, discourse graphs. So thanks for your attention. I'd be happy to talk more, with questions.

Yeah. Absolutely. So we are taking the the sort of tact that we think that large language models and other AI tools are extremely helpful and are best used in collaboration with humans who are doing some validation, who are contributing knowledge. So we have a a sort of large language model that's baked into this discourse graph that sort of reads elements of the graph and has given instructions about how to how to how best to help with sense making and coordination. So for example, you know, based on this one experiment, hey. Give me the premise approach results in conclusion, And the assistant, you know, uses our discourse graph schema, so you can still click into the answers. And so let's convert that into a narrative that would substitute for a research article and then, you know, include hyperlinks so that somebody can actually ground the l m's responses in a result. This is a lot like perplexity.ai, for example, but we would imagine that the database rather than being papers would be these unpublished research results, which I think could really speed up the so sense making, be it, like, before papers are published and also ground the large language models, answers question generation in something that is verifiable and gives gives the LLM the right layer of scaffolding to contribute to.

I'd be happy to riff off of probabilities, but maybe he had something more specific that he wanted to ask about. I'll just start talking. He can interrupt me. But, that, of course, with the lang the large language model, there's a a probabilistic element to exactly how it's how it's responding and what, you know, what, what terms it it pulls up. These graph queries are deterministic right now and that there are you know, this issue is informed by these three hypotheses, so that's the highest level of connection between these these discourse nodes. There's another layer of, like, indirect referencing that you can use for for graph search, And we are starting to surface related discourse notes through, like, a semantic embedding. And so that sort of similarity search has some, you know, probabilistic elements.

what I was specifically asking about was generating confidences and hypotheses.

Yes. So right now, as a pragmatic for pragmatic purposes, we are generating the confidences ourselves. So each person has their own sort of manual, description of how strongly a, hypothesis is supported by evidence. We haven't needed to formalize it beyond that, but as you as the graph gets larger and there's more people you don't know as well, it'll be important to either score or qualitatively assess those. So Right now, the qualitative assessment has been sufficient for us to answer the questions that are right at the edge of knowledge since we're not we don't need to represent all of knowledge with the discourse graph. It's really what's at the edge of knowledge and what people are interested in in working on right now.

Yeah. Great. Yeah. It seems like it does work really well in your local environment, and they're just gonna be different things that have to be tackled when you do scale it. So, yeah, you've got that in your head. That's the right orientation, I think.

Yes. So, one of the great strengths of this approach is that each person's contribution is attributable directly to them. So I don't know. Let's let's pick this this issue, for example. Like, each specific, like, piece of text that people created has a time stamp and a and a user stamp. And so you can see based on which of these discourse notes you created, not only a stream of, okay, like, what what did I accomplish over the past quarter, for example? How do I make sense of it? But also who used it and when and how. And so this has already contributed to researchers' sense of satisfaction and that they're contributing not just to their own advancement, but towards, you know, problems that their friends and colleagues are find important. So I my favorite, like, version of attribution right now is something equivalent to a GitHub contributor board where you get to see, you know, for the next CRISPR, the big, like, discovery, where in the timeline of of, like, attribution were you? Like, where did your experiment show up that was important for this hypothesis, which turned into blah blah blah blah blah. So you get you may not even need to have some ranked, you know, attribution contribution table, but rather you just have a very granular and accurate way to to trace where your contribution fit into the these these larger discoveries.

It can. So that original application is for, systematic reviews and meta analysis papers. And so, rather than those being one off massive efforts, the thought is that you can now make sort of updatable sustainable meta analysis reviews this way. So so Joel has filled out his discourse graph with enough, I would say, attribute level information to say, like, okay. I want this study, but only the ones that are, you know, controlled controlled studies, randomized controlled studies, or I want this one that, you know, just where the studies are just based in China versus The US. So you can do it sort of a more systematic comparison. The I think that the system the schema is really lets you figure out and, like, what the next element that would be needed for to to complete your meta analysis paper and make it much more feasible for other people to to make individual contributions to it. But it's possible that you could just feed in a whole subgraph of the LLM of the discourse graph, give it to the LLM, and they would do a decent job of making a meta analysis or at least, like, a first draft.

Yes. I mean yeah. He doesn't he doesn't do it through an LLM. He uses it through, like, human brain synthesis, and it it works works great. You mean talking? Yeah. Yes. Through writing. Yeah. Right? Writing writing based on this, you know, extensive, you know, graph based knowledge base. And we use it we use the LLM so far to to, like, summarize the state of, like, our answer from from our, like, general club on this targeted research question. And then the LLM can also, you know, suggest the next steps.

Yeah. Absolutely. And we can share, you know, like, our whole discourse graph and see how how it how the LOMs would would do with it. Be very happy to be part of that.

Yeah. Sorry also if this is super obvious and you already said that in technical language or that I'm not explaining it a lot. But, yeah, basically, what I'm trying to say is that, like, in in the real world, the context that I see is that two sides, two polarities are actually not they are failing to agree in the most basic things. So I'm I'm guessing that in sometimes, the evidence that is supporting a claim actually needs to be supported by other evidence that needs to be supported by other evidence. So it's almost like you need to go to the root of, like, okay. In which things we do agree? Like, what is the, like, the layer one of truth? And then building from there until you even get to branches where evidence is more subjective, and and you cannot even prove it, like, fully. So you have different approaches to the same answer. So I'm just wondering because in the visualization that you showed, it it it seems like it's all, like, it's all connected, but, like, also spread. It's not like like a train of thought or, like, a tree of of truth, just to to speed it up. Thank you.

Yeah. So you can make nested layers of claims for sure. And so we're we're experimenting with some some sort of, like, zoomable interface so you can sort of see different layers of abstraction. And there's, the work on argumentation mapping in sort of, like, public discourse is one of the original motivations for Joel for for creating this this software. And in practice, for us, the way that the path forward that I see is that you have a small community of people who have agreed to talk to each other or even, like, genuinely interested in the same question, and they sort of hash it out and they get to some unknown that they both have competing claims for that they both generally wanna try to find the answer to. And then this schema helps them find what the right level of granularity is, like you were saying, going up and down, probably mostly through informal methods, and then you formalize the part where there's really, like where you're really clashing. I my feeling and suspicion is that that will work more effectively than posting the, like, the objective, look, I've proven it to you, approach. But, of course, both both are going on out in the world.

Just to explain, maybe what I'm saying is, like, crazy, but I'm I'm assuming as well that with cryptography, you could validate each level. So you don't need to actually show the whole line, but you could at least validate that they are, like, a proof of knowledge for that claim. Just as I'm I'm just saying it as a user experience tool.

Yeah. Absolutely. And I think that's also true for if you wanna turn this into, like, a distributed peer review system for new science. You said, like, there's been some, like, attestations for this, this, and that, so don't you don't need to look at the details, but now we're at this stage. I think we should definitely try it. I also like leaving space for different people having different interpretations of the same evidence because I think that's extremely common once we're at the edge of knowledge, which essentially all of the questions in in our graph. Different people in the field have a different interpretation of.

Yes. Absolutely. So one of the reasons why these type of approaches have not taken off before is because they're too much work beyond the researchers' own sense making process. And so this is a key part of the design of this discourse graph software. So let's pull up what those design features were. So the first one is to enable the researchers to seamlessly create this formal discourse graph alongside our informal notes, and these is these are, like, some of the, you know, plug in tricks that we use to to make that possible. And second is that we, let users leverage their formal discourse graph to improve our thinking and writing. So it's genuinely useful to us week to week when we're trying to, like, save our knowledge for our future selves or to even share the, like, the TLDR for in our weekly updates. So that lets that has sort of shifted the trade off balance within our lab such that it's just it's helping people track their own project and and keep their own mental model, like, clear and traceable. And then the the side effect is that you have the searchable graph that's, like, good for the system of science. I think just the UX in room research is too clunky for another lab to just pull up off the bat. So this is what this user experience pilot is. Essentially, we just make it simpler, make it, like, minimal port in another, like, note taking software. But again and again, we I'd like scientists to say, well, this is what I've been trying to do anyway, but it, like, it it's it's, you know, storing it in my head didn't work, and, of course, it can't be transferred by other people. And we've been, pleasantly surprised by how good an LLM is at taking unstructured notes and plots and turning them into discourse graph elements if you give it a specific enough prompt. Interesting. Even just, like, the image itself, it does a pretty good job of giving you the first pass, like, results, which I'd be happy to to to show.

for scientists to do actual science, to actually use the method they're supposed to be using?

Yeah. Or to, like, make it it could, at the very least, encourage people to have their observations be reproducible because that if that was as important as it being double, then then you knew that your sort of, you know, history of contribution would be influenced by the extent to which other people could reproduce it. That would already that would all already change the sort of incentive structure for what you're contributing. But we should definitely have that conversation because I like, right now, I'm leaning towards this sort of, like, utopian open source internally motivated version, but we we should talk about the the versions where there's sort of more explicit incentives to to make these contributions to.

I don't know if Matthew could, like, pick one that he seems to find more interesting or wants to answer more. Maybe that would be better.

Sure. Yeah. So this question about, like, if you don't have a now have a facilitator in the middle, so I love that my role has transferred towards facilitator. I feel like the system will be fully successful if I can transfer my role out to a researcher again. I spend 80% of my time just plugging away at the at the venture or or coding. And this is a the facilitator is a it would be a great role for for an LLM, I think. They're not expected to do the whole work. They're just helping to point people one way or the other. Absolutely. And your second question about combining with quadratic voting and retroactive public goods funding, absolutely. That's that's in the our ideal road map. You know, when Protocol Labs was supporting us, it was part of the the, like, the grand vision that we haven't concretely attempted, but I think it works perfectly in, this sort of open source idea, open source ecosystem where you make the contribution and then in hindsight, the impact, the like, how important how how possible the discovery was if you, like, got rid of your contribution would be would correspond to how much money you got. So then you're you just do the work that's important to you, and and you get supported as a result. That's very much, the future that I wanna be doing research in.

That's fantastic. Thank you.

Yeah. So so this talk itself is in an open notebook in Room Research that you can access and look through, with this this QR code. And so looking through that, it has, like, contact information and a a couple of places where we're documenting our ongoing experiments. And, again, if you have a use case for forming a research group that could use a discourse graph and are willing to give user experience feedback, these coming two years look like where we're gonna be running an extended series of pilots that we'd love to, have your contribution in.

Thanks a lot.

Looking forward to talking more.

Thank you.