The SpecOps Method
We're joined by Mark Headd author The SpecOps Method: A New Approach to Modernizing Legacy Technology Systems. We talk about good modernization practices like the strangler fig pattern as well as ways teams can use artificial intelligence to develop specifications that can be understood and validated across both technical and programmatic teams.
Music Credit: Tumbleweeds by Monkey Warhol
Top Keywords
- book 0.011
- modernization 0.010
- legacy 0.009
- code 0.008
- legacy modernization 0.007
- spec 0.007
- work 0.007
- system 0.006
- government 0.006
- need 0.006
- software 0.006
- change 0.005
Transcript
Hello. I'm Ryan Cook, and this is Civic Tech Chat, a show that looks at the way technology, politics, and policy impacts the world around us. The tools we use, the way services are delivered, and how we talk about and set policy all shape our society. We'll gather around and have a chat about these things together and more. Before we get started, I do wanna let you all know that we've started a Discord for the podcast. There will be a link with an invite down in the episode description. Do feel free to go check that out. It's a small community right now, but hoping to grow it. It's a great way to reach out to me and let me know things that you might want us to cover or to just hang out and talk about civic tech. Mark, thanks so much for joining us here on Civic Tech Chat. Could you introduce yourself and tell us a bit about what you do?
Sure. My name is Mark Head. I am a senior director of technology at a digital services company called AdHoc. I currently and have, for many years in different roles, worked, with and for, federal and state agencies, helping them implement technology solutions to better serve people.
What would you say is your personal why? The thing that drives you to get out of bed each morning and do what you do.
Yeah. I ask myself that often. So for pretty much the entirety of my professional career, I have I've worked either in government or near government or with government. Almost all of that time, has been spent working with technology. I came into government, and I'm gonna age myself here. What I what I refer to as the second great wave of technology change came happened, that was the Internet. The first wave being telephones, which happened about fifty years prior to that. And, in the last wave, we're going through right now, which we'll talk about more today. But when I started in government, the internet was just happening and it was changing, really changing the way government, operated. And so for most of my career, I've been, trying to help government figure out the best way to use technology. And it's a really hard problem, but it's extremely rewarding as anyone who works in civic tech knows if you can help, if you can help do this work well, you can really impact people's lives. So I I guess to answer your question, that's that's what gets me up in the morning. That's what gets me out of bed.
That answer is that maybe a great way to segue us into our topic today as you kinda mentioned this, like, next phase of technology as you talked about. And, we're gonna talk a lot about, modernization because you have a a book that's come out called The Spec Ops Method. And, you know, writing a book is something that takes a lot of time, a lot of energy, passion. And, you really dig into digital modernization and the urgent need for new approaches, and, you call for urgent action. Why did you write a book like this one?
Yeah. And I I I asked myself if I would write another book knowing what I know now about how how much energy and time it takes. I was so naive when I when I started, but, I I felt a real urgency to to write this book, and and I and I feel continued urgency to talk about this issue. 2025 was a a real watershed year. I mentioned previously, the the third wave. I call it the third wave of technology change, that the telecommunications, revolution that brought telephones into people's homes took decades to to to actually happen. Governments had a fairly long time to adjust to it, and it it did change the way they operated. You know, you had telephones in government offices. You have government phone banks. All that happened over many, many decades. When I was coming into government, the second wave was happening. This was the Internet wave, and things moved much more quickly. And governments had a really hard time keeping pace with that change. And in many ways, we're still playing catch up. I don't know if you've seen the latest numbers on, the web accessibility audits of federal agencies, but we're still struggling with those sorts of things. The way I describe it to people is we're now in third wave of change, the AI age, and the change the rate of change is much more rapid. So to me, the imperative is how do we ensure that governments, do what they need to do to prepare for the changes that are already here. But in 2025, some really important things happened. Right? We had a we had a really significant change in the federal workforce. About 317,000 people left federal employment in 2025. Now that was that's above the average. If you look at the average over, the last, I don't know, ten years or so, it's about a 150,000 retirements per year, for for many reasons, which we won't get into on this podcast. There are a lot more of those in in 2025. Taken together, if you if you net out the the in the the new employees that came in, about 11% of the federal workforce left in 2025. That's a big deal. The other thing that happened in 2025 was, AI coding agents really came into their own, and it happened extremely rapidly. I only saw the change in the last six or seven months of 2025, but a a really fundamental change has taken place. I always describe it that way. I always describe it sort of in the present and past tense now because it the change has happened. And the debate that's kind of going on in the world of software development is really, in my opinion, just coming to grips with that. You know, we're going through the various stages of acceptance, if you will. So it really it really became a thing that I, you you know, I I described it to someone recently as, I don't know if you've ever seen the movie Close Encounters of the Third Kind, but there's a character in the movie that's played by Richard Dreyfuss, and he sees the Devil's Tower, mountain, in in everything he's doing. He's making it out of his mashed potatoes at the dinner table. I was seeing this issue that I read about in the book. I was seeing it everywhere. It was in everything I saw. And I just felt this really urgent need to, to write a book about it and and hopefully talk more about it and get people thinking about it. That's why I wrote the book.
And your reason why you mentioned that, like, around 10% of the workforce retiring, you know, a higher than normal amount, but, a lot a lot of folks. Right? You actually, in the book, talk about some specific examples of, categories of folks leaving that are problematic. Like, for example, you mentioned COBOL programmers. Yeah. An average age of 58 and, actually, 10% of that workforce also on average retiring each year. Can you talk about the urgency of that situation as it applies to kind of this bigger picture and why it's important for folks to pay attention to it?
Yeah. The the people that are leaving, the if you look at the the demographics of government employees, I'm I'm I'm I'm referring to federal data from the office of personnel management, but we can easily and comfortably extrapolate to many state governments. The the the workforce about 15% of the workforce is in their sixties. Less than 7% is in their twenties. The people that are leaving government, the people that are retiring, even though, you know, even though 2025 was an outlier, that trend is still not very favorable. Right? The people that are leaving are the people that have worked there the longest. They have the most institutional knowledge about how systems work, why they work the way they work. They are they have such valuable information that's walking out the door. I talk a lot about this in the book, but what's what's what ends up happening is, in some instances, these folks go back to work for government as contractors. Some of them into their seventies, because there simply are no other people that can do the work that they do. So, you know, we've known about this problem for a while. This is not a secret. It's been talked about in the world of civic tech, I think, for a good bit, and people have been, uncomfortably aware of it. But what happened in 2025, you know, accelerated the trend, and I think made it more urgent. So we are we are we are living through a time when the people who understand the most critical systems in government are walking out the door. And the imperative that I try and, articulate in the book is before we let that happen or before it happens to a greater degree than it's already happening, we need to take steps to capture the knowledge that they have, to understand the systems that they've maintained for decades, before it's too late.
It's it's hard to talk about, staffing without also talking about budgets. They tend to go hand in hand as it were. And, one of the things you point out in the book is that, 79% of federal IT spending goes to effectively keeping the lights on, you know, operations and maintenance, which really that doesn't leave a lot left over to make new things or to incrementally replace something or to even just make improvements to something that's there other than just keeping it turned on. What role do you see that reality having in creating and then sustaining these problems that we've been talking about so far?
Yeah. The, and I'm not the first I've been saying this for a while, but I'm not the first person to say it. And I won't be the last person to say it. But the the way we budget, for technology projects in government, is is one of the reasons why we are where we are. And I I do go into this, as you say, a bit in the book to talk about the way these projects are funded. I think mostly it's a reflection of of of an older view of the technology projects as more traditional infrastructure. You know, there's a there's a there's a point at which you pay a whole bunch of money upfront to build a thing, and then you sort of pay, a lot less annually over the usable life of that asset. And we know as people, you know, working in the civic tech community that that's the wrong way to look at these projects. We need to we need to view them as products, and we need to fund them as products. And we need to have, funding to enhance features, and and meet the needs of real users. And, you know, the reality is that that's, just not the way most governments budget for technology. I think I I think an important thing to understand is that and this goes back to what I was saying about the waves of change. This is the way we've always budgeted. Right? The budget process works in government as it was designed. It was designed to work the way it works. And there are very good reasons why it works the way it works, where we have an annual appropriations process. You know, we have accountability, through, audits and, you know, things of this nature. There's a good reason for it existing the way it exists. And I think a misconception that that some people have, particularly people that are new to government, is that, oh my gosh. This this system's broken. It doesn't work. It it it actually does. It works the way it was designed to work. It just doesn't work well in an era when technology is changing by the month. Right? Wasn't set up to move that quickly. There was a time, for example, when the the telephone was making its way into everyone's home and it fostered this different expectation for people about how they're going to interact with their government. And then eventually you saw, as I said before, you saw telephones turn up in government offices, you saw government phone banks, all of these sorts of things. That took many, many years. And the budget process and the and the policy making process all move at that pace. But starting with the second wave, with the Internet wave, and certainly now with the third wave, the AI wave, those those processes do not move quickly enough for for the rate of change that we're seeing.
Do you see that, adaption to rate of change as being something that's uniquely government organizations as a problem, or do you see that as it's just big organizations struggle with this kind of thing?
That's a great question. I I definitely think that there's that other big organizations, large bureaucracies, you know, particularly in regulated industries, will have the same challenges. You know, this is an historic pace of change that we're seeing. So yes, there will definitely be, struggles with other large organizations and and they have their own bureaucracies to navigate too. They have their own processes that where they're accountable to shareholders and and other stakeholders like that. With government, it's a little bit different because, a lot of the things that slow down the budget process or make it, I should say, make it more deliberative are things that we want. We want people to have a chance to participate in that process. We want there to be transparency in that process, and we want there to be accountability. And in many times, those things manifest as more time. And for much of our history, that was okay. Increasingly though, as technology becomes such a fundamental part of how government operates and since it's now changing so rapidly, again, I'm not the first person to say this, but it's definitely time for a reexamination of how do we accommodate those very, very important, principles, but also move more quickly and become more agile in how we, we adopt technology.
One of the things you talked about before, we were talking more about the kind of the labor situation, was the importance of institutional knowledge. I think that was a a term you'd use. And, in the book, you make a reference that I'm a a fan of where it's, Marianne Belotti's writing. I I hope I'm pronouncing their last name correctly. About, but they mentioned how legacy systems are there and become a legacy system because they managed to work that whole time and maybe even work really well, for some for some or all of that time. How do you suggest ensuring that respect for that institutional knowledge, is there as modernization work takes place?
Yeah. It's a great her book is it's on the bookshelf next to me, Never Far From My Reach. It's a seminal book on legacy system modernization. And it's a it's a very different way to think about legacy systems. Right? She she suggests that the reason these systems have been around so long is because they're good, they're successful. They achieved their end for a long, long time. Maybe they've outlived their usefulness, but for a long time, they they were very valuable. And I think, you know, in addition to it being just a very different way of thinking about these systems, I think it gets to some of the reasons why it's so hard sometimes to replace them because we have, as you mentioned, we have this whole body of institutional knowledge that's been built up around them and all sorts of internal business processes that are tied to them. They they're very hard. It's a legacy modernization is probably the most difficult thing to do in government technology. It's extremely hard. And I talk about in the book, I talk about the the track record of projects that have have tried to do it and have failed. You know, it's it's a long list. You don't have to look far for examples of where it has gone wrong. So I think to me, the reason her book was so useful in framing my thinking on on on what I wanted to write was because it really centers, this idea of knowledge being the valuable asset. I think that most of the time when we think about legacy modernization project and you talk to people who are involved in network, the goal is we wanna turn old software code into new software code. And what I'm proposing in the book is that that is wrong. That is the wrong way to think about it. The right way to think about it, in my opinion, is we need to capture the institutional knowledge around that system. We need to understand in detail what the system does and why it does that. Specifically, like the policy directives and the other the the legislation that drives what that system has to do. That is the and should be the focus of legacy modernization. Code conversion, converting to new code, that's incidental. That is incidental to the knowledge capture that should be at the heart of legacy modernization. And I think that's a still a fairly controversial statement. I may take some some some heat for that, but I believe that that's the right way to start thinking about these projects. And her her writing on it is eloquent and and engaging and and definitely worth worth the read for any any listeners.
I know I've, seen this myself in in my day job trying to do some modernization. I imagine you've seen it along your path and many others out there in the audience that, you mentioned in the book this, second system effect that I think you referred to as being from the mythical man month, which is also a fantastic book. But the idea is, like, as you're doing the work, folks who are kinda helping you figure out what it should do start to kinda put things they wish the old thing did onto the new thing. And then, oh, like, it you know, we're making this new thing. It should go ahead and fix this problem too. What kind of impact can that behavior have on a modernization effort?
Yeah. I mean, the the most direct consequence of that is, scope creep, is, you know, the accumulation of risk. I talk a lot in the book, and this won't be a surprise to you or to anyone who's ever worked on legacy modernization, but they they typically follow a pattern of, big bang deployments. So the the the plan is that you're gonna build a replacement system, and there will be some point in time when the old system gets shut off and the new system gets turned on. And invariably, that approach does not work. I talk at length about an example in the book from, the government of Canada and their Phoenix payroll system, But there are many others, that follow that same pattern. And I think it's because so to your question, Ryan, I think the the reason why that can be so tempting is because I think it relates to the budget process we talked about. I think it there's often a sense of an overarching sense in these projects of this is our only chance to do it. We we get this one shot to to to fix things, so let's fix everything. And, you know, to some degree, I think it goes back to, you know, product thinking. Thinking about this as a product that's evolved over time based on the feedback you get from real users rather than we're gonna just throw everything in the kitchen sink into this modernization, and all of our problems will be solved. It's very tempting to think that. And I'd I'd be I'd be I'd be over I'd be I'd be missing something if I didn't say that. I think oftentimes, that sentiment can be a product of, like, top down, like, you must to the team, you know, you must fix this. You must fix this. That that's never a recipe for a a successful project. But
We've, talked a bit about the the varying problems that could happen as one embarks on these projects. And, you started to talk a bit about, maybe something that's a good segue into some solution type thinking, like the as you mentioned, spec ops. This idea that using the institutional knowledge as your as you talked about in your prior answer. Your book with that same title spends a lot of time talking about that method. For folks that maybe haven't picked the book up yet, what is it, and why should agencies and teams look at working that way?
Yeah. So I and I said before, I think the the traditional way of thinking about legacy modernization is how can we convert the old software code, whether it's COBOL or Java or Assembly, whatever it is, and convert it to a new software stack so that we can be done. And in the usual course of those projects, there are documents many, many, many documents created, you know, system architectures, requirements documents, all sorts of things. But they almost always exist only to serve that transition from old code to new code. So the the title of the book is, spec ops the spec ops method, a new way of thinking about legacy system modernization. Because what I propose in the book is that instead of focusing on code conversion, updating the code, we should focus on capturing the knowledge that's that's currently embedded in that code. Right now, in these legacy systems, they are the source of truth. Whatever the COBOL says is the source of truth for how the system works. And that has a lot of disadvantages. Right? Because it means that if if if you or I are are are not an engineer, we're a policy expert, we're a domain expert, we can't inspect that source of truth on our on our own. We have to talk talk to an engineer and more specifically, we have to talk to, like, a COBOL programmer. So the spec in spec ops comes from something called specification driven development. And I mentioned earlier that this is something that's coming out of the the rapid development of these AI coding tools. It's an approach to building software that is based on a foundation of a detailed specification. So before you build anything, you first describe in detail what it is, how it's supposed to work, and why it's supposed to work that way. And then from there, you can have AI tooling build this thing for you. So in practice, what it what it shakes out as is you spend less time coding because your AI agents are doing that for you, but you spend a whole lot more time clearly articulating what is being built. The the ops in the title comes from a concept called, GitOps, which is a little bit esoteric and maybe some folks aren't as familiar with that practice. I do talk, I do have a chapter in the book where I talk about it. In essence, it's just a way to describe the state of a Cloud environment and that documentation, usually people have heard of something called infrastructure as code. Really, that's just a bunch of files that describe the state of a cloud environment, and and that's the source of truth. It's not the actual running environment. That's just an implementation of the truth. If you want the truth, you go to the documentation. So the really, the book sits at sort of the crossroads of those two ideas that, you know and and when when these changes were happening is that I was working with AI tools myself and I and I, you know, I'm my thinking is obviously informed by the work I've done in the past working on legacy systems. It it just occurred to me that this this we've been doing it wrong. We've been thinking about this the wrong way. And I had one of those moments where you're like, you know, you're you're you're talking to everyone you know. Every single conversation comes back around to this idea that you've got burning in your head, you know. I found myself at the grocery store, you know, talking to people, talking about this and, you know, I joke with my wife about it, but, you know, she says, you know, there's probably some guy at the at the produce counter going, hey, man. I'm just looking for the organic bananas. I don't know anything about stuff. I'm sorry. But every conversation kept coming back to this. I felt this real urgency to, to to talk about it and to write about it because I myself had worked on, some really painful legacy modernization projects. So some of them I talk about in the book, but others, I haven't talked about. And I really feel like, you know, it was, it was really important for me to, to try and get this idea out there, but it's really a function of these new methods of, software development, which, you know, some listeners may not be as, hands on with, but I'm I'm sure that, people are being exposed to some of the the debate about this, but it it really has fundamentally changed in just a very, very short period of time how software gets developed. And there's a whole bunch of debate in the software development community about it, but, you know, my take is we've crossed the Rubicon and we're not going back. And how we deal with that as people that work with and around technology and government, I think that's there's still some debate about that, but I don't think there's any debate about whether or not the fundamental nature of software development has changed. And that's kinda what underpins the book.
As we talk about those specifications, that knowledge that then would get used for modernization effort, kinda using the method you're describing, what would you say are key aspects to make good specs or specifications?
Yes. That's a great question. So a software specification has to serve two purposes in in the spec ops method. It has to be detailed enough in its technology, details for an AI coding agent to use it to build something. So, Ryan, if you were to write a software specification or a new podcasting software solution. You would need to write enough details in it so that your agent would know, what am I building? What software stack am I using? Am I using any sort of special approaches? Am I interfacing with any other systems? But the reason why I think this approach is is so valuable and so important is because if we do this correctly, it allows people who are not software engineers, who are domain experts, who are policy experts to inspect the source of truth and to verify, yes, that works. That's exactly what it's supposed to do. So if you were a, someone who's a domain expert in public benefits, you can read a software specification and understand whether or not something was, described correctly or incorrectly. And I provide some examples in the book of this. The way it works now, if you're a domain expert in public benefits and you wanna understand how the system calculates, based on a specific income type, more often than not, you've got to go ask an engineer and the engineer has to then go and inspect a code and say, okay, I see what's happening here. It looks to me like this is how the system accommodates that income type. The the real value of spec ops is that it brings people who are domain experts and policy experts, the people who know the most about how these programs are supposed to work, it brings them into the conversation about the system and how it's supposed to operate. And, that's one of the things that these all these changes in the world of AI coding has enabled. It's enabled a lot of things. And some of them are not good, but one of the very good and valuable things is that with this approach, it allows people who are the real experts in understanding how policies and programs are supposed to operate. And it gives them a seat at the table in determining how these systems are built and operate.
In the book, one of the things you advocate for, kind of in that AI debate of how do we use it is, beyond just the use of it that you see people talk about in the Internet that generating new code, right, based on prompts. You talk about using it to read old code, kinda going on that thing where these tools are very good at, like, taking lots of text and summarizing and drawing, find finding patterns, that sort of thing. Why do you see that use as a being a particularly compelling use case for this kind of tool?
So to me, the the it is what I propose, to respond to the trend we talked about, at the top of the discussion, which is the people that know the most about these systems are leaving government in in record numbers. And even though there are many things we can do with AI tools, we can create funny videos, we can create, you know, images, we can do all sorts of things. One of the things I think that will have the most, return on investment for governments is to use these tools to better understand their existing code bases, particularly when the number of people who understand those code bases are leaving. So I propose in the book to use these tools to generate these specifications that describe how a legacy system works. And while we have access to the people that are still in government to help us verify that, yes, that works the way it's that that is correct. Or, no, that calculation is wrong, but, it's fixed down the line, in another system or by another, you know, manual process. All of those details that that underpin how these legacy systems work, many of them are captured in the heads of the people that are leaving government. So we can use these AI tools to generate new code, and and and maybe that's the way we build replacement systems. It doesn't have to be, but I believe it is a very effective way for governments to better understand, the software that they have. And if I could, Ryan, I'll share a quick story with you. It I haven't told anybody this yet. This is the first time I've ever shared it, but I won't name the state. But I worked with a state not long ago, where I was brought in. There was a the the the, the motor vehicle department was having conversations with the agency that handles, drug and alcohol treatment programs. There's an obvious connection there. Right? Because if someone gets a violation for impaired driving, their part of their sentence might be to complete a drug and, alcohol program. So there's there's a data interchange point there. And, you know, in some side conversations, the folks from the motor vehicle department said, you know, we're having these conversations with this other agency, but we can't get any traction. We don't understand why nothing's being done. And through some investigation, what we came to learn was that there was a an old process that batched up a whole bunch of data from the drug and alcohol treatment agency for the motor vehicle agency, and no one understood how it worked. No one knew who had done it and they were they were terrified to touch it because that they thought they were thought they were gonna break it. These are the kinds of things that it was written in a in a very old scripting language. It was it was using a very integrated approach and it was a lot of really, really, high anxiety about touching it. So what I'm proposing in the book is this is a perfect opportunity for us to even if we don't change that code, it's a you know, initially, this is a chance for us to use these tools to better understand it. What is it doing? Is is it doing something we're not, fully appreciating? How would we propose replacing it? These are all questions we can start to ask with these tools that don't necessarily involve generating new code. It's really just a way to document how do these existing systems work. And I'll tell you, I'd love to tell you that that's an outlier. I'd love to tell you that that's a very, unusual circumstance in government, but it's not. There are many, many legacy systems. Now they're not all huge, you know, payroll systems like the ones I talked about in the book. There are tons and tons of small one off legacy systems that are not documented, that are doing critical functions and gluing systems together that we just need to understand better.
So I guess done in a semi ideal sense, what you might have someone do is say set up, an AI tool of their choice. Maybe using like a CLI tool or again or an ID, whatever it is that they prefer. Have it run and with some prompting that's like, hey. Like, help me document what's going on here and probably much nicer descriptive language. You then would get something that you could have both, like, engineers look at and go, oh, does this make sense with, like, my institutional brain? But, like, now I I'm I'm looking at something going, oh, like, is this correct or not correct? Not like, tell me everything you know as the question, which is much harder. And then you can also take it and hand it to a program person and go, hey. Like, do the rules that it's talking about here and the code make sense? Are they correct? Yes. And maybe there's even goofy stuff where they're like, oh, well, yeah, it does that. But then, like, I open the spreadsheet and I fix it and send it back. And and then, of course, you can choose whether you wanna keep that or not, as you make a new thing. Is that is that the kind of flow you're advocating for?
Exactly. It's a it's an iterative process where both engineers and domain experts can can use AI tools to better understand this code. For example, using the scenario I just described, we could have engineers examine this code, whatever code was running this this process, to better understand, oh, I see what's happening. It's pulling down a dataset from the mainframe and it's batching it up into a zip file and it's placing it on this FTP server and then another process comes along. Okay. I get the details. From a domain perspective, you know, this specification would have enough detail for somebody to come in and say, oh, wait a minute. We we're not allowed to share that data. This this particular field is not shareable because it's there was a there was a law passed two years ago that prevents us from sharing. These are the kinds of conversations, that this approach enables. It isn't just I'm a domain expert. I understand what we're legally allowed to share or not share, but I can't look at the source of truth. I can't look at the code and make a determination as to whether or not from a policy perspective that it's doing what it's supposed to be doing. And that is really the big change that has happened, and to me is is is a thing that, look, legacy modernization, there is no magic here. This this is an AI is the magic solution to all of our legacy modernization problems. It's not. I say this in the book, but it's it's basically that what we've seen occur over the past year or so has, has converted something that was almost impossible to something that's now just difficult. Right? It's still hard work. We still got a a lot to learn. But if we can give non engineers, new domain experts and policy experts, a a a seat at the table, we will be able to capture the knowledge we need before we see more and more people, walk out the door with that knowledge in their heads.
Earlier in our conversation, you made mention to the term get ups, which is someone who's done the kind of tech side of things a lot of my career that, hearing you talk about also seeing it in the book perked my interest. In particular, you cite the benefits of it having that declarative nature, having things like versioning among its other benefits. How do you see those principles being generalizable to kind of organizational practices for modernization?
Yeah. I mean, as someone who develops software a lot, I I'm I I, I I'm probably more attracted to version control than maybe your normal person, maybe maybe the guy in the grocery store who I was talking to. But the the idea that these specifications that describe how these important systems work are living documents, are the you know, version control is important when documents change. If documents never change, we wouldn't need version control. So to me, the the one of the powerful, connectors with with GitOps is that you have these documents that are constantly being refined, and they're constantly being iterated on. Not only as we add new features because, you know, perhaps the law changes or our policy changes and we need to the system needs to work differently. We're we're seeing that now play out in in states with HR one and some public benefit systems. But also as we learn more about these systems as we we we can complement our knowledge, We'll always be refining it. These are these are living documents. They should and and everybody that works in in these domains, whether they're engineers or or policy experts or frontline people, you know, people, you know, doing all sorts of things. They all have an ownership stake in that because it represents the system that that supports, the work they do. So, you know, it's I'm admittedly, I'm probably more enamored. Me, you and I are probably more enamored with the the idea of get offs than than maybe a a lay person. But the other thing I think it does is that, you know, GitOps is a is a is a mature practice. It's a mature form of DevOps. So the fact that the spec ops method I described in the book shares some of the same DNA with with GitOps to me as another validation point of, yes, this is an approach that that works. And we haven't gotten into this yet, but it it also sort of gets to the idea that the tools we need to do spec ops work are not these elaborate, you know, expensive tools. Git is an open source tool that is pretty ubiquitous, and you can get for free. That's another part of the book that I'm really excited about because when I when you think about how we do this work, we don't need to go out and purchase an elaborate, you know, software solution or tool to do it. We we have the tools. You need an AI model, but most governments have access to those things now. You need you need a version control system like Git, and you need a just a few other things, and then you can do this work. It is there isn't huge barriers to doing it. So for all those reasons, I think the GitOps, connection is a is a really powerful and important one.
As we're we're talking good practice, another one that your the book references is the, strangler fig pattern. For folks that are, reading the book and kind of learning about this, what is that pattern and why do you consider it a good practice?
That's a great question. The strangler fig pattern is, is a is a term that was coined by a gentleman named Martin Fowler who's a who's a prolific and and very respected, software developer and engineer and writer. And it refers to a type of plant that, grows up around an existing plant. And as it grows, it encases the old plant and eventually the old plant, I guess I guess, dies. So it's a little dark. It's a little bit of a dark analogy, but, the idea is the the reason why it's it's a very common concept in legacy modernization. It's not the only, pattern that that people use, but it's it's it's a very popular one, a very well known one. And if we bring that idea into the world of legacy system modernization, the approach dictates, a phased approach where you incrementally replace a legacy system. So you have some period of time where you have your old system that's still operating, and then you have this new system sort of growing up around it. And then eventually at the end, the new system encases the old one and the old one can be, decommissioned and turned off. It's I think the reason why it it resonates with a lot of people in, the world of civic tech and specifically in in the communities that have done legacy modernization is because these these these efforts can take a long time. They can take years. And, it it it is a way to accommodate the time it takes to do these, to do these projects. And, for me, it was it was important to to at least touch on this concept in the book and to talk about how spec ops can be used in in connection with it. It wasn't it wasn't in conflict with what we consider to be a very, very common way of doing legacy modernization, but the name comes from a, very strange plant.
You also spend a lot of time talking about the importance of subject matter experts validating the specs as we've kinda referred to earlier in the conversation. And it seems like there's maybe some effort for teams to get to a place where that's possible between, say, your technical teams and your program teams and and other stakeholders. Where it's almost like you have you have to even get to a place where you can understand how you're talking to each other. Yep. How can teams get to that place of whether you call it, like, shared language or a place of understanding?
Yeah. It's a great I mean, we're now we're getting into how do we do spec ops successfully. How do we do it well? You know, like any project, it requires, it requires a clear mandate. It requires, good leadership. You know, I I don't think it's as there was a time when I think, you know, just the whole notion of a cross functional team was was unique, but I don't think that's the case anymore. I think it's fairly common now. So, I think, you know, I've had many conversations over the past six to eight months with people that don't consider themselves engineers. They consider themselves policy experts and domain experts. And, you know, these folks are often they don't feel empowered or incentivized sometimes to ask questions of the engineers, or or or they don't feel like they're given the opportunity to say, hey. Wait a minute. It does is that does that work the way it's supposed to work? Or how can we validate that? Or, you know, these are the kinds of, I think I think, opportunities that spec ops helps surface, to really give those folks a a seat at the table. But, you know, the people that have the most expertise and the most knowledge in those areas are are are are are also some of the people that are leaving government. So I think bringing those folks into these project teams and empowering them to, you know, be the arbiters of correctness is is super important. Whether whether you're I would say that whether you're working on an active legacy modernization project, like, we have to replace this system by a substance date or as we were talking about before, we just wanna understand this work this code better. We just wanna document it as a as a insurance policy or an investment we make for the future. But it it's a very different way of thinking about, a technology project where usually it's engineers who are the arbiters of correctness. And in this model, it's the domain experts and the policy experts, the people that understand how programs work. They're the arbiters of, a a a a a, a certain aspect of correctness in the system. They're they're not going to try and say, hey. You need a a a, you know, high availability or, you know, five nines of availability. I'm gonna I'm gonna weigh in on that. But they are the ones who are gonna say, you have to calculate this benefit this way or if if this system works correctly, this application should be approved and this one should be died. You know what I'm saying? They get a full they get to see the table now to to to validate those things without having to have an engineer translate for them.
I've seen in shops that there can be a difference between, like, ceremonial signing off. Like, I kinda look at something, just kinda stamp it, versus, like, a deeper, more meaningful review of the thing. As I I guess we're kinda continuing in this topic of doing spec ops well. As your team kind of, say, pursues that, how can you encourage that deeper review to happen when kind of teams interact?
Yeah. I mean, that's a that's a great question. I think it's it's probably applicable to non spec ops teams as well. You know, like any effort in government, in the kind of work that we do, you know, getting the team composition right, getting the right, folks on a team, getting the right leadership per team, making sure the remit is clear. Those are all super important. I think that, you know, I do think that the the folks so we we talked a little bit about how policy experts and domain experts, you know, now become sort of first class disciplines. And that requires a little bit of a minds mind shift change for them. I've like I said, I've talked to many of them and there there is there are times when they're like, you know, should I be asking this? Is this should I be buttoning on this question here? I I I think it will require, some practice. I think it also requires, engineers and technical leads to understand that, you know, that there are other people who are now first class participants in this process, but there's a whole chapter in the book about getting started. I wanted this to be very practical. Like how do I actually start to use this? I do think this will require some practice and I do think that picking out the first project that you tackle with this approach is important. Right? It's it it needs to be one that's that's consequential enough so that it doesn't look like it was nothing, but it can't be, the most important system in the government on the first go. But getting that practice in and building up that culture, you know, this this is not dissimilar to what happened when, when DevOps, changed things. Right? There were plenty of old school system administrators who were used to managing the infrastructure from their command line that did not wanna hear about, you know, continuous integration and automation. They were like, what's that? That's what I do. And that required a real culture change in in, technology teams and and really changed the way the people that write software participated in those in those efforts because they now became, a different level of participant than than they than they had been. I think the same is gonna be true of spec ops and other, similar methods where it will require some practice, it will require some culture change. But the good thing is that we do have we do have a little bit of history to look back on in terms of, like, how did DevOps become the norm? Because there was a time when if a developer walked into a, you know, a meeting and said that we should do DevOps, you know, you had a whole bunch of, you know, people that wrote Perl scripts and were doing sysadmin from the command line. We're like, uh-uh. We're not doing that. So I think we'll get there. But it it does require some practice, which I would I would definitely urge people to look at the getting started section of that, of the book.
As you talk about culture change, something you do point out in the book is that at some projects, you end up with this kind of, like, good news only culture, which Yeah. You can argue a lot of that comes back to, like, do people feel, you know, psychologically safe to say something is to say bad news. Right? Yeah. To whoever needs to hear it, whether it's kinda up a chain or to a particular person who's working on a thing. How can you how how should teams try to help foster an environment where folks can feel like, I can say that this is not right, rather than just having that let that roll? Yeah.
You know, I think some of that was at least some of it was a function of, the the nature of software development. The the big change we've seen is that the cost of generating code has come down dramatically and will continue to come down. And when code is really expensive and you find out after that code has been generated, maybe even after it's been deployed, that something doesn't work. It's a big deal. It's expensive. It's painful to change that. Maybe maybe as, the cost of writing actual code comes down, that changes. And, you know, we're teams aren't as hesitant to say, hey. There's a there's a mistake in the system. But it's okay because we'll just update the spec and then we'll generate a new version of code and we'll we'll fix it. But I do think some of that was is a relic of it's expensive to generate code or or used to be. And, you know, some some of these not some of them. Many of them? All of them are political in the sense that, you know, careers have been won or lost on on projects like these. So there's a there's that whole other dimension to these projects that, you know, makes it sometimes more, advantageous to not share bad news, to not be the bearer of bad news. But, I really hope that, as we continue to see the cost of developing software come down, that there'll be a a a a greater incentive to for for lots of people to step up and say, hey. Wait a minute. That that doesn't look right. That looks wrong to me. Let's let's fix that.
I I think I hear a connection there where, for example, in, like, DevOps space. Like, we've talked we talked about how, like, using c I d c continuous integration, integration, systems, allows you to make effectively make change cheaper, less risky. Yep. If you change fast, you can undo mistakes or you can fail forward. As the production of code becomes cheaper, it's kind of a similar effect. Right? If work is incorrect, well, it's not that big of a deal to fix it or to redo it or to pull it back. So it seems like maybe those concepts connected. It's, not just about organizational risk, but even personal risk in the organization. It's it goes down as it's easier to make change. Would you say that that's kinda fair?
I would. I mean, I think, I also feel like there's a connection back to our previous conversation on the the the we were talking about trying to fit every change into the system. You know, the the the high cost of developing software, created a lot of, you know, perverse incentives, I think. We we see a lot of those incentives play out over these projects, and they're the cause of a lot of dysfunction. I'm not naive enough to believe that, you know, the changing nature of developing software won't create new problems. I'm sure it will. But I do feel like you're right that, you know, one of the benefits of DevOps was it lets you iterate more quickly. So the cost of a mistake is, is reduced because you can fix it. You know, as we move into, and we are unavoidably moving into a time when AI tooling is used to generate code, we will probably see more of that, where changes is easier and and the state of state are less costly. And maybe that will help folks who are, domain experts and policy experts step up and share when something's not working right.
With, the last, few minutes we have here in our conversation, I wanna ask a couple of kind of feature facing sorts of questions. For one, I kinda wanna speak directly to a group of, like, real folks that are probably out there listening to this. You know? Imagine there is an IT manager or a director, you know, or other folks in that leadership structure that they're hearing this, and they're like, you know what? Yeah. I wanna I wanna advocate for this approach in my work. What could they cite as ingredients for something like I think in the book, you mentioned, like, a ninety day spec ops pilot as being a virtuous thing to go for. What are, ingredients for that?
Yeah. I I I I really wanted that chapter of the book to be something people could pick up and start to use. And I I'm actually I'm I'm working with, a loose coalition of people who are trying this in in various, governments today. So, I think as we said before, I think picking the right pilot project is is really important. You're and this is true, you know, this was just as true when people wanted to move to the cloud, and and they needed a pilot or they wanted to, you know, adopt DevOps or user testing, whatever that was, whatever that that point of change was. Somebody had to figure out, well, we I I need to advocate for this to be a regular practice in my organization. So I've gotta pick a project where I can show that it works, and it's gotta be meaty enough so that it will land with the the, you know, leadership or stakeholders. And they'll say, oh, yeah. That that actually I can see the value in that. But it can't be so big and complex that I fall on my face the first time I do it. So I try and lay out in the book, a a a set of conditions for for picking the right project, but it really depends on the environment that someone's in. We talked earlier about the tooling. You need your tools. They these are not elaborate expensive tools. They're probably all already available in whatever agency you're working in. I list out the tools, in the chapter. And then putting your team together, picking the right kind of people. One very important kind of person that you need is you need a domain expert. You need someone who's gonna be able to so Ryan, when you point your AI tool at your legacy code base and you generate the first version of a a specification, you're gonna inspect it and say, yeah, this looks right to me. This looks like it makes sense. AI tools are very good at generating, correct looking, content. It needs to be validated. You need a domain expert to look at it and say, yes. That's right. That that makes sense. That matches this provision of the statute. Or no. That that that changed. It wasn't updated here, but it that has definitely changed. We need to fix that. Having those kind of people available, is is is super important. But beyond that, you really don't need anything else. You need an AI tool. You need a version control system. You need a way to track track, track work like a Trello board or, you know, GitHub issues. You don't need anything really elaborate. It's probably already all available at your agency. You just need to pick a good, pilot project, and you need to find your domain domain experts. And, and you're off and running.
Mark, thank you so much for joining us here on Civic Tech Chat. From this conversation, I have no doubt that, practitioners and and others out there are gonna have nuggets they can bring into their work or just interesting things into their day.
Brian, thanks for having me. I really appreciate it.
Visit us on the web at civictech.chat, or subscribe to us for content updates wherever it is you download your podcasts.
