Author: Courtney Nash

Transcript

Nash: We’re going to kick off this talk with vultures. We’re going to talk about predators a lot. In the early 1990s, or the late 1900s, a painkiller called diclofenac became available in a new and affordable generic form in India. Farmers quickly realized that this generic form of diclofenac was incredibly useful in treating pain and inflammation. It’s a widely used human drug, but they were using it in their livestock. This is a country where there’s lots of livestock, and in general, they’re not to be eaten. They are there to help out, to exist. It was adopted almost universally across India, which had at the time, approximately 500 million said livestock animals. What no one knew at the time was that even small residue amounts of diclofenac, when consumed by vultures, would cause kidney failure, killing the vulture in a matter of weeks. The crisis that followed was historic in its speed and its scope.

The three most common species of vultures disappeared from India. They all died in less than a decade. The region’s primary scavenger became carrion. It quickly became obvious that India was actually dependent on vultures to clean up livestock animals as they passed away. Cows, sheep, lots of other livestock. Without the infrastructure, which had previously been the vultures, to process these carcasses, the shock of the vulture collapse led to carcasses piling up all over India. A public health emergency followed, the depth of which researchers are really now just starting to fully disentangle.

This was the consequence that came out of that. This was not 500,000 people that died in India total, this was an extra 500,000 people that died, above the standard population death over the course of those 5 years, that was attributed by researchers to the disappearance of the vulture population. These other deaths came from other sources. Increased rates of disease, like rabies, and decreased water quality due to contamination from carcasses piling up in the local water supplies. This is a very fancy graphic of that.

In addition to the loss of life and death, the vulture crisis was costing India billions of dollars. The cleanup that the vultures had simply done in the past had to be accomplished with human crews, vehicles. Eventually, they had to develop expensive new facilities like rendering plants, which if you want to sleep at night, don’t Google animal rendering plants like I did. There were two researchers who primarily dug into this, Saudamini Das and her co-author N.M. Ishwar, and they estimated that each vulture, singular vulture in India, was providing between $3,000 and $4,000 of economic value. Probably more than that, but that’s what they could really tie to the factors that they were able to study.

This is one example of intertwined cultural, biological, and agricultural systems. Another one was the Great Hunger or the Potato Famine in Ireland, and also the Four Pests Campaign that led to the Great Chinese Famine in the middle of the 20th century. What’s really key to this is that vultures were an unexamined, unknown source of resilience within the agricultural landscape in India. We’re going to kick this off with that little story, and we’ll talk about some of the kinds of systems where you’re looking for resilience as well, day in and day out.

Background

I have a background in cognitive neuroscience. Then, I ran off and joined the internet and worked at all these places. In 2020, I started something called The VOID. The VOID is a large database of public incident reports. The goal of this when I first started it, and still to this day, was to make these incident reports available to anyone to raise awareness of how software failures impact our lives, and to increase understanding, and try to help us learn from those failures, and make the internet a more resilient and safe place. This is Honeycomb. Fred Hebert writes a lot of their incident reports.

My favorite part of this is where it says things are bad. You may be familiar with that phenomenon. There are over 10,000 public incident reports in the database at this point, from I think about 2008 up to present day, in a variety of formats. Big, long, comprehensive postmortems, but there’s also things like tweets, and media articles, and conference talks, and all those things. There’s a lot of metadata in The VOID. In the past, I’ve done a lot of analysis on that.

Why the Interest in Automation and AI?

I started to get very interested in what I could learn about automation. That’s what we’re really here to talk about. Much of my work prior in The VOID has been looking at quantitative data around things like duration and severity, trying to dispel some common beliefs and myths around things like MTTR. There’s a bunch of my previous work that had done that. There’s a lot of reasons why I started wanting to dig into automation and AI. Let’s start off with an example you might be familiar with. Anybody know what this is? Knight Capital? On August 1st, 2012, Knight Capital Group, a financial services company, released an update that ended up losing the company $460 million in 20 minutes, severely impacting the valuation in numerous other industries and companies.

By the next day, Knight Capital’s value had plummeted by 75%. Within a week, the company basically ceased to exist. It was an extinction event for that company. This is a quote from the Security Exchange Commission’s report on that Knight Capital event. I’m not going to go into a great detail of this, but I wanted to talk about this quote where they talk about being able to keep pace. John Allspaw has done a really great deep dive into this write-up. He wrote about our sharp contrast between our ability to create complex and valuable automation and our ability to reason about, influence, control, and understand it in even normal operating conditions. Forget about the time pressured emergency situations we often find ourselves in. I have some links and references at the end if you really want to dig into this. The SEC went into all the ways that they wanted to think that Knight Capital screwed up.

The biggest one being how an automated process could lead to something like this. Two caveats. I hate that write-up. Don’t go to that thinking that’s like some source of truth or anything. It was fascinating to watch a body like that have to dive into technical details and start to wrap their heads around something as complex as automation in financial services. That’s the first thing. Go read John’s blog post. The second thing is, I’m not anti-automation or necessarily AI. I’m just a smidge more pro people. Automation is not going away. It’s a necessary and generally beneficial thing for businesses, for all of us. What I’m here to challenge is our current mental models about automation and AI, which are founded in some not just outdated but originally misguided ideas.

What Is Automation?

Before we go down that road, we want to talk about what do we mean by this. Let’s try to have at least a shared definition, because it’s arguable that all software embodies some form of automation. It’s a computer doing something for you. That’s fine. I wanted to have a shared definition that I could use for the research that I did, based on what people assume automation does. Here’s the definition. Automation refers to the use of machines to perform tasks, often repetitive, in place of humans, with the aim of accelerating progress, enhancing accuracy, or reducing errors compared to manual human work. How many of you feel like that’s not bad? My goal is by the end, I’m going to convince you we’re all wrong. We’re all wrong, because I’ve wandered into this with that belief and assumption as well.

Functional Allocation and the Substitution Myth

Let’s get into the nerdy cognitive science and research stuff. The underlying and often unexamined assumptions about automation is the notion that computers and machines are better at some things and humans are better at others. Historically, this has been characterized as HABA-MABA, humans are better at. It used to be MABA-MABA, men are better at. We got that one figured out, supposedly. Also known as the Fitts List, based on the work of Paul Fitts. He was a psychologist and a researcher at The Ohio State University in the mid-1900s.

More recently, researchers who are starting to dig into this have described this as functional allocation by substitution, or the substitution myth. These were researchers such as Erik Hollnagel, Sidney Dekker, David Woods, trying to dig into the idea that we can just substitute machines for people. Research from other domains has recently really started to challenge this substitution myth and functional allocation. One of the primary ones comes from aviation and automated cockpits. That research, I’ll get into it here in a bit more, but really often found that automation, contrary to what we think and assume, contributes to incidents and accidents in unforeseen ways. It tends to impose unexpected and unforeseen burdens on the humans responsible for operating in those systems with automation.

We go back to the Fitts List right here. There’s fixed strengths and weaknesses that computers and humans have, and all we have to do is just give separate tasks. We’ll just budget this stuff out over here and this over here, and we’ll all just go off and do more stuff, and everything will be happy and great. Right? Not right? Not how it usually works? It’s certainly not always the experience of a lot of people who work in complex software systems. I mentioned a couple of researchers, Sidney Dekker and David Woods. This is from a paper that they wrote in 2002. I’m going to just step through a few of the things that they talked about having the consequences of this assumption built into your automated systems. As you’re thinking about this in your world, you could think about things like CI/CD deployment pipelines, or load balancing, or anything that automatically does those things for you.

The first one is this, that as designers, either us designing our own little bits of automation, or people who are putting automation into tools, tend to have the desired outcomes in mind, and only those outcomes in mind. They don’t really know necessarily or understand or think about the other consequences that could come out of the automation that they’re designing. The second one is, it doesn’t really have access to all of the real-world parameters that we have. This is something that you have probably already heard quite a lot about, about autonomous cars and other kinds of systems like that, that their model is their model that we give them, but it’s not entirely the real-world model. Of course, people are trying to work on developing much more real-world models, but we’re certainly not there yet. What happens is, without that larger broader context, it actually makes it harder for the humans who have to solve problems when they arise in those environments.

If you’ve ever not known what dashboard to look at when the shit’s hitting the fan, that’s a very simple way of thinking about this particular problem with our assumptions around how we build automation into systems. The third one is that automation just doesn’t take stuff away, automation adds in ways that we may not have thought of as designers of automation. It transforms people’s work and tasks, often forcing them to adapt in unexpected and novel ways. Digging through logs, I’d mentioned that before, but you have to start looking in other places, you don’t actually know what’s happening, and you don’t have any access to what that automated system is necessarily doing.

Then the last one is, it doesn’t really necessarily replace human weaknesses. What it can do actually is it creates new human weaknesses, or new pockets of problems. You’ll see this in the development of knowledge silos or pockets or silos of expertise. Some people have developed a certain degree of expertise with that system, but no one else has because they haven’t had experience with it. Amy the Kafka wizard is on vacation when things go sideways and nobody but Amy knows how to fix it, that’s because she’s the only one who’s actually developed that set of expertise with that automated system, and for everyone else it’s effectively a new weakness that they don’t have in the face of that system. That’s one piece of work done.

The Ironies of Automation

This next one is the ironies of automation. This was work done by a woman named Lisanne Bainbridge in the mid-80s, probably before Alanis Morissette wrote this song. Lisanne Bainbridge was a cognitive psychologist and a human factors researcher. A lot of her work was in the ’60s to ’80s. She obtained a doctorate for work on process controllers, but then went to work on things like mental load, cognitive complexity, and related topics. Her work was based primarily on direct observation of and interviews with experts, in this case pilots. She spoke to them. She watched them doing the work that they needed to do. This was as we were beginning to increasingly automate cockpits in aviation that pilots were then having to work with on a daily basis. She found this set of ironies of automation. In the paper that I referenced right there, it’s not like there’s a solid one to seven list or something. I’ve glommed them into four just to try to make things a little bit easier for us to get through here.

The first irony is that humans are the designers of automation, and yet their designs often have unanticipated negative consequences. I’ve already mentioned this, but she really saw how that worked for pilots working with automation in cockpits. Second, it’s monitoring turtles all the way down. I don’t know if any of you know Shitty Robots. She just has this whole project. It’s amazing. Human operators still have to make sure that the automation is doing what it’s supposed to be doing. Even if they don’t necessarily have access to exactly what it’s doing or why it’s doing that. The automation has supposedly been put in place because it can do the job better, but then you have to be certain that it’s doing that job better than you. The paradox of not knowing what to do or knowing what to look for, these systems are generally impenetrable to that real-time inspection of what it is they’re doing.

Then, the worst part of this is that when that automation does fail and the human who was supposed to be monitoring it does have to take over, they often lack the knowledge or the context to do so. This is the Amy the Kafka expert example, writ large. It’s due to the fact that proper knowledge of how a system works and then therefore how the automation within it works, requires frequent usage of and experience with that system. Now you’ve been put over here and the automation is over here, so how do you have experience with that system? When you have to come in and deal with that, you actually are at a disadvantage, given that.

This is the irony three that’s leading into the irony three. When it does fail, the scale and the expertise and the knowledge required for the tasks to deal with that is often much larger and much more complex than humans who have no experience with that stuff. They don’t really necessarily know what to do. This is one of my favorite all time. It was like the food cart for the airport. It was just going crazy. This one guy finally just rolled up and bulldozed his whatever cart thing into it. It creates new types of work and it often leaves humans left to cope with a situation that’s actually almost worse or more complex than the one that was supposed to have the automation in there to do that work for them in the first place. The last one, this is Homer. This is the power plant one. He’s like, it’s as big as a phone book, and he’s supposed to be reading the instructions.

A lot of times we’re told, don’t worry about that, but when things go wrong, just use the runbook. Runbooks, all of those things can’t cover all the possibilities of the things that the designers couldn’t imagine when they were designing the automation and writing the runbooks about it. As we all know, they’re often not updated regularly. Again, the human is supposed to monitor the monitoring or the automation and then fill in the gaps. We’ve taken them out of that situation and told, go do something else. To debug a system with automation in it, you need to know the system overall, but what the automation is doing and how it’s doing it. You can see how this whole thing starts to collapse on itself and become fundamentally more complex than anybody intended. This is the quote, my favorite Bainbridge quote, “The more advanced a control system is, the more crucial may be the contribution of the human operator”.

Research from The VOID (Thematic Analysis)

I thought to myself, I probably have some data on this stuff. I’ve got over 10,000 incident reports with lots of words in them. As I had mentioned in the past, I had done a lot of qualitative research from incident reports in The VOID. I’ve looked at reports on duration and severity and the relationships between all of those things. This time I decided to take a different tack. The work that I did on the data I could find in The VOID falls under a category of research called thematic analysis. This is something that’s very prevalent and common in especially social sciences and also in areas that just have a huge amount of unstructured data, also known as text. 10,000 something incident reports with people talking about what went wrong and what happened. This is the fancy pants diagram, but the idea is you read all of the data that you can, and then you’re going to code those data.

Then you’re going to look at the codes, go back, revisit that again. Does this fit? Does this not fit? You’re creating your own model right here. Then eventually you look at those again, and those start to cohere, hopefully coalesce into themes. You can think of codes almost as like tags for similar items of text, not necessarily just individual words, but conceptually similar things. Then, as you roll those up to themes, those capture hopefully a prominent aspect of the data in some way. This is something, like I said, that’s done a lot in sociology work, anthropology, but also not just social sciences, but large bodies of unstructured text. I did not use a large language model to help me do this. Irony, I know.

Here’s what I did have to deal with. Over 10,000 incident reports and me. I did this work. I did not have, I wish, a round of lots of folks to help me with this. I’ll explain why I didn’t use a large language model. I had to get through, how do I get 10,000 down to a number that I can manage to sift through myself? Also, not every report in there is going to talk about automation. I had to find the ones where when people write up a report, they actually talk about that. Some of that was easy to knock away. All of the ones that were like media articles, like, no, the Facebook went down.

Those were not necessarily included out of the gate. In the end, what I did was I did a search query through all of the data based on keywords that I solicited from experts in their field. Folks who have deep, dark experience with their systems going down involving automation. I was like, what keywords should I include to find things that might have had automation involved when things went sideways? This is how we ended up with things like automation, automated, obviously. Like load balancing, self-healing, retry storms, these kinds of things. There’s probably more. The idea was to get a sample of the population.

If any of you ever took your psychology or social sciences class, hopefully that will make some sense. We’re not going to look at everything, but we’re going to look at a subset of it, assuming that it’s a pretty good representation of the larger whole. We took this 10,000-plus set of incident reports, and that query set reduced it down to just shy of about 500. Then, the next thing I had to do was actually read all of those. The good news was I had actually read a lot of them, but then I had to go back and read them very carefully and start looking for this.

Here’s the biggest reason why I didn’t use a large language model or other automated forms of doing this, as it requires expertise with the subject matter to a degree that you could read something and be like, that was an automated process. This gave me a set to look at, but I still even whittled some of them out. There were things where tooling that automates stuff broke, but that wasn’t actually the cause of the incident. There were incident reports for like Chef’s automated something, something broke for Chef, and they’re like, this happened. If that makes sense, I had to whittle that down. You couldn’t just turn a large language model at this and expect it to have that level of expertise. Maybe they will get there.

Right now, I can tell you that it requires a human to look at and read all of these to start doing this coding. I did an initial pass, and then, I didn’t really know what I was looking for. I was just looking for incidents that looked like they had automation in them. This is the way this process works. You do this, you keep reading, you start to notice patterns, you start to notice things that share similar effects, or context, or what have you. The codes start to develop from there. As I started to get what looked like a set of codes of automation, which I’ll show you, I went back and re-read everything again. Was this right? Did I see that again? There was a lot of reading, re-reading, refining.

The only thing that I did not do in this process that I would love to be able to do at some point or go back and do again is from a purely academic perspective, if you were to try to go publish this in an academic journal, I would have had a set of other coders, other reviewers come through and look at and decide if those codes actually were what I said they were and they were showing up in the way that I said. I would have a score of interviewer reliability. For full transparency and for any academic nerds, I didn’t do that. I do hope to.

What did we find? These are the set of codes that came out of this work. Then, here’s a fun pro tip. If you were working in an environment where people demand quantitative numbers, you can take qualitative work like this and you can put a number on it. If you have to give somebody a number, this is the way you do that. This is the literal methodology for doing that. You go read a bunch of stuff, you set the codes to it and you say, 77% of those codes had to do with automation being a contributing factor in the incident. We’ll walk through these just a little bit more here. These don’t add up to 100. The reason they don’t add up to 100 is because multiple codes could be present in a given incident. This isn’t supposed to be a pie chart where one thing is only one piece of it. That’s actually going to become important shortly. The vast majority of the time, automation was a part of the problem and it took humans to solve it. Those are the first two pieces of that.

Automation in some way, shape, or form was one of the reasons why that incident happened, potentially, definitely usually more than one. There is no root cause. Then, humans had to come in most equally as often, three-fourths of the time, to try to resolve that situation. Here’s the other codes that came out of this. Automation, work on automation, development of automation, fixing automation, adding more automation, that’s an action item code. When automation came up as an action item in the incident report, that’s where that came from. Detection. I have problems with my own self with this one because I think automation being involved in detecting incidents is a lot higher.

The only way I could do that, if somebody said, our automated monitoring system picked up an issue, and if you don’t tell me that, then I don’t know. I honestly suspect that we detect a lot of incidents because of automation. I think that number is maybe not the most reflective, but I think it’s also just the nature of the way people write incident reports. They’re three or four degrees removed, a public incident report, from what actually happened. A lot is assumed and a lot is not said. The other two places, the other two codes for automation in software incidents in The VOID was that it was actually part of the solution or maybe even it was the solution. It’s very small where automation detected the problem, automation had caused the problem, and automation solved it. It’s like very small. I think there were two. They were really happy about that, the folks that found those.

Then, the last one is when automation would hinder the remediation. How many of you have had this experience? Yes, this one’s fun. I call this one the gremlin. They all have their own little names and personas. This is the set of the ways in which automation might be involved. These were the codes and the quantitative summation of those codes. This is the one. It’s Dave and HAL. It’s hidden behind the text. If you remember one thing, it’s this, 75% of the time, you all still have to come in and fix something when automation makes it a lot harder.

Automation Themes

These are the themes then that came out of all of that. The first one, and this was what I was trying to get at in that slide where the numbers don’t add up to 100, is that automation can and often does play multiple roles in software incidents. It’s not nearly as clear cut as we want to imagine it as when we design it and the things that it will do.

Then when it doesn’t work as designed or intended, it often is part of the problem, requires humans, and it frequently becomes a contributing factor without the ability to resolve those independently. That’s the first big theme. The second one is that automation can unexpectedly make things worse. Not only does it contribute to, but it behaves in ways that make incidents worse. This can take the form of things like retry storms, interference with access to automated logs or data, unexpected interactions with other components that then expand the surface area of the incident. It could mean you can’t get into your database building. Facebook, that was a fun one. It can really show up in incredibly unexpected ways and ways that then require a lot more work and effort to try to figure out while you’re in the midst of trying to resolve an incident.

Then, this was the biggest one. Humans are still essential in automated systems. We still have to be there, not just to do the work that we do day-to-day to make those systems work, but to help solve problems and figure out what to do when the thing that was supposed to do our work stops being able to do that work for us.

Better Automation Through Joint Cognitive Systems

That’s really not very optimistic and fun, you say, and I know. I like to believe that there is hope. Despite all of these challenges and the ways that I’ve brought up that automation can be a total pain in our butts, it’s not going away. There are ways that we have found it to be beneficial. As I said, I’m not anti-automation. I’m really just more pro people. What can you all do? What can developers, DevOps folks, site reliability engineers, but especially people who are building and designing automation and AI tooling, what can you do? I’m advocating for a paradigm shift, an adjustment to our mental models, as I said at the beginning. Instead of designing and perceiving automation and AI as replacements for things that humans aren’t as good at, we should view it as a means to augment and support and improve human work. The goal is to transform automation from an unreliable adversary, a bad coworker into a collaborative team player. The fancy pants academic research term for that is joint cognitive systems. Let’s talk a little bit more about that.

I’m going to talk through this just really briefly. As I’ve mentioned, automation often ends up being a bad team player, because we’ve fallen prey to these Fitts style conceptualizations of automation. Failing to realize that it adds to system complexity and coupling and all these other things, while transforming the work we have to do and forcing us to adapt our skills and routines. David Woods and some of his colleagues have proposed an alternate approach or an alternate set of capabilities and ways of thinking about working with machines, this is their un-Fitts List. It emphasizes how the competencies of humans and machines can be enhanced through intentional forms of mutual interaction. Instead of, you’re better at this and I’m better at that, and we’re going to just go do these things separate from each other, we’re going to enhance the things that we do intentionally and thinking about that as mutual interaction, joint cognitive systems.

I think that the biggest one that I just really want to point out about this, and then I talked a little bit about this at the beginning, is the bottom one really. Machines aren’t aware of the fact that the model of the world is itself their model. It’s all this fancy stuff around ontology but in the way that machines model the world that we give them and the way that we model the world that we exist in with them. Starting to rethink what that looks like versus you go do this and I’ll go do that, and other things. That’s the machine side on the left and the people side on the right. We have all these other abilities and skills that we’re not limited in in the way that machines are constrained. We have high context, a lot of knowledge and attention driven tasks that we do. We’re incredibly adaptable to change, and typically because we have so much context expertise, we can recognize an anomaly in those systems really easily. We have this different ontological view of the system than machines do.

Text stuff number two. The paper is called, 10 Challenges for Making Automation a Team Player in Joint Human-Agent Activity. This stuff was written a little while ago but the words agents still work in this context if you think of them as machines, computers, automation, what have you. They make an argument, a case for what characteristics you need to have a joint cognitive system that supports this work that we do with machines. They provide 10 challenges. I always get asked, what are the most important of these that I think we should focus on? Here they are. To be an effective team player, intelligent agents must be able to adequately model the other participants’ intentions and actions vis-a-vis the joint activity state, vis-a-vis what you’re trying to do, aka establishing common ground.

If you think about the way you work with team members, let’s say in an incident or in trying to figure out some complex design of a system or something like that, you have to make sure that you understand each other’s intentions, that you have common ground of what it is you’re trying to do and how you think you should get there. This is a really important part about how joint cognitive systems, whether it’s your brain and my brain or my brain and a computer brain actually successfully work together, not how they necessarily currently work together.

The second one is that us and our agent friends, our computers, our automated systems must be mutually predictable. How many of you feel like your automated systems are mutually predictable? The last one is that agents must be directable. When we have goal negotiation, which is further down here, number seven, if we’re going to have goal negotiation, then one of the other of us has to be able to say, “I’m going to do that, you do that”. That’s not usually the case with our automated systems. Not currently. It’s the capacity to deliberately assess and modify your actions in a joint activity.

Conclusion

This is the set of references. I just want to conclude by begging people who are designing automated systems, who are working towards AI in especially developer tooling environments where the complexity is incredibly high, to really dive into this stuff. Take this work seriously. This is research that has changed the way that healthcare systems work, that nuclear power plants work, that airplane cockpits that we fly in every day work. That is my challenge to you and my hope that we can rethink the way that we work with automated and artificial intelligence systems that is mutually beneficial and helps make the work we do better.

Questions and Answers

Participant 1: Have you seen any examples in all of the incidents that you’ve looked at of companies adapting in the way that you’ve recommended they adapt?

Nash: I would argue almost exclusively no. Not that they’re not adapting, because we know that adaptive capacity in these systems is what we do and how we manage these things, but because they don’t write about it. It’s a really important distinction. I started this talk off with an example from an incident report from Honeycomb, because I consider that, and the work that Fred Hebert does, analyzing and writing up their incidents to be almost exclusively the gold standard high bar of doing that. There are organizations that do adapt and learn, and they don’t talk about it in their incident reports. They talk about the action items and the things that they do. Honeycomb, a few others, talk about the adaptations. They talk about what they learn, both about their technical systems and their sociotechnical systems. The short answer is no. I wish people would write about that more, because if I could study that, I would be a really happy camper.

Participant 2: I work in building automated tools, and I have colleagues who work in building automated tools. The thing that really makes me just slam my forehead down on my desk every so often is the one where we have an outage or an incident because an automated tool presented a change nominally to the humans who were going to review the change, and the human went, the tool knows what it’s doing, and they rubber stamp it. It looks ok, but they’ve lost the expertise to actually understand the impact of that particular config change or whatever it was. There doesn’t seem to be a common pattern of saying, other people wrote this tool. They are just software developers like you, and this thing might have bugs. This is before you put AI in it.

Nash: Yes, not even bugs, you may not know the extent of what it does, and you don’t know all the conditions it’s exposed to.

Participant 2: Correct. Given that, in terms of this compact idea, like giant flashing lights saying, don’t trust me, I’m trying to help you, but I could be wrong, would be great. How does that actually happen from what you’ve seen?

Nash: The other question I commonly get is, what examples have you seen of this? The answer is none, because no one’s tried to really do this yet, that I know of. I don’t know what that is, because it’s generally very context-specific. The goal would be for the tool, the automation, to tell you, this is about to happen, and here’s the set of things that I expect to happen when that happens, or here’s what we’re expecting to see. That’s part of the being mutually predictable and then being directable. Yes, do that, don’t do that, some of that. I think, giant warning lights, I might be wrong, but I’m trying to help you. That’s also what our colleagues are like. It’s being able to introspect what it is that the tool is trying to do. No, I don’t just mean documentation. I mean the way that we interact with that thing.

The other thing I do want to bring up is something you mentioned, which is another term in the industry that comes up a lot in autonomous driving situations is automation complacency. I didn’t talk about that in here, it wasn’t in this set of research that I brought to this talk. It is also this notion that as we say, “Automation will just do this stuff for you”, then you’re like, “I’m going to not worry about that anymore”. It’s not just the lack of expertise with the system, it’s like, no, I trust that to do that, you become complacent. This is how accidents happen in a self-driving car situation, but it’s also a phenomenon within these systems as well, and like, “Yes. No, don’t worry about that red light, that one comes on all the time”. It’s in that same sphere of influence.

The biggest thing I would ask for people designing these is, when some form of automation is going to do something that it could give you as much possible information about what it’s going to do and what the consequences of that might be, and then you get to have some thought about and input into what that would look like.

Participant 3: I am working in the AI agent space, so I am dealing with this stuff every day. I think among the pillars of the joint collaboration that you listed, don’t you think that the mutual predictability is somehow the most problematic one because actually it has to do with the tooling test in some sense? You are assuming basically that you are predictable for the machine and the machine is predictable for you. It’s a loop. You know that there are some advanced LLMs which started to fake it in some extent in order to escape from your expectations. That’s something which is about a mutual trust. I see it personally as a problematic point.

Nash: It is. Some of these aren’t as separable. You can’t just be like, I’ll do this one thing. If you’re mutually predictable, if something’s not predictable enough, you have to be able to engage in goal negotiation. Then other things here, like observe and interpret signals of status and intentions. When the predictability isn’t there, what other avenues do you have to try to understand what’s going to happen? The goal is to be mutually predictable, but even we’re not all mutually predictable with each other.

Participant 3: Also, maybe engaging in a goal negotiation is not enough. Maybe you should be able to assess the results of the goal negotiation to say, ok, we understood the same thing. That’s really challenging.

Participant 4: A lot of this puts in mind something many of you have possibly seen before. It’s an IBM presentation slide from about half a century ago, which reads, a computer must never make a management decision because a computer can never be held accountable. Or as a friend of mine framed it to me, a computer must never fuck around because a computer can never find out.

Nash: Accurate.

Participant 4: It seems to me, someone I’ll admit as being a little bit of an AI pessimist or whatever, that there are a lot of cases where that lack of accountability is really more of a feature than it is a bug. Even less of a pessimistic framing, there’s a lot of instances of like AI boosterism where a complete lack of human involvement being necessary is touted as a goal rather than something concerning about it. Do you have any insights or input on how we can apply this framework or paradigm shift you’re talking about in cases where bringing up those sorts of concerns are very much not wanted?

Nash: Get venture capitalism out of tech? All I have are spicy takes on this one. Like late-stage capitalism is a hell of a drug. As long as our priorities are profit over people, we will always optimize towards what you’re discussing. The goal is to fight back in that locally as much as we can and where we can, which is why especially I make the appeal not so much to people who are trying to make broad-based cultural AI things, but people who are building tooling for developers in this space who then build things that impact our world so heavily to care about this stuff. Capitalism’s hard, but hopefully locally folks could care about this a bit more and improve the experience for us.

See more presentations with transcripts

.
From this page you also have access to our recorded show notes. They all have clickable links that will take you directly to that part of the audio.

Transcript

Nash: We’re going to be talking about comparing apples with Volkswagens. I have a background in cognitive neuroscience. I was fascinated by how the brain works in terms of how we learn and how we remember things. Along the way, this funny thing called the internet showed up, and I ran off to join the internet. I worked at a bunch of places. I am now at a company called Verica. There, I started this thing called the VOID, the Verica Open Incident Database, which I will get to and I will talk about.

What Is Resilience?

We’re in the resilience engineering track. This is obviously a topic near and dear to my heart. You’re going to hear this a lot. We’re going to talk about, what is resilience? I have in this VOID that I’m going to talk about lots of incidents. I’m going to share with you some things that people say about resilience. This is actually not an incident report, I lied. The first one is from Gartner. They said that cloud services are becoming more reliable, but they’re not immune to outages, which we would all agree with that. The key to achieving reliability in the cloud is to build in redundancy and have a clear incident response. We’re munging words already here in this space. How about what Facebook said? To ensure reliable operation, our DNS servers disable those BGP advertisements, so on and so forth. Duo is committed to providing our customers a robust, highly available service. An issue exposed a bug in our integration test suite. Microsoft, although the AFD platform has built-in resiliency and capacity, we must continuously strive to improve through these lessons learned. Also agree. We’re using a lot of similar words and we’re conflating things that we don’t want to conflate. I think as engineers, we like to be precise. We like to be accurate in the terms we use and the way we use them.

This is my general feeling about where we’re at as an industry right now. We talk about these things, but I don’t think we really mean the same thing. I don’t think we all know what we mean when we do that. The definition of resilience that I’d like to offer is that a system can adapt to unanticipated disturbances. This is a quote from a book by Sidney Dekker, “Drift into Failure.” I’ve highlighted the things here. Resilience isn’t a property. It’s not something you can instill in your systems and then it just exists and you have it. It’s a capability. It’s an ongoing capability. It’s actions that make resilience. You notice the things that he talks about in here is capability to recognize the boundaries, to steer back from them in a controlled manner, to recover from a loss of control, to detect and recognize. These are all things that humans do, that you do to keep our systems running the vast majority of the time. There are things you do when they stop doing what we hope they will do. What I’m here to really talk to you about is our efforts to try to measure this thing that we still haven’t even collectively defined or agreed upon. We’ve already got it. We’re already starting on shaky ground here.

Can We Measure It?

Let’s talk about metrics with this. This is the gold standard. We talked about this, meantime to resolve, to restore, to remediate. It’s probably something you’ve all heard. How many of you use this in your work? I want to talk a little bit about the origins, the history of this term. Many people might be familiar with this from the DORA work. MTTR is one of the four key metrics that they talk about in terms of high performing teams. That’s not where it came from. Actually, MTTR came from old school line manufacturing, widgets, things that you make, and you make them over again. Then sometimes, either the process or the conveyor belt or the parts break down, and you have this very predictable time window that you know, over time, it takes us this long to fix widget x or that part of that. That’s where mean time to repair came from, a predictable conveyor belt style production environment. Does that sound like anything that any of you deal with? No? This was the formula that came out of that environment. It’s very straightforward. It works. It works in that environment, but this is our environment. This is the reenactment, actually. It’s not the original. This is from Will & Grace. It’s one of my favorite episodes of, I Love Lucy. I think it really captures what we’re dealing with in our reality.

MTTR: What Is It Good For?

I want to talk about the way we talk about MTTR. These are actual terms or ways that people talk about MTTR from their incident reports, or from the internet. MTTR measures how quickly a team can restore a service when a failure impacts customers, allows enterprise level organizations to track the reliability and security of technical environments. Allows teams to set standards for reliability, accelerate velocities between sprints, and increase the overall quality of the product before it’s delivered to the end user. MTTR captures the severity of the impact, shows how efficiently software engineering teams are fixing the problems. Specifies the severity of the impact, or perhaps offers a look into the stability of your software as well as the agility of your team in the face of a challenge. It also encourages engineers to build more robust systems.

There’s that word again. Is one measure of the stability of an organization’s continuous development process. Helps teams improve their systems’ resilience. Evaluates the efficiency and effectiveness of a system or service. Measures the reliability and stability of the software that is delivered. Helps track reliability. Helps teams to improve their processes and reduce downtime. Assesses how resilient the software is during changes in runtime. Helps track the performance of both the Dev and Ops sides of the house. Can be a great proxy for how well your team monitors for issues and then prioritizes solving them. Low MTTR indicates that any failures will have a reduced business impact. Serves as a direct indicator of customer satisfaction. Directly impact system reliability, customer satisfaction, and operational efficiency. These are laudable things to want to know or understand or be able to measure. Could we all agree that one number couldn’t possibly tell you all those things? Yes. I have even worse news. That number doesn’t tell you what you think it tells you, and I have the data to prove it.

MTTR In the Wild: Data from the VOID

Now we get to talk about the VOID. The VOID, the Verica Open Incident Database is something I started almost two years ago. It came out of research I was doing for product, for this company Verica. We had a lot of things that were focused on Kubernetes and Kafka because those are really simple, and no one ever has any problems with them. Along the way, I wanted to see what was happening in the wild. I started collecting incident reports for those technologies. Then I just kept collecting incident reports. Then John sent me a whole lot of them. Then people kept sending them to me, and the next day I had like 2000. Now we have over 10,000. These are public incident reports. Have any of you written an incident report and published it on the internet? I read them. We collect metadata on top of these publicly written incident reports. Maybe about 600 organizations that are in there, some small, some large. Large, gigantic enterprises, small 2-person startups across a variety of formats, so retrospectives, deep post-mortem reviews, those things are in there, but so are other things: tweets, news articles, conference talks, status pages. I have a broader research goal that is why I have all of these things. We collect a bunch of metadata, the organization, the date of the incident, the date of the report, all of these things, if they’re available in these reports. The last one being duration. If it’s there, if it’s in the status page metadata, or if the author of the report tells us that, we are going to use that information. I want to talk about some of the limits of this duration data, because that’s the foundation of MTTR. You take all of these, and then you average them over time.

Duration: Gray Data

John Allspaw has done a great job of describing these types of data in general, these aggregate metrics as shallow data, but duration is a particularly gray version of shallow data. I want to show you how murky those shallows are. The problems that we have with the data we’re feeding into this metric, is that duration is super high in variability but low in fidelity. It’s fuzzy on both ends, like how do you decide when it started or when it stopped? Who decided? Was that automated? Was that updated? Did it ever get updated? Could you have a negative duration incident? Yes, you could. It’s sometimes automated, sometimes not, all of these things. Inherently, it’s a lagging indicator of what happened in your system and it’s inherently subjective. When you average all of those together, you get a big gray blob, which doesn’t tell you anything about what’s actually happening below.

Let’s get into the weeds now. There will be some statistics. Everybody has seen a normal distribution? We all know what this is. It’s a standard bell curve. The mean is smack in the middle of that. If you have a normal distribution of your data, then you can do all kinds of cool things with the mean and standard deviations, and all this really great stuff. Your data aren’t normal. Nothing’s really normal when we do. These are actual histograms of duration data from incidents in the VOID. To make these histograms, we just bin the durations that we find, so everything under an hour, we count those up. Everything under two hours, we count those up. You all are pretty used to seeing histograms, I think. These are your data. I’m not making these up. There will be a simulation. These aren’t simulations, this is real, what you’re telling us is happening. Every single company’s incident data, if you share them with me, look like this. I urge you if you haven’t shared them to go and look at them, because this alone means that you can’t take the mean, and that the mean is meaningless. If that alone doesn’t convince you, then we’ve created a super cool new product. MTTR, it does all these things. It’s the ShamWow of engineering metrics. We made a new product called TTReduce, and it’s a 10% reduction in all of your incidents, magically.

You go out, you buy this product, you install it, you apply it. We’re going to compare MTTReduce to no MTTReduce. This is a Monte Carlo experiment, it’s very much like an A/B test, if you’re familiar with that way of thinking about your website data or whatever, you’re going to test a new feature or something, you compare them to each other. We have a control group, which is all of your incidents without the TTReduce product applied to it. Then this experimental group, we’ve shortened all of your incident durations by 10%. Then we’re going to run a ton of simulations where we take the mean of those data, and then we compare them to each other. If 10%, we should be able to detect that. When you subtract the experimental group from the control group, you should be 10% better. The next graph that you’re going to look at is going to show you those curves, for a bunch of companies. We’ve run the simulation. We’ve compared the mean of the control group to the mean of the experimental group. It should be a nice little curve, right around 10%. This is actually what they look like. The red one I will explain, but all of these have these slumpy, lumpy curves. Right around that 10% mark, you have plenty of other cases where actually your duration got worse to the left here, and lots of cases where you actually think you’re doing better than you did, which is not an environment you want to exist in. Because we measure things to make decisions. Why measure it if you’re not going to do something with it? Can you make a decision based on these data? I wouldn’t want to be in charge of that. The red line is one particular company that sees a huge amount of traffic and a huge amount of internet data, and they have thousands of incidents. Who here wants to have thousands of incidents? Because that’s the only way you’re going to get close to that kind of fidelity, and even then, they’re still wrong sometimes. We know that it doesn’t help. The only way to have more data for MTTR is to have more incidents. That’s not the business that we’re in. This is the talk, it’s like comparing apples to Volkswagens. Average all the sizes of all your apples and take all your Volkswagens, average that together. This is what you’re doing when you’re trying to use MTTR as a way of understanding what’s happening in your systems.

If Not MTTR?

This is always the next question. It really is incident analysis, but it’s a different focus of incident analysis. My point is that your systems are sociotechnical systems, so you need to collect sociotechnical metrics. This is totally possible. Is it hard? Yes. Is it work? Yes. Is it worth it? Absolutely. In fact, I harbor a real belief that companies that do this kind of work, have an advantage. I don’t have the data to prove that yet. I’m not sure I will. I believe that it’s true. These are some of the sociotechnical metrics that you can collect. Cost of coordination is one of my favorites. This is Dr. Laura McGuire’s dissertation work that she did, and that she’s been expanding on. Since then, it’s things like how many people were hands-on involved in the incident, across how many unique teams, using what tools, in how many Slack or whatever channels? Were there concurrent incidents running at the same time? Who was involved in multiple incidents at the same time? Were PR and comms involved?

Did you have to get executives involved? This all tells you so much more about the scale, the impact, the intensity of that particular incident. There are other things like participation, the number of people reading write-ups, the number of people voluntarily attending post-incident review meetings. Are they linking to incident reports from code comments and commit messages, architecture diagrams, other related incident write-ups? Are executives asking for them, because they’re starting to realize that there’s value in these? Also, things like near misses. Are you able to look at the times where Amy was looking at the dashboard, and was like, did you guys see this? This is weird. Is that really happening? Then you fix it before it goes kablooey. That’s pretty cool. That’s a whole source of information about the adaptive capacity of your systems, knowledge gaps, assumptions, like misaligned mental models, and this term that I use a lot, which is safety margins.

I want to talk just a little bit about how you can learn some more of these things. Then I want to give you some examples of what people have learned from this way of thinking about their systems. This is a diagram from a researcher named Jens Rasmussen, it was from 1997. It’s not like new, and it’s not really old either. He arrived at this model, which I will describe, knowing nothing about technology. This is not about software systems, but it’s pretty spot on. The notion he has is that you have these boundaries of different kinds of failure that you can cross in your system. You have your economic failure boundary up here to the right. Are the lights on? Are you paying people? Are you Twitter and you’re shutting down your entire Google Cloud thing today, just to see what happens for fun? You’ll know if you’ve crossed that boundary, pretty quickly. This one down here on the bottom right, the unacceptable workload, unfortunately, we’re all a little familiar with this one as well. In order to achieve what your organization’s goals are, are you pushing things too hard, too far? Are you burning people out? Do you not have enough people to do the things that you want to do? It’s a reality that a lot of us have to deal with. Then the one that we’re really talking about in particular here is this big left hand one, the acceptable performance boundary. That’s the one where when you cross it, things fall over. You’re in some space in here, you have a point at which your system is operating. The other thing that I want to really convince you of here is that you don’t know where that is. At any given time, it might be shifting around in this Brownian motion. The only way you really understand where those boundaries are or where you were, is when you cross them. This is the opportunity that you have to learn from these kinds of systems. If you just look at MTTR, you’re not going to know why you were here, and now you’re here. I want to talk and give some examples of these kinds of safety boundaries, these insights that people have learned from more in-depth sociotechnical analysis of their systems.

Safety Margins

Fong-Jones: It’s trading one mechanism of safety for another. We traded off having the disk buffer, but in exchange, we’ve lost the ability to go back in time to replay more than a few hours of data. Whereas previously, we had 20 hours of data on disk.

Hebert: That’s the concept of safety margin. Ideally, we have like 24 to 48 hours of buffer so that if the core time series storage has an issue where it corrupts data, then we have the ability to take a bit of time to fix the bug and replay the data and lose none of the customer information. This speaks to the expertise of people operating the system, and that they understand these kinds of safety margin and measures that are put in place in some areas, when something unexpected happens to people who have a good understanding of where all that stuff is located are able to tweak the controls and change the buttons and turn some demand down to give the capacity for some other purpose. That’s one of the reasons why when we talk about sociotechnical systems, the social aspect is so important. Everything that’s codified and immutable is tweaked and adjusted by people working with the system.

Fong-Jones: Going through the cast of characters is really interesting, because I think that’s how we wound up in this situation. Two years ago, we had one Kafka expert on the team, and then I started doing some Kafka work, and then the Kafka expert left the company. It was just me for a little while. Then the platform engineering manager made the decision of, ok, we think we’re going to try this new tiered storage thing from Confluent. Let’s sign the contract and let’s figure out the migration. We thought we had accomplished all of it. Then we had one engineer from our team sign up to finish the migration right, like already running in dogfood, make it run in prod. Then when we started having a bunch of incidents.

Hebert: There was this transfer of knowledge and sometimes pieces fall on the ground, and that’s where some of the surprises come from. It’s not something for which you can have quantitative metrics. It’s something you have qualitative metrics for, which is, do you feel nervous around that? Is this something that makes you afraid, and getting a feeling for the feelings that people have towards the system is how you figure that one out.

Nash: I’ll play one more, which is from incident.io, which is another company in the space providing tooling. It’s Lawrence Jones, who’s an engineer there. It’s talking about patterns and the knowledge they’ve accumulated over time and the patterns, these instinctual things that they’ve figured out that only these humans running these systems can recognize, and then how they’ve been using those patterns and incidents to help them muddle through that faster, then figure out what’s happening a little bit better.

This is from an engineer at Reddit. I don’t know if anybody remembers the GameStop stuff that happened a while ago, and that Reddit got just absolutely hammered by this. They did a really fantastic thing, they called it an incident anthology. They wrote up a whole bunch of their incidents. They also talked about some of the patterns across these. In particular, they talked about just the process of doing that work and of what comes out of it.

Garcia: In terms of telling these kinds of stories, it serves a purpose that is not really served by anything else that we do, even post-mortems, or even documentation, because this is basically memory. We have memory as people, and we share that memory across people by telling stories. We need to do that as engineers. The only way that we can actually do it with people across different organizations is by writing it up, and having that collective memory grow.

Nash: That’s the task at hand.

Key Takeaway

The takeaway from this talk is that it’s people that make resilience. It’s not this technology or that tool or this piece of automation. It’s the collective work that we do together, the knowledge that we build over time, this adaptive capacity for our systems. You can’t measure that with something like an aggregate metric.

Questions and Answers

Participant 1: How would you suggest getting teammates, organizations on board with more of these qualitative metrics in companies that are traditionally very quantitatively focused or care more about what was the dollar cost of this incident?

Nash: There’s a slide from a talk that a fellow named David Lee gave. He’s given it now at a couple of places. He gave it at DevOps Enterprise last year. He gave it at the Learning From Incidents Conference. He is in the office of the CIO at IBM. Not a small company, very heavily focused on quantitative metrics and measuring things. The Office of the CIO is a 12,000-person organization. He’s done an amazing job over the last year and a half now, of doing the kind of incident analysis at a really small scale. When he started, they actually did it as a skunkworks project. It was not sanctioned by leadership. There was no big program for it. They’d had some experience at doing this and approaching incidents this way. He started just analyzing a few of them and sharing that, socializing that with engineers who were involved. The engineers who were involved were like, this is really good stuff. It was very much a grassroots, from the ground-up type of way of doing it. I do believe that is the way that this works. I don’t think you’re going to have a lot of success trying to tell executives, we need to go do this. This is the vegetables in the smoothie way. If you look around, there will be like-minded fellow travelers. You could find them, and then do this on a small scale with them. The value will be demonstrated, and slowly but surely, you can build that. That’s exactly what David did in the office of the CIO. They now have a situation where they have a monthly learning from incidents meeting. They have a rotating trained group of folks. One of them presents a case from an incident, it’s not always the most severe one, it’s the one they think they’d learn the most from, they get upwards of 200 people attending these. At the executive level, they finally got buy-in. The CIO herself would show up with her direct reports, and then suddenly everyone was paying attention. Hundreds of people come to multiple meetings, hundreds of people watch the recordings. Now what he’s doing is he’s quantifying qualitative data, which you can totally do. Social scientists have been doing it forever. My advice is to start really small, demonstrate the value amongst fellow travelers, and then you can build it and scale it from there.

Participant 2: In our organization we collect metrics about incidents and issues, there has been this thing about the collective memory or you know something that maybe on the back of your head is a blowup, because if someone is not paying attention, and your team is not aware, or if someone leaves with the knowledge. What are the top actions that you derived from the metrics that you ultimately shared the knowledge and improved the way that the teams handle their incidents?

Nash: I really think the cost of coordination stuff is really valuable here, because a lot of times the effort to manage incidents is obfuscated from the rest of the organization. Things like MTTR contribute to that obfuscation. If you start collecting metrics on like, actually, that really high severity or really long incident, it might have only taken two people, and they worked on it. Then there was this other thing over here, and that took 15 people and someone from PR. That’s where you start to show to your organization the cost of those incidents. You could probably get the cost. If you’re amazon.com, you know how much money you lose if your website’s down per second. These are these other associated costs, and you can quantify that too. You can say our incidents are involving this many people. That’s one of the top things that I would start focusing on. Then highlighting themes and patterns is a really great way of doing that. A couple of organizations that I know of that are doing that are doing it in a really powerful way. One of the things that David Lee talks about in his talk, is the big initiatives that actually came out of this. They had one where they did end up rearchitecting some. They had another one where they just changed the way people worked together on something. The patterns that you see across your incidents, those themes can sometimes highlight these things that you’re trying to make a case for. Oftentimes, it’s giving you data for rearchitecting something or fixing something that you as the engineer is like, I know this is going to blow up. If you can use your incidents to show that you’ve learned that, then now you have evidence.

Participant 3: Did you ever find any incidents where some teams were trying to maliciously gamify those metrics and push the blame to other dependencies, other folks, and not lose their trust.

Nash: I don’t really find evidence of that in the incident report. I do know of this as a phenomenon. Any metric becomes a target, Goodhart’s Law. We do see this thing happening, or where people will do really ridiculous things to essentially pervert the incentives or gamify the MTTR, make it look like their team’s MTTR is better than others. There’s also weird, behind the scenes horse trading stuff that goes on. I think that is very much a phenomenon that can happen, especially with these aggregated metrics where they don’t mean a lot but you can figure out how to push them around.

Participant 4: You had mentioned near misses. It seems like that would be very difficult to quantify, because the whole point of a near miss is it’s a near miss.

Nash: I wasn’t suggesting you count your near misses.

Participant 4: How do you turn it into a metric, in a systemic way, actually, other than two people having a chat on it?

Nash: I actually wouldn’t recommend trying to quantify near misses. You might be able to quantify the information you get for them. Really what I’m suggesting measuring your misses is getting that qualitative sociotechnical like, “I thought that system x did this, but you think it does that.” You can then surface those as themes or patterns. Clint Byrum at Spotify is doing a really good job of this, where they’re like, this incident matches the theme of squad confusion, or a misalignment of how things work between this squad and that squad. I think if you can roll things up to like themes or patterns, then maybe you can get to some form of quantification. I’m definitely not saying take all of your near misses as the numerator and your incidence is the denominator or the other way around, and use that as a metric. Please don’t do that. Also, you can’t count all your near misses. It’s a lot of work to even investigate those. I think it’s just picking the ones that you have a spider-sense about, that will tell you that. The other reason I like near misses is they don’t have all of the trauma. They’re successes, but they have all of the underlying stuff of incidents. Oftentimes, you can get these details out of them, especially if you’re in a pathological organization, where there’s finger pointing. In this case, it’s like, you can call up the hero worship stuff a little bit and be like, “Jane figured this out. Why? What was that? What was happening there? How can we take advantage of that? Or what do we need to do to not let that go over the rails next time?”

Participant 4: My point was the fact that near misses means that you missed them and you don’t know what they were. Then, when they happen, then you start identifying near misses.

Nash: It might be semantics.

See more presentations with transcripts