MMS • Kelsey Hightower
Shane Hastie: Good day folks. This is Shane Hastie with the InfoQ Engineering Culture Podcast. Today, I have the privilege of sitting down across the miles with Kelsey Hightower. Kelsey is a principal engineer with Google Cloud and will be doing one of the keynotes at the upcoming Agile 2022 Conference. Kelsey, welcome. Thanks so much for taking the time to talk to us today.
Kelsey Hightower: Yeah, thanks for having me.
Shane Hastie: Probably a good starting point is who’s Kelsey? Do you want to give us a bit of your background and experience?
Kelsey Hightower: Yeah, I’m this self-proclaimed minimalist. I try to be very intentional about everything that I own, about everything that I do. Not only question everything, but try to figure out why. If I go back to like my humble beginnings, I think as a self-taught engineer, which is someone who has learned from teachers they’ve never met, one of those people who decided to get into tech around 1999, year 2000, right out of high school. I decided college wasn’t for me and I took that certification path: A+ certification, Network+ certification, and just found my way into tech and been honing my craft ever since.
Shane Hastie: What does it mean for an engineer to hone the craft?
Fundamentals of software engineering [01:09]
Kelsey Hightower: Honestly, I think it’s really understanding your tools well, their pros, their cons, all tools have limitations, and I think the fundamentals. Like the fundamentals of software, I think sometimes we get caught up in various frameworks, whether they’re web frameworks or mobile frameworks, but the fundamental is we are basically trying to manipulate data to allow people to interact with the world. Right? If you’re buying something you put in your shipping information, maybe a payment info, and we try to give you this experience so you can explore our catalog, and ideally that thing should be in stock and then we ship it to the right address. So I think the fundamentals of writing software is applying the right algorithms over the right set of data and making sure people have a great experience with the real world. So for me, I don’t put software ahead of the experience, it’s just a tool to get us to the experience.
Kelsey Hightower: That’s where a lot of my fundamentals arrive. So if I’m going to learn about networking, I’m going to go back 30, 40 years and try to review what the discussion was when people thought about connecting the world. I’m going to go figure out who is Vincent Cerf. I’m going to go figure out why was the internet necessary? What were the naysayers saying? How did it turn out? What were the pros and cons? So I kind of look at everything through that lens. There’s a lot you can learn from history and I think as a fundamentalist, you can always find the right tool to get you the best experience that you’re after.
Shane Hastie: Ooh, some opportunities for some deep conversations of that one; self-taught engineer versus the university pathways. One of the challenges that we’ve seen in the industry that we look at is there’s not enough people. We’ve got a massive, massive shortage in technology skillsets and in roles. How do we bring more people on board?
Tackling the skills shortage – hire your customers [02:48]
Kelsey Hightower: Yeah, I have to say, I’ve been an executive for a while and one thing I used to tell other executives, a lot of tech companies are afraid to hire their customers. We kind of believe that the person necessary to build these experiences, these products and services have to be these classically trained engineers. So maybe you’re looking for people with a computer science degree or someone who can sort a red, black tree in the best sorting time ever. Honestly, that’s not the type of experience that we’re putting out. We have the benefit of standard libraries where someone has already figured out the best way to sort of list. These things are necessary, but they’re not necessary for every developer to go down into the weeds of. What we need a lot more now are people who understand real-life, real-life interactions, and we need those people to help craft these experiences.
Just from my own personal experience and the people around me, I’ve seen people go from being a flight attendant to being a software developer because they’re learning the frameworks. Maybe they can’t tell you the differences between the various algorithms that are out there, but they know how to build an experience because they have experienced themselves interacting with sites, they have lived experiences. And so I think the classical, maybe the computer science degree, what you learned in school is a fantastic journey for a lot of people, but that doesn’t guarantee that you can actually go out and build the very best experience possible because it takes experience in order to reflect it in the outcomes. I think the way we close this gap is we have to understand the experience comes first, and so you’re better off making sure that if you’re going to be building… You know, sticking with the airline analogy, if you’re going to be building tools for people to buy tickets, you want someone who’s flown before.
So who better than a flight attendant or someone that used to work behind the Delta ticketing counter, to be able to say, “Wait a minute, if we’re going to build software to make it fast for people to get their tickets and their reservations together, here’s what I need for my workflow.” That’s usually the magical piece between an okay experience and a fantastic experience is that real world workflow. And I think I’ve seen more of that happening where there’s companies like The Home Depot who have programs that take people out of their stores into a training facility that then go on to work in the IT department, improving the experience, not just for customers, but for their fellow colleagues that are still working in those stores. So I think we’re starting to understand what it takes to build software for other human beings.
Shane Hastie: The title of your keynote is Engineering with Empathy. I heard some empathy in that conversation, but I’d really like to go deeper on that one.
Engineering with empathy [05:19]
Kelsey Hightower: up skills as a software engineer and all I was worried about at that point in my career was just acquiring more skills in order to increase my pay, right? Like that was what most people were doing at that time. I remember working at that financial institution and maybe it’s like the second month in, and apparently we were having lots of outages. Every month or so there’s two outages where the payment system is down. On this particular outage, one of our payment systems, this is the EBT card. So for people on food assistance in the US, they don’t have a visa or a MasterCard, this is government assistance, and to give them dignity, it comes on the same type of plastic card that other forms of payment come on.
So this system is down, the CTO walks in a room and he lets everyone know, because there wasn’t a sense of urgency. People were just mucking around, check the logs, restart this, restart that. Now time is going on and he reminded us this system that we’re working on, on the other end of all of these nauseous alerts turning red on our projected screen, there’s a woman or someone with their family checking out at the grocery line with their children, basket full of groceries,aAnd unlike our other customers, they don’t have another form of payment. So every time that they swipe that card, they’re going to get a decline and they’re going to have to go home empty handed.
So he asked that if we could just take this a little bit more serious, if we can figure out a way to prevent these type of outages from happening over and over again, because there are people who really count on this system being up. And that was kind of my first experience of really realizing what I was actually doing. I wasn’t just a Linux system administrator, I was someone who was responsible for keeping this payment network going so that people can buy their groceries.
Shane Hastie: That deep story there of starting to think about who is the person at the end of the product we are building. Sometimes I jokingly use the term, the victims of our product, as opposed to the users. User as a horrible, horrible term. The only other domain where we talk about users is drug addiction. These are our clients, these are our customers, our phase on the art clients, the people whose lives we change. How many engineers get it and how do we help them get it?
Rethink the interview process [07:39]
Kelsey Hightower: You know, I think just the way we measure results. Think about the interview process. We don’t ask people to tell stories in interview processes, we ask them to get on the whiteboard, and maybe write some code. Can you remember rise the right flag to find the right log statement that you’re looking for? We ask a lot of this low level technical trivia as if the machine is the most important part of this equation. I do believe that it’s still important. You can’t be a technologist without tech skills, this is a given, but hardly do you ever see someone really try to figure out where do you stand as a human being? I know how to evaluate a senior engineer, but do you know how to evaluate a senior human being? There are traits to becoming a great human being, to just have this connection of when is it like to make a tough decision? What goes into that calculus? Right?
Well, like when you’re designing an on call program, having people with insomnia get off of work, go home and then sleep with one eye open because they don’t know when the pager going to ring, you develop a culture that it’s okay if the pager goes off at 1:00 AM every night, because the same system keeps falling over and no one treats it as an important thing to fix because they know someone’s on call. That is not a good way to treat a fellow human being. That’s no way to treat yourself, let alone anyone else, but hardly do we ever talk about that as a standard course of defining what a great engineer is. I think a great engineer is also going to be working on their other skills that make them a better human being as well.
Shane Hastie: Great engineers work in great environments. What is a great engineering culture look like and feel like?
Building great culture is fluid and changing [09:12]
Kelsey Hightower: You know, different parts of my career I needed different things. When I first started my career, great engineering culture was you let me use the tools I needed to get my job done. If I prefer to work on a Linux machine, then you made sure I had the right tools in order for me to execute well. You know, these days after 20 years of being in this business, I want clarity. I want to understand what is the business actually trying to do here? Because when things are very vague, no one knows why they’re working on what they’re working on. We have no idea if this project successor fail is going to impact the business, and if so, by how much. And so it’s that lack of clarity, so there’s a huge disconnect between what the executive teams think. You know, you listen to your company’s earning calls and you’re like, I” didn’t know that’s what we were doing.”
Then you try to map your everyday work. You’re trying to map your output to that vision and the more blurry it is, the more confusing it is. I don’t think people understand what they’re working on and it’s hard to bring you a very best to a job, it’s hard to bring you a very best to a problem when you’re not clear if you’re even working on the right thing. And I think this is how people tend to get burned out. It’s always weird, right? I think we’ve all been there. You can put in countless nights and as long as the thing that you build and you complete is used and people seem happy and it feels like you’re making progress, you almost feel energized to keep going even though it’s unhealthy.
You can take the same situation, put in all of that work and then the thing you produce people say, “Yeah, we don’t really need it.” That makes you feel tired, burnout, and you just don’t want to go through that again so you become apprehensive to the next task because you don’t want to go through that feeling again. So to me, the best environment has been one with a lot of clarity. The reason why I leave with the clarity is because once you have clarity, then I think individuals know what they can bring to the table, right? And say, “If that’s the problem, maybe I need a new set of skills because I don’t have the skills necessary to solve that problem.” Instead of pretending, instead of guessing, I can kind of raise my hand and say, “look, I don’t have skills in that area. I think I know how to go acquire them.” Maybe it’s a bit of training, maybe I can do some self study on my own and look, we’re in a connected world in year 2022. You can just reach out on social media, go to a meetup and ask questions and start to close that knowledge gap for yourself in pretty short order.
Shane Hastie: As a leader of technical people; is it model, is it inspire my teams to have that learning mindset? And even what is teamwork?
Kelsey Hightower: The last time I was a director of engineering that I took over a team that was kind of in an unhealthy state, and I was always patient. The first act was moving out of my office with the window onto the floor in the same cubicles with everyone else. And then it was what are we doing as a team? And how can I help? I remember one of the problems we had at this particular company, we used to screen scrape amazon.com for pricing information on their products and services, because our core product helped you manage your inventory and gave you a recommendation on how to price that inventory to move on third party marketplaces. It turns out scraping is not an official API. Amazon can just change a header or div and your whole tooling falls apart and then your tool stops working. The team needs to run around in circles, trying to put out the fire, trying to figure out what the scraper needs to be adjusted so we can get pricing information again.
One day I was like, “How long have you all been doing this?” And they were like, “Look, Kelsey, our product people assume that information’s going to be there.” And so what I asked the team to do is commission a study. How many customers rely on that particular feature? Is there another way to get that data? If so, we need to wind down this scraping and alleviate these emergencies and fires because they’re reoccurring and this isn’t healthy. I promised them and I went to the CEO, “We have to change this.” I went to the product team, “We have to change this.” And it went from “we have to change this” to “we are going to change this” and “we changed it,” and the team just saw that we can do whatever we want to do, but I need you to bring your very best self.
Kelsey Hightower: To do that, I had to just study how individuals was working. And I remember there were two people, one person working in a Unix-Linux team writing Java, and one person working on Windows on C#. Neither of them liked their respective areas. One person just wanted to go back on the Unix side to writing in C# and working on Windows, the other person wanted to go from working on Windows back to the Linux world. I was like, “This is too easy. Starting tomorrow, you all just swap and there you go.” And their managers are like, “Kelsey, why are we doing this?”
It’s like right now, you’re dealing with people that are working at half speed, unmotivated, and they’re wishing to be somewhere else. We’re going to erase that line and let’s just see what happens. And of course, as soon as they move their desks, they’re like I am where I’m supposed to be and now they’re focused on being the very best version of themselves. So I think as any leader, individual humans need individual things and you have to understand them as a whole person, and then you try to make small adjustments to let them be the best version of themselves. I think that’s always worked for me.
Shane Hastie: We don’t teach leaders how to do this.
Kelsey Hightower: Which is unfortunate because a lot of times we… And sometimes it’s the right move, but often it’s a mistake. We take the best engineer on the team and we try to make them the leader. That best engineer may have never worked on their human skills, right? They may be so used to just working solo, “Give me the problem, I’ll solve it by myself.” And that is not necessarily the best outcome for a team because they may not understand what’s required for someone who doesn’t have as much skill as them. They may have the wrong level of expectations, they may not understand value in others because they may only value the traits that they have and so they tend to hire for people that are just like them. And so you come up with these kind of monotone teams, and then you just kind of get this singular culture.
No one’s really thinking outside of the box, you almost like this one person; you know, hive mind over everything else. And so that’s not necessarily the best thing that you want for a leader and sometimes to those people, you have to explain to them. A 10X engineer is not someone who just working 10X better than everyone else. I think a 10X engineer is the type of person who can come in and make 10 other people better than they were before, and that’s usually the ultimate challenge for someone who could just say, “Give me the problem, I’ll do it myself.” And if I say, “Well, what if you couldn’t do that this time? What if you had to give the problem to another group of people and you had to help them along the way until they did the problem themselves, all in hopes of next month, next week, the next day, they would be better versions of themselves?” And I think that’s a challenge that is a good sign of leadership and the potential to grow into that role.
Shane Hastie: At higher levels in the organization, how do we create this culture that is a learning environment where it’s safe and easy to do this?
Changing the incentives to enable a learning culture [15:56]
Kelsey Hightower: Now the truth is it won’t be safe nor easy and I think that’s the hard part of it. When you are spending time on these human things and your team, your role definition, doesn’t know how to even evaluate, then it’s going to be hard at promotion time because you may say, “Hey, look, I’ve leveled up these five other people, but I didn’t do a bunch of things myself.” You may be penalized and so I think it’s a bit challenging if the organization doesn’t realize how valuable it is to have someone like that. So I think what we have to do is, we have to change the conversation around what we expect from a leader, like what does leadership look like? I mean, you literally have to sit down and say, “Hey, maybe, we got this all wrong.”
When a leader shows up, how do we know how to recognize it if we’re probably bad leaders ourselves? So you kind of have this kind of self-fulfilling prophecy around we want better leaders, but we’re the ones judging them, and maybe you don’t have the right rubric that you’re applying to that leadership. So I think what we have to do is for each team, each team may be different, is to really reward those situations and narratives where someone says, “I started to think and I recruited co-founders who now own the thing, and the thing is now better than when I was doing it by myself.” And that goes back to the saying, “Some people have ideas and some ideas have people.” And we need to do a better job of recognizing when something becomes bigger than just one person and then reward that person for having the courage to hand it off and let the whole team take part in this execution.
Shane Hastie: These are big shifts for many organizations. I’m thinking of our audience who are technical influences, many of them first time leaders, very often that’s highly skilled technologists thrown into a people leadership role. What advice have you got for them to really make the shift for themselves?
Advice for new leaders [17:46]
Kelsey Hightower: You know, I think it’s hard because I have a certain style and my style is really to understand the person, I mean, really understand the person and it’s awkward. I remember giving an underpaid engineer a raise. This woman came from the support organization and learned how to write code. I think she was making half of what everyone was making, but she was definitely in the top three of our engineers. As a people manager, I got to go look at all of the salaries proactively. I don’t need to wait until this person threatens to quit. I don’t need to wait until this person finds out from someone else at lunch how much they make versus what they make. I have to proactively be looking and thinking about these things. I remember identifying that and said, “This isn’t right.” Then you go to HR and say, “Listen, instead of hiring someone new, I have to make things right by this person, because right now, things are not right by this person.” That’s literally part of my job.
Then when you sit down with this person and they’re wondering if they’re in trouble, you go into the office, you close the door for privacy and you sit there and tell them, “You’ve been doing a fantastic job and here’s how, in detail, that there’s a problem though.” They don’t know what you’re going to say, “And you’re underpaid. You’re dramatically underpaid. I want to fix that today.” You slide them a piece of paper and you look at their eyes just widen at the fact that their salary has literally doubled between the time they came in, had a coffee and you asked them to jump into this meeting. All kind of things start swirling in their head about, “Maybe I’m better than I thought I was.”
They just go back to their desks, smile cannot be hidden from their colleagues and they start hacking away. And I think there’s this sense of I belong on this team and they understand my value and that’s part of the deal. But it’s these small nuanced things: that awkward silence when you have to tell people good news, that awkward silence when you have to tell people bad news. All of that is part of leadership and as long as that person knows, “I am literally trying to make you better. You’re great at some things, you need to improve in some other areas, but no one is perfect. But if I find an opportunity to make the things you’re good at great, my job, and I promise this to you, is that I’m going to invest heavily in making sure we don’t miss that opportunity. And the things you’re bad at, I’m just going to let you know you’re aware of them and then you can decide how much help you want improving them, so they don’t wash out the things that you’re good at.”
And I think that is what’s required. So there’s a level of patience. You might as well throw the word scale out of the conversation because you can’t scale human interaction when you’re in charge of individual people. This is why it’s hard to manage a hundred people with one person. You can’t do it. This is why I think you need help as you kind of go down the stack, making sure that your people have real leadership and not just management.
Shane Hastie: Moving back to early in our conversation, you were talking about how in your growth path you have looked back. So you mentioned if you wanted to understand communications, you were looking at the history of the internet. What are the things that are happening today that people like you are going to be looking at in 20 years time and saying, “What was going on then?”
Reflecting on society today and how it will be looked back on in the future [21:01]
Kelsey Hightower: For me is just where I am in my career. I’m the type of person that’s probably fortunate enough to be facing early retirement. The question I asked myself, is this the society we actually wanted? This process of buy and sell and trade, buy, sell, trade, honestly, is that what the whole universe was formed for? To buy, sell, and trade things? I don’t know. I think the fact that we’re just collective species of people that have a shared fate, regardless if we acknowledge it or not, we have to trust each other. You trust the pilot to land the plane safely. You trust a person preparing your food to prepare it safely. So we already have a web of trust amongst us, we’re just not intentional about it.
I think as a society when you look back and say the fundamentals of life is that everything around you has the opportunity to help or hurt. Then as humans with the conscious, with the ability to understand and to remember, we get to make that choice every day, you can help a hurt. You can either say hello with a smile, or you can look at them crazy as I don’t want you to talk to me, you get to decide, and it’s not super easy, but I think we have to be very intentional about the world that we create. So when I look around, whether it’s technology, whether it’s just giving someone some encouragement, or being there to talk to someone, these are the decisions that I think are more important than the 24 hour news cycle. Everyone has to have something to say about everything, but where do we think? When do we take a pause to think? And that’s the part where I look back and say, fundamentally, you need time to process information.
Even for the people that are super technical listening to this, even machines need time to process the information, right? They will even context switch so instead of receiving the data, they will process it and then they’ll get back to the thread to receive more data. Well, humans also need time to process. That means when you have an interaction you ask yourself, “Was that the best interaction possible? Was I my very best person during that interaction? What can I learn from the interaction for the next time?” And so you have to process this stuff, you really need to think, need to think critically and it’s okay to then share once you’ve had a bit of time to think and then make corrections as necessary. So to me, when I look back over, at least my time on earth is there has to be more than buying, selling, and trading things. And that’s what I wish people would kind of get back to and just say, “What is your role in this overall web of life?”
Shane Hastie: Deep thinking there. Kelsey, thank you very much. If people want to continue the conversation, where do they find you?
Kelsey Hightower: Well, I spend most of my time on Twitter. My DMs are open. I make plenty of time for one-on-ones where I can learn things from you and I can share things that I’ve had time to critically think about. If you’re going to be at Agile 22, I’ll be there for a couple days and I’ll be walking around the floor. But catch me on Twitter, DMs are open.
Shane Hastie: Thanks so much.
From this page you also have access to our recorded show notes. They all have clickable links that will take you directly to that part of the audio.