Transcript

Shane Hastie: Good day folks. This is Shane Hastie for the InfoQ Engineering Culture podcast. Today I am sitting down over a relatively short distance for a change. Our guest, Craig Box, and myself are both in the same time zone, which is quite unusual. We’re both in New Zealand.

Craig, welcome. Thanks for taking the time to talk to us today.

Craig Box: Thank you. Shane is a pleasure.

Shane Hastie: My starting point on all of these conversations is who’s Craig?

Introductions [01:16]

Craig Box: Oh gosh, that’s a very existential question. You mentioned that we’re both in New Zealand. We’re having a chat beforehand to figure out how our paths have come and gone with various countries. I think that most people who grew up in New Zealand decide that they want to go off and see the rest of the world. It’s a beautiful place here, but it’s by virtue of its separation from the rest of the world. I had started with a career in IT here because when I went through university as a person interested in technology, I didn’t want to be a developer. I didn’t enjoy the idea of locking myself in a dark room and writing code and not talking to people. That led me to, first of all, customer-facing IT roles and then implementation and using programming skills and ability, so on, what would later on become what we think of as DevOps, but was called system administration at the time. Moved to Canada and did that for a couple of years there before deciding that it was far too cold and we wanted to go somewhere warm and ended up in London, which is of course is famous for its warm weather.

But it was there that I got involved with cloud, first of all in doing deployments on cloud and that led me to working at cloud consultancy companies and then eventually onto Google. I came at a very interesting time and that we were starting to talk internally about this secret project that was taking the internals of Google and the lessons that had been learned from many years of doing deployments internally and open sourcing that and making that public to people in the form of a project which was later known as Kubernetes. I was at Google for nine years and that was basically through the entire life cycle of Kubernetes, from just before its launch to becoming the juggernaut that it is today.

Since leaving Google, I’ve worked at a couple of different companies, but now I’m working with a team of people that I was effectively working with while I was at Google. One of the benefits of these community open source projects is that you can collaborate with people at different companies and quite often you’ll find yourself moving between those companies as well. Working at Solo now, we’re a vendor that I’m working largely with the open source project called Istio, which I was one of the leads on when I was working at Google.

Shane Hastie: So a technical person not interested in writing code …

Craig Box: I know. It’s heresy, isn’t it?

Shane Hastie: But still deeply involved in the technology. You’ve fallen into the space that is often called developer relations. Tell us about that. Why do we need that?

Why do we need developer relations roles? [03:31]

Craig Box: It’s interesting because to some degree I don’t think we need it at all, but a lot of people feel that they need to have it as part of perhaps a marketing strategy or some sort of community outreach, and therefore it’s just the thing that exists, especially the larger vendors. It goes by many names. People don’t necessarily like to use the word evangelism, so they use the word advocacy instead. There is an idea that you are going out and having conversations with end users and saying, “This is what we’re building and this is what we see as a trend in the industry,” bringing back insight from them to a development team.

That’s very much what the traditional approach was in the world of boxed software perhaps. I think the first people who traditionally think of as a developer relations team were the Macintosh team at Apple around 1984 because the idea there was that Apple would make computers and they would sell them to consumers and the computer wasn’t very much use unless the consumer also purchased some software. There was a different audience, a much smaller audience of developers who would make that software the customers would then go on and buy. There was a need for a different team to help those people find out what they needed in order to be able to make the software and go on, make the platform better for everybody.

That follows through for modern platforms. The teams at Google who looked at things like Chrome and Android for example, you could say it kind of makes sense to have that. But for a lot of the vendors we talk about today in the cloud space and anyone doing B2B software, you’re selling a thing to developers, but those developers are your customers. You have a field technical team who go out and talk to them and if your thing is open source, your engineers are participating in these upstream communities themselves.

They know everything that’s going on and they know what people are doing and where they’re failing and so on because that’s all public. It’s interesting nowadays to say, “We’ve got all these people who are nominally in the same-named role, but there’s such a wide spectrum of things that they can be doing.” I think it’s very different depending on which organization you’re at and where they are in the life cycle of the product they’re supporting.

Shane Hastie: If we do need somebody, if we are in that position of almost the platform and we need somebody doing this, what are the skills that you bring to the table?

Skills needed for effective developer relations [05:35]

Craig Box: A lot of the people who I’ve worked with in the past, they are programmers with abilities that I could never hope. There’s no point in playing that game. I look at some of the bug reports like, “Oh, I could probably try and work this out,” but ultimately the people who built the thing are going to be so much better at it. But the idea of going out and explaining it to an audience, especially an audience who isn’t deep in the technology or doesn’t have that background, that’s really a skill that I happen to have and a lot of other people don’t versus many skills that they have and I don’t. I think that’s very useful and being able to do that with a technical background, even going out to audiences, it’s very easy to go to a crowd and say, “Hey, I know the guy who wrote the software. I know the people who do this. I know the engineering team.”

I remember saying, “I didn’t get any smarter the day I joined Google, but all of a sudden I had access to the internal bug database and I could go and look things up and see things.” That really gave me a better understanding of how things were done that I couldn’t necessarily explain to the users at the end, but I could say, “Look, well there’s more to it than this,” and being able to explain to people what’s going on and talk to them with this sort of shared history of the fact that I used to do this kind of implementation work. It’s been a few years since I’ve actually done end-user work in terms of DevOps or anything like that, but I have done this, I have been through this cycle.

It’s useful to be able to report back and some teams use the different people as a customer zero and say, “You go out and take our platform and try and build something cool with it yourself and figure out if it fits a need.” Again, the people who build these things, they are very, very smart, but sometimes they’re just a little bit too close to the problem.

Shane Hastie: This leans towards developer experience. What does good developer experience look like?

What does good developer experience look like? [07:16]

Craig Box: That’s a tough question. Again, there is a sub-section of developer relations, at Google, it was called developer programs in engineering. You hear people talk about it defining APIs and SDKs and all that kind of thing. That is a thing that I think should be considered the product of a team who are building a platform that’s focused towards developers. I don’t think you can say, “I’m going to just have my dev rel team do that for me.” You need to be deep as part of that team and you need to assume that the product managers are having those kinds of conversations, and if they’re not, it’s a bit of a red flag really. I think they need to know because that’s the audience they’re building.

I can’t speak specifically to what makes a good experience for any given platform or product because the problem space is just so big. I think that even the audience of people having an idea of what it is that is relevant to them and what background they have, some people come in with a bit more knowledge and some platforms are an evolution of a previous idea. It’s a lot easier to pick up newer programming languages if you have experience with similar older ones, for example. But then there are people who are coming into the industry fresh today and they don’t have that background. They’ve never looked at these other languages or platforms. I think you need a combination of both. I think you need to be able to explain something in a way that is easy to adopt for someone who doesn’t have that but expresses power in a way that is easy to peel back layers of.

I saw a comment recently, which is to say it’s easy to hide complexity. If you have a complex product that is in large part because the people who build it know that you might need this one day but you definitely don’t need it on day one, so you need to have a way of hiding it from people. Versus not having the product be able to do these things, in which case people play with it for a couple of weeks and then they hit a brick wall because it doesn’t help solve their problems anymore.

Shane Hastie: You touched on product management. What does product management look like in products that are being built for developers?

The nuances of product management for developer-focused products [09:06]

Craig Box: It depends in the sense that if it’s within a single company, it looks like building any other product I feel. There’s a modern idea of what we now call platform engineering, which is sort of the idea of building a product only for the internal team that you’re looking at. Your developers are your customers, especially the big company. You are now running a product organization that’s saying, “I’m building out this infrastructure which I’m going to let my developers use.” That’s an interesting way of saying, “All product management ideals and so on should be brought to this discipline of SRE or sysops or whatever you want to call it.”

The thing that is interesting, especially when you start talking about developer tools is the fact that so much of the tool space we deal with today is open source and a large amount of it is built by more than one vendor. If you are someone like GitHub and you’re building something which is effectively proprietary, a service run by a particular company, they can do product management in the traditional sense whether or not their product is closed or open source. But when you’re looking at something like Kubernetes that’s built by hundreds of different companies and tens of thousands of people, I think in those cases you just kind of don’t have product management.

You have people who apply that skill to things like GKE, which are Google’s hosted version of that. But it’s hard to say, “Well, what that looks like when you’ve got so many collaborators on the space.” It’s easy to say, “Here’s some code, here’s some engineering.” It’s a lot harder to say, “Here’s documentation, here’s planning, here’s the rigor that you need around all of this stuff.” Because again, it’s harder for the people who are all contributing to this to find budget to apply that skill as well.

Shane Hastie: Let’s dig into open source community. You’ve worked and are part of a number of fairly large open source projects. How do those communities be effective?

What makes open source communities effective [10:56]

Craig Box: In a self-organizing fashion, I think. You get people who are interested in community and who want to deal with topics like governance and especially with open sources or licensing issues or legal aspects. You get a lot of people who are interested in technology and have that as their particular bent who are willing to participate in those ways. These multi-vendor communities have come up with approaches as to how they will deal with things, as to things that are fair in a company situation where you have people who are competitors who all want to come together. Quite often you’ll have a foundation of some sort which will own the trademarks or the copyrights and particular things so that all the vendors feel safe in participating.

But then you also find that the, as I mentioned before, there are things like documentation and administration of a project where it sort of comes down to luck to some degree as to whether you find someone who’s willing to do that. In large part with the Istio project, which is the open source project that I’m primarily involved in, it was just that I knew that this needed to happen and I kind of was interested in product management to some degree and said, “Well, look, I’m going to apply the skill or this interest to not Google’s internal version of it,” because that wasn’t my role there. But I had a little bit of a rope being in the developer relations department to say, “Well, hey, I think that this needs to be applied to the project and that someone needs to come in and help run the meetings. Someone needs to figure out how the vendors communicate with each other and how their roadmaps align and how we plan everything.” If you find a person who’s willing to do that and has those skills, then hold onto them because they’re few and far between.

But a lot of companies, like if you are one company building an open source project yourself, it’s very easy because you just do it internally. But when you get to the point where you have these large multi-vendor things, not a lot of the vendors are going to staff that role. I think some people might say, “Hey, it’s missing and it would be useful to have it there,” but it is definitely something that you notice the absence of.

Shane Hastie: Digging further into the open source communities and collaboration, this is the culture podcast, what is the open source culture?

Factors that influence the team culture in open source initiatives [12:56]

Craig Box: I think it’s somewhat representative of the people and you talk about, there’s a paradox, I can’t remember the name of which basically says it’s going to represent the culture of the organizations who contribute to it. If you start with something that is largely built by a single vendor, all of the norms in that project will largely reflect what the norms were of the company who contributed. Over time things will take something on largely reflecting the energy and effort that it put in. If we talk about Kubernetes for example, a lot of the early team there was a lady named Sarah Novotny who was one of the first community managers who worked at Google on the Kubernetes project. She put a lot of processes in place around the project and the CNCF that allowed collaboration and allowed vendor-neutral contribution and so on. That really carried through to the way that project operates. They have very strong norms in certain ways and they have a very big culture. You get a lot of people who come together and pick up the same T-shirt at the conference and so on and go to the Contributor Summit.

Then you get some other projects which are sort of reflective of the fact that the companies who contribute to them are just that, they’re companies. You get people who are building products based on this technology and they all want to treat it like I had someone call it a clearinghouse recently or a trade body effectively around a particular thing.

Then at the other end you get people’s passion projects. You get apps, you get Linux and desktop software that go on it that people just install and run and they find something broken about it. They say, “Hey, this is something I want to participate in.” Those communities are largely just individuals.

In the cloud space that I’m in, there’s not a huge number of individuals. There are people who are looking to get involved and build up their skills and maybe be hired by one of these companies, but the reality is that these large open source projects are built by people who are paid to build them for businesses because they are a complement to something that they do. That has been the case for many years now. If you look at the Linux kernel, the people who contribute the most to it are vendors, people like Oracle for example, may be surprising, I think they were one of the most prolific contributors quite recently. Then you get vendors of people who make network hardware and storage hardware and so on. They are the people who are writing the drivers and moving things forward so they can sell more network hardware and storage hardware. It’s not just people in their basement who are working on open source anymore.

Shane Hastie: You are on the governing board of the Cloud Native Computing Foundation. Tell us a little bit about what is the CNCF and why?

The Cloud Native Computing Foundation [15:21]

Craig Box: Why indeed. Yes, so the CNCF was created as a way for Google to spread Kubernetes wider than just itself. If it was a thing that only was a Google project, then it would never get the people who are competitors to Google’s cloud business, they wouldn’t get behind that. Google obviously said, “All right, we want this thing to become an industry standard, and for that to happen, it needs to have a certain neutrality about it.” They worked with the Linux Foundation to set up a project there, which is the Cloud Native Computing Foundation. It was seeded with Kubernetes as a technology, but it now has upwards of 200 projects, I think, that are hosted in the CNCF.

There are a few different levels of the way the organization works. The largest sponsors basically get to have a seat on the business board, and there are a couple of seats that are reserved in that governing board for the projects and for the technical community. I was recently elected to one of those as a representative of the project maintainers. I’m saying, “I can come here and give the voice of the people who are working on building the software back to the CNCF in issues like how they spend their budget and how they market themselves and so on.”

Largely the technical decisions are made by a group called the Technical Oversight Committee who are a group of 11 members, I believe, who are, again, elected from the community to accept projects and to decide the direction of the foundation and should it be looking at projects in this space or that space and so on, and how should they interact and relate. There are end user members as well. They’re the group who organize the conferences, the KubeCon conferences that have been going on for many years now and a bunch of other events. They do a great work in terms of international, making sure there’s events in upcoming countries, making sure there’s diversity programs and so on. There is mentorship programs. There’s a whole bunch of stuff that’s done there made possible by the fact that there is such a commercial value to all of the software and that there companies who are willing to all put in and fund the organization.

Shane Hastie: How do you get such diverse stakeholders to collaborate?

Craig Box: You’d promise them that it’s going to be a fair and neutral playing field, as I understand it. The goal was from Google’s perspective to have this become the standard because it was a pattern that they knew how to operate and that they thought they would get more people coming to their cloud to run workloads at a time when they had a smaller market share. That obviously worked out very well. There were a bunch of other people who are interested in taking that, what was more superior technology at the time that had the sheen of Google’s internals associated with it. Then there are also other vendors who are like, “Well, we might not want to adopt the thing that Google built because we compete very heavily with them, but our customers are asking us for it and it has that demand.”

Bringing it back to the beginning that is working in developer relations for Google at that time that was going out and trying to convince people bottom-up that they should want to run Kubernetes. Then they think, “Yes, I like that,” and they go to Amazon who were their cloud provider at the time and say, “We want to run Kubernetes.” Eventually Amazon builds out a very good hosted Kubernetes implementation and that gets them on board with the project and joining the CNCF and so on. But it is largely down to customer demand and the way that Amazon operates. Then it is the political neutrality and the reputation of groups like the Linux Foundation, that everyone will be treated fairly and so on, that allows those things to happen.

Shane Hastie: Thinking our audience, as a technical leader, technical influencer, technical practitioner, why would I want to get involved in one of these communities?

Why get involved with open source communities? [18:56]

Craig Box: You might not. Again, it comes down to your own particular interests. Even within the communities that we deal with, there are people who just want to write code. They want to take a technical issue, solve an interesting technical problem. There are people who want to write documentation and don’t want to understand the code, but who want to effectively ask one of those technical people how something works and translate that. There is every possible opportunity, especially in something, again, as broad as the Kubernetes community, which is testament to how people like Sarah, I mentioned before, set the community up is that it’s a welcoming space for everyone.

The CNCF and the Kubernetes project especially put a lot of effort into being welcoming for new users. There are a lot of mentorship programs. There’s a lot of money from the Linux Foundation to sponsor people, programs like the Google Summer of Code that kicked this off maybe almost 15 years ago. Now, there is a Linux Foundation program that’s similar that gets new people in who don’t necessarily have the opportunity to just walk into a job like you might if you were in California, but who are interested in building their skills in these spaces.

A lot of effort is put into make sure it’s a welcoming community, that there are opportunities for people at every level. There are things that you can do if your company wants to get involved and has a product based on this, but equally there are things you can do if you are a student and you want to get involved in technology.

Shane Hastie: What’s the future direction?

Craig Box: That’s a tough one, and I say that because it feels in large part that these problems have been solved within the space that we’re in. If you think about the curves of disruption over the last few years and so on, everything gets to a point where desktop operating systems are mostly done. It got to the point where everything was good enough and then the internet came on and it needed to be a new thing. Cloud native was a new thing on replacing the idea of building on top of also, but replacing the idea of virtual machines or servers and so on. I’m not sure I could say, “Hey, there’s some amazing new development that’s going to happen in this particular space.” What I think will happen is some other space will come along. The basic unit of deployment or the building block of cloud native technologies is usually referred to as the container, the Linux container, well, Windows now containers as well.

But then it’s possible that something like WebAssembly comes along and says, “All right, well, we have a new way of taking code and running it.” Ultimately, we’re still doing what we were all doing 50 years ago, which is writing code and then deploying it to some computer somewhere. Maybe it’s the one in front of you, maybe it’s tens of thousands of machines in a data center in Virginia somewhere, but we’re still just writing code and running it on computers. It might be that there is a new methodology for doing that that gets better. It might be that the singularity and the AI stuff that we all promised ourselves that we wouldn’t talk about obviates the need for that somehow. But unless those things happen, we’re probably still going to be writing code and deploying it on computers. The means may change, but whether or not it happens within this community or whether something else comes along, it’s hard to say.

Shane Hastie: What haven’t I asked you that I should have done?

Craig Box: Who does that lovely hair of yours? That’s what I would’ve asked.

Shane Hastie: I polish it.

Craig Box: I can’t come with a piece of wisdom that I would apply to everyone. I could say that there are now so many more things to do, like the things that I do and possibly yourself also, Shane, that you couldn’t say when you were in school, “You can study to do this thing,” because it didn’t exist. I’m not sure what to tell my children to learn today.

I saw a historian being interviewed recently who’s saying, “Well, in the Middle Ages, and even as recently as a couple of hundred years ago, you can teach your kids to farm and to sow seeds and so on, and that is a skill that will be useful to them and see them through 20 years from now.” We don’t know what the skills of 20 years from now will look like. I can tell you that there were no developer relations people. There was no practical worldwide web and so on 20, 30 years ago, and that’s the thing that’s enabled my job and probably the job of many people who are listening to the show. It’s really hard to say, “This is a thing I think you can do.” But one thing that especially working in open source taught me is that it’s the relationships, it’s the people, it’s the way everybody works together. All of that will still apply and all of that will still be relevant no matter what the underlying technology changes are.

Shane Hastie: We’re in New Zealand, the Māori term, “He tangata he tangata he tangata!”. It’s the people, it’s the people, it’s the people.

That’s a great point for us to pull this to a close. Craig, if people want to continue the conversation, where do they find you?

Craig Box: This used to be so easy. I used to just tell people to follow me on Twitter, and I’m not sure that that’s a thing that still people are comfortable doing and so on. But I am on the remains of Twitter. I am CraigBox there and you can basically find me by that name on most of the modern social networks and so on. I am a elapsed podcaster with a couple of young kids at home. I haven’t been able to keep up with that as much as possible, but you can definitely find my work around and Twitter’s probably as good the places anywhere to look for me.

Shane Hastie: Wonderful. Thanks so much for taking the time to talk to us today.

Craig Box: Thank you very much.

Mentioned: