Author: Jessica Andersson
MMS • Jessica Andersson
Transcript
Andersson: Once upon a time, there was a startup. This startup was running code in production, because that’s how we get our product out. They have been running for a couple of years, and they started to identify the need for all the things DevOps. They knew that they wanted to keep developer focus on delivering software value, and they knew that they wanted to do all them things cloud. Recognize this story? I think this is common for a lot of people.
They realized that the solution was to invest into a base platform and platform engineering in order to solve the problem of CI/CD, runtime, and observability. It’s been evolving ever since. I’m here to tell you that you need a platform regardless of the size of your organization. I would even go as far as to say, you already have a platform, you just don’t know it yet. You need to treat your platform as a product. It’s no different than delivering the software product that your company is probably surviving on. Trust is a currency, and you need to treat it as such, because otherwise you will fail. Being a small company, the tradeoffs that you make will be the key to your success.
Background
I’m Jessica. I’m the product area lead for developer experience or engineering enablement at Kognic, the startup that I’m working at. I’m also a CNCF ambassador and a speaker at conferences. Before joining Kognic, I was a platform engineer working at another company and delivering platform as a service to more than 40 teams globally across the world. We were focusing mainly on Kubernetes as a platform and logging as a service. We had other things that we provided as well. I first heard of Kognic from a friend of mine who worked there when he sent me a job ad with a motivation, like, “You do cloud and Kubernetes. How about this one?” Who can resist that? It read, “Wanted, Head of DevOps”. Let’s give it a chance. I like my friend. He’s nice. I start reading the ad and I realize what they describe is something that actually sounds fairly similar to what I’m already doing.
I set up a meeting with our co-founder, Daniel, and we met up in a cafe looking like this, actually. We took a cup of coffee and we talked about it. I told him about all the things that I’ve been doing with trying to empower and enabling different development teams to solve their needs more efficiently. He told me about their situation and how far they had gotten, that they had some software running, but they realized that they need to invest more into making it continue to run smoothly over time, and solve some of the common needs of how to operate it in production. I told him my other hot take that I don’t dare to put in text, is that, I think a DevOps team is a glorified operations team, and I don’t work with operations. It’s nothing wrong doing that, but I feel that I really enjoy being a platform team more because I can affect a lot of people and I can try to make their lives easier.
Empowering and Enabling Product Teams
As a platform team, making people’s lives easier, someone wrote on Twitter, “Being a platform engineer is the closest that I will probably ever be to become a 10x developer”. This was all the rage when I was looking for this job. We also know that by empowering and enabling our product teams, they can focus on the things that they care about a lot, which is delivering product value. We tried to look at the needs for this head of DevOps, the reason why this ad was out. What Daniel described the need of the company was that they needed to do quick iterations because they didn’t really know exactly what the product would end up being in the long run. This is a startup. You’re trying to figure out what is the perfect fit for my product. They wanted to do quick iterations. They also wanted to have a lot of flexibility in changing direction if they discovered this is not the way we want to go. They wanted to maintain the developer focus on delivering value. They didn’t want all the developers to understand everything about Kubernetes.
I don’t want them to have to do that either because there’s a lot of things to know. They also knew that they wanted to keep a cloud native modern tech stack. We saw on the hype scale that cloud native was right up there with generative AI. We also talked about developers and operations versus DevOps. I’ve done a talk only about this thing previously. I think the main thing is that when you have a situation where you have a developer and operations team as we had at my previous company and we transformed that into DevOps, we have several reasons of doing so. We discovered that having the developers focusing only on pushing out new features and code changes was very disconnected from operating them in production because we started to see a lot of issues such as failures in production. We got long time to get new things pushed out because operations were busy firefighting. They were pushing back when developers wanted to deploy more things.
Operations had a problem getting the feedback back to the developers and prioritize the fixes for solving the things failing in production. It was more or less just throwing things over the wall and hoping it all works out. It did not work out.
At my previous company, we did a large effort in trying to transform into DevOps and have all the product teams work with an end-to-end full application lifecycle workflow. When we talk about end-to-end, like only in the full application lifecycle, we can also talk about how we get there. We get there through having empowered product teams. If you haven’t read Empowered Product Teams by Marty Cagan, it’s still a really good book, and there’s a lot of great ideas in it. You don’t have to read all of it. There’s a lot of blog posts that summarize some of the main points, or talk to someone smart around you that actually read it. That also works for me. Check it out if you haven’t. Marty Cagan describes empowered product teams as being about ordinary people delivering extra-ordinary products. You want to take any product teams, empower them so that they can focus on delivering great products.
Difference between, as I mentioned, the developers pushing features and empowered product teams can be described as, product teams, they are cross-functional. They have all the functionality or all the skillsets that they need in order to deliver their part, their slice of the product. They might have product managers, they might have designers, engineering, whatnot, they need to deliver their slice. They are also measured by outcomes and not output. Output is, I did a thing, yes, me. Outcome is, I made an impact. There’s a difference in that. We want to optimize for making good outcomes rather than a lot of output. They’re also empowered to figure out the best way to solve the problems that they’ve been asked to solve. This is a quote from the blog post that describes exactly this. It says that solving problems in ways our customers love, yet work for our business. It’s not all about only focusing on making customers happy, about doing it in such a way that the business can succeed, because we’re all here to earn money.
Very much related to this is “Team Topologies” by Matthew Skelton and Manuel Pais. I will focus on it because I think this is strongly related and this is something that we looked at on how to structure our teams. Stream-aligned teams have also a slice of the cake. They have like a you build it, you run it thing. They have a segment of the business domain, and they’re responsible for that, end-to-end. Then you have the enabling team. I said, I offer engineering enablement. There’s a reason why it’s here. They work on trying to help and unblock the stream-aligned teams. They are there to figure out the capabilities that we need to make in order to improve the life of the stream-aligned teams.
Then we have the platform team, and they are supposed to build a compelling internal product to accelerate delivery by the stream-aligned teams. We’re here to empower and enable our stream-aligned teams to deliver good outcomes to create business value. As a small company and a small organization, I argue that you probably can’t afford to have both an enabling team and a platform team. In our case, we decided to combine these two. We decided that we should have both engineering enablement and platform engineering within the same function.
Given what I told you about empowered product teams and trying to focus on good outcomes, do you think we should have a DevOps team or a platform team? It’s a given. We’re going for the platform team. That’s why I’m here. Back to the coffee shop, me and Daniel, we talked for longer than we set off time. We’re both big talkers. In the end, we think that this might be something and we decided to give it a go. In June 2020, I joined Kognic with the mission of starting up a platform engineering team and try to solve the problem of empowering the product teams to deliver more value.
By the time we had the platform team hired because there was new hiring going on, the engineering team had grown to 31 people. Four of these were working in the platform team. That means that about 13% of our headcount were dedicated to platform and internal tooling. The reason why I tell you is not because 13 is a magic number, I just thought it could be nice to know, like put it in numbers. Tell us what you really were doing. Being a small company, this is 13% of the capacity, whatever you want to call it, that we had to deliver new value, and we thought that we could actually gain this in more empowerment.
Implicit Platform
We had a team. We could start with a platform. We didn’t start from scratch, because, as I said, we were running code in production. It turns out, if you’re running code in production, you already have an implicit platform. The first thing that I had to do was try to figure out what is already here, what do we have right now. This platform, it exists, but it’s less structured and less intentional. This is what happens when you don’t do an intentional effort in trying to build your platform. There were some really good things in place, but a lot of it had happened in the way of, we need something for this, let’s do it, and then go back to the things that I’m really supposed to be working on. Meaning that we had Kubernetes running, but it had not been upgraded since it was started. Yes, that’s true. We had other things that were working good enough to solve the immediate need, but maybe not good enough to empower the teams fully. We had a lot of things in place.
They were done with the best effort, good enough approach, and we needed to turn this around. I’m going to show you what we had in our implicit platform. This is not me telling you this is what you should run in any way, but I want you to have this in mind. We use Google Cloud Platform as our cloud provider. We use GKE, Kubernetes running on top of it for our runtime. We had CircleCI for our CI. We are writing code in TypeScript and Scala and Python. We had bash scripts for deploying to production. We also had InfluxDB and Grafana for observability. You can see that there’s nothing here about logs because I don’t think we work with logs at this point in time.
A Base Platform
What do I mean when I say platform? Because this is what we were hired to fix. This is where we started out and what we wanted to build. I define this as a base platform. This is, for me, a new term. Focus on solving the basic needs. We’re talking about CI/CD. You want to build, package, and distribute your code. You want to have it run somewhere in what’s equivalent to production for you. You want to have the observability to be able to know in case something goes wrong so that you can operate and maintain your applications. Without these three, it’s really hard to have something continuously deployed to production and keep working in production. I see those as the bare necessities of platform. With this concept in mind, we want to take our implicit platform and turn it into an intentional platform.
Our platform, it changed a tiny bit, not much. You can see it’s basically the same picture that I showed you before. We took Google Cloud and we introduced infrastructure as code to make sure that we have resources created in the same way and that it’s easy to upgrade when we want to make changes to how we utilize them. We improved the separation of concern for cloud resources. There was a lot of reusing the same database instances for staging and production and other things. We also took Kubernetes and we upgraded it and applied security patches, and then continuously maintain it. Kubernetes does several releases per year, so it’s a lot of years to keep up. CircleCI, we were running a monorepo where all the code was, for both TypeScript, Scala, and Python. We broke it apart and we reduced the build times a lot. We went from 40-plus minutes to less than 5 minutes.
We also introduced GitHub Actions in order to have smaller, more efficient jobs because our developers really felt those were easy to integrate with. We didn’t even provide it as a platform. It was just something they started using and then we adopted it. We removed InfluxDB and replaced it with OpenTelemetry and automagic instrumentation of applications. When I say automagic, I really mean automagic. If you haven’t looked at it, it’s as good as they say, and I didn’t believe it until we tried it. Then we removed the bash scripts for deployments and we introduced Argo CD and GitOps for better version controlling, and control and easier upgrades and rollbacks of applications.
Platform as a Product
How did we go about getting to this place? We treated our platform as a product. It’s an important part. I think the first thing you need to do is to understand your market. This is your empowered product teams. What do they need to do? How do they work today? What pain points do they have? You need to understand, what would be the good direction to go with this? You need to iterate and validate your solutions. You can’t just go away for a year and do something and come back and deploy it, and hope everyone is happy. Because you need to always constantly work together with the teams in order to make sure that you have something good. If you’re new to working with a product mindset, I can recommend looking at something that is called a Double Diamond, where you have two different phases. One is you go broad for problem discovery, and then you narrow down on a problem solution.
Then you go broad on a solution discovery and then narrow down on a solution decision, and then you iterate on that. When we look at our platform team, we do similar things as our empowered product teams, meaning that we try to be cross-functional. We try to figure out, what capabilities do we need in our team in order to deliver the things that we need? We are looking at cloud infrastructure. We need that skill. We also need containers and Kubernetes skills because there’s a lot to learn about those.
Observability is a big thing. It’s good to have that skill. Also, the enablement and the teaching. You need to teach your empowered product teams to adopt and work with these services that you provide. You need to be able to show them new ways of working. You need people that can actually both teach and communicate with other people. Communication is actually one of the bullets that we had in job ads for skills we’re looking for. You also need product management skill combined into this team. Obviously, since we’re doing product. If you want to learn more about working with product thinking for platform teams, I can recommend checking out this talk by Samantha Coffman. It was at KubeCon + CloudNativeCon EU in Paris. The recording is up on YouTube. Check it out. She does a really good description of it and she has really concrete examples of what it means to figure out what the real problem is rather than fixing the symptoms.
Finding the Right Problems
Talking about figuring out what the real problems are, remember what we wanted to achieve? We wanted to achieve quick iterations, flexibility to change direction, and maintain a developer focus on delivering product value. Given that, we wanted to figure out what is holding us back from achieving that. Let’s start with understanding the market. Autonomous teams and empowered product teams, I have trouble separating those terms. Where does one end, where does another start? Autonomous teams is something that we have talked a lot about at Kognic. One thing that it says is that autonomous teams can deliver, beginning to earn value, and with minimum supervision. It’s similar to empowered product teams: do all the things that you need to solve the problem, but also with the caveat of minimum supervision. They’re in charge of defining the daily tasks and the work processes that need. It’s very similar.
If we think about autonomy and the free choice, we can’t force our product teams to use our platform because then we are removing the autonomy and the freedom to choose. As a platform team, it’s very important that we try to make it enticing and something they want to use. We can have things as our platform as a default setting, but maybe we can’t hinder them from choosing something else. To achieve that, we want to create a paved road for the developers. What does that even mean? What is paved road? We want to empower and enable our product teams or our stream-aligned teams to deliver valuable outcomes. For every decision and every action they need to take that doesn’t build towards that, we can view that as a cost that takes away from the outcomes that they are able to deliver. If we provide a paved road, something that is easy to follow, they don’t have to make a decision of, how do I want to deploy to production? They can follow the already paved road of that.
Then we can solve the basic needs of building, running, and operating applications. We allow our teams to focus on the things that make a difference, and we reduce that cost. This paved road should be effortless. It should not cost them energy to stay on the paved road because then your paved road is not that great. As a platform team, I mentioned Kubernetes has a lot of upgrades. How many of you do migrations continuously, because I feel like we are always in a migration from one thing to another? If we as a platform team make those migrations feel cumbersome or heavy to do, people would try to like, “Maybe I don’t have to migrate. Maybe I can do something else. Maybe I don’t have to follow this thing that platform is now forcing me to do again”. You need to make those things effortless so that people can maintain on the paved road without adding more energy.
Is autonomy limitless? Are there boundaries to what you’re allowed to do as an autonomous team? Who is responsible for setting those limits? If everyone decides to go for their own solution, it will be really hard to be a platform team. Like you say, this is how you should deploy your applications. Then the team goes like, I think I have a better way. There are two reasons for that: either your platform is shit or your culture is shit. Both of those is something that you should try to figure out as soon as possible so you can address it. I also think that there are occasions where autonomy is good, but like if you have people running around freely, just doing the things that they find really interesting, it will be very costly in the long run because it will happen that you have to do something. With a very diverse codebase, it’s super hard to handle that as a platform team and as an organization. The longer you wait to figure out where the limits for your autonomy goes, the harder it will be to address it once you decide to do it.
There are things that might be good to put as not optional for the paved road. When I talk about that, I usually think about compliance and I think about security. Everyone loves compliance and security. I’m sure you do because I do know that I do. A paved road or a platform is something that can really help you figure those things out. If you make it easy for the teams to be compliant and be secure by building innately into your platform, you can reduce the load for them to do so, and be able to focus on the things that they want to do, like valuable outcomes. I think there are situations where the paved road might not be optional and that you can build it into the platform in order to solve that.
Back to finding the right problem. If we build a paved road that enables quick iterations, flexibility change while allowing product teams to focus on product value while staying empowered, if we want to do that, then we need to figure out what is holding us back from doing so right now. We need to figure out what the right problems are. We knew that with our limited resources, being a small team, 31 people, 4 people in platform, we needed to figure out what we wanted to focus on and be intentional with what we invest to. We want to take our implicit platform, apply strategy in order to make it intentional. We wanted to reduce the pain points and the time sinks, and improve developer experience to increase our ability to deliver product value.
Problem Statements
I have some problem statements that we can use as a tool when asking ourselves what should we focus on. The first thing is like, teams are blocked from performing the tasks that they need to do. Maybe they have to wait for someone to help them in order to move forward. This is, of course, bad. I’m only listing bad things here. The second one could be like the task that the team performed takes a long time and they are hindered from moving on until it’s done. The third thing is the tasks teams perform are unreliable and prone to failure. I’m going to give you three examples of where this applied to us. The first one was DNS. DNS was not failing, but DNS was blocking. When our teams wanted to deploy a new service and they wanted to attach a DNS to it, they had to go and ask one or two people that can help them create a DNS record and give it back. They were blocked from moving on until they got that support.
Something that was taking a very long time, I mentioned before, we had a monorepository with a lot of long builds. You had to wait for the build to build your package so you could deploy to production. We had build times of over 40 minutes. This was taking a lot of time and hindering people from moving forward. When it comes to unreliable and failures, we had deploying to production with bash scripts. Because there was a lot of hidden functions within this bash script that was not clear, it was a black box to the developers, and it failed several times a week. It was becoming painful. The team members were not sure how to figure it out themselves. They couldn’t know for sure, even if it seemed to go fine, if it actually also got reproduced in production. It was prone to errors. It was unreliable.
This was something that they were not able to solve themselves. They were hindered from moving forward. We looked at these tasks and we figured out what we should focus on. Hint, we took all three of those and tried to look at. We tried to look at our implicit platform. We tried to figure out where can we streamline it, where can we upgrade it, and where can we improve it in order to remove those pain points and time sinks? When we have tackled how to solve those problems, we also need to figure out how to roll this out to the teams, and how we can get them started using it, and how can we gain adoption of a platform.
Trust is Currency
Which nicely leads me to the next section, which says, trust. Gaining adoption is closely related to the amount of trust your product teams have in you as a platform team. As eng says, trust is currency and you should treat it as such. You have to gain some currency before you can spend it. Credibility is a currency and it is what you earn and spend as a platform team. When we talk about trust, trust goes up and trust goes down. When I say up, I mean up in the organization. You have to keep your trust with leadership because they are the ones that decide to continue to invest into your platform team. If you look at the budget, you’re just cost. If you’re unlucky as well, you also get the cloud bill on your cost part of their budget, and then it looks like you’re very expensive. You need to build trust with your organization that you are actually improving things so that you can keep doing it. It’s already been talked about, and DORA metrics is something that you can work with in order to show some kind of improvement and show what value to deliver.
This link goes to the “Accelerate” book which is written by Dr. Nicole Forsgren, Jez Humble, and Gene Kim. If we think about DORA metrics, they are four metrics and they are focused on deployment frequency, how often do you get new things into production? Lead time for changes, how long does it take that you start working on something until it actually reaches an end user? Mean time to recovery, if you have a failure, how quickly do you recover? Change failure rate, how often does a change lead to failure? Those four metrics is something that you measure your empowered product teams on but can be a nice indicator on your effect as a platform team. If you think about down, you want to have trust with your product teams in order to gain adoption of your platform.
If they don’t trust you to solve their pain points, then you need to figure out why they don’t trust you and what you can do to change that. I would suggest starting with something easy but painful or time consuming. Make that work so much easier for them, and then go on to the next thing. Start small, start building up credibility, because when you have built some trust, people will start coming to you and then you will have an easier time understanding their point of view. For us, something that we did, the DNS thing, we introduced external DNS into Kubernetes which means that you can use Kubernetes configuration in order to allocate a DNS record. This was very easy for the developers to understand how to use and it was very quick for them to start using it as well, meaning that from one day to another basically, they were no longer blocked by anyone ever when wanting to change DNS.
Once you have tackled some of the small things, you can go on to the bigger things, and then you will probably spend some credits so you can earn them back again. Based on my experience, these are some of the things that you can do in order to earn or spend the credits. When we talk about earning credits, we can talk about removing pain points, and really anything that is painful for developers will do. As a platform team, it’s good to be approachable and helpful. You want people to reach out to you so you can learn about what they are doing. Something that we do for this, is that in Slack, we have a team channel that is team platform engineering in which we have a user group that is a goalkeeper, platform engineering goalkeeper. Teams know that they can ping this goalkeeper with questions regarding the platform and to get help to figure out how they can solve something in case something breaks and they need help understanding what went wrong, they can do that.
If they want help understanding how they can utilize some part of the platform, they can do that. By being very approachable and helpful, and by approachable, I mean there are no stupid questions, we know this. Also make sure that they understand it. Be nice. Be kind. If someone comes to you with a question and you go like, here’s a link to wiki. Do you think they will ask you again? They will probably be like, no, they don’t want to help. If you go like, we wrote this part about it but if there’s anything that’s unclear, please let me know and we can work on it together. That’s more approachable. That is something that makes people want to come back and ask again. You can still give them the link to wiki because like, check the documentation. You can do it in a kind way so people want to reach out again.
You want to be proactive. You want to be able to fix some things before people have to ask, especially if it’s something that you really should know, like something is broken in your platform. It would be nice if you know it before some developers come and ask you, why is this thing not working anymore? You need to understand the team perspective, like, where do they come from? What do they know? What do they not know? What do they want to achieve? In spending credits, we have enforcing processes. Sometimes you can’t avoid it, like compliance, like security. It costs you credits. In my experience, teams really don’t like when you tell them you have to do this just because. Be mindful of how you enforce your processes.
Also, blocking processes. Empowered teams, they know that they are empowered. They know they’re supposed to be. If you take that away from them, you’re blocking them from being empowered, they’re not going to like it. Migrations, we can’t get away from it, but depending on how you perform them, it will cost you credits. It might even cost you credits even if you do it well. Assumptions, I know everyone working on my platform team is really smart and capable at what they’re doing. I also know that there are several times when we made assumptions of what the teams needed and how they wanted it to work, and we were wrong. It’s very easy to take your world view and project it on something where it doesn’t really fit. Make sure that you validate your assumptions and understand the team perspective in combination with your assumptions. Otherwise, you might be spending credits.
I want to tell you, this could be our trust credit score over time, from June 2020 up until now. Please don’t mind the time warp. It was hard. I don’t have any numbers as well because we haven’t been tracking this, but it could look like this. On the y-axis, we have the trust credits. On the x-axis, we have the time. You can see that we did a really big drop in trust. This is when we were making the assumption, enforcing a process, not talking to the teams to anchor or understand their perspective before migrating everyone to a new way of working. I’m talking about how we introduced Argo CD and GitOps instead of a bash script for deploying into Kubernetes.
For us, it was clear that everyone wants to work with GitOps, because you have it version controlled. It’s very nice. You have control always running. You can follow the trail of everything. It’s easy to do rollbacks and all the things. We knew this. This was clear. The whole industry is talking about how this is a good way of working, but we did not anchor it with the teams. We did not understand how they viewed working with the bash script and interacting with it. We took something away from them, we forced something on them, and we did not make them understand why.
In the long run, we actually managed to gain back some trust on this because this change and the new process that we enforced, it proved itself time over time. In the long run, we gained more trust than we spent. I would rather not have that dip if I were doing it again, because I think it could have been avoidable, and I think we could have mitigated it through spending more time understanding the developers and making sure they’ve anchored the change before performing it. In the long run, great investment.
Small Teams and Tradeoffs
Speaking of investment, as a small team, you will have to do tradeoffs. The link goes to the CNCF landscape. The CNCF landscape is a map of all the open-source projects that is under the Cloud Native Computing Foundation umbrella. I don’t know the number, but if you zoom in, it looks like this. There’s a lot of project icons, and they are structured into different areas. Being a small team, you will not be able to use all these tools. You need to figure out what you need for your use case. You need to be mindful of what you take on, because if you take on too many things, it will be hard for you to maintain the speed, and it will be hard for you to adapt once you have to really work with the business value. Let’s say you’re working and you have some slack time, and so you go like, “What should we do now? How about this cool thing over here that I found? I think this would be a really nice feature for our product teams. I think they would really love it. Let’s do that”.
Then you start. Then you make everyone start using it. Then you get a new version, and you have to migrate everyone. Then suddenly the business comes and says like, we need you to fix this thing, because we need to add this capability into our platform. You go like, but we are working on this nice-to-have thing over here. You have must needs that you will not be able to address because suddenly you have filled all your time with nice-to-haves. Be mindful of what you take on. Be mindful that the open-source community is both a power and a risk, because there’s a risk of drowning into a lot of things, but it’s also the power of standing on the shoulders of other people and utilizing what they already have done. Have reservation, but make use of the things that you must have. Ask yourself, what can you live without? What do I not need?
For us, we realize we can live without a service mesh. A service mesh is a dedicated infrastructure layering for facilitating service-to-service communications within Kubernetes or other things. It’s really nice. You can get these fancy maps where you can see the graph of services talking to each other and all the things. You can do network policies, all them things. Really nice, but not a must for us. We don’t need it. In a similar way, we don’t need mutual TLS between applications in our Kubernetes cluster, because that’s not the main concern for us right now. Caveat, I really love Backstage.io as a project, but we don’t need a developer portal. It can be extremely nice to have. It can solve many issues that you have, but as a small company, we don’t have those pain points that motivate people to start using Backstage.
We don’t need to invest into a developer portal. Design system. Starting out, design system is like clear standards and component libraries that you can reuse for the frontend. Starting out, we did not want to invest into this because we didn’t see the need. Actually, in the last year, we have started to invest into a design system. It’s really valuable. We started out with the components that were mostly used throughout the application, and we started by standardizing those. Not every component is in the design system, but the ones that are, are used a lot, which is really nice for our frontend developers and our designers who can work together in collaborating on how they want the standard to work. Starting out, ask yourself, what things can you live without? What is on your nice-to-have list, but maybe not worth investing into?
Summary
With this knowledge, if you want to get started on your own paved road, what should your paved road contain? When you know what your business needs are, what your team needs are, how you can continuously build trust with your organization and your teams, and what tradeoffs you’re willing to make, then you’re ready to start paving your own road of empowered product teams. Do remember, you already have a platform. Start investing strategically into it. You should treat your platform as a product and unleash new capabilities. Trust is a currency, and you use it to gain adoption from your product teams. Tradeoffs is a key to success. Pick the right ones, and you can win again and again.
Questions and Answers
Participant 1: You started with your personal journey, and I appreciate that a lot. Forgive me for saying so. It seems to me like the deal was already in place when you joined the new company. You didn’t have to fight for a consensus. You didn’t have to convince the company to start building a platform team. Myself, I’m in a different situation. I’m still fighting that battle, and I am exactly against the things you were saying. We are very small as a company. Maybe we just need a bigger DevOps team. Based on my experience, based on my reading, it seems to me like to win these arguments, what one would have to do is, small POCs improve value, but you also need a little bit of help from the top down. I need to manage upwards. I need to manage downwards. I’m looking for some advice, basically.
Andersson: Eb was talking about change, how to drive change without having the authority. He talked about finding allies for your cause, and I think finding allies in the leadership is what you need to do. Maybe not someone directly in your line, if you have trouble convincing them. Find someone else in leadership that will listen to you and that can be an ally for you. Then we had a talk about bits and bots and something else around DevOps, and she talked about how the different team structures can look. I think she made a really good case for why a DevOps team is not the way to go. She had a really nice dumpster picture on the DevOps team and everything. Check that talk out if you haven’t. I think you can use that as a motivation for why a big DevOps team is not the solution.
Then I think, yes, it really helped to have our co-founder, Daniel, convinced before joining the company. He knew they needed to change something. He wasn’t sure how to approach it. Talking together, we could come to a shared vision of what that would look like, which was very useful.
Participant 2: Luckily, for us, we’re on the right path towards this, so we just started something like this. I’m trying to know, did you have something like a roadmap from the beginning? How long did it take you to achieve this?
Andersson: I’m a really bad product manager because I never had a roadmap that I successfully maintained, unfortunately. Luckily, no one really blamed me for it either. What we did mainly was, it took about a little bit over half a year from me joining before we actually had a team in place. There was a lot of recruitment and getting to know the company first. That’s where I spent the first time. Then when the team joined, we started to remove the blockers because there were some things that the teams were not able to do. Those were often quicker fixes, so we started with that. Within a year from June 2020, we had changed the DNS thing and we had changed the GitOps thing, but we had not started on the monorepo and the build times. Half of the things I’ve told you now was basically within the first year. This second half spread out over the last three years, but also all the things that I did not mention happened in the last three years.
Participant 3: If I’m a developer within your company, what does the platform look like? Is it a website? Is it documentation, API? If I want to deploy something on your platform as a developer, where do I go? How does it work on a day-to-day basis?
Andersson: As a developer wanting to use our platform, it’s back to the keynote thing. If you want to introduce a Kanban for a team that never worked with Agile, or Jira, or one of these things, start simple. They used an Excel sheet. We used mainly GitHub repositories where it was like, this is how you can copy-paste code to get started. It’s not a fancy platform in any way. It’s more like, here’s where you put your Kubernetes configuration and here’s what you can copy-paste to get started. Here’s where you can clone a GitHub repository to get that base application. It’s a little bit crude still, but it’s still making it more streamlined and everything. Boilerplates, we are currently working on rolling that out. It takes a while. Boilerplates are part of the fancy part of the platform. Bare necessities is just, make it run.
See more presentations with transcripts
MMS • Jessica Andersson
Subscribe on:
Transcript
Shane Hastie: Good day, folks. This is Shane Hastie for the InfoQ Engineering Culture Podcast. Today, I’m sitting down across many miles with Jessica Andresson. Jessica is in all the Swedish summer. Jessica, welcome. Thanks for taking the time to talk to us today.
Jessica Andresson: Thank you for having me.
Shane Hastie: My normal starting point is who’s Jessica?
Introductions [00:27]
Jessica Andresson: Yes, I wish I had a really good quick answer to that one. So I’m currently in between jobs intentionally, but I identify as a platform engineer and also a leader of platforms. I’ve been doing both and looking forward to getting a little bit hands-on again, come this fall. I’m very passionate about enabling teams and accelerating developers and trying to empower teams to deliver a lot of value. I think that is something where you can have a big impact. That’s something that I’m very passionate about. I’m also very involved in our local communities around cloud native, and I’m also a CNCF ambassador. It’s a lovely community. I’m really happy to be a part of that and be able to help that community evolve as well.
Shane Hastie: Jessica, one of the things that we were chatting about before we started was platform engineering as empowering and enabling others. You in fact said this is the opportunity to be the 10x engineer. How does that come about?
Platform engineering is about empowering others [01:29]
Jessica Andresson: Yes, I think that it was a lot of talk many years now that everyone should strive to become the 10x developer, and it was a lot of pressure and did it mean to build more code? Did it mean to ship faster, or whatever did it mean? And there was someone writing on Twitter a while back that being a platform engineer was the closest I ever felt to become a 10x developer, and this resonated so much with me. So I think for my own leading star, it’s not so much about becoming 10x of myself.
It’s about enabling 10 other developers to help them provide the value they want to do faster and more efficient and removing a lot of their pain points. And I think being on a platform team and working with platform engineering, that’s the chance you have to make an impact on that. As a platform team, we have always had the goal of empowering other teams to focus on things that they do to create value for their company or the organization. And so being able to help them to get there is really cool.
Shane Hastie: So what does a good platform engineering team feel like from a culture perspective?
Jessica Andresson: They are perceived from others as very approachable and helpful, I think. It’s always a little bit of track of being helpful because you could always end up being a little bit too helpful and spend all your time just answering questions, et cetera. But if you don’t have that approachableness and having people come up to you, it’s very hard to understand the problems of the other developers and the pain points that have, and they need to be able to feel like they are not asking these stupid questions or at least that they can come to you with these stupid questions and not get dissed or something, because they obviously don’t know the answer, and if you’re going to help them, if you’re going to empower them, you need to help them get to the point where they can solve themselves.
When it works really well, you should almost not notice that the platform team is there because the development teams should be able to do all the things they want to do themselves. I think it’s very important to have this non-blocking self-service mindset. And so I think that a really good platform team is probably perceived as helpful but also not really there because everything’s running so smoothly.
Shane Hastie: How do we create that platform team effectively?
Creating effective platform teams [03:44]
Jessica Andresson: I think you have to start somewhere. Everyone has to start somewhere. And when looking back on my own journey at Kognic, I realized that starting somewhere means that there is already a platform, because there’s always an implicit platform even if you didn’t plan for it, because if you’re running software in production, you have a need for some certain common needs such as build time, CICD, runtime, observability, et cetera. So those that did not work with the platform explicitly, they try to make the things work out as best as they could, and once it worked good enough, they went on to building the product as they were trying to do from the start.
And so as a platform team, you have to figure out what is there, and then you can try to transform it and stabilize it and make it work more streamlined. And from there try to go on and build it up. And when it comes to culture and what leads us to a trust and adoption of your platform, I think, is that you need to show deep development teams that you can solve their pain points. You need to show them quite early on that you are there to empower them and not to restrict them or be the police of what they can do and not do. So making it easy for them to do the things they want to do while also making sure that you aren’t blocking them so that it can move forward. And then you have to take one segment, make it work well, and then you take the next one and then you keep going. It’s an endless repeat on that one.
Shane Hastie: So how do you build these strong trusting relationships?
Building strong trusting relationships [05:24]
Jessica Andresson: Actually, my partner Carl Engström, he used to phrase that trust is a currency and that you have to treat it as such. And so in turn then you have to use trust in order to gain adoption. And if we look at trust as something that we can earn or spend, we can look at what things can we do in order to earn trust and what things will we do that will cause us to spend trust, and then we can keep a track of that balance and try to make sure that we never reach a negative balance on our accounts.
And so some of the things that I’ve identified as something that can help you earn more trust is removing pain points. I know I keep coming back to it, but it’s so easy to forget that in the daily life, the developers do things that they always think is slightly painful, but it’s not painful enough that they are going to raise a flag or come screaming at you whatnot, but it’s still slightly painful, slightly annoying, and it’s repetitive. So if you can find those and you can remove those, then that will create more trust from your developers and build that relationship.
We talked about being approachable and helpful and I think that if you keep being approachable and you’re keeping helpful and not be mean to people that ask stupid questions, they will keep coming back, and they will go back and they will tell their team that, “Oh, I had this issue and the platform team was super nice and helped me out,” and the next time someone else in that team has a problem, they are not afraid to reach out because they knew that their teammate got a nice reply. And it doesn’t mean that you have to always jump very fast trying to solve everything without thinking. It’s just about, “Hmm, could you explain more about your problem? Ah, interesting. Now we’ll look into that. I will get back to you,” and then you get back to them when you have answer or not. It’s about being treating all with kindness.
I think also if you are proactive while being approachable, you would probably hear things. The team stopped explicitly saying, “This is a problem,” but you realize I can do something to improve this situation. And if you proactively do that to improve this situation, that will also help people that trust and create a better collaboration. And I think, so easy to forget, but understanding the team’s perspective. They don’t stick with the same technology as you do day out and day in. They have other things in their plans that they want to achieve. You’re measured on different success rates. So understand where they come from, understand their perspective, and what they know and what they don’t know, and that will help you have a common language when you talk, and that’s very important in order to create a good collaboration and build trust.
And if we’re looking at what you can do in order to spend trust, which I think is also easy to forget that there are actually things that will decrease the trust that the team’s have in you. And I think that probably this one to understand it’s like enforcing processes. In my experience, all teams hate when you try to tell them what to do, they always want to try to figure out themselves. It comes back to autonomy. I think that teams like to feel like they have the chance to decide themselves, but when you enforce a process or you tell them you have to do, like they see you remove that choice from them, and you’re pushing them into a box that they did not know that they wanted to be.
Of course you can counter that by anchoring decisions and all those things, but still enforcing a process will cause you to spend some amount of trust. If it’s much or little, that depends on how you do it. If you introduce blocking processes, I think you are also spending trust. If the teams can’t do something themself, they have to go to a gatekeeper in order to get it forward, that’s going to cost you. And it feels given, but worth mentioning.
Platform teams in general spend a lot of time on migrating and upgrading. We move it from one tool to another, we move it in a new version, some standard got deprecated, we have to get a new standard and whatnot. All migrations cost you, depending on how well you do it, depends on how much you spend on it. The best migrations are the ones that the teams don’t have to care about at all. You sometimes you can get away if they feel like they’re getting something extremely valuable out of migration, but it’s still like a small cost, I would say. And I think diverse spending is probably assumptions. Like we are really good at our jobs, we know what the teams should want.
So if we make an assumption that they really want this thing and then we make it happen and they didn’t really want it, then that’s going to cost us a lot. And so we’re back to anchoring change and understanding their perspective and all those things. So I think understanding the team’s perspective and assumptions goes hand-in-hand.
Shane Hastie: You mentioned that platform teams have different success measures to product development teams. What are the important success metrics or measures that platform teams should be looking to?
Success metrics for platform teams [10:24]
Jessica Andresson: So I think that it’s very easy to look at the DORA metrics when it comes to this. It’s hard to implement, but it’s easy to look at, because you have to figure out what those means to you. And so we have four main metrics or core metrics that I usually get to look at. So that is deployment frequency, lead time to changes, mean time to recovery, and change failure rate. And exactly how you measure those will be unique to your company.
There are some guidelines and some standards that you can look at, but you will probably have to make some slight adoption to your organization. But I think the important thing is not that you measured exactly the same as everyone else, it’s that you measured exactly the same all the time for yourself and that you compare against yourself over time, because that’s how you will figure out if you’re making any difference.
I heard of a lot of people measuring things such as developer efficiency and developer happiness and those kind of things. From my point of view, it’s probably not that easy to do, but we were playing around a lot with the idea of continuously sending out a survey. Kognic uses Peakon for employee satisfaction and those kind of things, and we had the idea that maybe we could send out something similar maybe every half year or something. We never got there though, because other things were very important.
Shane Hastie: The challenges in software engineering in my experience and what I see and hear are almost always people-related, not technology-related. How do we as leaders in that space engage people well to reduce those challenges as much as possible?
Most of the challenges we face are people challenges, not technology [12:13]
Jessica Andresson: As you say, the people problem is always the hardest one, and I find it very interesting to hear when I talk with other friends outside in other organizations is that we all seem to struggle with quite similar problems regardless of what product we are building. And so it’s often comes down to communication, alignment, and all those kind of things that usually are the same that show up frequently. And I honestly don’t have a good answer, I’m sorry, but I’m thinking working on making sure that everyone knows what we’re trying to achieve, what direction we’re supposed to go, and what is the most important right now, like alignment, that’s something that you can start with.
And then working on how you communicate around alignment so that people have the right information when they need it and then they are not flooded with too much information that doesn’t change anything for them so that they feel like … Because you can end up in a state where you feel like a lot of things are changing while they really are not because of how you communicate, and the other way around where people feel like something needs to change but no one is doing anything, and that’s because maybe you’re having those meetings in a different setting with different group of people and you don’t communicate to others. We’re on it but we don’t have the answer yet. So from my point of view, those are probably some of the hardest questions, and I think we will struggle with them.
Shane Hastie: A lot of our audience are people who are moving into leadership roles for often the first time. You’ve spoken about being an engineering leader and also being the practitioner. What are the big steps? What are the big changes when you move from that individual contributor practitioner into the engineering leader space?
Advice for new leaders [14:03]
Jessica Andresson: For me, I was very lucky because I had a chance of doing very gradually. So when I joined Kognic about four years ago, I was the first platform engineer/leader. There was no team. And so I started out very hand-on but also we trying out thinking about a strategy, trying to build a plan, hiring a team, and then getting started with a very small team while still working hands-on on the platform. And then over time, I took on more responsibilities and in the end the team kindly told me that they were not expecting me to contribute to the backlog anymore, also known as “Please don’t mess anything up.” And I think that was a very nice way of getting to know the different aspects a little bit at a time. And If anyone has a chance to do that, I probably would recommend it because I really enjoyed that process.
And I think something that I struggled with a lot at this pace when I was no longer hands-on in the team but still very much invested into what was being built technically is to not be able to have the full picture in my mind, because I was used to know how every system worked and how all things were connected together, and to go from not being able to do that while still being involved in the product and where it was going and what problems we were solving was very hard for me.
And I think also, because I’m one of those people that tend to have a lot of control over things around me, I know where all the things are in the house, I can pull up a lot of details from my mind, and not being able to have all that context was a bit of a struggle. But that’s probably very personal for me. I think everyone will discover their own struggles. There’s also so many things that you don’t think about, like how doing salary review is probably more of a salary season than a salary review period for instance, and all those things that tend to be a lot more around the practical details of running a team of employees rather than building a platform. That’s probably one of the reasons why I’m moving back to hands-on instead.
Shane Hastie: What advice would you give somebody who’s stepping into that space?
Jessica Andresson: Probably try to figure out what you want to achieve. There was a great talk at QCon London 2024 regarding how to drive change without authority, and the speaker said that he wanted to get into management because he wanted to tell people what they should do, and then he realized that’s definitely not what I should do. I should rather try to drive the change and try to influence people, because we are back to telling people what to do, it’s usually not successful.
And I feel that if you try to get into leadership or a platform team or engineering in general, figure out why you want to do it and what you want to achieve with it, and then double check if your perception of what it means matches reality. What I did before taking on this role was that I had a contact that had done a lot of technical hands-on herself previously, and then moved into an engineering leadership position.
And so I asked her out lunch and asked if I could ask some questions about that transition, and ask all those concerns like will you feel like you don’t connect to the lingo after a while? Like you lose connection to the technical aspects? Is it fun? Is it a lot of questions about handling conflicts? How do you handle those things? Because suddenly you have to do the things that are not fun as well because there’s a lot of things in leadership that are not super fun. They might even be very hard, such as if you have big conflict or something is not working out the way it’s supposed to do and no one else is there to handle it, suddenly it’s your problem. When you are an IC, in most cases, someone is there to take care of those things.
So I had the chance to talk to her for a long time and ask all my questions and try to figure out what it would mean to be a leader and to probably voice some of my concerns and get some advice around that. And that’s probably a really good advice if anyone is looking to move into engineering leadership, to try to find someone and have that conversation.
Shane Hastie: Where is platform engineering heading? What are the trends? What’s next?
Trends in platform engineering [18:38]
Jessica Andresson: That’s a very good question. I think it’s just wasted hype, like the highest hype level. I know for sure that people have been doing platform engineering for a very long time, but I think it’s just the recent two, three years maybe that we had the words for it so that everyone could share the same words for what they were doing, because a lot of people have been working with this for a very long time, and I’ve had knowledge sharing with other companies back before we even called it platform engineering. But I think in the future, it will probably have to try to deal with the awkwardness around what does platform engineering mean? What does DevOps mean? What does empowered teams and autonomy and all those things mean? Because right now, I’m a little bit worried that platform and engineer will follow the same road as DevOps did.
That means that you have platform teams, but they are just the new operations team with a fancy title, because that’s a lot of what I feel happened with DevOps, to be honest. I think a lot of companies will be able to drive platform as it’s meant to be. That means like internal developer platform, empowering teams to reach their goals, but also worried that a lot of companies will do a little bit of that and a lot of operations, because that’s what they’re used to, that’s what always worked, and you always have these things that you don’t have a better solution for, so why not just throw it on the team that seems that they get all those kind of things and that. Used to be the DevOps teams, in quotes, and now it might be the platform team, I worry.
Shane Hastie: So, hold the space carefully.
Jessica Andresson: Yes, I think so. And you need strong leaders with a plan, and leader in this case might be an engineering manager, it might be a product manager, it could be anyone, but you need strong leaders that can fight for maintaining the ID of running a platform team with the goal of empowering development teams. Because if you don’t have that, you will end up being the sink where all the things go.
Shane Hastie: Jessica, a lot of good advice and interesting ideas here. If people wanted to continue the conversation, where can they find you?
Jessica Andresson: They can find me on LinkedIn and X and Bluesky. Mostly I’m solidtubez. I think we’ll add a link somewhere because that’s hard to spell. I’m happy if anyone would like to reach out. I love chatting about these topics, and I would really love to hear what everyone else is working on as well because these are a lot of my ideas. I think there are a lot of synergies with others, but everyone has their own flavor and you always have to adapt slightly to whatever situation you’re in. So I think everyone has a unique story to tell about this.
Shane Hastie: Thank you so much.
Jessica Andresson: Thank you.
Mentioned:
.
From this page you also have access to our recorded show notes. They all have clickable links that will take you directly to that part of the audio.