Month: July 2022

Transcript

Shane Hastie: Good day, folks. This is Shane Hastie for the InfoQ Engineering Culture Podcast. Today I’m sitting down with Lilac Mohr. Lilac is an engineering leader at Pluralsight. Lilac, welcome. Thank you for taking the time to talk to us today.

Lilac Mohr: Thanks for having me, Shane.

Shane Hastie: Let’s start a little bit and who’s Lilac, what’s your background?

Introductions [00:25]

Lilac Mohr: As you said, right now I lead software engineering teams that are working on the Flow product at Pluralsight. Flow’s the software delivery intelligence platform. So we look at development workflows holistically. We look at all the different activities that make up an engineer’s day, how engineering teams collaborate, how well they’re doing unpredictability of being able to deliver on their commitments, and the general efficiency of the workflow.

And it’s really fun because we get to use our own product and continuously improve the product and the process. And prior to this job I led other engineering teams, but I was also a developer for a very long time. So I’m still very connected to the software engineering side.

Shane Hastie: What are some of the things that get in the way of engineers being productive today?

Challenges engineering teams are facing [01:20]

Lilac Mohr: We try to stay away from the word productivity, we don’t want to measure engineering productivity. We want to really have a holistic view. Make sure that we’re looking at engineers as people, at what makes their day good. At the end of the day ask them the question, “What were the blockers you had today? And then what were things you were really excited about today?”

And as an engineering leader, I do have to talk about deliverables, being able to deliver with excellence and the outcomes, but then it comes down to the people who are really behind the code. Every line of code comes from an engineer, and it comes from a collaborative process. And some of the things that I’ve found have been creating friction to that’s delivery. Right now I think with the Great Resignation it’s very difficult for me to keep a healthy team because people are leaving, so I think that’s where the culture plays in and it’s important. When someone leaves, it puts a big burden on everyone who’s left. So I want engineers to feel like they can keep moving forward and don’t have to take on a lot of extra work.

I think that’s a big challenge that all engineering teams are facing right now. And as something we do with our Flow product, is we’re always talking to our customers and trying to understand what it takes to have great engineering teams. We ask them the question, “Are your team set up for success?” And we get a wide variety of answers. It’s actually a difficult question to answer. A lot of times engineering managers focus on the process and the tools and say, “If I have the right tools, if I have an agile process, then my teams are set up for success.”

And that’s not digging deep enough, in my opinion. I think that a lot of times that’s based on an old playbook. Everyone is still talking about agile transformation as they have been the last 20 years. And with all the attrition that we’re seeing, you need to really focus on people. You need to take a broader view of everything that goes into making teams healthy.

Shane Hastie: What does a healthy team look like and feel like today?

What makes a healthy team [03:39]

Lilac Mohr: I think it comes down to developer satisfaction. And actually, I’m really big into metrics, and I think part of that is looking at the data itself, for how they’re able to deliver what their day looks like. But then part of it is also being able to survey them and ask them, “How are you doing? How are you feeling? What’s your confidence level right now?”

So I think it’s a combination of those factors. And I think a healthy team is a team that genuinely likes everyone on the team. They like working together, they like solving problems. They feel empowered to be able to make decisions that are closest to them, the engineers are the ones that are the closest to the code. And they want to feel that they understand the value of the things that they’re producing, and that they’re empowered to be able to make those decisions.

Shane Hastie: As an engineering leads, how can you influence this? How do you create this space?

Leaders enable the culture [04:43]

Lilac Mohr: It all comes down to culture. And I think that visibility is really important. I think that when you have the right metrics and you have the visibility into those metrics, then you let individuals make decisions about their team, what’s best for their team. And I think that drives the culture.

I think that it’s also important to have gratitude and have a culture of constantly acknowledging people’s hard work and their contributions. And not just coming from leadership, but encouraging them to acknowledge each other and have that space where you can provide feedback across teams, up, down, all around. It just becomes part of the culture. And the example of how we do this at Pluralsight and at Flow is we have Gratitude Fridays, where on Friday everyone posts the co-worker of the Week and recognizes someone who just moved the needle a little bit on making their experience that week fantastic. And it’s really fun to see everyone just acknowledging each other.

And I know, doing a skip level review with one of the new engineers, I remember him telling me, “At first I was really sceptical and thought it was really cheesy to do that. But then after someone tagged me and acknowledged me for the help that I provided that week, it felt really good, and I get it now.” I love hearing that type of feedback and seeing people just appreciate each other and enjoy working together as a team.

Shane Hastie: The perspective that you just mentioned from that engineer, “It felt really tacky,” it does kind of sound like that. As an organization, as a leader, as a co-worker, how do you prevent that from becoming tacky?

An example I can give is one organization that I spent some time with used a weekly 15-5, where they asked five questions, 15 minutes, “How are you doing?” And if anyone seemed even vaguely dissatisfied, a very senior leader, out of genuine care and concern, would immediately get hold of that person and say, “What’s going on? How can I help?” But to the engineers this felt intrusive and creepy. So they stopped reporting it was tough, everything was shiny, so they didn’t get the personal phone call.

Creating trust and enabling openness [07:18]

Lilac Mohr: Yeah, I think you have to create that trust in each other. And I think that a lot of those types of initiatives need to come from within, not from leaders saying, “Hey, this is the great new thing we’re going to do.” Maybe they can get it started and then make sure they have that feedback loop and that ear to the ground to ensure that it’s being carried on by the team members, that they appreciate it and that they share it with each other. And I think that’s important.

For example, like what you mentioned in one-on-ones. Everyone talks about how important it is to get that feedback in one-on-ones and how employees are doing, and making sure as a leader that you can address some of that’s discontent. But there’s ways to do it that are healthy, and I think there’s ways to do it that feel weaponizing. It’s about creating that safe environment. And being able to ask, if someone shares some information with you about how they’re doing, make sure that you ask them, “Is this okay if I share this with anyone else?” Sometimes you want to keep that relationship private.

I think as humans, when someone says that something’s wrong, you want to immediately react and say, “Well, I want to fix it.” Or you want to make them feel better. Sometimes all you need to do is just validate them and say, “What, if anything, would you like me to do about this? Or do you want me just to be a sounding board just to listen?” And sometimes people say, “Right now I just need you to listen.”

Shane Hastie: One of the things you mentioned earlier on, Lilac, was the importance of metrics and getting that feedback about what is actually happening in the team, the way work is flowing and so forth. Couple of things if I could explore with you there. One, what are some of those important metrics?

The importance of metrics [09:13]

Lilac Mohr: First I’d like to zoom out and just talk about the importance of metrics in general. I think a lot of times engineering leaders are very comfortable with using metrics to measure systems. We want to know how is the database operating, is the system up or down? Things that are less human. And then when they think about their humans they get a little bit uncomfortable about using metrics.

And I think that discomfort probably comes from focusing on a single metric. I think that the holistic approach is what you really need to take, where you’re not looking at just a single dimension. We all know that if you focus on a single metric, that’s what you’re going to get. And sometimes you are optimizing one little piece, but it actually creates tension somewhere else. Or maybe you’re focusing on the wrong thing. A lot of times that’s called the streetlight effect, where maybe you dropped your keys on the way to your car and the dark, but where you’re going to look is under the streetlight because that’s where you can see them.

Examples of metrics that can be useful [10:20]

Lilac Mohr: At Flow, we really try to create this holistic view, where we’re looking at engineering metrics down to the primary level is your code commits. So making sure that engineers have good habits around code, and those habits need to come with a purpose. The reason that we want to commit frequently is because we don’t want to hold onto code until it gets stale. We want to have that safety to check in the code and then be able to quickly get some peer review on it and get the process moving. We want to make sure that we’re freed up to code frequently. How frequently you commit code is not about how fast you’re working, it gives you an outlet as an engineer to say, “I don’t have time to do my work because I keep getting pulled in these different directions,” or, “I have meetings all day.” And it allows you to have those types of conversations with your leader.

That’s at the primary level, just things you should be doing is making the time to be able to code and doing frequent commits. And then the next level is collaboration, which I think is the most critical to code quality. A lot of times we think about code quality as lets have tools that analyze our code for that quality. But the quality really comes from helping each other on a team. Doing that peer review and doing it effectively, and being able to get all that work in progress moved through the system.

We take a look at pull requests, at the type of comments you’re getting on pull requests. How long they’re sitting in queues before another pair of eyes takes a look at them. And then also making sure that we’re not just rubber-stamping them as they go through. It’s a great way to learn and to help each other in that quality, so that’s the second level. And then the highest level that we track in Flow is at the ticket level, which is, these are the things that your team committed to doing in the sprint or in this chunk of time. Are you able to deliver on those, and what are the blockers to flow? Where is that work getting stuck in queues? Where are there unnecessary handoffs, anything else that we can improve in that higher level process.

I think all those things stack on top of each other and help you get an understanding of how, as a team, you’re able to deliver code. And important part of using those metrics is using them to guide discussions. They’re not supposed to be used to punish teams, to make teams feel bad about their process, to blame anyone. It’s all about being able to identify things that sometimes your intuition doesn’t realize that some of that friction is there. It gets conversations going. And I think that if it’s framed that way, then those metrics are enabling.

Shane Hastie: How do we ensure, and as a manager, how do I make sure that we’re not bringing that blame culture in, that we’re not using those metrics in the ways that would be so easy to do, but as you say, are not right.

Empowering people to own their own improvements [13:49]

Lilac Mohr: I think visibility is the key, in having access to the metrics. We have an engineer who was using Flow at her previous company. And she said that her leader would come in and show the Flow metrics to the team and provide some insight, but they didn’t have access to that data themselves. And she was curious, she was like, “Well, if they’re tracking this information about me, what else does he know? What else is in there?”

I think that what we’re trying to do with our customers is encourage them to let individual contributors have access to their own metrics. And we found that by doing that internally in our own team, a lot of great insights come out. Sometimes what we do is we just say, “Okay, just explore the metrics and come back with one thing. One thing that you found that’s interesting, maybe something unexpected, or something that you want to improve on as an individual or as a team.” And we’ve found amazing things come out of that because it’s open, it’s not coming from engineering managers. It’s not something punishing like, “I found this thing we can improve upon.” It’s asking them to be empowered, to look at their own data and make improvements.

Shane Hastie: One of the things you mentioned earlier on that I’d like to just delve into a little bit more again, if I may. You spoke about how team performance does come when people like each other and work effectively together. How do we prevent that from becoming a monoculture?

Avoiding monoculture [15:28]

Lilac Mohr: I think that’s an important topic. And a lot of times we find our teams do have a subculture. The teams, we let them pick their own names and they’re really proud of who they are. And I think that probably needs to be balanced with the culture for the entire engineering organization, because one of the problems is a lot of times it creates problems with inclusion. For example, an engineer who maybe is older, who’s on a team with younger individuals, they talk about different things. And I’ve heard some of this feedback. They talk about things that he feels he can’t relate to. So how do we create that sense of inclusion on a team and still allow them to feel really close to each other [inaudible 00:16:18]?

Pluralsight does really well because we talk about inclusion and diversity a lot. We just make it … It’s not a special topic we discuss, it’s just part of how we think. So we’re constantly asking, “Am I being inclusive?” And being able to stand up for each other and be able to provide that feedback and say, “I feel that this person was left out of the conversation,” or, “Can we choose an activity that might be more inclusive to do as a group?” And by talking about it a lot it makes it safe. And it makes it something that, when it’s not calling people out, it’s just pointing out something that we all are aiming towards. We all want everyone to feel good.

And I know that personally, as a female software engineer in a very male-dominated industry, I’ve been doing this for about 25 years. So back in the late ’90s, a lot of times I was at startups where I was the only female. Sometimes the only female engineer, sometimes the only female in the entire small organization. And there were a lot of awkward situations where I didn’t feel like I belonged. And coming into Pluralsight and being able to say, “I belong here” was really powerful for me. Because looking back, I didn’t always have that feeling. And I think it comes from just creating the importance around belonging, where everyone really wants everyone else to feel like they should be here. And it’s huge.

Shane Hastie: One of the things that we see, and I’d love to know your experiences around this, becoming an engineering leader, we often take the best technologist and we throw them into a leadership position. And one statistic says 58% of new managers receive absolutely no training.

How do we help people who are transitioning into those leadership roles become leaders? Because the skillsets are very, very different from the individual contributor to the leader.

Helping new leaders [18:30]

Lilac Mohr: It’s definitely a challenge. I know in the Flow organization, we do have that culture of taking the best engineer and saying, “You’re going to be the leader,” and it doesn’t always work out. A lot of times we have seen those leaders go back to being individual contributors. And we make it okay, we make it okay even within the organization to do that because otherwise they would leave the organization to become ICs.

But part of it is making sure that they get the satisfaction that they did from being able to generate a lot of code to help their team, to be able to focus on other high-performing engineers and be able to raise the level up of the entire organization. I think it’s just that mind shift. It’s very difficult to let go off code because we enjoy writing code, but how can you be more impactful? Instead of just grabbing tickets and then becoming a bottleneck for your team, how can you coach others to be able to get to the next level with our own careers?

And then also, how can you keep your fingernails dirty in the code, because that’s what you love. Maybe doing some pairing with engineers to help them get their own levels up or be able to be involved in architectural discussions. Ways that you can still help at a technical level, but you’re helping not just in the code, but helping the team move up and improve their own career paths and being able to deliver.

Shane Hastie: In your position as an even more senior leader, what are you doing?

Transitioning into more senior leadership [20:14]

Lilac Mohr: For me, the transition from individual contributor to a leader of one team, to a leader of multiple teams, it was tricky because you’re responsible for more. You know that ultimately, I’m responsible for all these outcomes, but I have to resist the urge to micromanage. I have to trust people.

I think that I would focus that effort where I want to dig in and be able to know everything that’s going on, to focus on coaching and focus on that higher level of enabling people and putting that trust in people and using data. We always come back to the data. The data to help me understand what’s going on with my teams, but also helping those leaders do the same thing, where they’re not micromanaging, but they’re driving and understanding where they can be helpful on reducing friction on their teams as well.

Shane Hastie: And cycling back to another point that you made that’s hanging over us all at the moment, the Great Resignation. Keeping things flowing through it and retaining great people, how do we do that?

Advice on retaining people [21:27]

Lilac Mohr: The first thing is you need to value your people. It sounds really easy, but a lot of people don’t genuinely care about people at the level that they should. I remember I was interviewing for a leadership role for a very small company, and the owner of the company made some offhanded remark about engineers are a dime a dozen. And I remember having a very strong reaction to that comment and trying to think about it. Was my reaction due to me thinking that engineers need to be treated like special snowflakes? Is it because of my experience with hiring new engineers and how difficult that was?

And I think what it came down to is you can’t have a successful company at any level unless you really value all the individuals there. And you’re not going to have individuals put in their best work unless they know that they’re cared for, that they’re valued. I think it all comes down to that, putting people first. Not thinking about people as resources but as human beings, and I think that that helps with retention.

Shane Hastie: And when we do lose people, as inevitable in these times, how do we let them go well?

Lilac Mohr: We need, as a group, to celebrate the opportunities that they have, and leaders can set the stage for that. I think that if a leader handles someone leaving well, then the rest of the organization feels like it’s okay, it’s not something that they need to panic about. And I think that a lot of times I question my own leadership abilities in different areas, I think everyone does. But some feedback I’ve heard that made me feel really good is that I can be a calming force amidst all the change. And I remember talking to one engineer who admitted that they were thinking about leaving. So it was about a year ago they were thinking about leaving and seriously considering it and not happy. And they decided to stay because of me, because of a discussion they had with me.

To me, just knowing that even if I saved one person and made them feel like they can have a successful career staying here at Pluralsight, that gives me validation that I did my job as a leader. You can’t save everyone, but if you can make a difference, even in just one engineer’s life, I think it’s worthwhile.

Shane Hastie: And then the flip side of that, how do we bring people into the organization well?

Onboarding people well [24:14]

Lilac Mohr: A lot of times we’re looking in the wrong places we’re looking for, “Let’s bring in the top talent,” instead of thinking, “How do we bring in really good people and then be able to grow them?” I think that mind shift helps us bring in people who feel like they’re going to belong. We know that they can have good problem-solving skills, that they’re going to work together with their team members, that they’re hungry to learn to solve these problems. And I think those sometimes make the best engineers, especially in the long-term, because then you get to cultivate them and be able to watch them grow and learn.

I think that’s definitely the first step, is thinking about who you’re hiring and why. We know that diverse teams are better performing teams. Bringing in that diversity and then making sure that it’s not checking a box, it’s bringing in the best people, and then making sure they feel like they belong, I think is really playing the game for the long-term. Making sure that you’re building a healthy team, not just for now, but into the future.

Shane Hastie: Lilac, thank you very much for taking the time to talk to us today. If people want to continue the conversation, where do they find you?

Lilac Mohr: They can find me on LinkedIn. And we’re constantly hiring, so don’t hesitate to reach out.

Shane Hastie: Wonderful. Thanks so much.

Lilac Mohr: Thank you.

Mentioned

Introduction [00:01]

Thomas Betts: Hi, everyone. Before we get to today’s episode with Marco Valtas, I wanted to let you know that Marco will be speaking at our upcoming software development conferences, QCon San Francisco and QCon Plus. Both QCon conferences focus on the people that develop and work with future technologies. You’ll learn practical inspiration from over 60 software leaders deep in the trenches, creating software, scaling architectures, and fine tuning their technical leadership to help you adopt the right patterns and practices. Marco will be there speaking about green tech and I’ll be there hosting the modern APIs track.

QCon San Francisco is in-person from October 24th to the 26th, and QCon Plus is online and runs from November 29th through to December 9th. Early bird pricing is currently available for both events, and you can learn more at qconsf.com and qconplus.com. We hope to see you there.

Hello, and welcome to another episode of The InfoQ Podcast. I’m Thomas Betts. And today, I’m joined by Marco Valtas. Marco is the Technical Lead for cleantech and sustainability at ThoughtWorks North America. He’s been with ThoughtWorks for about 12 years, and he’s here today to talk about green software. Marco, welcome to The InfoQ Podcast.

Marco Valtas: Thank you. Thank you for having me.

The Principles of Green Software Engineering [01:07]

Thomas Betts: I want to start off our discussion with the principles of green software engineering. Our listeners can go find these at principles.green is the website. There are eight listed. I don’t think we need to go into all of them, but can you give a high level overview of why the principles were created and discuss some of the major issues they cover?

Marco Valtas: The principles were published around 2019 by the Green Software Foundation. They are very broad on purpose. And the need for the principles is basically that, how can we frame … Well, that’s how principles work. Principles help us to make decisions. When you are facing a decision, you can rely on a principle to guide you. Well, what is the trade-off that am doing? That’s basically what we have on the Green Software Principles.

They are generic in a sense. Like, be carbon efficient, be electricity efficient, measure your carbon intensity or be aware of your carbon intensity. I think the challenge that it posed to all development is like, okay, when I’m doing my software development decision, what trade-offs I’m making in terms of those principles. Am I making a trade-off of using a certain technology or doing something a certain way that will incur more carbon emissions or electricity consumption and so on and so forth?

Thomas Betts: Who are these principles for? Are they just for engineers writing the code? Are they CTOs and CEOs making big monetary decisions? Operations coming in and saying, “We need to run bigger or smaller servers”?

Marco Valtas: I think they are a target for folks that are making decisions on the application level. You can think about the operations and the software development itself. But operations, usually you are going to look on energy profile, like the data center that you are using, what region you are using. So operations can make big calls about what hardware you’re using.

Those are also in the principles, but the principles are also to help you to make a decision, at a very low-level. Like, what is the size of assets that you are using on your website? How much data you’re trafficking through the network, how often you’re doing that. If those apply to an operations decision, that will work. If those principles apply to a development decision, they will work. And that can be said for a CTO position. If you’re making a decision that will have an impact on any of those carbon emissions, electricity consumption, that can help you to make a decision.

Thomas Betts: I know one of the topics on our InfoQ Trends Report this year, and I think last year was the first time we put design for sustainability for one of the architecture trends that people are watching. And it’s something that people have been thinking about for a few years now, but I think you said these came out in 2019.

That’s about the same time where it says these are how you think about those decisions, and the idea of the trade-offs you’re saying architects are always concerned with. It’s like, well, the answer is, it depends. What does it depend on? 

Varying carbon impact across data center regions [04:03]

Thomas Betts: Can you go into some of the details? When you say about a data center choice, what difference does that make based on which data center you use and where it’s located?

Marco Valtas: For data centers, you can think about cloud usually, not considering your own data center, but each data center is located in a geographic region. And that geographic region have a grid that provides energy, provides electricity to that data center. If that data center is located in a region where there’s way more fossil fuel energy being generated than solar or wind, that region, that data center has a carbon intensity profile of a larger carbon intensity. If you run your workloads on that region, usually you are going to count more carbon intensity.

But if you run the same workload in another data center, let’s say that you change from one cloud region to another cloud region that is on the energy grid, that region has way more solar than fossil, you are using way more renewable energy. So it lowers down the intensity of your workload. Considering where your data center is, is one of the factors, especially if you look on how the grids are distributed.

Thomas Betts: Is that information that’s easily accessible? I think there’s a general consensus. I live in the United States. The Eastern regions are very coal oriented, and so it’s very fossil fuel. But if you move more into the US Central regions for Azure or AWS, whoever, you get a little bit more of renewables. But is that something that I can go onto a website and say, “I’m running this in US-EAST-1, compare it to US-CENTRAL and tell me what the carbon offset is”? That doesn’t seem like a number I see on any of the dashboards I go to.

Marco Valtas: You won’t see it on any of the dashboards. We at Thoughtworks, we created one cloud carbon footprint tool, which actually helps you to look on the footprint by region. But the data, unfortunately, the data is not easily accessible from the cloud providers. We can go over a tangent around how cloud providers are actually handling this information, but the way that we do on our tool and most of other tools that are there, in the case of the United States, you can use the EPA Report on regions from the United States.

If you go to Europe, there will be the agents of Europe, and there’s other regions where you can get from another public databases, where they have reports around, what is the carbon intensity per watt hour in that region? You plug that and you make the assumption that that data center, it’s on that carbon intensity level. It gets complicated or complex if you consider that maybe the data center is not using that grid’s power. Maybe the data center has solar in itself, so it’s offsetting a little bit. It’s using a little bit of a renewable.

But that starts to be really hard because you don’t know, and the providers will make that information easily accessible. So you go for estimates.

Cost has a limited correlation to carbon impact [07:01]

Thomas Betts: All the data center usage, the number that people are usually familiar with is, what’s my monthly bill? Because that’s the number they get. And there’s correlations. Well, if I’m spending more money, I’m probably using more resources. But all of these have a fudge factor built in. That is not a direct correlation.

And now we’re going to an even more indirect correlation if I’m using more electricity here and it’s in a region that has a higher carbon footprint than I have a higher carbon footprint. But I’m in an 87 and I could be at a 43, whatever those numbers would be.

Marco Valtas: Cost is interesting. Cost is a fair assumption if you have nothing more to rely on. Especially on cloud resources, because you pay by the resource. If you’re using more space on the hard drives, you’re going to pay for more. If you’re using a lot of computing, you are paying more. If you’re using less, you’re going to pay less. It’s the easier assumption that, well, if I cut my costs, I will cut my emissions. But as you said, there’s a correlation, but there’s a limit to that correlation. You actually find that limit quite quickly, just moving regions.

Recently, I recorded a talk on XConf which will be published later this month, and I plotted against the cost of running some resources on AWS and how much carbon intensity on those resources. That correlation of cost and emission is not a straight line at all. If you move from the United States to Europe, you can cut your emissions considerably if you look just on the intensity of those regions. But you are going to raise your cost. That actually breaks the argument like, oh, cost? If I cut my cost, I cut my emissions. No, it doesn’t work like that.

If you have only cost, sure, use that. But don’t aim for that. Try to get the real carbon emission factor that you have on your application. That’s where our cost stands.

Taking a pragmatic approach to optimizing applications [08:57]

Thomas Betts: How do I go about looking at my code and saying, “Well, I want it to be more performant”? Anyone who’s been around for a while has run into processes that this is taking an hour. And if I change it a little bit, I can get it to run in a minute, because it’s just inefficient code. Well, there’s those kinds of optimizations, but then there’s all of the different scenarios. Like, how are my users using this system?

Am I sending too much data over the wire and they’re making too many requests? How far away are they from my data center? There’s just so many factors that go into this. You said it’s a holistic view, but how do you take the holistic view? And then, do you have to look at every data point and say, okay, we can optimize it here, and we can optimize it there, and we can optimize it here?

Marco Valtas: This is what makes this such a rich field to be in, and why I really enjoy being part of it. There are so many things that you can think of, so many actions that you can take. But in order to be pragmatic about it, you should think as an optimization loop, as you have an optimization loop in your performances of your application. You try to find, what is the lowest, what is my bottleneck? What is the thing that is more responsible for my emissions overall?

Let’s say that I have a very, very inefficient application that takes too long to answer. It sorts the data four times before delivering back to the user, and that is obviously my computing side of things are the worst culprit on my emission. So let’s tackle that. And then you can drill down over and over and over again, till you can make calls about what kind of data structure I’m using. Am I using a linked list? I’m using an array list. I’m using what kind of loop. I’m using streams.

Those are decisions that will affect your CPU utilization, which translates to your energy utilization that you can profile and make some calls. But again, it gets complicated. It gets very distributed. The amount of savings depends, so you need to go for the low-hanging fruit first.

But yeah, measuring? There is some proposals from the Green Software Foundations, like the Carbon-Aware SDK and the Software Carbon Intensity Score, which you can get some variables and do a calculation, try to measure your application as a whole. Like, what is my score?

The good thing about those scores is not just being able to see a number, but also compare that number with your decisions. If I change something on my application, does this, relative to the previous state, it gets better or it gets worse? Am I doing things that are improving my intensity or not? And of course, there’s counterintuitive things.

There’s one concept which is called the static power draw from servers. Imagine that you have a server running at 10%. It consumes in energy to be just on. And the counterintuitive idea here is that if you run your server at a 90% of utilization, that won’t be the same as increasing your 80% in energy consumption, because you have a baseline just to keep the server up. Memory also needs to be powered, but it doesn’t draw more power if it’s busy or not. Sometimes you need to make decisions of using a server more than spreading across servers. Those are trade-offs that you need to take.

Thomas Betts: That’s one of those ideas that people have about moving from an on-premise data center to the cloud, that you used to have to provision a server for your maximum possible capacity. Usually, that’s over provisioned most of your time, because you’re waiting for the Black Friday event. And then these servers are sitting idle. The cloud gives you those capabilities, but you still have to design for them.

And that goes back to, I think, some of these trade-offs. We need to design our system so that it scales differently. And assuming the run at 50 to 90% is a good thing as opposed to, oh my gosh, my server’s under heavy load and that seems bad, you said it’s kind of counterintuitive. How do we get people to start thinking that way about their software designs?

Marco Valtas: That’s true. Moving to the cloud gave us the ability of, use just the things that we are actually using and not having the servers idle. I don’t think we got there, in the sense that there was a blog post saying that it calculated around $26 billion wasted in cloud resources in 2021, with servers that are up and not doing anything or are offering resources. We can do better on our optimization of the use of the cloud.

Cloud is excellent too, because you can power off and provision as you like, other than on-prems. Bringing that to the table and how can you design your systems to think about that, it starts with measuring how complex an application can be. It’s kind of unbounded. The way that you’re going to design your application to make best use of the carbon resource is going to go through hops. And you should measure how much energy, how much resources are you using?

Some things are given in a sense like, well, if I use more compute, probably I’m doing more emissions. But then there’s more complex decisions. Like, should I use an event-based architecture? How microservice versus a monolith will behave. I don’t have answers for that. At my company, we are researching and trying to run experiments and get to some other information around, well, how much microservice architecture actually impacts on your carbon emissions?

And then the big trade-off is the last, I don’t know, 20 years on software development, we optimize to be ready for deployment, to minimize uncertainty on our releases. And be fast in delivering our values. That’s what continuous delivery is all about. But then you have to ask your question, how much of those practices, or what practices are you using that will turn out to be less carbon efficient? I can think about an example.

There are several clients that I work with that have hundreds of CI servers, and they will run hundreds of pipelines because developers will be pushing code throughout the day. And the pipeline will run and run the test, and run sometimes the performance test and everything. And the builder will stop right in the gate of being deployed and never be deployed. There’s a trade-off to be made here of readiness and carbon emissions.

Should you run the pipeline, every push of code? Does that make sense for your project, for your company? How can you balance all this readiness and quality that we develop throughout the years with the reality that our resources are not endless? They are not infinite. I think one of the impressions that we got from cloud was yeah, we can do everything. We can run with any amount of CPU, any amount of space.

I couldn’t tell you how many data scientists were happy of going to cloud, because now they can run jobs of machine learning that are huge. But now, we have to go back to the question, is this good for the planet? Is this consumption of resources unbounded something that we need to do, or we should?

Carbon considerations for AI/ML [16:42]

Thomas Betts: You touched on one thing that I did want to get to, which is, I’ve heard various reports and you can find different numbers online of how much machine learning and AI models cost just to generate the model. And it’s the idea that once I generate the model, then I can use it. And the using it is fairly efficient. But I think some of the reports are millions of dollars, or the same energy to heat 100 homes for a year to build GTP-3 and other very complex models. They run and they get calculated and then we use them, but you don’t see how much went into their creation.

Do we just take it for granted that those things have been done and someone’s going to run them, and we’ll all just absorb the costs of those being created? And does it trickle down to the people who say, “Oh, I can just run a new model on my workload”? Like you said, the data scientist who just wants to rerun it and say, “Oh, that wasn’t good. I’m going to run it again tomorrow.” Because it doesn’t make a difference. It runs really quickly because the cloud scales up automatically. Uses whatever resources I need, and I don’t have to worry about it.

Marco Valtas: One of the finer principles is, everybody has a part on sustainability, and I think that still holds true independently. What are you doing? We can get into philosophy about technology and see, are you responsible for the technology that you’re producing? And how is the ethics behind that? I don’t want to dig into that, but also we don’t want to cut the value that we’re trying to achieve.

Of course, GTP-3 and other models that are useful and important for another parts might be a case where, well, we’re going to generate this amount of emissions, but then we can leverage that model to be way more efficient in other human endeavors. But can you abstain yourself of the responsibility based on the theoretical value they’re generating? I don’t think so.

I think it’s a call every time, not knowing your emissions. In time, this is going to turn to be something that everybody needs to at least have some idea. As we do our recycling today, 10 years ago, we never worried about doing recycle. Nowadays, we look on the packages. We see, well, even separating your trash, you go like, why this vendor design this package this way that is impossible to recycle? Because this is glued in that, right?

We have that incorporated to our daily lives. And I think in the future, that will be incorporated to design decisions on software development, too.

Performance measurements and estimates [19:12]

Thomas Betts: I wanted to go back a little bit when you talked about measurements, because I think this is one of those key things. You said there are scores available and I hope we can provide some links to that in our show notes. People do performance tests, like you said. Either part of their daily build or on a regular basis, or just I’m watching this code and I know I can make it better. I’m going to instrument it. There are obviously lots of different scales.

This is one of those same things, but it’s still not a direct measurement, that I can’t say my code has saved this carbon. I can change my algorithm here and get the carbon score. I can get the number of milliseconds that it took to run something. Again, is that a good analogy, that if I can get my code to be more efficient, then I can say, well, I’m doing it because it saves some carbon?

Marco Valtas: If it is something that you are targeting at that point, yes. About how accurate is the measurement? That’s hard. The way that we set up our software development tooling, it doesn’t take that in consideration. So you’re going to have rough estimates. And it can say, well, I’m optimizing for carbon, instead of I’m optimizing for memory or something else. That’s definitely something you can do.

When we use the word performance though, it has a broad meaning. Sometimes we can talk about performance like, well, I want my code to run faster, or I want my code to handle more requests per second because I have this amount of users that are arriving at my end point. Does not necessarily means that if we’re making more performance in those dimensions, you are also making more performance on the carbon dimension.

What it boils down to is that carbon intensity will become, at least the way that you can incorporate nowadays, it’s like a cross-functional requirement or a nonfunctional requirement. It’s something that you are aware of. You might not always optimize for it because sometimes it doesn’t make business sense, or your application will not run if you don’t use certain technologies, but it’s something that you are aware of.

And you are aware during the lifetime of your software, if you’re doing certain changes, how that carbon intensity is varying. There’s a good argument like, oh, well, my carbon intensity is rising, but the amount of users I’m handling is increasing too, because I’m a business and I’m increasing my market. It’s not a fixed point. It’s something that you keep an eye on it and you try to do the best that you can for that dimension.

Corporate carbon-neutral goals [21:38]

Thomas Betts: And then this is getting a little bit away from the software, but I know there are a lot of companies that have carbon-neutral emissions. And usually that just focuses on our buildings get electricity from green sources, which got a lot easier when a lot of places closed their offices and sent everybody home, and they stop measuring individuals in their houses. Because I don’t think I have full green energy at my house for my internet.

But I don’t think a lot of companies when they talk about their carbon-neutral goals, as a company are looking at their software usage and their cloud data center usage as part of that equation. Or are they? In the same way that companies will buy carbon offsets to say, “Well, we can’t reduce our emissions, so we’re going to do something else, like plant trees to cancel out our usage,” is there something like that you can do for software?

Marco Valtas: Offsetting something that you can do as a company, not as software, you can definitely buy offsets to offset the software emission, but that’s more a corporation decision. In terms of software development itself, we just want to look on how my software is performing in terms of intensity. In general, we use the philosophy that reducing is better than offsetting. Offsetting is the last resort that you have, before when you get into a wall where are you trying to reduce.

Two philosophies of green software engineering [22:53]

Thomas Betts: The last part of the Green Software Principles, there’s eight principles and then there’s two philosophies. The first is, everyone has a part to play in the climate solution, and then sustainability is enough all by itself to justify our work.

Can you talk to both of those points? I think you mentioned the first one already, that everyone has a part to play. Let’s go to the second one then. Why is sustainability enough? And how do we make that be motivation for people working on software?

Marco Valtas: If you are following up on climate in general, it might resonate with you that we are in an urgent situation. The climate change is something that needs action and needs action now. The amount of changes that we can make to reduce emissions so that we reduce how much hotter will get Earth and how that impacts on environment in general, it’s enough to justify the work. That’s what is behind it. It’s putting the idea that this is enough of a worthy goal to do what we need to do.

Of course, cases can get very complex, especially on large corporations in, what is your part? But sustainability is just enough because it’s urgent in a sense. This is where sometimes organizations might have a conflict. Because there’s the sustainable business or making a profit or making whatever your business grow, and there’s sometimes sustainability, which is not exactly in the same alignment. Sometimes it will cost you more to emit less.

That’s an organization, that’s a corporation decision. That’s environmental governance of the corporation itself. What is behind this principle is basically that idea where focusing on sustainability is enough of a goal. We don’t think that other goals need to be conveyed in order to justify this work.

Thomas Betts: Well, it’s definitely given me a lot to think about, this whole conversation. I’m going to go and reread the principles. We’ll post a link in the show notes. I’m sure our listeners will have questions. If you want to join the discussion, I invite you to go to the episode page on infoq.com and leave a comment. Marco, where can people go if they want to know more about you?

Marco Valtas: You can Google my name, Marco Valtas. It’s easy. If you want to talk with me, I’m not in social networks. I gave up on those several years ago. You can hit me on marco.valtas@thoughtworks.com if you really want to ask a question directly to me.

Thomas Betts: Marco, thank you again for joining me today.

Marco Valtas: Thank you, Thomas.

Thomas Betts: Listeners, thank you for listening and subscribing to the show. I hope you’ll join us again soon for another episode of The InfoQ Podcast.

Mentioned