Article originally posted on InfoQ. Visit InfoQ

Transcript

Venugopal: These are some typical growth trajectories of successful companies. The hockey stick being the most sought after and popular one. The software systems that worked well during the initial phases of a company, phase A, will not be sufficient as you prepare to scale your business in phase B. The exponential growth phase, phase C, requires drastically different software capabilities than A or B. Successful companies, like the ones that live to see exponential growth, outgrow their software systems one way or the other. I’m making the case that legacy systems are a byproduct of success. Despite being a byproduct of success, legacy systems have a bad rep. Just the word legacy evokes strong emotions, and for good reason. We associate legacy with technical debt, painful migrations, high maintenance costs, and poor developer experience.

For the long-term success of your company, build the muscle to renovate legacy systems. While legacy systems are a byproduct of success in the past, success in the future depends on your ability to not let legacy systems get in the way of the growth for your company. That brings me to my first takeaway, that legacy systems are inevitable. Don’t let them weigh you down. Make them work for you instead. In fact, this takeaway is the inspiration for my talk.

Background

Welcome to Renovate to Innovate: The Fundamentals of Transforming Legacy Architecture. I’m Rashmi Venugopal, a staff engineer at Netflix. I spent the last decade building and operating reliable distributed systems at scale. During that time, I’ve been fortunate to work with, learn from, and grow amongst some of the brightest minds at Microsoft, CMU, Uber, and most recently at Netflix. First, we’ll unpack what legacy systems are and why they exist. The focus of this talk is technical renovation. We’ll cover what it means, when a technical renovation is applicable, and discuss strategies for effective renovation.

Legacy Systems – What?

What is the first thing that comes to mind when you think of the word legacy? Old, unsupported, unchangeable, and no tech. Let’s see how you all match up with OpenAI’s word cloud for the term legacy. I see a few in here, not bad. As you can tell, the term legacy is quite overloaded. Let’s spend a couple minutes to get on the same page about what legacy means in the context of this talk. I define a system as legacy if it is incapable of keeping up with business requirements. After all, your software systems exist to serve your business goals. Let’s make this more concrete and talk through some symptoms of legacy systems. There are numerous dimensions of complexity in software engineering. Legacy systems usually have substantial complexity in one or more of these dimensions. Working across a large number of teams and people slows engineers down. This is because of coordination tasks. That is organizational complexity.

Operational complexity is when there’s insufficient automation, testing, monitoring, observability leading to high operational costs. Cognitive complexity is when institutional knowledge builds up. Documentation becomes outdated or people turn over. Why is complexity such a bad thing after all? As complexity goes up, innovation velocity goes down. There’s a direct correlation between complexity and innovation velocity. Product and project managers expect productivity to scale linearly with complexity. Engineers, we know better. Our past experience has primed us to be more pragmatic. It is a sign of a legacy system when the reality of how long it takes far exceeds expectations in a bad way.

Another sign of legacy system is degraded quality of experience. Quality of experience measures the overall satisfaction of end users when they interact with a system. I’m sure we’ve all experienced the very real frustration of waiting many seconds for a page to load. Amazon has an infamous study where they quantify the impact of latency on their business. They find that every 100-millisecond increase in latency impacts their sales by 1%. A dip in the quality of experience despite your best efforts to tune them is a symptom of a legacy system. To recap, I consider a system to be legacy if it is incapable of keeping up with business requirements.

Legacy Systems – Why?

Now that we’ve covered what legacy systems are, let’s talk about why software systems become legacy in the first place. The most obvious reason is the rapid pace at which technology advances today. Who here has used two or more of these devices? Systems that were once considered cutting edge struggle to keep up with modern industry standards just a few years down the line. Technology choices become outdated. In addition to this obvious reason, there are two schools of thought that explain software degradation. The first school of thought is the bit rot theory. It states that software gradually degrades over time due to incremental changes to itself or its surroundings. An unused code path is an example of bit rot, so is code duplication. A lack of documentation or a loss of knowledge is yet another example. In theory, bit rot can be kept in check with good software engineering practices.

In reality, bit rot accumulates over time. The second school of thought is the Law of Architectural Entropy. It states that software systems lose their integrity when features are added without much consideration of the original architecture. The primary driving factor for architectural entropy is the real and unintentional tradeoffs that engineers have to make in order to deliver results faster or meet deadlines. Imagine the growth of a successful e-commerce company. In the early stages, they’re focused on establishing a thriving business. Evolving their architecture to be perfect is just not a priority. In fact, changes to the architecture is driven by business needs.

In this example, every new feature is added to the existing monolith, steadily increasing the architectural entropy. In the real world, software systems are affected by all of these phenomena. This explains why outdated and legacy systems are more commonplace than we’d like them to be. Now that we’ve agreed that legacy systems are commonplace, let’s ask ourselves, do we always proactively renovate legacy systems? I wish we did. The inevitability of software degradation on one hand, combined with the lack of renovation of legacy systems on the other, leaves us with systems that are difficult to maintain, understand, and extend. These are the systems that are very likely to get in the way of growth and success for your organization’s future.

Technical Renovation – What?

That brings us to technical renovation. What does technical renovation actually mean? I define technical renovation as the act of upgrading or replacing outdated systems and technology to improve the software’s state of affairs. Every time I bring up technical renovation, I get asked, how does refactoring fit in? Why is technical renovation different from refactoring? I’d like to address the elephant in the room with a closet analogy. Refactoring is like organizing your closet. Organizing involves moving things around. You make it easy to access all pieces of your clothing. You might even get rid of some stuff to make room for more things.

This whole process has a side effect of reminding you what you already have and it potentially influences your future wardrobe investments. Renovation is when you break down the walls of your closet to replace a regular one with a walk-in one. Renovation goes beyond just moving things around. Renovation is when you make a drastic change to shake things up and the end result gives you capabilities that you did not have before. Renovation is usually a much larger undertaking and therefore occurs less frequently than refactoring. While this talk is about technical renovation, I just wanted to pause to say that refactoring is valuable. It is a valid strategy to maintain a healthy codebase and there’s many benefits to maintaining a healthy codebase.

Technical Renovation – When?

Now that we’ve discussed what technical renovation is and how it’s different from refactoring, let’s review some scenarios for which technical renovation is applicable. In other words, if technical renovation were a hammer, what do the nails look like? As your business needs evolve, attempting to reuse existing systems to solve for something drastically different doesn’t typically end very well. Here’s an example of a business-driven renovation. Netflix evolved from a DVD distribution company to a streaming service. The capabilities required to deliver DVDs is drastically different from the capabilities required to stream video on-demand. The systems that served Netflix well in the DVD era isn’t going to be sufficient to run a successful streaming service.

The point being, drastic changes in business needs eventually call for a renovation. Technical renovation is also a valid strategy for an ecosystem-driven change. When the ecosystem changes, the underlying assumptions built into the existing systems are challenged. If you’re going from hosting REST APIs to now serving data behind a GraphQL gateway, a renovation is in order. Technical debt occurs when you borrow from the future to make a tradeoff for the present. Even with the most well-intentioned engineers, there are scenarios when technical debt accumulates, and accumulates to a point of no return. Unexpected longevity is one such example.

Sometimes software systems turn out to be more successful than anyone imagined they would be. While that specifically is a good problem to have, the unexpected longevity accumulates significant technical debt. That makes technical renovation a viable option to improve the state of affairs. These are some nails for the hammer that is technical renovation. Time for our next takeaway. Use the right tool for the right problem. Leverage renovation for the scenarios similar to the ones we just discussed. Refactor your code as often as it makes sense to do so.

Technical Renovation – Strategies

Let’s talk about how to approach a technical renovation next. I’d like to share four strategies to consider as you embark on your renovation journey. I found these strategies useful to do renovation right. My first strategy is evolutionary architecture. Historically, architecture is viewed as something that has to be developed ahead of time, even before a single line of code gets written. It’s also perceived as something that’s set in stone, never to change. In the world of modern technology, this pre-planned approach to architecture doesn’t keep up with the evolving needs of your business. Here’s an alternate approach to consider for your renovation initiative. Evolutionary architecture emphasizes incremental changes because complex systems cannot be fully designed upfront. It advocates for evolvability. When your priority changes, your tools should change with it. How do we make evolvable architecture a reality?

Step one, identify a set of fitness functions that represent the desired qualities of your end state, such as performance, scalability, security. Once you’ve picked the quality that matters the most for your business, use that to inform engineering decisions. In that process, ask yourself some hard questions. Does performance really matter? If yes, by how much? Will users actually notice the difference between a 1-second page load and an 800-millisecond page load? The point is, don’t optimize prematurely. As a rule of thumb, if you don’t regret any of your early decisions, chances are you overengineered. Excessive abstractions or overly generic solutions and premature scalability are some common pitfalls to watch out for.

Step two, invest in continuous delivery. Create an infrastructure that you can use to execute fast. Automate the steps between developing, testing, and releasing a feature. Step three, make small and incremental changes. Making changes in the Big Bang fashion is difficult to get right. Incremental changes makes it easy to course correct as you go. The crux of evolutionary architecture is to make small changes, release them often, and use feedback loops to see how well you’re doing against your fitness functions. As the business requirements evolve, your fitness functions are going to change. Lean into continuous delivery and incremental changes to keep up with your evolving needs.

Speaking of incremental changes, my next strategy breaks down an incremental approach to renovation: make it work, make it right, make it fast. I’m sure most of us have heard of this quote from Kent Beck, we apply to writing code. I’m making the case that this applies more broadly to software engineering, including your renovation initiatives. The first part, make it work, is all about getting the assurance that your problem can be solved one way or the other. From a coding perspective, make it work is all about giving yourself the permission to write ugly, unreadable code. Even if it means you have to hardcode inputs along the way, that’s fine.

In the case of your renovation, use this time to validate your technology choices, handle the common use cases, and eliminate some bad solutions along the way. If your integration is just barely held together by duct tapes, so be it. Now is not the time for perfection. I consider a proof of concept as a good outcome of the make it work phase. It’s time to make it right once you’ve established the validity of your solution. From a coding perspective, you would prioritize readability, adding tests, or even refactoring. For your renovation initiative, make sure your edge cases are accounted for, your fitness functions are being met, and even test against some real-world users to see how your solution holds up. I view a minimum viable product as a good outcome of this phase. A working solution that’s a natural extension of your proof of concept. That brings us to make it fast.

From a coding perspective, make it fast feels like a performance thing. It gets interpreted very literally. How do I make this piece of code run faster? For your renovation initiative, however, make it fast is so much more than just performance. It’s adding documentation. It’s integrating with continuous delivery. It’s setting up observability and monitoring. All this speeds up your development process and is very much in the realm of make it fast. The output of this phase is production grade software that’s ready for prime time. This structured approach to tackling the different aspects of a technical renovation helps break down a daunting endeavor into trackable and manageable milestones. You’re set up to overcome analysis paralysis because you’ve given yourself the permission to just focus on making it work.

Then you iterate to make it right and ensure that your fitness functions are met. If you care about performance, and performance is one of your fitness functions, now is the time to get it right. Lastly, optimize for speed of execution. This structured approach also gives me the clarity I need to move fast without breaking things. My third takeaway is a combination of the two strategies that we just discussed. As you renovate your systems, build incremental and evolvable software that is capable of aligning with changing business needs.

On to the third strategy. Deprecation driven development focuses on what we gain from deprecating as opposed to what we lose. I’m making the case that removing code is as important as adding code. Systematically removing obsolete technology is a prerequisite for healthy software systems. Weigh the tradeoffs before you renovate a feature. Be honest about the return on investments especially when it doesn’t justify the effort required to migrate them, because not all features are equally important. When you encounter a feature that is not critical to the success or the growth of your organization, consider leaving them in the legacy system.

Better yet, deprecate them because the cost of maintaining is often higher than the cost of building them in the first place. Netflix winding down DVD.com is a good example of a product deprecation that was driven by a similar tradeoff, the tradeoff that’s between the cost to maintain and the benefits to business. As the number of DVD members continued to shrink, it became increasingly difficult to justify the cost of providing the best-in-class experience for DVD users. Once the decision to deprecate was made, this clarified engineering priorities. We didn’t invest in DVD related technology the year leading up to the deprecation. No points for guessing what my fourth takeaway is. Removing features is as important as adding new features. Be ruthless about deprecating features that don’t serve your business, because they either weigh you down with high maintenance cost or make your renovation journey endless and expensive.

My fourth strategy is intentional organization design. As your company grows from phase A to B to C, renovating your organization is just as important as renovating your software. Intentional organization design is all about identifying the optimal collaboration model to drive the best business outcomes. The goal is to make it easy for ideas to flow through the organization. In addition to the flow of ideas, organization design also has an impact on architecture. Conway’s Law explains the synergy between the two. It suggests that the way teams are organized influences the architecture of the systems they create.

For any organization that’s undertaking a technical renovation initiative, it’s a good idea to first take a step back, assess the org structure, identify any changes that might be worth making. It could be to streamline communication or minimize cross-team collaboration. Let’s look at an example of organizing teams to make this more concrete. Design A involves grouping engineers based on their function, like frontend, middle-tier, backend. Design B groups engineers based on a common product deliverable, but as full-stack teams. Each design has its pros and cons. A optimizes for engineers to ramp up quickly and provides space for them to become experts at their craft.

If every new feature requires making changes to all three parts of the stack, they will need somebody outside to coordinate and assign tasks and manage dependencies. If you happen to be optimizing for minimum cross-team collaboration, Design B might be more efficient for you. In summary, you have to choose to strengthen the communication paths that are most important for your organization, because every communication path can’t be the strongest.

The Growth Mindset

While this brings us to the end of the renovation strategies that are important to consider as you work to transform your legacy systems, I’d like to talk about an important piece of the puzzle that’s required to bring all the work for a technical renovation to actually come together, the growth mindset. The strategies I shared may seem ambitious. They’re intentionally aspirational in the spirit of shooting for the stars and landing on the moon. Also, because there are no silver bullets for technical renovation. The ideal approach is highly context dependent. Your strategy and decisions should be debated on a case-by-case basis and accounted for the unique circumstances and goals for your organization.

Recap

Start with the right perspective, the perspective that legacy systems are inevitable in successful companies because they are a byproduct of success. For continued success, don’t let them weigh you down. Invest in building the muscle to renovate legacy systems. Use the right tool for the right problem. Refactoring solves some problems, but not all. Invest in technical renovation when it makes sense to do so. When you do invest in renovating your systems, approach that with a focus on building incremental and evolvable systems that keep up with your business. Removing lines of code is as important as adding lines of code. Regardless of what the growth trajectory of your company looks like, you can rest assured that the path to successfully transforming legacy systems will be bumpy. Embrace the growth mindset, seek feedback, learn from your mistakes, and enjoy your renovation journey.

See more presentations with transcripts