Article originally posted on InfoQ. Visit InfoQ

Transcript

Martins: I’m a person that talks a lot about the future of JavaScript and emerging technologies. This time, I got a challenge to talk about UI architecture.

My name is Willian. I work at Netflix as a TV UI engineer. I’ve been doing this for 20 years. Doing this for that long takes another exposure to the ecosystem to the point that we can start to spot some patterns on the evolution of UI systems in some of the solutions that they try to bring to the ecosystem at this time. Although I have 20 years of experience, it looks more like 5 times 4, because every 3 to 5 years, the UI changes a lot. The ecosystem changes a lot. I want to set some expectations. We usually come on this conference to get inspired about the future and to learn with each other’s experience. This session is about looking back into the history that I experienced with my lens, to bring some observation of the evolution of UI systems over the years and share some of my observations and got you’s I had on this journey. I will try my best to present these observations and opinions as agnostic as I can be, because each ecosystem is unique and has its own quirks.

Outline

First, I’m going to walk you through the history of UI advancements I experienced over the years, and the problems that each of those cycles try to solve. Then I will show my perspective on scalability, on UI architecture. This section will set the tone for the third and last session of my presentation, which is my take on how you should organize your UI systems to be scalable and easy to change.

Do you remember this photo, and the thoughts that it brought to the UI tech social media? If you have more than 5 years of experience, I think you’re going to relate to the perception that UI ecosystems evolve every 3 to 5 years. Experiencing this a lot of times, sometimes you start to think that perhaps we’re going to go back to PHP days, as we saw here. If you have enough experience to your ecosystem, regardless if you’re coming from mobile or web UI, I suspect you face some scenario like this. Someone’s proposing you need to do something for improving your UI, and people start to throw hype names into the solution for the problem. When you’re like the third guy, you’re going to throw away from the window because you’re trying to change your architecture, not going for fancy stuff.

The History – Multi-Page Application (MPA)

Let’s get to the history. I started working tech in 2004. My first job as a UI developer was in 2007. From that time to now, I roughly spot like five cycles of UI architecture evolution. The first one was the MPA, multi-page application, without JavaScript. That was the beginning of web development that day. To better explain the difference of those architectures, I will use this user flow as an example. This is our example for the explanation we’re going to do right now. Let’s say that this is a generic signup flow for an app that shows media based on user preference. Any resemblance with Netflix signup flow is just a mere coincidence here. The first screen is just like a home screen. You can see some value props and style folder. The second screen like, condition put screen. The third one will be the plane selection screen. The fourth one will be the payment screen.

Then after you’ve done everything, you go to the member home screen, where you can filter media and load them for usage. What is common on the UI architecture at that time? As simple as it can be. You fully render the page. Any user action goes for a round trip to the server and receive via Get and Post requests. Then the server validates and then receives back the full-page re-render. So far, so good. There is a problem with this approach. Let’s suppose that in the second screen, the user inputs an invalid email, so the request needs to go all the way to the server that will validate the entry, detect the misentry, and then return back with the validation error. Only after the user fixes the issue, then they can proceed to the next screen. This pattern exists to this day, and static site generator leverage this for build fast and non-interactive websites. Those characters, as I said before, was an issue for general user experience. We don’t want to send a full request-response to get an error, while those validations, of course, can be treated on the client side instead.

MPA + Vanilla JS

That leads us to the second evolution on that architecture that I can say like 1.2, which is leveraging vanilla JavaScript for this validation. The community quickly understand that JavaScript is a powerful tool to avoid those round trips and also validate on the client side. More than that, JavaScript is an essential tool to improve the user experience. With that in mind, the developers at that time decided to create a new layer of abstraction on the web, and at that time a JavaScript layer emerged, like a behavioral layer for the web. Now we don’t need to do a full round trip to fix a small issue like that one on user input. Even better, we could disable the send button and submit events only when the user fixed those issues. Again, that was not enough. At that time, something you got to traction on the ecosystem. I think you relate to those two. In 2004 Google created the first clear app, in 2005, the second one.

That was the beginning of web apps. They leveraged on something that was created in 1999 by Microsoft, and was under use as a leverage for building web apps. I’m talking about AJAX, and AJAX stands for Asynchronous JavaScript and XML. I know we don’t use any XML nowadays, but that was the name that came at the time. I want to bring your attention to the fact that Google, and as well the Kayak team, because they created e-commerce at more or less the same time. They look at this function that wasn’t standard back in the days, and they create the next advancements for the foundation of async request that we have today. Now the web can house a dynamic web app. At that time, the browser APIs were not standard. The browser war was real, and the developers, most of the time, had to decide whether the website will be compatible with Internet Explorer, Netscape, Mozilla Firefox, or Opera. Things like CSS rendering or event listeners, as well as element retrieval, were hard to maintain cross-browser.

Once you overcome that issue, you have to reason about how to organize your code. At that time, we didn’t have any bundlers, transpilers to help with this task. Usually, the code organization was the following here. We leverage the full-page rendering to load one JavaScript per page that help somewhat visually the behavior of each page, but at the other side, issues were still there. I would like to mention three libraries that further enhanced the UI ecosystem and made the development UI more advanced at that time. In 2006, the community brought us three different libraries.

The main goal they had at that time is standardizing UI manipulation with a series of API for DOM manipulation, event handling, styling, and even animation. Yahoo UI, MooTools, and mini jQuery were releasing segments between each other, and greatly simplified how we used to write web applications, because we don’t need to care about the browser compatibilities. To give an example, this is how we would do some basic operation using MooTools. This is the same using jQuery. I didn’t find the documentation for Yahoo UI 1.0, but they tried to solve the problem in a similar manner, although their approach to the API side slightly varies from one to the other.

The result of introduction of these frameworks made the development of the UI, not only concerned about fighting against the browser API ecosystems, moreover, with the introduction of these user created plugins, we started to experience the early signs of component modules. That was a standard to create some custom components behavior like date picker, drop-down. However, a downside of these advancements was increasing complexity of the UI logic, lead with some of commonly referred as spaghetti code. I think you saw some of this post back in time.

Rich Web App Era and the Death of Flash

Then in 2007 with the release of iPhone, the web ecosystem started to shift away from previously Rich application running over Flash to more web driven Rich application. I call that the start of the 30-year of the UI architecture evolution. The web app started to evolve to be more client driven, rather than server driven. In this era, we started to experience the fast growth of UI techniques to improve UI apps. In this era, we started to migrate from fully multi-page applications to a hybrid of single-page application and multi-page application. I will talk on the contribution that this brought to the architecture ecosystem.

The first technique I want to uncover here is the custom route technique. We used to leverage the fact that browser do not navigate away with a hash fragment to create a client route using hashbang, as a perceived prefix to the sudo route. Our web apps, essentially, the first request will always go to the server using that technique, and then, once the user has loaded the website in their machine, all the other requests will not hit the server until it’s really necessary. For bots like Google crawler, they would interpret the hashbang and will replace the correct string to reach the server, to retrieve the page style. This type of approach increased the usability of web apps, since it had faster page load times, and avoided a round trip to the server as much as possible.

This brought an issue that was often overlooked back then. I’m talking about memory leaks. In this case, the most common case of memory leak was the fact that we added an element to the screen. We attach events, and if you didn’t forget to remove the event before the UI in the next screen, the DOM element will be stuck in there, and you’re going to never be cleared from memory. To solve that problem, DevOps leveraged a technique called event delegation. Event delegation relies on the event propagation, how the event propagation behaves on DOM. The event propagation in DOM works in the following manner.

First, you have event capturing phase, which is stop at the target element. After that, it bubbles up to the document carrying all the event target information. The event delegation technique consists of attaching the event to the document whenever your event leads to the document, and switching to the desired effect based on the targets that it’s bubbling up. Another approach derived from event delegation was the increase in usage of the mediator pattern to control the UI application logic. Let’s take an example on our member page where we have our media that needs to be filtered back at Jaeger, for example. The code example will be more or less like this. I have the simple implementation of a mediator object. Of course, I’m committing an anachronism here because this code wouldn’t run in 2007, but I think it’s better for us to learn in modern JavaScript.

Essentially, you would use this in your document, in every component that needs to respond to that event, and should listen to the mediator instance. The components that would emit the event would persist it to the mediator, like that. The component that responds to an event, would do like that. I think it’s part of the pattern here, in these two last techniques. It easily created like a source of events in the source of event that you can instantly react to, but we still have the problem of the spaghetti code. In 2009, Require.js was launched as an attempt to decrease the spaghetti code and create modularity on their web application. It consists on creating model definitions with define functions and import then with the require function.

This would be how you encapsulate your models. The search box will look like this, and the items will be like this. This is how your application would start at that page. You set the dependencies and then require will inject that to here. With these advancements, the Rich web apps improve the architecture in code organization. The lack of opinionated framework made each application carry too much institutional knowledge, making it hard to transfer the knowledge from one project to another. The single source of event came from the delegation mediator pattern, simplified the reasoning about the data flow and improved the resilience of the single-page application by decreasing the instance of memory leaks by design.

The MVW Era

In 2010, the ecosystem received a new wave of frameworks that was very opinionated. These frameworks followed what I can call like MV-whatever. That’s why the W is there. These were some examples of MVW framework available at that time. Ember, Knockout was forming the MVVM, Model-View-ViewModel pattern. While Backbone followed the MVC pattern. To show an example of the MVC implementation, we use Backbone.js. If you can check what will be the difference between Backbone, Ember, and Knockout, you can go to the MVC website and check. There are a bunch of same app implementations in different frameworks, and you can check the difference there. In this case, I would not need the mediator, since the Backbone provides the event Pub/Sub system, instead. You need to have just a model to carry all the information. The search box doesn’t change much from what we saw before, just glue code from Backbone. For the items, you need to create abstraction from one single item and another one for the collection, and that’s how it’s going to work.

The Reactive UI

The last piece of history I want to show you was the latest big advancements we saw on the web wide ecosystem. In 2010, roughly at the same time as the previous framework I mentioned before, we had some different approach on how the UI should be designed. Frameworks like React, Angular, and Vue, introduced a concept of reactive UI, where the user actions on UI cascades back to the UI. First, Angular introduced the two-way data binding, where the effects propagate in both ways by design. Although that brought some improvements to the UI design, the lack of well-defined constraints that was later added on Angular 2, made the overall design of a large application hard to follow. Even damaging their performance, so in some case, due to the way that digest system worked. Let me talk about React.js. Who here has 11 years of experience or less? For those that have 11 years or less of experience, the odds that you’re born and raised professionally under React are high.

I can assume that the impact of writing HTML in a JS file was minimal, and for others here was logical and natural. Before React, there was not such a case. JSX now is our option in most of the modern UI. React made popular the fact that the component is a pure function, which means that you should remove all the side effects from the component code. This brings some investments for the development cycle, since there was less code logic to be tested on that layer. Since React sells itself as just a UI library rather than a fully opinionated framework, there is a rich ecosystem of supporting libraries to enhance or complete React with things like, a conduit for data flow, side effect management, event data fetchers, even animation for your React application. Right along with those, a bunch of supporting libraries. We saw as well, some improvements on the library itself, over this last decade.

We came from class components to mixins. Who remembers those? High order components, function components, hooks, now server components in the compiler. Let’s suppose you have a system that was built around about 2014, we are saying that this system is more or less 10 years by now. Imagine trying to keep up with those major changes on the React ecosystem over the past decade. The biggest toll that we pay, initially, from the migration of the MV-whatever mindset to the React mindset, is that the previous advancements we had before were incremental, whereas the React approach was a total shift from the previous paradigm. Even currently, we have a new wave of promising frameworks maturing into the ecosystem, like Solid.js, and a shift to rely more on signals as a foundational of the next advancement of the Reactive ecosystem.

Scalability and Resilience

How can we prepare our current system to be more scalable and resilient to those changes? They realized that you could potentially write a good, scalable, resilient UI application using even Backbone or Angular. We have to be conscious that evolving the application building blocks are essential for being relevant in the market. The talent pool is bigger on more popular technologies, and we can leverage the open-source advancements for popular frameworks as well. The important part here is how to design, organize your application, to be scalable, resilient, while having good heuristics on when to introduce those new technologies to our stack.

Before I jump into the specific content matter, I would like to present my interpretation of scalable UI systems, and what I mean by that. If you check the definition of scalability in the dictionary, you are going to see a definition similar to this. The capability to be changed in size and scale, and the ability of computing process to be used or produced in a range of capabilities. One thing I want to distinguish here is that the semantics of scalability on UI systems is not a one-to-one map to discard it on the backend system, since it has more than one semantics on the UI system. I will share my desired scalable UI system.

The first aspect of scalable I want to cover here is the scalability of your team. That doesn’t mean to scale your team by hiring more people. The technology and design of your UI system should enable your engineering team to do more with less friction, that will give the scale of your team. Another aspect of the team scale is that if your UI system operates with well-defined guardrails, most likely you can have a full stack engineer or less experienced engineers working with your system with little to no guidance. One example we have at Netflix is that we have server-driven UI, that is really good at doing one thing, one type of flow, and when we have a new experience that feeds that technology, a TV engineer like me can do the work for the flow for TV, mobile, and web, because the code base is just one.

While, if we do this in a more traditional way, we will require more engineers. I’m not even mentioning the code repetition that this will avoid. A scalable design enables the team to change features fast. Moreover, enables to isolate models that don’t perform well, easily. It is an easy way to add, and more important, remove features and migrate domains as needed. Last but not least, the design of an application is designed in such a way with a certain mindset, easy to delete, that is easy to achieve resilience in the application because of the modularity or the preservability that this mindset will enable.

Tips and Tricks

What are the tips for a scalable UI design? Everything I’m going to say is based on my experience so far. The first aspect of a scalable UI is, leverage design system as the foundation of your UI architecture. We often forget that the easiest way to revamp an application, is doing bottom-up. The design system is a powerful asset that creates not only standards for UI, look and feel, but creates the ubiquitous language among the UI engineers, design, and even product stakeholders. We as a UI engineer tend to jump straight to the backend implementation and integration when we plan our system, but you should not forget that design is the other end of this integration. Another good aspect of the design system is that it creates a structure to the design output such a way that you’re going to receive less surprise, and deviation for the status quo should be intentional, not accidental.

This will be the only book I will recommend, “Atomic Design.” This book by Brad Frost, describes how the UI is composable from a small set of complex components that are divided by small and simple components. The idea here is to start creating a pattern library from the smallest item that’s not even available in this image here, which are the tokens. I can categorize tokens as values that cannot stand alone, things like size, color, typography. This will help start the foundational work and kickstart implementation of your pattern and library. Then you can continue finding more unitary components with minimal to no UI logic, like buttons, cards, image holders, and create components out of that. Then once you have that done, you can go for more complex components and so on.

If you decide to implement design system, I’m going to give you some tips. Start small. You’re going to have a lot of questions, corner case scenarios that need to cover the beginning, a lot of constraints and buy-ins you need to do. Start from the smaller components to bigger components. Decide the type of template and layouts you want to apply to a project. Start from categorizing what you have already on your project. Theming your UI might be important as well, since theming can make it faster to reskin, redesign, or even create a white label version of your application, if it’s needed. Do not underestimate the power of having a CSS magician in your team.

We often look to good JavaScript developers when we are hiring for UI engineers. A good CSS magician can pay the tax for dozens of JavaScript and full stack engineers to fast implement those UI with ease. Talking about a CSS magician and complexity, if performance is a must have on your application, please avoid at all cost using styling solutions that creates a runtime like CSS-in-JS, a solution that applies styling at runtime. Once your design system is mature, you can integrate with your design tool like Figma, and make code generator tools out of it. If you feel adventurous, you can even create an ML model to generate design out of a sketch. I saw that before. System will cover like this part of your boundary between UI engineers and design product team, that check the morphology of the UI system.

When you start designing how your UI system components should relate to each other in the system as a whole, you need to detect the follow type of category of components. The first component category of your application is the foundation. Is your app wrapped in a Docker container? What’s the server structure? What’s the build pipeline? What are the types of communication that this app will have with the downstream services? Are you going to have a BFF? All these type of surround concerns of your application are considered foundational. Usually, this category can be managed by SRE team, core team, or foundational team. The core encompasses all the features that has to be present on the whole application, things like Bundler, TypeScript config, test infrastructure, route structure, what library you’re going to use to deal with side effects, data digest system, pattern library integration.

All those support libraries that you need to create your app is part of the core. Depending on the size and the complexity of your system, we might have another horizontal domain, which is the platform, and all those two domains had some gray zones. I consider this categorization test more an art than a science. These two categories together should be sufficient to have the application running without any functionality. It’s going to be ready to be built upon it, like when you use the Create React App CLI command. Everything that’s built on top of your framework is what I call feature. The feature is built in such a way that it avoids, as much as possible, disrupting other features. We should not assume that the feature is only a vertical of your application. You can have a feature that is horizontal as well.

I said too much theory on this, but it’s a core selection. Why am I saying all of these? Not that you’re aware of those categories, we can assume some stuff. The foundation is what you can build any application on top of it. It’s not necessarily UI, you want to call the backend as well. The core is what you need to build your application on top of, and you have a clear separation between those two. The foundation will help you to have a homogeneous implementation, and observability as well. It will help you to swap foundation if it’s necessary. That’s why the ideal foundation, or ideal facade between the foundation and core is just a web platform. When it comes to the features, we should be sure that they are easily detachable. I can give you some heuristics. The goal here is to improve for code deletion. Whenever we are in doubt, what’s the best approach for your system, you should aim first for the simplest one, but if they’re comparable in complexity, go for the easiest to delete.

The second aspect, try to bubble up as many UI logic that’s not presentation logic, away from the UI views. The [inaudible 00:37:21] the better. If you have a canvas that has a self-containing logic, like a sub app, it’s fine to have it there. In this case where the domain is unique for just that component, only processing the result will be part of your overall UI state. It’s better to have the logic close to the UI. The third aspect is, make sure all the component related code are colocated. It’s not a good experience when a developer needs to add a feature to an application, he needs to navigate to three, four, or five different folders and parts of your application to register the feature to be available.

Removing a feature for your application should be as simple as deleting a folder and reboot the whole application. The last aspect I want to add here is the registration process. The ideal scenario would be, you create a folder. Your core application, you give a registration function or a CLI tool that can be interpreted in build time, and it will be responsible for wiring up your feature to the whole app. If that’s too complex, because of maturity of your app, try to minimize as much as possible the place that a developer needs to touch in order to register the feature to the application.

What should we do with that logic? I told you to bubble up from the UI in the previous section, same to the server. Let me explain this to you. Remember our initial app example. We can describe each page as a node in this application. When the user hits a route, the request goes to the server that will do the initial validation, and it’s going to see what’s going to be the information that’s going to be in that page. It will send that to that page. I will call pages, node, from now on. All the data fetching and manipulation that needs to be sent to the UIs is called by this BFF layer. If there is any fields that needs to be presented UI, needs to be defined on that layer as well. If the field has any validation logic, because it’s an input that the user needs to be entering on your app, you need to define that in the server as well.

All the actions that need to be mapped back, should be sequential steps on the flow, should be mapped on the server as well. Once your preload is done, the server will send the response back to the UI that will look at these fields, actions, and validations, and the UI will know how to present those data. Once the user has any action to the client, the client will say, “I’m in this node. I have this action. I have these fields. What’s next?” Then the server will say back to you, the next is this page.

Summary

In 2018, Eric posted more detail on the payload that we use at Netflix, which I just described. If you don’t want to create a server from scratch for it, you can use a library called XState, that can handle the state machine for you. You can make this layer just above the core of your application as well. If you want to have a client owned application or offline application, you don’t need to do this on the server, you can do this in the client. The difference here is that instead of the HTTP request, you’re going to just communicate via internal API. The beauty of this type of approach is that we define a protocol that defines the communication between the UI and the server.

The benefit for having this kind of protocol is the speed that this brings to the development. Usually, when the developer team receive a design spec, we already know all the components in that page, because we have the design system for that. If you have a more advanced design system, you can have a code generator tool for the UI of that screen. If the design is not in place, but you have the flow, the flow is defined, we can sit down and create a state machine with the backend engineers. We can work in parallel using that protocol as an integration point between both. The discussion is more about like, what’s the data validation? What are the fields that we have? What are the actions we have in this page? The list goes on. If the flow is the same for the other UI platform, the implementation on BFF can include all these fields for those clients, create just one flow for all of them. Then we can apply this implementation just once on the BFF. Although this is a desired outcome, I think you should expect this as a star alignment in a corner cases scenario.

Conclusion

Throughout these years, the UI ecosystem evolved, and much of the design decisions that we made in the past, now is part of the web platform. As the UI systems became more complex, the community came with ways to organize the web. To scale the website without increasing the developer experience, and then development complexity. The community steered the development patterns towards the reactive approach. This is what we have right now. This is the approach we have used for the last decade. I think, they’re going to have this for a while. We need to come up with an architecture that can help us to avoid refactoring as much as possible. That’s why I propose this feature-based development. Of course, in the near future, if the ecosystem changes, but we have a great separation of concern, and we need to do the refactor, or rewrite, the nature of loose coupled components can help you do this with less pain.

Questions and Answers

Participant 1: Give me thoughts on Web3 and how it will affect the development landscape in the future? Web3 in the sense of the decentralized architecture of the internet that they’re proposing, so like blockchain technologies.

Martins: I think there are some use cases for that. To me, for instance, smart contracts, banks can leverage these, and things like that. That, I think, is a subset of the overall web ecosystem that can benefit from it. Maybe peer-to-peer communication can leverage the nature of Web3. At least from my opinion, it’s not something that will be brought, unless they come up with some out of the box idea for that.

Participant 2: Any callouts or thoughts as a team, maybe shifting from their design system being on the client to being server driven.

Martins: You mean the whole design system is being driven by the server. Essentially, you’re saying the client will say, I need those components and the server provides that.

Participant 2: Yes. Currently, we’re testing out doing a POC on it. Any callouts that we should be aware of while testing that.

Martins: I think we should have in both places. It depends on the nature of the application. For example, if you have an application that you boot just once, and the user is going to stay stuck on that page for a while, and the application is a small thing, that might not be the use case for this. For example, if you have the case flow that you need to go back to the server, back and forth, and there’s no other way to do that, I think that’s useful. We have similar patterns at Netflix as well.

Participant 3: If you have a design system that’s been fragmented, there are four versions of the design system, and you have a thousand screens using different versions, how would you approach a rollout of everyone going to the same version, going to a more scalable design system?

Martins: I experienced the introduction of design system through different companies, and all of those, they decide to use design system as a byproduct from the redesign. What we had back then is you have everything that’s on your client, it’s there, and then you start to leverage those things. I think that’s a great question for a CSS magician to solve, because, for example, let’s suppose that it’s just a theme difference. I think we often make things more complex than they should be, because at the end of the day, the design is just the look and feel.

The look and feel can be greatly expressed by CSS. I’ve been working in TV for 4 or 5 years, and I came back to do sidebars on web, and everything changed. There’s a lot of CSS stuff that I’ve never seen before, and it’s evolving. My approach to that from the outside, will be like, let’s see how much the CSS can cover. Because you can version your CSS. You can decide, for this use case, you should receive this one, and you can acquire via a CSS class. I think the same thing that we do for theming. It depends on your CSS technology. I know that I have bash on CSS-in-JS. For instance, if I have the CSS-in-JS, you can do this on JavaScript. There is a lot of avenues.

See more presentations with transcripts