Month: July 2024

Article originally posted on InfoQ. Visit InfoQ

Subscribe on:

Apple Podcasts
Google Podcasts
Soundcloud
Spotify
Overcast
Podcast Feed

.
From this page you also have access to our recorded show notes. They all have clickable links that will take you directly to that part of the audio.

Article originally posted on InfoQ. Visit InfoQ

Platform engineering is about accelerating and empowering developers to deliver more product value faster over time. According to Jessica Andersson, most companies don’t invest in platform engineering until they reach a certain size. At QCon London she presented how their startup adopted platform engineering, what strategy they took, and what they have done to gain platform adoption from developers.

Andersson mentioned that they launched their platform engineering team when the company was two years old and already had software running in production. As soon as you have software in production, you have an implicit platform, regardless if you know it or not, she said. This implicit platform is built in bits and pieces to solve whatever need the teams have at the moment and once it works well enough, whoever built it goes back to building the product, she added.

Starting out with platform engineering they first identified their implicit platform and then started to turn it into an intentional platform:

Basically we took existing things and streamlined, upgraded, and secured them. It’s important here to acknowledge that the implicit platform was built with the best knowledge available at the time and to remember that this was not the main focus for those building it.

It’s important to avoid replacing all tools just because they would not have been your first choice; replace the ones that are insecure or hinder your platform, Andersson said.

The main goal of the platform engineering team is to increase the ability to deliver product value, Andersson said. You can do this through removing bottlenecks and reducing pain points and time sinks:

Some indicators we look for in order to identify where we need to spend more effort are:

• Teams are blocked from performing tasks
• The tasks teams perform take long time
• The tasks teams perform are unreliable and prone to failures

There are always trade offs you need to make; being a small team we definitely can’t take on everything, Andersson said, but we aim to solve the most common and urgent needs.

Treating your platform as a product means building it like you would any other software product. The platform has users, problems that it solves, and a lifespan throughout which you need to take care of both the software and your users, as Andersson explained:

I often see platform teams forgetting about the users when it comes to migrations or switching to new tools, and they deprecate the old thing without providing a seamless transition to the new thing.

Andersson mentioned that you need to keep a focus on what your product is. Working with a product manager in the team is important to maintain that focus:

I try to focus a lot on solving the right problems as well. As a platform engineering team in a cloud native environment there are infinitely interesting problems you can work on; the question is whether you should. So finding out what are the right problems for your organisation and your users is important in order to spend time on those problems.

Platform engineering aims to improve the developer’s experience, Andersson said. Internal developer platforms can help you build a good foundation for good developer experience, helping your teams focus on building excellent products, she concluded.

InfoQ interviewed Jessica Andersson about platform adoption and building trust.

InfoQ: What have you done to gain platform adoption from developers?

Jessica Andersson: Some activities we’ve seen that build trust and thus encourage adoption from our teams are:

• Remove pain points – we show our teams that we care about them and can improve their experience
• Be approachable and helpful – people more likely to reach out to us again, spreading the word to their teammates
• Be proactive – while being approachable we might hear about pain points the teams don’t realise we can solve for them, if we fix it we will gain more trust
• Understand the teams’ perspective – if we understand where they come from, what they know and not, then we can communicate in the same language and better understand their pain points

InfoQ: What role does trust play in supporting a platform?

Andersson: Trust is important as everything builds on it. Adoptions, information, communication: it all comes back to trust.

I believe that as a platform team it’s very important to build a high level of trust with the product teams as it will determine how successful your platform will be.

Article originally posted on InfoQ. Visit InfoQ

Transcript

Hyde: In grade school, I was given an assignment to write a talk. We all came in the next day, and I was waiting for my turn. Someone before me talked about jumping up on a chair and trying to jump off and getting a fear of heights with a chair, which seemed silly. Indeed, it was a humorous talk. Then someone else went next, and it was a very light-hearted thing. I had this very consequential thing that I had prepared. I don’t remember what it was at the time. I froze. The teacher pulled me out into the hallway and said, what do you mean, you don’t have the talk? You haven’t done the assignment? They have no way to know that, in fact, I did. I had written it, and I wasn’t ready to share it. I took the L, and to this day, didn’t pass, didn’t get that assignment in sixth grade or whatever. At any rate, I’m going to try again today. I was pretty miserable in grade school. That might be unusual. Maybe no one else has hated middle school, but I didn’t. My sister at the time talked to me about educational theory. She had read Piaget and various things, and she had this idea that I should homeschool. I went to my father and I said, I want to homeschool. My father said, that’s a terrible idea. This is not my father. This is Jean Piaget, from the Wikipedia article, who is known for educational theory or behavioral development in children. The image prior was from the logo, programming language, which I had learned around the same time and had an Apple IIe, and taught myself Logo and BASIC and that sort of thing. My sister was my first mentor, and I would not be where I am today if it weren’t for her, as well as my brother. All of my family have been incredibly influential. You don’t always get to choose that. You don’t necessarily have an older sister, and you don’t necessarily know to go out and find mentors. It doesn’t necessarily take a ton of effort, but the reward is huge.

Context Setting

This is a talk about risk and failure on the path to staff engineer. I want to share my progression and the things that I’ve tried, many of them which were not successful, and frame how that turned out and how I conceptualize that, how I think about it. If you leave this talk with nothing else, I want it to be this one slide. I want us to develop a framework to recognize risk and failure, and see that in others, and anticipate when there are other folks who are about to make mistakes. We should have a framework to mentor them out of that and bring them back from the edge. Part of that, I think, is by sharing stories such as I’m going to do here, and make it ok.

Do the Work

When I’m not working, I sometimes make robots. They’re often terrible, horrible things that work poorly and are wildly inefficient. It’s something I like to do and it’s a little bit of my non-professional work, although, maybe there’s overlap as well. It keeps me off the streets. This was me a little while back. My sister and my father and I, we made a compromise. I didn’t go to homeschool, whatever that would be, and I didn’t stay into public school. I had already been studying classical music, and so I entered the North Carolina School of the Arts, which is a school with five schools inside of a university, in a sense: dance, design and production, visual arts, music, and drama. Despite what conceptions you may have about music school, or band camp, or something of the sort, it is an immense amount of work. I entered high school for performing arts and got down to the task of doing that work. I had already been taking private lessons, but I took courses in music theory, composition, master classes, performing, and that sort of thing. I also realized that there are a lot of people there who had an immense amount of talent and an incredible drive, and had been studying and training since literally they were infants, to do this work. I took myself to New York City and the culmination of my career in performing arts was that I auditioned at the Juilliard School. There is nothing quite like walking through those doors. It’s a career defining moment to hear the person before you audition at Juilliard. I realized that I didn’t have the chops for that. There were a lot of people that had been preparing for a very long time. It’s a very cutthroat industry, or there just aren’t a lot of paying gigs. Tanya Reilly talks about the foundation of staff engineers is the technical chops, and there is no substitute. You have to put that work in. You have to learn your domain. It’s required. There’s high effort. It takes an immense amount of work. It pays off, because, if nothing else, it forms the foundation for future work, doing staff work, and that sort of thing.

Encouragement Matters (Nominal Value, Low Effort)

Has there been someone in your life who was a mentor and who has led you forward on your career? Offer it to other people. I graduated high school and I left performing arts, and didn’t know what I was going to do. I lived in Jamaica Plain in South Boston. I worked as a baker at a coffee shop called the Coffee Cantata, that is not the Coffee Cantata, but that is the one which allegedly Bach composed the work at, or performed at. The owner of that coffee shop where I worked, was also a musician, a classical pianist. One day, I was walking through the Arnold Arboretum which was in that area. The Arnold Arboretum is actually a research garden, as part of Harvard University. It’s a very beautiful space. I was walking there and a friend of mine at the time showed me this martial art that she had been training, capoeira, and I knew right then and there that that’s something I wanted to do. I began studying actually under a student of the man here in the middle, but that at that point in my life, I was making up my way and I wasn’t in college. This was an important piece of that progression, and learning, showing up and taking those classes. Then, two or three years into that, some other folks came to me, some friends of mine, people I trusted, and they said to me, “You need to go to college. This is the moment. If you don’t do it, you’re going to forget how.” I listened to them, and I applied. I think they probably sent some emails as well. I got into college. Again, encouraging people when they’re stumbling or they don’t know what they’re going to do is really useful, really important. It’s also not something you can necessarily choose. Some folks don’t have mentors, and they don’t have sisters and older siblings. Again, I don’t know that it takes effort other than maybe be in the right place at the right time, which if you’re a wandering non-college student you may not know to do. It matters.

College, and Graduation

I got into college. I went to Oberlin College in Ohio, south of Cleveland, tiny little town, tiny little university or college. I decided to study engineering, which totally makes sense because Oberlin is a little liberal arts college, and doesn’t have an engineering program. I started taking the prereqs and doing the chemistry 101, and whatnot. Also, Oberlin has a beautiful library. It has a lot of cool seating. It’s a really nice space. This is a place where I actually spent a fair amount of time studying physics 101 and whatnot. College was the best seven years of my life. I did a five-year degree program. I went to Oberlin and transferred to the Washington University School of Engineering. I took some time off and graduated late, which is a whole separate story about not being in your graduating class and being older than people expect you to be when you’re a new college grad. I did the work. I took a whole bunch of engineering courseload, linear systems, and numerical methods, and controls theory, and whatnot. I graduated with a Bachelor of Science in Engineering in the exceedingly well-known field of system science mathematics. Which if you say you have an ME degree or you have an EE degree, people know what that means. If you say you have an SSM degree, or a systems engineering degree, you find job postings for network system administrator.

Networks Matter (Moderate Effort, High Value)

I graduated with a Bachelor of Science in Engineering, and then nothing happened. I was sitting at home in my childhood bedroom. I was applying call to jobs on monster.com. I had $200 in my bank account. It was 6, 9 months after graduation, and I was broke. I had nothing except an engineering degree. I did what we all know to do now because we’re all fairly far along. At this point, I sent some emails, and I emailed my advisor from college. I emailed some classmates from my graduating class, from my degree program. Sure enough, someone connected me with a hiring manager and I interviewed and I got the job. Networking, probably most folks by now know that. If you take it for granted and you say, why is Caleb talking about this? There is someone out there who doesn’t know that and hasn’t heard that, and they need to. Encouragement matters, networking is exceedingly useful. It’s also not that hard to do. It’s a lot more enjoyable than just applying to your 200th job on monster.com, and not getting any responses. I don’t think it’s worse than the alternatives in terms of effort, but it’s how everything happens. Not everyone knows that. You have to learn it. It’s a skill set. A lot of these things are skill sets.

A Turning Point

Second question, has there been a point when you were about to go off and join the circus or something, and someone pulled you back and said, stay on track? Maybe they’re like, you want to join the circus, great, do it. Has there been a point where you reach the terminal point or decision point early in your career? Not so many people. I was wondering about this. I didn’t know what to expect. I think I learned later in my career that this is not true of a lot of people. You have classical musicians for parents, and they get you on that path early on. Or you go to the best schools and you take internships and you graduate and all that, and that’s great. There’s a lot of people out there for who that’s not the case, so non-traditional paths and non-traditional careers is just the natural way of being in the world. I do think it’s easy to take it for granted that you go to a good high school and you do internships in college and you go to a good college, and then you graduate and get the degree and all that. It’s not true for a lot of people.

I got a job at Sprint, the telecommunications company. Sprint is based in Overland Park, Kansas in the Midwest. I got down to work. I hadn’t done internships. I hadn’t done the traditional professional life, and I didn’t know how to do it. I got down to work of like, learning the job of showing up and not falling asleep in meetings when it’s alphabet soup. The conversations are just like, good gravy, telecommunications has so many acronyms. Literally, people just talk 45 minutes in acronyms, and none of it makes sense. You learn how to show up and be there and be present. Apparently, that involves a lot of khakis, and blue jeans, and shades of blue. It became this whole theme. We were like taking these photos because we’re like, “You wore blue today? Me too.”

Networks are Sticky

I was at Sprint for many years. Telecommunications is a deep engineering field. There’s a lot of really interesting engineering, goes into how call setup happens in sub-second latency, and how power propagation works. There’s a lot of complexity and nuance to it. It was interesting for several years. Then I realized that all I knew was telecom. I knew people at Qualcomm. If I wanted to go work at Motorola or whatnot, I could probably do that. Or at least I could email someone or make a phone call. It’s this insular thing. They’re sticky. Networks are sticky. I didn’t necessarily want to stay in Telecom. I didn’t want to stay in the Midwest. I didn’t necessarily want to wear khakis and blue all my life. It took me a little while to figure out how to extend my network and how to bridge outward. We took some more photos, and we got together. This crew actually was my innovation lab for a few years. We got together and we’d meet once a week in a supply closet where we kept telecommunications hardware and whatnot, and we would whiteboard ideas and work on patent disclosures. We challenged each other. I put this idea up on the board that we could radiate power from a cell tower to a cell phone, and Andy on the right there said that’s a silly idea, that’s never going to work. We were this innovation crew, ad hoc group of folks at the time. It was around the time that “The 4-Hour Workweek” came out. Then later I stumbled upon Seth Godin’s work and the idea of how to develop a skill set that’s unique no matter where you are, whatever company you’re at, or whatever field you work, and how do you make your skill set useful. We disclosed a patent disclosure. We disclosed a whole bunch of patents and kept a spreadsheet. We had about a 1 in 3 success rate. From a very small window of my career, I actually have a whole bunch of patents to my name. The point is, it was successful. Our goofy little ragtag crew that was meeting in a supply closet, was making headway and learning some interesting stuff.

Exit Strategy

I started formulating this idea that I needed to branch out. I needed to learn a new skill set and develop a broader network. I began to teach myself software development. I picked up Python. I went to the Python Kansas City meetup for many years. I was co-organizer for many years. I taught myself software development, version control theory, data structures and algorithms, social coding. I did the work again. I then participated in Devopsdays Kansas City. I helped co-organize it one year. I spoke at it the following year. At that point, I had developed a network outside of telecommunications and outside of the Midwest. It worked. I got out. There’s no substitute. I taught myself a new field, software development, and it was nights and weekends. I don’t want to do it again. If someone asks me like, what did you do? I did the work. There’s no substitute.

Choose Your Maximizer

I was also at the time reading Bogle and the idea that I wanted to develop an emergency fund, a savings fund. I became an inherent of buy and hold. Despite the synopsis of the talk, I don’t bet. I don’t really know anything about betting, in the sense of like Vegas, or off-track, or whatever. That’s a strategy, and buy and hold is a strategy. I developed a savings fund, and developed a network outside of Sprint, and outside of telecommunications. I graduated college late, and spent many years at Sprint. Some of them I felt like I wasn’t getting anywhere. I decided to optimize for compensation. I decided to be a jerk, to be a capitalist, to make money, and build up a safety net fund. It worked. This is, in fact, my year-over-year compensation for the period of time when I was focused on this, and it worked. I wasn’t living paycheck to paycheck. I was able to take more risks, and take longer with my job search and that sort of thing.

I can’t say that this is the ordering of things that someone else should choose. Because, being a jerk, being a capitalist means you’re a jerk. It means you’re missing out on other opportunities or other companies’ generative cultures. It’s useful, because now I have freedom. I don’t have to worry so much about simple things like how much a coffee costs. A lot of people do. I don’t have to worry about saving face. I don’t have to wear khakis. That worked. I reached a point where I was comfortable with that, and so I changed my maximizer function, at that point. I wanted to focus on finding fit. I wanted to optimize for fit. Also, all of these different things are skill sets. We talked earlier in the staff panel about what is the skill set that you might have as a staff engineer that you don’t as a senior engineer, and project management is one of them. How to interview, how to negotiate salaries, how to change jobs, maybe change back, leave a company, and come back and make more money. These are skills and this is a skill set, and you can develop it. You can share that skill set with others. Really, to this day, one of the most consequential outcomes for me in my career was that I had a friend who was interviewing and she came to me and said, they’re making me an offer, but it’s so much. I said, you need to make twice that much, you should go back to them and tell them that. It was several conversations over a couple of days or a week or something, but it worked and she doubled her salary. To this day, I find that I was able to share a technical skill set which I had developed for myself with someone else, and help them.

I was also not that happy at the time. I took this photo, in fact, to mark the point in time because I had gone through a couple of companies in fairly short timeframes, and they were not always the most generative cultures. I was responsible for some of that, for some of that toxicity. I wasn’t happy about that. I entered this company, Foghorn, as the 11th full-time hire. I had this vanity English license plate as a result of being an early hire at this consultancy. They were based in the Bay Area. They’ve since been acquired. I worked at Foghorn for many years. It’s a consultancy in cloud infrastructure. We are a client service delivery firm. We delivered projects to engineering customers of ours, and I worked on large scale engineering cloud infrastructure projects in AWS and GCP. I built up the GCP Premier Partnership status myself. I learned project management. I learned how to manage a team of engineers and deliver work to a client, which is also a team of engineers. I got to see their code bases. I got to see how they do CI/CD, all of these different companies, large biotech companies in the South Bay. You can probably think of their names. Smaller companies, which you will probably have never heard of, because they were tiny. I developed that skill set. I was a senior engineer when I entered Foghorn, and learned these non-traditional engineering skills like project management as well as like how to organize teams and put up with people who are really good engineers, but don’t show up for work.

Be Your Own Advocate – High Effort, High Value (Moderate Opportunity Cost)

We got to this point where my boss, the director of service delivery was basically going to level up, and we were interviewing externally for his backfill for a director of service delivery. Just as a point of clarification, in client services, director is an engineering management title. It’s a line engineering title. It does not mean Director of Engineering in the more traditional sense. I was a senior engineer, and we were scheduled to interview someone here, actually. I think I was going to talk to them remotely, but they were scheduled to show up. Then they didn’t show up for the interview, which baffled my boss at the time. I said, “Why don’t you give me a chance? I don’t necessarily plan to go into management but it would make sense for me as a logical next step in my career path. It’s a skill set you need. You’re struggling to fill that role.” He went back to the leadership team and they came back and they said, yes. I got the role of director of service delivery for a period of time. I wasn’t there for too long after that, but in a fairly short period of time, I got exposure to managing people, hiring and firing, how to have conversations with a direct report who is a higher skill level than yourself. Also, conversations with engineers who are belligerent. All the while also delivering the client work, the project work, and the engineering work. A new skill set and a new exposure.

I decided that I didn’t want to stay in engineering and management. I actually came to appreciate that it is a distinct skill set. It is something you can learn and practice. I have a lot of respect for managers and engineering managers because of that. I wanted to stay on the technical side, and so I decided not to continue doing that. I left Foghorn. I also wanted to get out of consulting. In consulting, you have to establish your technical expertise quickly at first glance. You also don’t have long term ownership. You advise companies and you help them deliver their work, but ultimately, they decide and they own the work, and so I wanted to get back into product engineering. I left Foghorn. I was continuing to train capoeira. Capoeira gave me a voice. It gave me the ability to feel comfortable in front of people and in crowds, and to be assertive, even if you’re not sure, even if you’re not confident, to still be assertive. It also appeals to me. It keeps me active. I don’t get bored like I do when I go running or something of the sort.

I said to my boss, give me a shot. If I hadn’t said that, they would have just kept interviewing externally, and trying to hire. I also learned through other means, through my martial arts, to be confident, and to speak loudly or clearly. This also takes a lot of work. There’s also no substitute. Again, if you don’t, at least ask. If you don’t propose that you take on a new role or a new responsibility in your current role, you’re less likely to get it. Higher reward. The opportunity cost is that you’re still optimizing for yourself, not for the business. You might not be focused on what’s most meaningful for the business, which, ultimately, is why we’re here, why businesses exist.

Optimize for Fit (Moderate Effort, High Value)

I kept training. I learned to do new things. Then I got out of Foghorn. I left the consultancy and I got another job. I started to, again, change my maximizer, earlier from optimizing for comp and from optimizing for position or title, and now looking at, which company is the best, finding the right company or the company that I want to work for. One that isn’t too toxic, or too boring, or whatnot. The next step for me was this progression to finding new roles. The culmination of my social media career and why I don’t really promote it is this tweet which got zero replies, zero likes, anything at all. I still to this day think about it, because it’s so true. If you’re going to take risks, you have to expect some of them to fail, and you have to be willing to admit that you failed. You have to be ok with that, and figure out how to do that. You have to develop a skill set. This is the next few steps. I was at Foghorn as senior, and I moved to the director of service delivery, an engineering manager role that also had duties of client consulting or project delivery. Then I left Foghorn and got an IC title as senior engineer. Now, here’s the thing is that moving from senior to engineering management could be considered a lateral move, or maybe not lateral, but in a sense, like a negotiation because I didn’t have management experience. They were taking a chance and I was taking an opportunity to gain more experience as a director. When I decided to leave consulting, I wanted to go be an IC again, an engineer again. I took a title as a senior engineer. If you were scanning my resume, you could be forgiven for thinking that I took a demotion, or a step down. As I see it, it was two lateral movements, but it requires explanation, and that doesn’t come off in the wink of an eye or if you’re scanning resumes. Narratives are important and being able to provide context, communicate that context is important.

Then I took another role. I left the company after a fairly short period of time and took a role as a staff engineer. That is clearly a promotion. It was a pay bump. I did move up in IC levels from senior to staff, and yet, it was a short tenure. That senior role was a short tenure. If you’re scanning a resume, you might think that was a failure. I have gotten questions about that as I interview, is, why were you only there for so long? The point here is like, if I tell you I moved from senior to staff, it seems like a clear win, but there’s nuance, and that gets sussed out when you’re interviewing and whatnot. Then I failed to execute as staff engineer, and I was laid off. I was unemployed for three months, and I collected unemployment for the first time in my life. There is a stigma around unemployment, being unemployed, but companies pay out of your salary. They pay taxes into unemployment funds, so you should be ok taking that if you need to. I did fail as staff. I could caveat it. I could tell you like, it was not a generative culture, or they were not working on interesting stuff. That doesn’t matter, because I’m not there to influence the decisions. In any sense, I wasn’t successful delivering or executing on a vision because I’m not there anymore.

Culture is Your First Priority (Enormous Effort, High Value)

That period of time, at the consultancy, and then afterwards, me consulting, is like language immersion. I learned so much about engineering orgs, and code bases, and all of that, and project management. It’s not low margins, but you have to control cost carefully. You have to talk about utilization rates and say, we expect engineers to spend 83% of their week working on billable work. You have those kinds of conversations. It’s pretty demanding. It can be pretty exhausting. I would joke with my manager that I was going to study for the PMP exam in the 50-to-60-hour block of my week, because there’s always more to do. The consulting was exhausting. Then the work after that was successful in some sense, and not always successful, and not always enjoyable. I’ve, again, changed my focus, my maximizer function, and now I consider that culture is king. It’s all social coding. A few years ago, GitHub had that as their tagline. It’s all about optimizing for the business and optimizing for what the business needs, and figuring out how to make that happen. There’s no shortcuts. There’s no substitute. It requires having a lot of direct, difficult conversations. Being willing to say when you’re wrong, but also not tell someone else too bluntly that they’re wrong. That’s my work now. That’s where I’m going next.

Summary

If you had $200 in your bank account, and you went to Vegas, would you play it all on slots, or play it all at once on roulette? I don’t know. I can’t even ask this question because I don’t bet. Inaction is a form of risk. You have to decide what to do in any case. There’s different strategies for investing, and there’s different strategies for maximizing your compensation. I think I’ve spoken to all of these and spoken about each one. This is the idea. This is the framework of like, where on the quadrant of effort and risk and reward, do each of these things fall? Like I said, I started optimizing for compensation first, and then later for fit. In hindsight, that might not have been a great idea, but it did give me latitude and leeway. Again, if all that I’ve said today is obvious to you, and you’re like, why again, is Caleb talking about this? It’s because there are other people earlier in their career who haven’t heard it and they need to. We have to share our stories and be vulnerable in order for them to know that you’re authentic, and to know that you mean what you say, and that you have the bona fide or whatever to advise them. My name is Caleb Hyde. This has been a talk about risk and failure.

Questions and Answers

Participant 1: You started off by saying you need to optimize for compensation because that was needed since then. Then, later on in your presentation, you talked about optimizing for fit and then culture. With your experience now, would you say that you would recommend young engineers like don’t get attracted by optimizing for compensation, or do you still think that that was the right thing to do at the time.

Hyde: I don’t know. I can say with assurance that working intentionally to maximize my pay, and also along with that, like you’re standing in an organization in your respect, like you have a seat at the table, that was successful and useful. To answer the question without answering what you’re asking is, why don’t you just do all of that at once? Why don’t you timeshare your development effort between comp, and fit, and behavior, and whatnot? Sure, if you have the discipline for that, and if you know to do it. If you’re early career, you don’t have a skill set. All of these things are obvious in hindsight, because we learned the hard way and learned by doing them or because someone told us. Early on you don’t know what cloud infrastructure is. You don’t know what project management is. You don’t know how to tell someone they’re wrong politely, and all of that. I don’t have an opinion. I can’t say that it was better or worse. You have to choose where to spend your time, and sometimes the only thing to work on is the thing in your view.

Participant 2: Someone once said to me, basically, work with people who want to work with you, people who are working against you, don’t waste your time, don’t care. Under the general umbrella of choosing your bets, kind of thing. Curious if you have anything else to talk on that, in your opinion.

Hyde: Someone I worked with early in my career would always say, choose who you work for, who you work with, and what you do. I still quote them because that’s great advice. It’s concise. It’s pithy. It’s useful. It’s true. I would also never work with that person again. You learn from everyone you work with, for better or worse. There are no absolutes. It all depends.

Participant 3: You mentioned that you were working in software engineering, and then you said that you think that you failed. Do you mind sharing your thoughts on maybe how you failed?

Hyde: I think, more or less, what I was doing fits a pattern that’s pretty common, which is, I was steadfast in my technology decisions. I’m like, we need to do this. What we’re doing is a terrible idea, and whatnot. There was also an element of, they were asking me to do something that I had actually done several times before, a large scale Terraform migration and modernization effort, and I had opinions about that. Like, I’ve done that, and I found it a little boring. I also had a fairly clear idea about how to get it done from an execution perspective, like from a technical perspective of like, here’s how you implement a Strangler Fig pattern, and bite the elephant one bite at a time. Again, my delivery was terrible. Even though I have in fact done large scale Terraform modernizations and written software systems to migrate Terraform, none of that mattered because they weren’t working with me and listening to my advice, and ultimately, I was laid off.

See more presentations with transcripts

Article originally posted on InfoQ. Visit InfoQ

Microsoft recently announced the public preview of its new Geo-Replication feature in the Azure Service Bus premium tier. This feature allows continuous replication of a namespace’s metadata and data from a primary region to a secondary region, which users can promote at any time.

Azure Service Bus is Microsoft’s fully managed messaging service, enabling reliable and secure communication between distributed applications and services across cloud and on-premises environments. The new Geo-Replication feature for this service is designed to provide robust protection for Azure Service Bus applications against outages and disasters, complementing existing options such as Geo-Disaster Recovery and Availability Zones.

Unlike the existing Geo-Disaster Recovery feature, Geo-Replication replicates metadata and data. It can establish resilience between Azure regions, such as the East and West US. In addition, Availability Zones provide resilience within a specific geographic region.

Eldert Grootenboer, a senior program manager on the service bus team at Microsoft, writes:

The Geo-Replication feature implements metadata and data replication in a primary-secondary replication model. It works with a single namespace, and at a given time, there’s only one primary region serving producers and consumers. A single hostname connects to the namespace, which always points to the current primary region. After promoting a secondary region, the hostname points to the new primary region, and the old primary region is demoted to the secondary region. After the new secondary has been re-initialized, it is possible to promote this region again to primary at any moment.

Service Bus Premium Geo-replication (Source: Tech community blog post)

Microsoft’s competitors offer similar messaging services with disaster recovery features. AWS Simple Queue Service (SQS) ensures high availability and fault tolerance by distributing messages across multiple availability zones within a region. This means that messages remain accessible and are not lost even if there is an outage or disaster affecting a specific data center. Similarly, Google Cloud Pub/Sub provides high availability and disaster recovery by automatically replicating data across multiple regions, ensuring that messages are reliably delivered and available even if one region experiences an outage or disaster.

Clemens Vasters, a Principal Architect for Messaging and Real-Time Analytics at Microsoft, tweeted a bold statement:

#Azure Service Bus is the most powerful transactional hyperscale queue broker in existence.

Introducing multi-region, synchronous (RPO-Zero), or asynchronous namespace replication (all messages and message states!) as insurance against region failures.

Lastly, the feature is available in the Central US EUAP, Italy North, Spain Central, and Norway East Azure regions.

Article originally posted on InfoQ. Visit InfoQ

OpenAI recently published a paper about CriticGPT, a version of GPT-4 fine-tuned to critique code generated by ChatGPT. When compared with human evaluators, CriticGPT catches more bugs and produces better critiques. OpenAI plans to use CriticGPT to improve future versions of their models.

When originally developing ChatGPT, OpenAI used human “AI trainers” to rate the outputs of the model, creating a dataset that was used to fine-tune it using reinforcement learning from human feedback (RLHF). However, as AI models improve, and can now perform some tasks at the same level as human experts, it can be difficult for human judges to evaluate their output. CriticGPT is part of OpenAI’s effort on scalable oversight, which is intended to help solve this problem. OpenAI decided first to focus on helping ChatGPT improve its code-generating abilities. The researchers used CriticGPT to generate critiques of code; they also paid qualified human coders to do the same. In evaluations, AI trainers preferred CriticGPT’s critiques 80% of the time, showing that CriticGPT could be a good source for RLHF training data. According to OpenAI:

The need for scalable oversight, broadly construed as methods that can help humans to correctly evaluate model output, is stronger than ever. Whether or not RLHF maintains its dominant status as the primary means by which LLMs are post-trained into useful assistants, we will still need to answer the question of whether particular model outputs are trustworthy. Here we take a very direct approach: training models that help humans to evaluate models….It is…essential to find scalable methods that ensure that we reward the right behaviors in our AI systems even as they become much smarter than us. We find LLM critics to be a promising start.

Interestly, CriticGPT is also a version of GPT-4 that is fine-tuned with RLHF. In this case, the RLHF training data consisted of buggy code as the input, and a human-generated critique or explanation of the bug as the desired output. The buggy code was produced by having ChatGPT write code, then having a human contractor insert a bug and write the critique.

To evaluate CriticGPT, OpenAI used human judges to rank several critiques side-by-side; judges were shown outputs from CriticGPT and from baseline ChatGPT, as well as critiques generated by humans alone or by humans with CriticGPT assistance (“Human+CriticGPT”). The judges preferred CriticGPT’s output over that of ChatGPT and human critics. OpenAI also found that the Human+CriticGPT teams’ output was “substantially more comprehensive” than that of humans alone. However, it tended to have more “nitpicks.”

In a discussion about the work on Hacker News, one user wrote:

For those new to the field of AGI safety: this is an implementation of Paul Christiano’s alignment procedure proposal called Iterated Amplification from 6 years ago…It’s wonderful to see his idea coming to fruition! I’m honestly a bit skeptical of the idea myself (it’s like proposing to stabilize the stack of “turtles all the way down” by adding more turtles)…but every innovative idea is worth a try, in a field as time-critical and urgent as AGI safety.

Christiano formerly ran OpenAI’s language model alignment team. Other companies besides OpenAI are also working on scalable oversight. In particular, Anthropic has published research papers on the problem, such as their work on using a debate between LLMs to improve model truthfulness.

Article originally posted on InfoQ. Visit InfoQ

Subscribe on:

Apple Podcasts
Google Podcasts
Soundcloud
Spotify
Overcast
Podcast Feed

.
From this page you also have access to our recorded show notes. They all have clickable links that will take you directly to that part of the audio.

Article originally posted on InfoQ. Visit InfoQ

Transcript

Zhou: We’re going to talk about how we delivered millions of notifications within a few seconds during Super Bowl. I’m Zhen. How many of you know Duolingo? Duolingo is the most popular language learning platform. We have the most downloaded language learning app worldwide. Our mission is to develop the best education in the world and make it universally available. What do we have? We currently have 31.4 million DAU, which is Daily Active Users. We have over 100 courses offered in 40-plus languages. On top of that, we also have a very unhinged social presence. You may know us from a lot of things, and I assume internet memes is one of them. You may have heard of like Spanish or vanish. Some other stuff. Also, our marketing team is really hard at work. You may have seen some of the pretty wild TikToks or memes that revolves around Duo or our employees. That is really our marketing team hard at work.

Let’s switch gears a little bit and look at this number, 123.4 million. Does anyone have a guess of what this number could be? That’s actually the number of viewers of last Super Bowl. This year, the Super Bowl game between Kansas City Chiefs and San Francisco 49ers brought an average of 123.4 million viewers. That is the highest number of people watching the same broadcast in history. If you ask ChatGPT to create an image of a typical Super Bowl ad, it looks like this. Basically what you would expect from a ChatGPT image. It’s all over the place. It has cars, angels, neon signs, everything. Why are we really talking about this? This is because everything was fine for us engineers at Duolingo for a while, until one day we heard that Duolingo would run its first Super Bowl ad as it expands beyond TikTok. Our chief marketing officer expects the brand to be even more unhinged in 2024. Then we saw some design docs that revolves around Duo’s Butt, so we know what’s coming here. What really dawned on us was that we are buying a Super Bowl ad. This is a 5-second ad that was played during this year’s Super Bowl. At the time, we know that it has something to do with Duo’s Butt. There’s one last thing. The marketing team asked us to do a push notification at the same time. What could possibly go wrong?

Overview

We’ll take you on a deep dive into the Superb Owl service, which we named conveniently after the Super Bowl. First section, we’ll be talking about how we dissect this impossible task. Then we’ll talk about how we cope with the changing requirements. Followed by talking about the architecture deep dive into how exactly the service works. Last but not least, we will talk about what we learned and what we would have done differently.

A Dissection of An Impossible Task

Let’s start out with a dissection of an impossible task. When I use the word impossible, it may sound strong to you. It’s just sending notifications. When the marketing team came to us, they asked us, we want you to send millions of notifications in a few seconds, can you do it? We were like, hold on, let’s do some research. We did some research. We came back to them and said, I don’t think we can do that. Then, the next thing you heard was that them telling us, we already paid for the ad. Actually, at that moment, it wasn’t really up to us, we have to do it. They told us that they want to send 4 million notifications within 5 seconds. When they first came to us, we were actually a little nervous, because here you see how fast our notification typically goes out is at a rate of 10,000 notifications per second. If you do the math, that’s actually a lot of notifications per day. What they’re asking is to send notifications at 80 times faster speed. If you’re a site reliability engineer, or you work in the backend, there might be already alarms that are going off. This is what I’m talking about. Back in 2022, Coinbase launched its bouncing QR code like campaign during Super Bowl, and it was so popular that it crashed their site. We probably don’t want to do the same. On top of that, we also discovered an issue with our Android app, so that when we send out a notification to the Android app, it sends a request right back to the backend. This sounds a little bit similar to DDoSing ourselves. If you want to send 5 million, 4 million notifications within a few seconds, you probably want to avoid doing that.

In theory, leadership just came to us, “You should just ship it.” I don’t actually know if ship it is still a principle at Duolingo. In practice, this is a complicated problem. Because here you see, we want to build a system to send notifications fast, we also want to make sure that we do not DDoS ourselves. We also want to handle the organic traffic, because people actually click on those notifications and come back to the app to practice learning new languages. Our solution is basically somewhere in between those three problems. To make matters worse, we have people coming to us from different perspectives, product managers, or other engineers, they come to us with suggestions or inquiries. Can we add an icon to the notification? Can we send to more cities? Can you localize the push into different languages? Can we use the vendor for this? All of these questions. At this point, you might be imagining there’s like your favorite superhero from Marvel or DC to come and save the day. In this case, we would probably call it the coding man or something. There’s no coding man in real life. What we did was just very simple, we divide and conquer the task. I was in charge of building the system to send notifications fast. Two of my wonderful colleagues, Louise and Milo, they’re respectively in charge of making sure that we’re not DDoSing ourselves, and making sure that we can handle the organic traffic.

Coping with Changing Requirements

Let’s move on to the next section. I’ll talk about how we coped with the changing requirements. On the first day of the project, my technical program manager, Paul, came to me and asked me, “Can you give me a timeline on this project?” At the time, I wasn’t really a senior engineer. I said, “Sure, but where do I even start?” Because it’s such a complicated project. All we knew was, from August 2023, we know we want to send 4 million notifications in less than 5 seconds. Super Bowl happens the next year on February 11th, and they’re not going to move that date for us. How do we account for all these uncertainties in between? As you may know, to make matters worse, requirements are always changing. People are suggesting all sorts of things to do. You should do this. You should design this system in one way than the other. To give you an example, I said, they came to us saying we want to send 4 million notifications within 5 seconds. That was actually a lie. That actually changed. Because when they first came to us, they told us that they want to send notifications to 3 markets, with 2.6 million notifications in total, and they want to send all of it within 8 seconds. A few days later, they changed their mind. They were like, how about we round out to 3 million notifications? We want to send all that in 5 seconds. A few months have gone by, they came to us, they told us, we actually closed a new deal, and we now have the ability to show the ad in 7 markets. Wonderful, but that also means we want you to send to 4 million users. Because we’re in the United States, and people have a lot of digital devices, and potentially, we want to send to 6 million plus devices, which is 6 million plus notifications, all of that within 5 seconds. You can see how requirements can be changing all the time in a project.

Our solution was to figure out, what does not change? We figured that our operating principle does not change. One of our operating principles at Duolingo here is to test it first. We’re obsessed with A/B testing. We can’t A/B test this project, though, but I’ll talk just in a bit how we tested it. Our principle in this project is to not ship anything that we cannot test. Here’s a few examples. When they came to us and asked if it is ok to send to a larger audience, we said, yes, maybe, because we can probably test it. When they came to us and asked whether we can send notifications with an icon. We said, yes, because we can test it. When they asked, is it ok to add the ability to send notifications to different markets at different times? Sure, as long as we can test it. Is it ok to add 2 million users the day before, after basically we finalized the audience list? No, probably not, because we cannot test it. After that, we established the timeline. Back in August, we know that we want to send to roughly 4 million users in less than 5 seconds. We want to have the MVP ready by September 2023. We know that we want to make sure that we cannot DDoS ourselves by the end of December 2023, so that enough time could be given so that new client changes could be rolled out to users. By January 2024, we want to make sure that all testings are done, so that when Super Bowl really starts, we’re ready because we have tested everything.

Architecture Deep Dive

Let me take you on the core of this presentation which is the deep dive into the architecture of the system to send notifications. To send 4 million notifications in 5 seconds, it’s a lot of challenge. At Duolingo, we have an operating principle of, embrace challenges. To summarize the challenges in this project, we have three. The first one is speed. I’ll call it how fast. The second one is scale, which is how big. Third one is timing, which is what time. The first challenge is speed, of course. The speed is 800,000 notifications per second, roughly. If you think of the population of Boston, last time I checked on Google, Boston had a population of 650,000 people. I think that was back in 2022. Now it’s probably a little more. To send notifications to all these people, at our rate, it will take less than one second. It’s just that fast. The second problem and challenge is scale. At Duolingo, we host a lot of our microservices on AWS. Unfortunately, that cost us money. I wish that it was free. To build a system that sends notifications really fast, you also have to make sure that it’s scaled up to the point that it can do its job. You also want to make sure that your backend is scaled up fast enough to do its job. On top of that, you probably don’t want to spend too much money on cloud infrastructures. That is also a challenge at scale. The third challenge here is timing. For those of you who are into sports event, you’ll know that sports events are unpredictable, because people get injured or sometimes it goes faster or slower. Even the more predictable ones like the soccer games, they’re usually fixed, 90 minutes, but they have added time. It’s also the same case with the Super Bowl. Super Bowl has timeouts and everything. The problem with an ad campaign during Super Bowl is that you don’t know exactly when your ad is going to air. You see, our plan is to show the notification right after our ad airs. We have to build a system that is on-demand, so that when some marketer presses a button, potentially a big red button, then our notification goes out. We have to build such an on-demand system. We want to make sure that maybe if two marketers press the same button at the same time, no duplicate messages go out. That’s the challenge of timing. To summarize, we have the following technical requirements to build this system. We want to send notifications at the target rate. We want to get all the cloud resources we need just in time. We want to ensure resiliency and idempotency. I just explained idempotency here by the marketer example. I hope you all understand.

Here is our system. It’s basically an asynchronous system. We have a lot of our services on AWS. I’ll walk you through some of the use cases that we have, to basically show you how the system works. Let’s assume that it’s a few months before the Super Bowl, you want to upload a notification campaign. It starts out with an engineer creating a campaign with a whole list of user ID. The server after receiving that request will acknowledge the request and then asynchronously fetching all the data from DynamoDB. After getting the data, they will put the parsed data into S3, locks the result into CloudWatch. A few months have gone by, and maybe it’s Super Bowl Day already, you want to get ready because you want your backend infrastructure to be scaled up well enough to handle the incoming traffic, as well as being able to send out a notification. What happens then? The cloud operations admin will actually have to scale up the ASG, the automatic scaling group first, and then the engineer who is responsible, at the time it was me, would scale up the workers in the ECS console, just change a few numbers. The workers will then fetch data from S3, which we previously stored all the data, and store that data in memory. The data will be mapping from the user ID to the device IDs. After all that, the workers will log their complete status to CloudWatch. Maybe a few hours have gone by and someone was watching the ad, and the ad shows up and someone presses the big red button. What happens after that? The marketing admin hits the Go button. After that, the API server will actually send out 50-plus messages to the FIFO, First In, First Out SQS queue. The interim worker in between after receiving those messages, they’ll dispatch 10,000-plus SQS messages to the next queue. Finally, the last tier of notification workers will send notifications by calling the batch APNS and FCM’s API. They are Apple and Google’s notification API. After all that, after we’re done making all the network requests, the workers will log their process time into CloudWatch, so that we can analyze their performance.

Let’s come back to some of these technical requirements that we discussed earlier. Because if the system doesn’t answer all these questions, it’s still not working. The first question is to understand, does this send notifications at the target rate? It’s hard for me to convince you, but some of you who are familiar with SQS might notice that SQS in itself has an in-flight message limit of 120,000 messages per second. Just to send 4 million notifications within 5 seconds, would put us at 800,000 messages per second. How does that happen? Because actually, we simply use the technique of batching. We batched 500 iOS users together in one message, 250 for Android, and that put us under the limit. Second question we want to answer is, can we provision all the cloud resources we need in time? To solve this problem, we actually got in touch with a technical contact from AWS, who helped us draft an IEM, an Infrastructure Event Management document, which included detailed steps, like when and how you would need to scale up, as well as some of the more concrete steps such as things like the cache and the cache connection limits, and also Dynamo limits that you have to consider to address in these kinds of situations. Also, we also gave the Superb Owl service its own dedicated ECS cluster. I’ll tell you why we did that. Last question we want to answer is, can we ensure idempotency? To solve this problem, we actually cheated a little bit, we used a FIFO, First In, First Out queue from AWS SQS service. You see, the FIFO queue in itself, it deduplicates messages by message identifiers. It also has a deduplication window of 5 minutes. When multiple marketers press the same big red buttons, it’ll try to send the same messages through the queue, the queue will deduplicate the message for us. We basically just hand off that work. To keep in mind that the queue itself does have limited capacity for the FIFO queue. It has limited capacity of 300 requests per second. If you’re building a system like this for your own purpose, you also might want to use a cache or a table if you want to get past some of those limits.

Let’s get back to the system diagram graph. Does it send notifications at the target rate? Yes. Can we provision all the cloud resources we need in time? Yes. Does it ensure idempotency? Yes. Some of you may say that we’re probably done here. I’m inclined to say so, but actually not. Let’s look at this timeline of where we are now. We basically are still at September 2023, is when we built our MVP. Next subject is everyone’s favorite subject, which is testing. How we tested it, at Duolingo, we love to test things by A/B testing. We are obsessed with A/B testing every single detail in the app. This is honestly not a thing that we can A/B test, because we don’t want to leak our precious marketing creatives to our users. You don’t want to show the Duo’s Butt on something else, because that will ruin all the fun. We had to come up with our own way of testing the service. I’ll walk you through a few examples of how we tested the service. First, we tested the throughput of the service. This is what I call, how we tested the MVP. At the beginning, we tested with not really notifications, we tested with what we call silent notifications. This is effectively an empty payload that we send to the client devices and see what happens. It turns out, it seems that we ran into some bottlenecks. The bottleneck we ran into was the thread count. Here you see we actually built our service in Python. I know many of you might disagree. If you are an expert in Python, you will know that when you build multi-threaded programs in Python, you run into issues with global interpreter lock and all sorts of bad things. That’s the issue that we ran into. What could we do? We decreased the number of threads, and continued with the test, decrease it from 10 to 5 to 1, and the bottleneck seems to be gone. Then, we decided to test with a larger audience. Because at the beginning we tested with 500,000 users, now we want to test with 3 million users. Then the bottleneck came back. As you can see, we decreased the number of threads, but to get the same performance, we have to have a lot of tests. It was really problematic to scale up to that many test count. We decided to bin pack and put multiple processes within the same test. We experimented with a few setups and got to a point that we’re ok with.

The next thing we tested was the cloud resources. This is when we tested whether we can scale things up fast enough. The first thing we ran was pretty straightforward. We want to make sure that we can scale up the system to send the notifications really fast. In this case, it’s the Superb Owl service. It’s actually relatively simple. You just go in, you change one number, and it scales up, just like magic. Then, we want to scale up the backend. When we did that, go through a few more steps and understand what services we need to scale up. We found those services. We changed a few more numbers. They scale up just like your usual life. Then the third question we want to answer is, can we actually scale up both the Superb Owl service and the backend. It turns out that we ran into some trouble. Here, you see that when you have services in the same AWS ECS cluster, and when you try to scale up all of them, AWS is actually doing some maybe like bin packing problems and trying to figure out the optimal solution, or some good solutions. In reality, what we observed was that when we placed the Superb Owl service in the same cluster as the backend, it has to wait in a virtual queue. It has to wait for all the other backend services to scale up so that it can scale up, or vice versa. That’s not really the scenario we want to see happening, because we don’t want to see our backend services waiting, because our notification service is scaling up. We don’t want our notification service not being able to scale up because backend is trying to consume more resources. We gave it a dedicated ECS cluster, so that it got rid of that queue. Last but not least, we want to ensure that we can scale up both the backend and the Superb Owl service in less than 3 hours, so that we don’t have to waste money to have this service running around, maybe days before the Super Bowl. We did that. We tested. It works.

We move on to test on real users. I said that, we don’t want to leak the creatives of Duo’s Butt to our users. I never said that we couldn’t test on real users. Actually, we tried to test in plain sight using our new notification system. In October, we tested on 1 million users using a Halloween theme notification. In November, it was like a year in review theme notification. In January, we tested down 4 million users with a welcome back from New Year message. This is all us testing in plain sight without leaking the creative. One of our internal tools called Zombie mode really helped us in this process. Zombie mode is essentially a mode that we created so that when we turn it on, the user’s device will stop making the requests to the backend for a certain amount of time until it tried to check that flag again. It helped us in the scenarios where backend cannot handle a surge in traffic, and it needed time to recover. Thanks to that mode, we were actually more comfortable with testing with real users, because we know that we wouldn’t have extended periods of outages. One of our lessons learned here is to always actually send yourself a copy before sending it to the user, because you never know in what way your internal representation of the message can leak out to the user. You never want that to happen.

Last but not least, we tested within our microservice architecture. Here are some of the things that we saw in that IEM, Infrastructure Event Management document. The first thing that we considered was, is our memcache, Redis hitting connection limit, or can we use a proxy for some of the tables that we have? This is actually a real thing that we did. One of our services was having problems scaling up past 400H tasks because it was hitting the connection limit. It wasn’t really a priority to address, so people put it off. Until this point, you really need to scale up, so what we did was we put a proxy behind the datastore. It actually can scale up to however many tasks you want. Problem solved. Second problem is, will our DynamoDB throttle? DynamoDB claims to be infinitely scalable, but some of you may know that it doesn’t handle the spiky traffic really well, especially if you’re provisioning your resources. We considered, should we switch some of these tables to on-demand briefly, or should we overprovision it to a point that it wouldn’t throttle even with a spike in traffic? The last two are the things that we considered, but we actually never addressed, but I thought it would be relevant to mention here. One of the things that we considered was whether our retry policy was reasonable. Because you can see, we have so many microservices, and when one microservice call the other it might retry three times, and the other call the third one, three times, and it goes on forever, and it grows exponentially. You don’t want that to happen. Last but not least, we know that we are going to request a certain subset of our users. Maybe we could have cached some of their user data that we know that are going to be requested. Because of time limit of this project, we obviously didn’t get into the detail of all that. We got to a point where we were happy.

On the day of the Super Bowl, we have a pretty well written playbook. We have engineers on Zoom really carefully watching everything, scaling things up. We have marketers potentially at their TV, holding popcorns. That’s how I imagined what they’re doing. We have a square go button. Some of you might ask, why isn’t it round? I might explain it to you just a bit later. When Super Bowl starts, everybody was watching. We were watching for different reasons. Our marketers were waiting anxiously for the ad to show up. It didn’t show up for 20 minutes when it was supposed to. When it finally did, two marketers clicked on that square button, and one message went out. The campaign results were actually wonderful. Ninety-nine percent of notifications were out in 5.7 seconds, 95% in 3.9 seconds. There was a Figma blog post written about it. There’s also a Wall Street Journal on our Super Bowl campaign. I think someone on Twitter or X actually posted about our notification campaign as well.

What We Learned, and What Would We Do Differently?

It’s time to move on and talk about what we learned and what we would have done differently in this project. Some of the mistakes we wish we had avoided, relates back to the very fundamentals. We started out this project using Python. It turns out, it actually gave us a ton of performance bottlenecks. We thought about using Async Python. We tried it. It didn’t really work with the tooling that we had. Actually, we didn’t end up trying because at the time, our original notification stack was built in Python. We don’t want to reinvent all the technology, all the authentications, the connections that we establish with the outside APIs. We stopped there because we wouldn’t have enough time to test. Actually, one month after the project, my manager came out to me and said, have you seen this HackerNoon article from 2019 that says, “Using Go to build a system that sends millions of notifications within a few seconds?” I was like, that’s interesting. I wish I had read that article. It could have given us a lot more inspiration and lessons. That also uses Go, which is a completely different stack than what we used at Duolingo. That could have helped us. The second thing that we wish we had improved was to optimize memory. I told you that at the end, we want to send 6 million plus notifications. To pack all these 6 million plus device IDs and user IDs into the memory of a service, it was actually a lot. We ended up using 7 gigs of memory for each task. You may remember that we have a lot of tasks, which means that it requires a lot of cloud cost and intervention. My colleague Milo worked really hard on that, and we finally got it across. If we had time, we would have optimized memories and resource usage in general. Last but not least, we want to have better observability into the client behavior. I told you earlier that our Android app used to have a problem, when you send a notification to the app, it sends a request right back. That’s effectively DDoSing ourselves. That’s not a problem anymore, but we discovered new problems. We discovered that when someone clicks on the notification, it triggers multiple requests to the backend, that number may be 10, 20, 50. That’s not good. Because even if 5% of people click on that notification, and if that multiplier is 50, you’re effectively making two times our notification requests back. That’s a lot of requests back to your backend in a short amount of time.

Some of our lessons learned here are the following. The first one is to be open minded about the design, but be more rigorous with testing. I wouldn’t say our design was the best, it was really far from the best. We went with one of the designs that we could iterate and test on. We know that testing will be the core of this project. Because, as long as you can test it, you can make sure that the system works the way that it’s intended. The second lesson learned is to always build the system with resilience and robustness. As we just mentioned, multiple marketers can press the same button at the same time. You don’t want duplicate messages to go out to your users. This is just one example of the things that you should take care of when you design a system like this. The third thing is, things can always go wrong, so accept it. I’m not really saying to do nothing about it. Actually, we took a rather active approach. We wrote a very detailed handbook to tell engineers on that day what you should do on the exact time of the day. Also, it included backup plans. What if this A thing fails? What if this B thing fails? We even have a backup plan for the whole notification not going out. What should we do then? I think what we are saying here is to face it with a more active attitude.

Questions and Answers

Participant 1: I was curious about the number of instances in your ECS cluster, and if you guys try to vary that number or do some tests. Was it fixed at 5k as the max amount for the ECS cluster?

Zhou: We did experiment with different instance sizes, and everything. One of the things that we could have done better was to optimize memory. At the end, we were bounded by memory, because we tried to store all this information in memory. That led us in the decision of how we decided to select the instance types, which ended up being memory intensive instance types. Five thousand, I think is just one of the numbers that we were comfortable with that makes sure that we can send out notifications that fast. There could be a limit, but I think it’s mostly handled by my colleague with some negotiation with AWS.

Participant 2: When we talk about workloads that need to scale very rapidly and are very occasional, my mind goes to serverless. I’m curious, did you consider maybe using AWS lambdas for some of the workloads instead of looking heavily into your container cluster, and what was the role of serverless in general?

Zhou: I think that was definitely one of the suggestions that was being made early on in the project. I think someone suggested this to me and promised me, like even if it’s lambda, it can also basically pre-spin up all the things under the hood, and you know you’re going to get all these resources. At the time we just felt safer working with all the existing containers and the ECS services. I think, without seeing the machines spin up for you, it still puts us into the area of like, we’re going to be more concerned whether this is going to work or not. This is definitely valid, because this is basically like a one-shot operation and you pay for what you do. It would make sense in another scenario, if we could test it to use lambda.

Participant 3: Did your team worry about whether FCM or APNS was going to handle the load, or freak out? Did you have to reach out to anyone to make sure that that didn’t happen?

Zhou: Basically, the question was about whether we’re concerned that we’re going to get rate limited or throttled by FCM or APNS.

Yes, we were concerned. Also, at the very early stage of the project, we took initiative and we reached out, we asked some of our business people to reach out to the Google and Apple’s reps to understand whether we are going to get rate limited or not. We only actually green flagged most of these after we knew that it’s not going to get rate limited.

Participant 4: I’m curious about how you folks approach testing, because you talked a little bit about the cost. How many times do you have to actually keep testing that, and then actually multiplying the cost versus the time that you have to do testing? A little bit about how you approach the problem, testing times cost.

Zhou: Our philosophy was that, at least we want to minimize the risk. The costs in tests are relatively smaller, compared with the cost of the system not working at all, first of all. Also, we try to leverage the system as much as we can when we test on real users. As you can see, we tested on millions of real users in actual campaigns. In those campaigns, we would have spent the money anyway. That was basically no cost for that test. On top of that, we did run multiple cloud tests, basically, to try to scale up the infrastructure only, and not try to send anything. I think it’s worth it to do maybe 3, 5, or even 10 times, until you’re at the point that you’re comfortable with the system’s ability to scale up, because it’s actually our first Super Bowl campaign, and we don’t want to mess it up. We want to be as much risk averse as possible here. Cost is a little less of a factor if you’re considering we are already spending on the Super Bowl ad.

Participant 5: As the Super Bowl is a very popular event, were there any concerns about capacity not being available when you wanted to scale up because this was such a high spike usage?

Zhou: Absolutely. I think this is definitely something that we talked with our AWS representative. This is something we definitely raised early on, whether we are going to have enough memory, whether we are going to have enough resource allocation for us. That’s definitely something that you would have to discuss when you anticipate that your program is either compute or memory intensive.

Participant 6: For this project, I know you have to do a lot of scaling. Was there any strategy around optimization for the service itself? Did you use any specific tools for that?

Zhou: I think we are not known for performance as a language learning company, as you may tell. In this project we mainly focused on getting it to a point where we can send notifications that fast. The way that we tested was actually just pretty simple. We logged some of the process times to CloudWatch. We would process those logs afterwards and see how much time it takes for each function or each step to execute, and figure out the bottlenecks and speed them up. I think there are definitely more performance frameworks out there that we could leverage. With the constraints of this project, we didn’t go with all that. Definitely it’s a good question and concern. If we were to do this again, we’d definitely do it right.

I asked my colleague, JP, to initially make a big red button, make a round button. Later, actually, a marketer discovered that when you click on the corners of the button, which appears to be white, but it actually will trigger the same function as the square. You would call it, in video games terms, like the hitbox is still a square. That’s not good. As much as we want to avoid human errors, we definitely want to avoid that, so we made it a square, so everybody knows what they’re doing.

See more presentations with transcripts