Transcript

Shane Hastie: Good day, folks. This is Shane Hastie for the InfoQ Engineering Culture Podcast. Today, I’m sitting down with if Yiftach Shoolman. Yiftach is the co-founder and CTO of Redis. Today I’m in Australia and he’s in Israel. Yiftach, welcome. Thank you for taking the time to talk to us today.

Yiftach Shoolman: Great to talk with you, Shane. You pronounce my name correctly. Thank you.

Shane Hastie: Thank you very much. So let’s start. Tell us about your background.

Introductions [00:48]

Yiftach Shoolman: Okay, so I’m a veteran in the high tech area. I started my career early in 1990, and did ADSL system in order to accelerate the broadband. So everything in my career is somehow related to performance and then I realized that after the broadband become adopted, I realized that this is no more the bottleneck. The bottleneck is inside at the centre. I started my first company named Crescendo Networks that does, I would say, application acceleration plus balance capability at the front end of web servers and actually accelerate web servers. This company was eventually sold to a F5 in 2008 and one of the board member in Crescendo was Ofer Bengal, my co-founder in Redis. Back then we realized that after we solved the bottleneck at the front end of the data centre, the web server, the application server, we realized that the main bottleneck inside where the data is, which is the database, and this is where we started to think about Redis. At that time that was early 2010.

We started a company in 2011. There was a technology called Memcached which is an open source that does cashing for databases and we found a lot of holes in the Memcached. For instance, every time you scale it you lose data or it doesn’t have replication, et cetera. We thought “What is the best way to solve all these walls at the technology level?” Then we realized that there is a new project which is getting momentum. It is called Redis. It has replication built in. Back then it didn’t have  scaling capabilities but we managed to solve it somehow and then we created the Redis Company. Back then it was different name. It was called Garantia Data and we offered Memcached while the backend was the Redis infrastructure. Now when we launched the service and it was the first database as a service ever, I think, in the middle of 2012, all of a sudden people were asking us, “Okay, Memcached is great but what about Redis?”

We said to ourself, “The entire infrastructure is Redis. Let’s also open the interface of Redis and offer two services.” So we did it early 2013. We offered Redis as a service and Memcached as a service. It took us one year to understand that the market wants Redis, not anymore Memcached and this is the story I would say. Few years afterwards we asked Salvatore Sanfilippo who is the creator of Redis open source to join forces and to work for Redis Company because we believe it’ll help the community and to grow Redis. He accepted the offer and became part of our team, is still part of our team. Redis as a company, in addition to us contributing to the open source, the majority of the contributions coming from our employees, we also have a commercial product, Redis as a service by Redis the company. It is called Redis Enterprise Cloud as well as the Redis Enterprise Software that people can deploy on-prem.

Few words about the company, we are almost 800 employee distributed across multiple geo regions and of course continue in supporting the open source in a friendly open manner. We’ve well defined committee that is composed from people from registered company, a lady from AWS named Madelyn and a guy from Alibaba, Zhao Zhao and freed two leaders of the open source are from Redis, Yossi and Oran and another guy Itamar is also from Redis, so like five people, the committee of the open source in addition to all the commercial products that we have. But we are able to talk about developer culture. So I’ll let you ask the questions.

Shane Hastie: Thank you. So how do you build a collaborative culture that both supports the open source community but also does bring the for-profit motive of a commercial enterprise?

Supporting open-source and commercial needs [05:08]

Yiftach Shoolman: So let’s start with giving you some numbers. I cannot expose our commercial numbers because we are private company, but just for our audience to know. Redis is downloaded more than 5 million time a day. If you combine all the download alternative like going to register and downloading the code, launching dock here with Redis, et cetera, this is MOS and all the other databases together. Amazing, this is because of many two reasons. First, Redis is very simple to work with. Second, it solves a real problem. You cannot practically guarantee any real time application without using Redis, whether use it as a cache or whether use it as your primary database. We as a commercial company of many, many customers, were using it as a primary database. On our cloud service metrics that I can share with you, we have over 1.6 million databases that were launched, which is a huge number of course, all the public clouds like AWS, GCP and Azure.

This is just to show the popularity of Redis. We tried and I think so far we didn’t try it to be able to make the core of Redis completely open source BSD so people can use it everywhere, even create a competitive product to our commercial offering which you can see that all the large CSP cloud service provider provide ready as a service, which is fine. We are competing with them. We think we are better, but it’s up to the user to decide. This is a huge community of users but also contributors. If you look at the number of active contributors to Redis, they think it is close to 1000. I think we discuss it well in our conversation prior to the recording, just for everyone to understand what it means to manage an open source. It’s not just go to GitHub and open source it and people can contribute whatever they like.

When you manage a critical component such as Redis which is the core reason for your real time performance, you just cannot allow any PR out there to be accepted. First, some of the PR are not relevant to the roadmap of the border. Some of them need to be rewritten because they may expose to security breach or they may contain bugs, et cetera. Someone needs to mentor everyone who is contributing and we allocate our best people to do that, which is a huge work. It’s a huge work. Think about the group of your best engineers on a daily basis looking at the PRs the community guys are providing. They’re viewing them, providing comment and then making sure that this can be part of the next version of Redis.

By doing so, we believe that we increase the popularity of Redis, increase the usage of Redis and increase the ambition of developers to develop, to contribute to Redis because people would like to be a contributor to Redis because it’s very populated, well written, it’s a great technology and you want to publish in your LinkedIn profile or whenever you go to an interview that you are a contributor to such a great success. So this is how we do it.

Source Available licenses [08:17]

You ask on the commercial side. There are some capabilities that we keep only on the commercial license that we provide with Redis Enterprise. I think by 2018 we were the first company launched what is called Source Available License, which is a new version I would say of open-source software. It allows you the freedom to change it, to use it for yourself without contributing back, et cetera but it doesn’t allow you to create a competitive commercial product to Redis the company. We did it only on certain component of the open source, not the core, on the modules that extend the Redis capabilities to new use cases like Search and document with JSON and Graph and TimeSeries and even vector similarity search and AI. We put this under the source available license so people can use it for free in the open source wherever they want, but if you want to create a commercial product, we say you cannot use our code for that. This is how we monetize. We have the free deals of licensing, open source, source available and closed source.

Shane Hastie: You described putting your best engineers onto reviewing the open source, managing the pull request and so forth. How do you nurture and build that community and how do you, I want to say incentivize, motivate your teams?

Nurturing the community and motivating contributors [09:45]

Yiftach Shoolman: Yeah, great question. I think it is out to maintain, to be able to be successful for such a long time. If you look at the history of Redis, almost any other database out there used to pitch against Redis because the other databases found that Redis is so popular and can solve a lot of the problems that their commercial product was designed for. It actually put a risk on their business.

I think the magic of Redis with this large community of user is that we don’t stop innovating. We don’t stop contributing code and we build an architecture that allows people to extend it for their own use cases. By the way, like I mentioned with the modules that we decided to commercialize and at least the company, Redis, whenever we hire developers, we tell them, “Listen guys, if you find a problem that you think it is meaningful and you want to create a product out of it, the Redis platform is the right platform for you. Just do it and if we find it attractive in the market and we see good reaction from customers and community, we’ll support you. We’ll create a team around you. You will live it, et cetera.

As a result of this, we created a lot of very good ideas and product like this Search, JSON, the Graph, the IimeSeries, the Probabilistic, the AI. All these were created by the developers who found real problems and this is what we do internally. I know that there are other companies who are doing it internally for their own use cases based on the open source so these provide it as a platform. It’s a core platform, open core that allows you to extend it with your own core and you can decide whether you want this core to be open source, source available or closed source even. The platform allows you to do that. If it was a very closed platform, limited capabilities without any contribution from outside, et cetera, there are open sources like this, the adoption would’ve been very, very limited.

Shane Hastie: And you talk about hiring developers, how do you build great teams?

Building great teams [11:56]

Yiftach Shoolman: Great question. I think culturewise, first of all, in term of where to hire engineers, we don’t care. If there is someone that wants to work for us as a company or with us, the partner of our team, if we find a match between his desires and what we need anywhere in any place in the world so we are completely distributed. On the other end, we would like to maintain the culture of Redis which is humble design, humble people. I would say simple design but very, very unique in the way it was written, very clear, well documented. This is how we want our developers to react. Be very clear. Be very transparent. Expose the things that bother you and be very precise in how you document the issues that you encounter with. I think if we manage to maintain this culture for a very long time… I forgot to mention what I mentioned before, be innovative if you find the problems that you think it is a real problem.

Redis was designed when it started 2009. It is almost 13 years ago. The technology world has changed dramatically. The basic design of Redis is great like simple, shared-nothing architecture, scale horizontally and vertically as you want with multi processes, not multi-threaded, et cetera. Again, shared-nothing architecture but if for instance there is a problem that can only be solved with multi-threaded, this is fine. Let’s sync together how we can change the architecture of Redis and allow it to scale also horizontally. Okay, this is just one example.

If there are other, for instance, Redis is written in C and if someone wants to extend Redis with other capabilities with modules and it wants to use Rust because Rust is becoming very popular, it runs as fast as C almost, but it protects you from memory leak and under memory corruption bug, we provide a way to do it. You can do it. You can extend the code with fast functionality to solve any problem you want. The code will continue to be written in C, but you want the extension to be with Rust or other language like Python or Java or whatever or Go, go ahead and do that.

So this diversity I would say of programming languages that can support the core is very good for Redis. In addition, I would say there is a huge ecosystem that the community built around clients like clients that utilize the capabilities of Redis and are integrated part of any applications that people are developing today.

There are 200 different clients for Redis. Someone can say, “Listen this is too much.” I can agree with that as well. This is why for our customer we recommended I would say full set of clients across the different languages that we maintain together with the community. We guarantee that the bugs are fixed, et cetera. On the other end, if someone find is that what we provide is not good enough and you want to do something with other language, et cetera or just want to innovate something, go ahead and do that. We will support you by promoting it in the developer side of Redis, redis.io. If it is good, you will pick up stars very, very nicely and you will see adoption.

There are multiple examples in which people created additions to Redis, not only on the core server but also on the client side or on the ecosystem. This became a very popular open source software and some of them are also maintained commercially. They provide open source client for some language with commercial supports that they provide and they created a small business around it. Some of them even created bigger business world, which is good. This is how open source should work.

Shane Hastie: So looking back over the last 12 years and beyond, what are the significant changes that you have seen in the challenges that developers face?

The evolving challenges developers face [16:04]

Yiftach Shoolman: I think no question. The first, the foremost is the cloud. Today’s are developers who start developing in the clouds. They don’t even put the code on the laptop. I still think this is the minority, but the new wave of developers will go to some of the developer platform out there and start developing immediately in the cloud that was not there before, so this is one thing.

Second, they think from the database perspective, it is no more relational. In fact, if you look at the trends in the market, the number of new applications that are developed with relational databases declines. More and more applications are now starting new model data models that are provided by the No-SQL of the world or the multi-model of the world, like key-value document, graph, time series with search capabilities, et cetera. Unstructured data can scale linearly without all the restrictions of schema, et cetera. This is another trend.

The third trend, I would say, it’s real time. A lot of application need the real time speed today and real time has two areas, I would say. First I think the more significant one is how fast you access the data and this is where Redis shine. As a general concept, we want all the operation complex or simple and ready to build in less than a millisecond speed because otherwise you just break the real time experience of the end user. This is just one part. The other part is how far the application is being deployed from the end user because if for instance, the application works great and inside the data centre the experience is less than 50 millisecond which include the application and the database and everything in between, but your distance from your application is 200 millisecond, you will not get the written experience.

And I think the trends that we see today specifically by the cloud provider but also by a CDN vendor, is that they try to develop a lot of edge data centre and data centre across the regions worldwide in order to be closer as possible to the user because if you think on next generation applications, the ways will be deployed is completely globally between that hour and that hour, the majority of the user will be from Europe. Let’s build the clusters there. With Kubernetes it’s easy to do that because you can scale it very nicely and then shrinking down doing the night time and scale out your cluster in South America because this is where the users are there. But applications cannot work without data and the data needs to be real time, so Redis is used in these use cases and it should be globally deployed, et cetera. This is a sync game. Another trend that we see today in the market.

We mentioned cloud, we mentioned the multi-model, we mentioned the real time, we mentioned the globally distributed need. Last but not least, you will ask it, async AI will be a component of every application stack soon if not now. Whether this is features for feature store, feature stories, a way for you to enrich your data before you do the influencing of AI, use the store in order to add more information to the data, receive from the user in order to be more accurate in your AI results or whether this is vector similarity search which also supported by Redis. Just to make sure that we align with this definition, so vector allows you to put the entire context about the user in a vector and almost every application needs it in addition to the regular data about the user.

I don’t know your hobbies or things that related to context, what you like, what you prefer, et cetera. You can do recommendation engine. You can do a lot of stuff with this. Eventually if you look at the next generation of profile of user in any database, they will have the regular stuff like where do you live, your age, your height, and then a vector about yourself, which any application will decide how to formulate the vector. Then if you would like to see user with similar capabilities, you will look for vector similarity search inside your database. I truly believe this is a huge trend in next generation application and the way databases should be built.

Shane Hastie: I can hear screams about privacy and how do we do this ethically.

Privacy challenges from vector data [20:40]

Yiftach Shoolman: You’re right. It’s a good question to ask. How do you do it without exposing privacy? It’s a challenge. I totally agree it’s a challenge. By the way, from our perspective at Redis, we don’t really deal with that because it’s up to the application to decide which vectors they put in the database but once they put it there, we will make sure that if you would like to find similar users based on vector which is no one can read vectors. It’s a floating point, it’s coded. No one can understand from it anything about the user unless you wrote the application. I think in term of privacy, this is more secure than any other characteristics that you put on the user but based on the fact that the vector is there and under all the privacy compliance, et cetera, you can search for similar vectors in Redis very, very fast. This is what I’m trying to say.

Shane Hastie: On leadership perspective, how do you grow good leaders in technology teams?

Growing leaders in technology teams [21:42]

Yiftach Shoolman: As a technology company, we always have these guys in the technologies that want to be great managers and there is another type of people who want to be great technologists. They less want to manage others. They want to invest deeper in the technology and be a kind of architect. As a company we allow people to grow on both direction. It’s up to them to decide whether they want to be manager that mean less closer to the code, probably in some cases less closer to the technology, to the day-to-day technology, not in general and the other wants to continue doing it but less managing people. We allow people to grow on both parts and I think other companies are doing it as well, but we always want, even if you select to be a manager in the technology, we want you to be closer to the technology in order to take the right decisions.

I mean it’s a different space. It changes so quickly. One of the things that differentiate I think software programming than any other profession out there is the gap between great developers, I would say mediocre developer, not the poor one can be 100 time in execution. This is why we would like to hire great developers. We understand that. To manage these people, sometimes you need to be flexible because they are smart at their own perspective, not only on the code and technology but what is going on in the world. How we can be better citizen in our countries and in the world in general. By the way, the spirit of the open source is exactly that. It’s exactly that. Contribute to other. It’s not only yours. People can take it and do something else. Encourage people to do other use cases, not the things that you thought about only.

You should have this culture to accept these people because even if they grow… I don’t know in the morning, to run for four hours and they start. They work at 11:00 AM and they work until 5:00 PM but they’re excellent in what they do. The deliveries that they can provide is huge, huge, huge benefit to the companies that they work for. This is why we try not to put limitations there. I think our leaders, our managers, of course you need to stand behind the milestones and the promises, et cetera. One of the ways to do it is to be able to accept that some of your employees are exceptional and they work differently and you cannot use the same boundaries to everyone. This is a thing that we are taking care of.

Shane Hastie: Looking back over the last couple of years, there are industry statistics out there that developer experience hasn’t been great. Burnout figures are at 80 plus percent, the great resignation, all of these things happening, mental health being a challenge and general developer experience. How do we make these people’s lives better?

Supporting teams through developer experience [24:46]

Yiftach Shoolman: I think it’s a great question in here. I think also you will find out different answer from different people because eventually a great developer experience allows you to write your code even without writing code. You would agree like low code, you don’t need to do anything. Just few clicks here, few clicks there, no compilation, click the button, it is running in the cloud. I have an application. Not all developers want to work like this. Some of them want to understand what is running behind the scene, what is the overhead associated with the fact that you abstract so many things for me and they want to deep dive and understand the details in order to create faster application, better application or paper so I think you always need to give this balance. One of the thing, and by the way, I didn’t mention it when you asked me about the trends, and I think at least at the database space, there is what is called today a serverless concept.

I must say, and you ask me not to do it, but I’ll do it, that we started with this concept in 2011 when we started the company. We said, “Why should developer when it comes to the cloud, know about instances, know about cluster, know about everything that the outer world that is going there.” If you look at the number of instances that each cloud provider provides today, these are a hundred and every year there is a new generation, etc. Why should as a developer, I should care about it. The only thing I care about when I access the database are two things. How many operations per second I would like to make and what is the memory limit or the dataset size that I would like to work with? You was a database provider should challenge me according to the combination of both, how many operation, what is the side of my data set?

If I run more operation, you should be smart. The database as a service, you should be smart to scale out or scale in the cluster and eventually charge me according to the actual uses that I use. We invested, we started with it as I mentioned earlier, we don’t want the customer to deal with no cluster, how many CPUs, et cetera, care about your Redis and we will charge you only for the amount of request that you use and the memories that you use. This is becoming very popular. We are not the only one who provide serverless solution today and it makes the developer experience very easy because you just have an end point. You do whatever you want and eventually if needed you will be charged for a few dollar a month, so maybe more depending on your usage. That said, the other developers really like to fill the other way even if it is VM, even if it’s Docker to fill the software, to load it, to understand exactly how it uses the CPU, et cetera.

As a company, we also provide them a way to work with our software and do that and of course, they can take the source available element and do it themself. I think we are trying, I would say, to give a lot of options for developers across the spectrum not only serverless, but if someone wants more details, go manage it yourself. Understand what is running behind the scene. We will help you to run it better.

Shane Hastie: Yiftach, thank you very very much, some really interesting concepts there and some good advice. If people want to continue the conversation, where do they find you?

Yiftach Shoolman: So you can find me on Twitter, I’m @yiftachsh or you can of course find me on LinkedIn. Look for Yiftach Shoolman. Look for Redis. If you want to start developing with Redis, go to redis.io or redis.com. You will find a lot of material about this great technology and this is it.

Shane Hastie: Thank you so much. Thank

Yiftach Shoolman: Thank you. It was great talking with you Shane.

Mentioned: