Author: Olalekan Elesin
MMS • Olalekan Elesin
Transcript
Elesin: My name is Olalekan. I was the director of engineering, now I’m VP of engineering. The title says, elevate developer experience with generative AI capabilities on AWS.
We go through, we introduce Amazon Bedrock, code review assistant, agentic code generation, code summarization, and I have a bonus material which I would share with you. Personally, I’m a fan of football. I would show you an example of, I was watching a game, and then in 20 minutes or so, built a simple application.
Introduction to Amazon Bedrock
What is Amazon Bedrock? It’s a fully managed service. I’m not, as I said, maybe AWS hero, but I just want to introduce you to this. Not a marketing slide. I would also give a critique of my own of this service, or AWS services that I have used to elevate developer experiences at my workplace. Fully managed service comes with foundation models out of the box from leading companies, AI21, Anthropic, and things like that. Key features, Bedrock Studio, Knowledge Bases.
I’m not here to talk about Bedrock, but I’m just trying to give you an introduction to it and how we can use it to elevate developer experience, which for me is very important. If you have access to an AWS console, you look at Bedrock, this is what it looks like. It’s why Meta, which is also a very key contributor in the open-source community for the development of foundational models or large language models. This is where it gets interesting, everyday developer experience. For me, this is what I care about, not Bedrock.
Code Review Assistant with Amazon Bedrock
As an engineer, who loves code reviews? If you’re an engineer, you love code reviews, you’re from Mars, everybody else is from Earth. Let’s take that. I still do code reviews, and at every point in time I’m wondering, why do I have to do this? If you look at the coding flow, there is 19 hours of coding on average, pickup time takes about 9 hours on average. Review takes five days because, if someone says, yes, in this daily standup, I need you to review this. Don’t worry, I’ll pick it up. Who picks it up? Nobody. I also do the same, nobody picks it up. Then the deployment time is on an average 80 minutes. There is a lot of work getting stuck in the review phase.
The value that we as engineers create is that we are very excited when the value that we create with code, when the problems that we solve with code, gets into hand of customer. For me, this is very important. How do we reduce this? One day, I was super pissed at code reviews, and I realized that the engineers on my teams also were struggling with code reviews. I sat and said, how can I solve this? Then I came up with this architecture. Fortunately, it’s quite simple, so you look at a third-party Git repository, think of GitHub or whichever one. Created a webhook with API gateway, which goes to a lambda function, sends the information to Bedrock.
Once you create a pull request, it triggers the webhook. Goes straight, does the review of the code, goes back and comments, and does review. This is something that can shorten that nine hours plus five days to minutes, if not seconds. I was super excited on working on this, only to realize that people already built this on GitHub, and they’re putting in the GitHub Marketplace.
Then again, this is something that can take you from nine hours plus five days, into minutes. It doesn’t mean that the reviews are super perfect, but it gets you up to speed as quickly as possible. Now think of the webhook, and think of how you can extend this use case within your organization. You can also exchange Amazon Bedrock. You can change it to Mistral code assistant, you can change it to Claude code assistant. You can change it to anything that you want. It’s simple. Gets you up to speed as quickly as possible.
Agentic Code Generation with Amazon Q Developer
Who loves moving from Java 8 to Java 17 and Java 19 or 21? Who has ever embarked on that journey where we say, we’re running on Java 8, we have CVE issues, now we need to go from 8 to 95? Who enjoys that? I think there was some Log4j issue in, I think 2022, where lots of people had old Java versions that we had to migrate. This was really interesting for a lot of us. If I see this, I know I’m excited. Who’s excited, to do code migrations? Here comes Amazon Q Developer. Amazon has mentioned a lot about this. They’ve done migrations on their own, they’ve done 1000-plus migrations. I can tell you that this actually works, from my own personal test.
What I did not like about the Amazon Q was that it took us months to provision. The reason, you have to connect it to AWS SSO, which for most of us, in our organizations, we already have an SSO provider. What we did was not to use Amazon Q Developer, but we used GitHub Copilot. I cannot share the stats, but I will share with you what I observed with the engineers that used Copilot in my teams and engineers that didn’t use Copilot.
Every morning we use our instant messaging tool, and then I see at least four engineers sending pull requests every day saying, please review this. It turns out that these engineers are actually using GitHub Copilot. When we ran a survey with them, estimated again, they had between 15% to 30% efficiency gain by using Copilot. Then I asked myself, the colleagues in suits saying, in five years, engineers will go away. I don’t think so. I think in five years, engineers will become better at the craft. We will be problem solvers.
One of the things that I noticed with the engineers that used Copilot or AI assistants in writing code was that they became better problem solvers than code implementers. It can be Amazon Q. It can be GitHub Copilot. It can be Cursor. What I’m trying to say is that, as engineers, you can start introducing this to your workflow, as long as it aligns with the security policy of your company.
Let me see if I play this video. I’m actually a Manchester United fan. When I was preparing for this, was one of the football games on a Saturday afternoon. I wanted to watch the football match, and at the same time, I wanted to prepare for this. I wrote a prompt saying, give me this particular project that would perform this particular activity.
It generates the sequence of tasks as an LLM agent, and here I am for about five minutes just sitting and watching Manchester United. This is what we do as engineers. We tell the product manager and the engineering manager that, yes, I’m working remotely, just give me a bit of time. We’re chilling, sipping some coffee, whatever you drink. It’s October 1st, so you can get the idea.
Then, you wait for the LLM to generate the code. Definitely it generates code, but it doesn’t mean it’s accurate. It means that you still have to look through it by guiding it through the problem you’re trying to solve. What do I do after? That’s it. It generates my CloudFormation template. Generates the code itself, and I spend maybe another 10, 15, 20 minutes editing to make sure it aligns, and testing it. What would have taken me about three hours, I got it done watching a football match, and also in 20 minutes. Again, better problem solvers as engineers. Engineers are not going to be replaced in five years, we’re going to evolve and be better at the craft.
Code Explanation and Summarization
Code explanation and summarization. How many of us have people joining our teams regularly? Tell me if you actually enjoy explaining to them how the code works. Nobody does that. Is there anybody that loves to do that, to explain? I don’t enjoy it. When I joined HRS, I led the data platform team, and we built it from scratch together, the Kinesis data ingestion and data lake, and S3, and here we have about four years after someone was asking me a question, and I’d left the team. I was like, how do I start explaining how this works? I got tired.
Then comes code summarization and explanation. What I’m showing you right now is an architectural blueprint that you can give a try to. We have repositories that exist on GitLab, Bitbucket, GitHub, you can put that in an S3 bucket securely, making sure that there is no public access. That’s very important. Then put that in the Amazon knowledge base with OpenSearch, either the OpenSearch with instance, or OpenSearch Serverless, and then with a foundation model, the new team members can ask questions to it. Very easy.
Even within GitHub Copilot and Amazon Bedrock, you can also have that in there, in the code repository itself. They can highlight the code and ask you to explain. This way, you can easily generate documentation that’s really closest to the code as much as possible. You can have interesting automation in here.
Example, think about when you do a git push, you can attach a webhook to it to automatically update the documentation and make publishing it as an HTML, where they can go to and see. Think of platform engineering teams as well that are maybe publishing CDK templates or CloudFormation templates or Terraform templates. These can really help onboard engineers into what you’re building. This is something that you can get started with without writing as much code as possible.
Support Case Investigation
This is the bonus material. I work in a B2B environment. How many of us get someone saying there was a customer somewhere that complained about an issue in production, but our infrastructure monitoring never picked it up? What happens is, when a customer raises an issue, sometimes it takes a couple of days for us to find the underlying issue. One of the things that we realized was that the information about the issue reported by the customer exists in different systems. If you’re not using super complicated architecture, super complicated login systems, it might be difficult to find.
Then the customer service agent with the application support, the product manager are all pissed and wondering, why can’t you find it? Here is a simple architecture. Let’s say you have a case management system as an example. Someone logs in and creates a case saying, we have this issue in front of customer x, customer y, you can easily trigger a webhook similar to the git integration earlier, which invokes an agent. What that agent does is that it goes through your login system.
You can do that with Python or Java SDK, both with three SDKs, saying, search CloudWatch with CloudWatch Insights for this particular string. If you use Prometheus, you can use OpenTelemetry to query it. If you have a database that you have read only access, you can also put that in place to do a read.
Then, once you find all that information, what LLMs are good at is that they are good at really synthesizing unstructured data. What inspired this was what I described earlier, but I think late last year, just to give you a bit of background context, we had colleagues in our HR department. They run surveys multiple times every year where they ask colleagues, how do you feel about the onboarding process? How do you feel about this? This is give or take, they receive 240 responses, unstructured, in an Excel file. Who can go through 240 and still have not drank maybe five or six cups of double espresso?
What I did was to sit with the HR colleague, this is to say that, even with using LLMs as elevating developer experience, you can also extend to other parts of the organization. Sat with the HR colleague and said, let’s look at all these files. Let’s put them in a large language model, and let’s map the result of this file to our onboarding process. What would have taken seven hours, we did it in five minutes.
Again, developers are not going away. We’re here to stay. Tell the suits that. What I’m saying here is, support case triggers a webhook, API gateway invokes a lambda function, queries all possible locations that you have your logs, and then sends the information back into the support case application. What used to take about three days, nine hours, whatever time, with this approach, you can do it in five minutes.
Then nobody has to sit on my neck and start asking me, where’s the answer to the question. This is very important. Like I said, you can think about multiple use cases and try to reshape this based on your organizational context, your current problems, but I can tell you that this would elevate your experience as an engineer, because now you focus on the actual problem solving. It’s good to explain code to the colleagues, but what we love is to create value with software.
Questions and Answers
Participant 1: Have you gathered any experience using GenAI for when you have a refactoring, like the upgrade you mentioned, that you have to roll out across 200 services, and services are different enough that you cannot really script it. Have you ever had any use cases like that?
Elesin: Yes. One example was that we wanted to put a REST interface in front of an internal gRPC protocol. We put it in GenAI, and it messed up bad. What we did was to think about the problem we’re trying to solve and then write it by ourselves. We couldn’t trust GenAI in that regard. In some cases, it behaves really well. In this case, it was really terrible.
In fact, I could not believe the result myself, because then I had to sit on the call with the engineer, saying, can you please show me which went through? I was like, we have to do this on our own. I know, it happens. It’s not perfect, but what it does is that it gets us a head start.
Participant 1: If you have to roll that out across a lot of repositories, you could probably make the change once, show it to the AI and tell it to do the same thing everywhere else.
Elesin: At least what I know is that in Amazon Q Developer, you can integrate your own code and train the underlying model with your own code, but due to our own security compliance, we haven’t done that.
Participant 2: Can you give me an idea of what an AI code review looks like? You make a 2000-line code pull request, what does it say?
Elesin: One of the things that it looks like is that it doesn’t have context of the entire application itself. What we realize is that with the new models, which say they have 200,000 token size, which is about a book of maybe about 500 to 600 pages, you can actually give it your entire book itself, and then it’s able to understand the context. You can give it the entire repository, and then it understands the context. What it does is that it says this particular line behaves like this. This is the expected result, maybe adjust it in this way to simplify the runtime.
Then it comments one after the other that way. Very human readable and very understandable. Like I said earlier, I have a team that is using this integration as we have it, and every day it’s about four to five pull requests, anytime I open the instant messaging. On the other hand, we have teams that are not using this, so I did more like A/B testing with the team, and this is the result.
Participant 3: How much do you spend approximately on the tokens for inaudible 00:21:13. You might not need to measure it, because you’re free on tokens, I assume. Was it more in the dollars or the tens of dollars? Because the repository can get arbitrarily long when it comes to the tokens you need to consume to even embed it.
Elesin: I didn’t see it pop up in my AWS costs. I know that if I exceed the budgets, my FinOps colleagues will reach out to me. So far, no.
Participant 4: Could you share some rookie mistakes which you did when starting to build on an AWS Bedrock. What are the things which you would recommend if I’m starting out now, not to do?
Elesin: What not to do when starting out with Bedrock? Number one, check with your security, check with your compliance. Number two, and this is my personal understanding, anything that we consider as high proprietary company information, intellectual property, don’t put it in there unless you have the compliance check from your security department. I don’t think it’s more of don’t do, but I think it’s more to do. What I would say to do is to try it out once you have some validation, immediately. We’re in business travel as HRS, and then we’re also heavy on sustainability, which is helping companies to align their company strategy when it comes to business travel with building a sustainable planet.
One of the things that we have, it’s called a Green Stay, which is about sustainability classing for the hotels that work with us. That information is very voluminous. One of the engineers on the team asked me a question, how can we simplify this? I said, you don’t need to train an AI model, so put all this information in a Word document, because it’s public information.
Let’s go to the Bedrock UI, put it in there, let’s start asking it questions. We have the version of that chatbot in a development area where someone could log into and actually play with it. This engineer had no experience with machine learning at all or AI at all. Check security. Get started immediately. Get into validation as quickly as possible. Do’s, not so many don’ts.
Losio: You mentioned before, one example is, I have Java 8, I want to bring it to Java 17, whatever it was. The scenario, usually, you have Java 8, or whatever else it is, you have probably a project that’s been running for many years. A person that has been probably quite senior, old, whatever adjective you want to have, a project that maybe has been there for many years. Think of that scenario. You want to convince the team to do that, because it’s easy to say, yes, Amazon did it, they migrated 10,000 projects.
Automatically, if you go to the developer team that has been playing with that code for the last 10 years, probably are going to tell you, I’m not going to put my entire code base and see 50 changes in every single class, all committed together tomorrow morning. How do you convince your team to follow you on your machine learning journey?
Elesin: How do you convince experienced engineers to use this tool to accelerate? It’s a difficult one. In my area, we’ve all come to the realization that this journey is almost impossible without some automation in place. That’s the first part. The second is, I had to show. Like I said, we have a team that is using it, and the colleagues see the efficiency gain with the team using it. That’s one of the proof points. The first fact that we came to the realization that this is almost impossible for us to do. The second is that we have a team that is using GenAI in one way or the other, and we’re seeing accelerated value delivery with software.
The question is, why not go in this route? Right now, when I’m hiring engineers on the team, I’m actually expecting them to tell me that they use GenAI to solve a problem, and then we go into detail. To your point, it’s change management. It’s difficult. For me, the communication is that, as an engineer, it’s not about someone getting rid of you in five years. It’s about you evolving to solve problems faster and get value into the hands of the customer quicker. It takes time, but yes, once they see the benefit, then they begin to understand. Because I myself also demonstrate how to use it. I think that’s what I do.
Apart from the developer experience, I also do product managers as well. Who loves estimations? We have refinement, now we have to do estimate. Then you get into the meeting the product owner or the product manager says, I’m not sure, I’ve broken them down into user stories, but let’s start discussing. Who enjoys those kinds of meetings? I ran a session just to also explain to you that this also takes time, not only from an engineering perspective, but also from product ownership perspective. I had a session with our product owners and said, I know that this is a problem, and I know that you might not trust this software, but let me show you the benefits of this.
First, map out who are the users of this and let’s take the problem that you have, put it into this large language model and say, generate user stories based on this problem, acceptance criteria, and then you’re better prepared to have discussions with the engineers in the refinement sessions. Now we have better refinement sessions because this helps better preparation.
The development experience, from that perspective, it also took time. Because I’ve mentioned it to the colleagues before, it took them time to understand, now they understand this, now it’s accelerating. The change management is difficult, but it is a journey that we keep on trying and improving over time.
Participant 5: You mentioned so many use cases about generative AI. One case you didn’t mention is generation of test cases. Is that it can’t handle Bedrock, or is that you deliberately allow it?
Elesin: I actually did this myself. Because I’m in business travel, like I said, we had a use case where we had so much work for our QA engineer to do, and nobody was taking care of this. Because I understood the context of the problem we wanted solve, we simply took that and put it into the large language model, in this case I think it’s Bedrock, one of the models on it, closed source models, and it generated the test cases. It was then based on this test case that we created tasks for other people, so that we could parallelize the work. It’s possible, and I’ve actually done it myself. It does work.
Participant 5: If it does, then how can it generate test oracles for your program? How does it know that it is doing what it’s supposed to do? It doesn’t know. You feed in code, which could be wrong, but that the test case needs to be coming from the requirements, for example, that A plus B is equal to C, and you have programmed A plus B is equal to D, how the Bedrock or any algorithm would know that actually it needs to write the test?
Elesin: When it comes to generative AI, it’s generative. Then, for it to generate something, you need to give it a prompt. In my case, I understood the context of what was expected. There were policies with regards to the expected behavior of the system which I had clear understanding of. It was basically my clear understanding of the problem. I said, this is the expected outcome of the user, this is the expected outcome for the problem itself.
Based on this, generate test cases. Then we looked at the test cases, some of them made sense, quite a number of them didn’t make any sense. Then we ruled out the ones that didn’t make sense. The one that made sense, we simply took them and said, let’s create the work for the colleagues to do.
Participant 5: You need a human in the loop?
Elesin: Definitely, yes. You saw, it was starting from engineers, then people are there. There’s always a human in there. In this case, the human was me.
Participant 6: Maybe one question on the specific topic you had with the repository and embedding it. How did you handle the generally weird structure of repositories being deeply nested. Because normally if you embed a document, you have this one embedding for this one document. Did you start building a hierarchy, or is it something that’s just natively handled in Bedrock?
Elesin: I didn’t care. I simply uploaded it into S3 but I excluded the irrelevant parts. What RAG, retrieval augmented generation, does is that it converts all text form into some vector representation and then stores that in OpenSearch. For me, it doesn’t matter if it’s nested or not nested. It’s simply making sure that it goes into the vector database, in this case, OpenSearch.
If you are planning to do maybe something more complicated that might require hierarchical structuring, and I think there have been techniques like the graph RAG, graph which is connecting nodes and edges based on the retrieval augmented generation, you can also try that. For me, I wanted to get it to work as quickly as possible. First phase this. If it requires more fine-tuning based on the work, then you can add that hierarchical structure in there. First, don’t care about the hierarchical structure.
Participant 6: I was just wondering if it handles it internally.
Elesin: Yes, it did it quite well.
Participant 7: Did you measure that, because inaudible 00:33:26 with installing the stuff, at least for Java code, not using Amazon but customly just build embeddings and just started to search through the Java code, and it provided completely unsatisfactory results. Because if you just think about Java code, they are just tokens that are not relevant to the language directly and just there are indeed should be some techniques, you overcome these limitations. Basically, yes, it’s interesting. What were the results? Basically, at the end these new team members were satisfied.
Elesin: How do you measure the accuracy of the response from large language models. Because it’s not a classical machine learning problem, that is, you have an outcome or an output that you can predict. It’s difficult to say the accuracy is this. This is another point where you need the human in the loop to do a thumbs up or a thumbs down, and say, this was relevant, this was not relevant. In this case, it was 60% relevant for the queries that we issued. In some cases, the remaining 40%, it wasn’t relevant at all because it was missing the context.
Especially we saw this in the pull request example, where, when we first tried the pull request use case, we were only giving it the snippet of the code that was written, not the entire context of the classes that were involved. That’s the case. In the 60% of this time, about 60%, there was a thumbs up that the responses were relevant.
Participant 8: You mentioned about the test scenarios generation, which is nice, during the refinement sessions, they will help. Is it also helping scripting the tests, such as that automated test, because there are some development environments which generate unit tests very easily, but in terms of unit test or API layer automated tests, it would be really useful to have a level of automation where afterwards developers can make changes and make it workable. Maybe also connected question in terms of test automation coverage, because if it is intelligent enough to detect the areas around pull request reviews, maybe also in terms of coverage, it can be useful then.
Elesin: Can it generate unit tests? What’s the unit test coverage? Can it also increase?
The team that uses GitHub Copilot, my expected outcome for them was to increase the unit test coverage of the projects that they have in their purview, the target was 70%. This was really almost impossible for colleagues that joined newly to write unit tests and also still ship value with code. They optimized by using GitHub Copilot to generate unit tests, which then increased our test coverage from around 0 to about 50%. We’re still getting there, but we’re not there yet. It also increases the test coverage. I think they improved efficiency by 15% to 30%, so there’s unit test generation, which is now increasing the unit test coverage of the projects in their purview.
Losio: You mentioned at the beginning that you didn’t use Amazon Q, I think you said, just for the SSO configuration. I was wondering if that’s the only reason, or actually you got better results as well with Copilot. What’s your feeling about it?
Elesin: That was the reason. Now we have it enabled. The main reason, and the only reason we didn’t use it initially, was because of the access reason. Amazon SSO, we had Azure SSO already. Why maintain two things? Because that’s a security exposure on its own. That was the reason. Now we solved that.
The other side of GitHub Copilot is that we didn’t get statistics of the usage, so we didn’t know how many people were using it, and what they were using it for. Now we’re switching to Q. It gives us the cost visibility. It gives us how many times that engineers are accepting the recommendations from Q Developer itself. Gives us the number of times that recommendations are rejected. It also gives us visibility into how the cost is developing. We just started rolling that out. The main reason was the fact that we had to use AWS SSO, which at that point we didn’t want to use.
Losio: I was quite curious about the integration of different services in that sense. I was curious if there was as well a choice of like, try to pick up the best of every provider, or integrate different providers, how it works as well, or it’s more an effort to them.
Elesin: It was more of the security. Now, also to your point, we want to do a comparison to be sure of what to roll out at scale for the entire organization. Because, for us, we want to double down on this as much as possible. It wasn’t the comparison. It was more of the limitation at that point in time.
See more presentations with transcripts
Podcast: Leveraging AI Platforms to Improve Developer Experience – From Personal Hackathon to AI at Scale
MMS • Olalekan Elesin
Transcript
Shane Hastie: Good day, folks. This is Shane Hastie for the InfoQ Engineering Culture Podcast. Today I’m sitting down across 12 times zones with Lekan Elesin. Lekan, welcome. Thanks for taking the time to talk to us today.
Olalekan Elesin: Thanks a lot, Shane. Thank you for having me here. To the audience, my name is Lekan Elesin. I think Shane has done a very good job to pronounce the words quite well, to be honest.
It’s not an easy name to pronounce, I’m from the, let’s say the western part of Nigeria, but I stay in Germany. Been in Germany for about seven years. I’m happy to be here.
Shane Hastie: Thank you. So you’re from the western part of Nigeria, you’re in Germany, that’s quite a long journey in many different ways. Tell us about how you got there, what’s your background?
Introductions [01:14]
Olalekan Elesin: So interesting one. So my background is in computer science. I studied computer science as bachelor. So what happened, I think in 2013, I was… In Nigeria, when you graduated from university, you have one year to serve the country, and before my time, I was listening to Bloomberg News, and there was this story called Big Data. This was 2012, 2013, and it was called the sexiest job on the planet back then. And I felt like doing statistics, the odds of being a very, very good software, let’s say programmer was very slim. So I decided to journey into the path of what most people were not aware of back then, which is big data, and that’s how it started.
So computer science graduates turn out to learn how to, back then, a lot of O’Reilly, a lot of InfoQ summits as well, a lot of watching a lot of tutorials on InfoQ as well. My first job was as a software engineer, PHP, but then I learned scalar, also with tutorials from InfoQ, to be honest, as far back as 2013. And from software engineering, I switched to data engineering, and then had an opportunity to work in telecoms, e-commerce, and then education technology.
In 2017, I had an opportunity to move to Germany in an online marketplace, and after three years on the job, I wanted to get a new opportunity, so I decided to opt for product management. So I was data and software engineer, data engineer, product manager, technical product manager. And when I was the technical product manager for an AI platform at the company back then, at some point, there was a decision to reshape the company, or restructure the company, and there was no need for my position anymore.
So I spoke to my then head of engineering, saying, “I created a lot of tutorials internally for the data scientists to use, so is it possible for me to, let’s say, exclude the content that is really, really tailored for the company, and then publish this information externally?” Because it could be valuable for the developer community outside of the company back then. And he gave me the go ahead, so that was how I started writing about my experience using AWS services for machine learning.
And then at some point, someone reached out to me from AWS saying, “Wow, you’ve written a lot. People are appreciating what you’re doing. Are you interested in the AWS Hero program?” So that was how I got into the AWS Hero program. At that point in time, it was also in the middle of the pandemic, and I was getting tired of product management, and I had another opportunity to move back into technical leadership. No offense to all product managers out there, it was just not my thing, I tried it for about nine months. I had an opportunity to lead a data platform in my current company, HRS Group, which is in business travel. And ever since then, I grew from technical lead, to head of, to director, and then leading an entire business unit along with my product management counterpart. So that’s the journey so far.
Shane Hastie: An interesting pathway. Machine learning, product platforms. At the recent Dev Summit, you spoke about leveraging AI platforms for developer experience.
Olalekan Elesin: Yes.
Shane Hastie: Let dig into that.
Personal hackathon – leveraging AI [04:16]
Olalekan Elesin: Definitely. So the talk, it was a very interesting one, because even though I lead multiple teams, at least 5 or 6 teams, software engineering teams, including data and AI team, I still find myself writing code once in a while, on my weekends, trying to keep up-to-date with what is going on, the new technology, just making sure that I’m up-to-date, so I can have productive discussions with the team leads with me, and also with the engineers. And on a Saturday, while watching… For the New Zealanders, you might not like this, I was watching the football, football in the English Premier League, and I wanted to build something, but I wanted to build something that was new to me, and at the same time, I wanted to leverage, let’s say generative AI tools available.
So I launched my VS Code, and then I remembered I had Amazon Q Developer. So from that, I asked it to create the application from scratch, and I was quite surprised. From that perspective, it made me realize what would’ve taken me maybe 4 or 5 hours, going into two days, because I also have the toddler to run after, took me about 30 minutes while watching a game of football at the same time.
But at that point, I realized this could really elevate developer experience, because as much as possible, we like to be creative as engineers, and also as engineering leaders, we want our team members to be creative, and at the same time, to make sure that we get value to the customers as quickly as possible. So this was where the idea behind the talk started from. So for me, it was quite an interesting experience to see that I could watch football on the weekend, at the same time, build an application that I wanted to try out. So this is where it started from.
Shane Hastie: I recall for myself the first time using some of these generative AI tools to do some “real work”. There was almost a feeling of, “ooh – I feel like I cheated”.
Olalekan Elesin: Exactly. Exactly. It happened to me as well. This one was quite funny, so I cannot remember the actual workflow itself that I was trying to build, or the actual application, but I remember it helped me to write at least 200 lines of cloud formation code to deploy the infrastructure. And I felt like, “I don’t want to think about this”. So I put it to Q Developer, and said, “Please write this. It should be able to connect to X, Y, Z, A, B, C”.
And then it wrote it out, and I felt like, “Oh my, this is cool”. And I’m lazy. And I remember smiling, and my partner is in the house, she was like, “Are you sure what you’re doing is correct?” I’m like, “Yes, it’s working”. To your point, it felt like I’m lazy. I’m not sure who said that, that the best engineers are the lazy ones, because they find ways to make it work at scale, with minimal effort possible.
Shane Hastie: So from that personal hackathon almost, experiencing it from the first time, what’s involved in bringing this into actually doing work?
Shifting from personal hackathon to scaling within teams [07:17]
Olalekan Elesin: Good point. So for me, it was a hackathon, but then to scale it within my teams itself, it’s a different experience entirely. Because on one side, we hear people granting interviews, saying, “There will be no developers in the next five years”. On the other hand, we see that this can really be a collaborative effort between engineers and artificial intelligence, and it has a change impact on people.
And one of the stories I heard when it comes to change was, a couple of years back, there was the compact disk. Even before that, there was the video cassette or whatever it’s called. And I remember having this, and so many people built businesses around renting video cassettes, even where I come from in Nigeria. And all of a sudden, things moved from video cassettes… Even we had people manufacturing spinners or rewinders for these tapes. All of a sudden, we had compact disks, and now we had to have compact disks. That meant that there was a lot of change. A lot of things would’ve happened to people that built their businesses around that. Not necessarily thinking about the digital content, but really about the hardware.
And for me, that was a light bulb moment when I am speaking with my teams to make them understand, “You have a concern with regards to this particular technology, but think about it when it works in collaboration with you. Then you see that you’re more productive, and look at it also…”
For us at HRS, look at it from taking the customer view first. “Look at it from how fast do you want to get features into the hands of the customer to validate every assumption you have based on the line of code you’re writing. And if you have one assistant sitting next to you as a pair programmer, enabling you to do creative work, and also helping you to write the code, why not leverage that?” And for me, the hackathon is, for me, scaling it across multiple people, and working through that change mindset is one of the biggest concerns that I think a lot of people don’t talk about.
Shane Hastie: So what’s involved? How do we make that work?
Olalekan Elesin: So this is an interesting question, and it’s difficult to answer. But my view, because we also started scaling it with my teams right now, and also across the engineering teams at the company, is to educate people about the change as much as possible, and get them to pilot it. So what we’re doing right now is getting a few people that we believe are enthusiastic, so the scientific approach. Believe that if you have a few people that can use this to validate the assumptions, that it’ll help them become more… It would elevate their developer experience, or their experience when doing development.
And after validation, doing a lot of trial and error, a lot of expectation… I don’t want to use the word expectation management, but there’s no better word coming to me right now, because there’s this assumption that AI can write all the code. You have to be aware that when you use it, it will make mistakes like every person when doing things initially, and you have to correct it along the way. And that expectation management, and get people to experiment as much as possible, and as quickly as possible, this is, for me, what is important.
A handful of people, get them to try it, get them to set the expectations, and use those people based on their learnings as multipliers across the organization. Not trying to do the big bang, saying, “Everybody, now go use…” Whatever it is called. But we have a handful of people, we have learning experiences from them, and then those individuals are part of different teams where they can then share their knowledge and their experiences with the team members, and also scale that across the organization.
Shane Hastie: As a software engineer who’s defined themselves by writing the code, you’re shifting from coder to editor.
Olalekan Elesin: Yes.
Shane Hastie: That’s quite a big shift.
Engineers shifting from coder to editor [11:07]
Olalekan Elesin: Definitely, it is. It is. So I remember, my… I think it was my second year in computer science in bachelor’s, that we were introduced to assembly language, or something like that. And we got to the final year, someone was telling me I had to write C#, and I’m trying to bring the relationship between what I learned in second year to what I’m being taught in the final year. And even in the final year, I remember we had to write Java, and more and more, I realized that as computer science has evolved with the years, that people have been building abstractions, let’s say low-level languages.
And this is what is happening even from my perspective also, that there are abstractions. And at this point, the abstraction we are getting into is where everyone, as much as possible, that has an understanding of how to build stuff, that has imaginative capabilities, can leverage these tools to become editors, and not necessarily trying to write the code from scratch. For me, this is it. It’s, we build a craft, we like to write the code, but then we need to build another craft, which is becoming editors, and letting something that can do it better, to a certain extent, to write the code, where we guide it across how we wanted to write the code.
Shane Hastie: You did make the point that these tools make mistakes. They make them very confidently.
Olalekan Elesin: Because they don’t know better.
Shane Hastie: Yes. This implies that that problem solving, mistake finding is another, not necessarily new, but a skill we have to build more strength in.
The tools will make mistakes [12:43]
Olalekan Elesin: Yes, of course. Yes, of course. And while you were asking the question, I remember one of the first production errors I made while being a junior engineer, this was when I was in the telecoms industry doing my first job, we had production on-premise data servers, and we wanted to analyze the log files. And I had done a lot of tutorials about Hadoop, and I didn’t know that I needed to move the data from where it is, which was on the production server, into another server, or another virtual machine which was separated from the production environment. So what I did was to install Hadoop Hive on the production server that was serving millions of transactions, and we had a major incident.
It didn’t crash the business, but it was quite an interesting one, but the learning there was that I could make a mistake as a junior engineer. And this is what our folks out there have to realize, that these tools are still learning. And another key learning I took here from that experience I had was one of the senior engineers explaining to me how not to repeat that mistake next time, and how to solve the problem next time, and guiding me through the understanding of, this is what happened, this is how to solve it, this is how to make sure that it doesn’t happen all over again.
And this is what we see with these tools, where… There are some techniques called retrieval-augmented generation, where you can put your data somewhere so that it doesn’t go off tangent with regards to the recommendations it comes with. And there are some mechanisms where you can also fine-tune an existing model, which is helping it to understand better the context. But until that time where it is really affordable for us, or everyone, to train or fine-tune their own model, these things will make mistakes, and also, we should expect that. I think, for me, this is the biggest part of the expectation management that I do with the engineers that I work with on a daily basis, that it’s expected to make mistakes. You have to be comfortable to see that it can make mistake, and be willing enough to take the time to edit what it’s generating for you.
Shane Hastie: When that mistake is subtle, so it’s not going to cause a syntax error, it’s not going to prevent compilation, but it’s one of those, and we’ve had them for a long, long time, requirements bugs effectively, where, “Oh, it’s solving the wrong problem”. What’s the skillset that we need to develop as humans to find those mistakes?
Building the skillset to ask the right questions [15:11]
Olalekan Elesin: Excellent question. I think the skillset we need to develop as humans is asking the right questions. So when I interview candidates as well, when hiring for my teams, we usually ask them, “How do you deal with requirements that are not clear, or problem statements that are not clear?” And this for me is one of the things that differentiate engineers from developers. Engineers always try to understand the problem, developers think about code.
And this is why, for me personally right now, I’m not seeing this coding assistance as engineers yet. They might evolve to that level, but they are more development assistance. And the reason why I take it in the direction to answer this question is that being able to ask the right question, and this is what people call prompt engineering, is one of the skills that we need to add to what we do. If you have engineers that know how to ask the right questions, they would get a lot more productive by using these tools, because they can then ask it to refine, to change a bit, to adjust what the context of how they’re asking the question to get better responses, that they can then edit from these tools. At least this is what I do. So the skill is prompt engineering, but the underlying thinking behind it is being able to ask the right questions.
Shane Hastie: We hear horror stories of security concerns around these tools. How do we make them secure for our own organizations?
Tackling the potential security risks [16:42]
Olalekan Elesin: This is a very tricky one. First, I think at the QCon developer conference also someone asked a similar question. And my first answer is that security is not there to restrain our space, it’s to enable us to do the right thing. And first and foremost is to make sure that if in any tool that anyone is using in an enterprise setting, please ensure that it is aligned with your security objectives or security policies within the company. This is the first thing.
The second is that we need colleagues to think in terms of the customer, and I think in terms of the business itself. Customers want to have more and more control with regards to their data, so if I, as an engineer, I am responsible towards my customer, and I as an engineering team leader, person leading multiple engineering teams, we have a responsibility towards our customer, and my customer says, “I want my data not to be shared with some AI tool”. Simply don’t do it. Also because it has implications on the reputation of the business if that is not done right. So that responsibility, we all need to live with it, and understand that it has implications on the customer, has implications on the business, also has implications on how we do our jobs by ourselves. For me, this is it.
And I remember when I was building the data platform, and then we said, “We want to be security first”. After that, “No, but this is how to do data responsibly”. It’s the same for me, it’s security first, everything else comes after. As long as it’s satisfying the customer objective with regards to security, wake me up in the middle of the night… This is what I tell anybody, “Security first”. And fortunately, I’m also in Germany, so we’re super cautious about data security.
Shane Hastie: What are the, I want to say future risks, considering developer education, for instance? You learned assembler, are we going to teach new developers assembler, or are we abstracting so far away that we almost don’t know what’s happening underneath?
Developer education needs to evolve [18:47]
Olalekan Elesin: Education definitely needs to evolve to make sure it’s relevant for the generation that we’re educating. That said, anyone being educated needs to be curious with regards to, how did we get here? And that curiosity is another trait of really good engineers. So example, one of my own personal experiences was when I started using Apache Spark for big data processing. I remember going through the Spark GitHub repository just to understand how the people that contributed to that amazing project wrote the code. And this was just being curious.
I also remember going through some parts of Hadoop at some point, some parts of other libraries that I used at some point. And I think that irrespective of the fact that we need to educate people with what is needed right now, those engineers that will be chief engineering editor, or however… Editor engineer, however the title might look like, need to be curious to understand, how did we get here?
When I mentioned that I stumbled into product management, my responsibility back then was… Even though I was building an AI platform as a technical product manager, I needed to understand how this AI platform will enabled the company to have a competitive edge. And I remember something on the book by… I’ve forgotten his name, his last name is Porter, that wrote Competitive Strategy. And this was a book written, I’m not sure, 1990 something, just to study, to really now understand how people thought about competitive strategy a while back. And what I’m saying is, engineers need to be curious with regards to understanding, how did we get to where AI is enabling us to write code and solve problems faster than what people 20 years ago, five years ago, 10 years ago could do?
Shane Hastie: How do we train that curiosity?
How do we train for curiosity? [20:37]
Olalekan Elesin: Oh my God, that’s a really… To be honest, this is a question I ask myself every day. The only way that I’ve seen not to lose it is, we need to find a way to not lose the baby part of our brains, the toddler part of our brains. Where I see my toddler, and he’s asking me, “Daddy, why?” And sometimes I want to answer him, sometimes I’m like, “Maybe I need to help him understand the reason why”. And that curiosity, it’s a trait, difficult to train, but it can be coached.
So in some sessions when I’m having, let’s say one-on-ones with the engineering team leads that work with me, or the skip levels, and they make a statement about something that… A problem that they’re struggling with, the first thing I ask is why. And when I have that discussion, the next thing I do is, Iinform them, “I want you to always have this type of approach to problems”. And so asking why, so that you get as close as possible to the first principles, using the first principle thinking. I’m not sure it’s easy to train, but I know, for me, what has worked is being able to coach it with the people I have direct interactions with.
Shane Hastie: How do we help engineers be coachable?
Helping engineers become coachable [21:52]
Olalekan Elesin: Oh my. There is this book called Billion Dollar Coach written about Bill Campbell, and I remember… I think I stumbled on it two years ago, I think he passed on maybe about three years ago or so. I remember stumbling on the book, and I cannot remember in detail about his approach to coaching people and making people coachable, but what has worked for me, and one of the learnings that I’ve made mine is really explaining the why, and to help the engineers understand how I also think, and making them understand that there is a world before now. And the world before now to get into that means to think in terms of first principles. So what’s the minimum basic element of making this work? And really getting them to really, as much as possible, go back to rethinking, because that’s also the foundation of solving the problem.
Another way I differentiate between engineers and developers when I’m interviewing is, does this person really understand the business problem, or the customer problem they’re solving? And this is how I look out for curiosity in engineers again. And if I don’t see it, I’m happy to share feedback, “I need you to understand the business problem. I need you to understand the customer problem”. And then if I’m talking to skip levels, it’s the week from now, “Send me your detailed understanding of what you see as the problem itself that you’re solving, not the task that you’ve been assigned”. And as leaders, we need to constantly push our engineers in that direction to really realize, “What problem am I trying to solve? Not the lines of code that I need to write”.
Shane Hastie: Lekan, a lot of deep thinking and good ideas here. If people want to continue the conversation, where can they find you?
Olalekan Elesin: They can find me on LinkedIn. It’s only one, but my name is longer on LinkedIn, so it’s Olalekan Fuad Elesin, and I respond to messages. I have put an SLA on it maybe in three days, but as quickly as possible, in less than three days I respond to the messages. I’m also not as active on X, but I’m very active on LinkedIn, and this is where… I use my LinkedIn to share my thoughts and also engage as much as possible.
Shane Hastie: Thank you so much for taking the time to talk to us today.
Olalekan Elesin: Thanks a lot, Shane. Thanks a lot for having me. Super interesting discussion.
Mentioned:
.
From this page you also have access to our recorded show notes. They all have clickable links that will take you directly to that part of the audio.