Author: Loiane Groner

Transcript

Groner: We’re going to talk a little bit about API security. Before we get started, we have to understand why we have to do this. For application security, how many companies are handling this today is, you do your planning, development. Developers will push the PRs, going to do the build. There’s usually a QA environment, testing, UAT, many companies will call this different things. Then, that’s when you’re going to raise your security testing request. You’re going to ask the InfoSec team, please test my application. Let me know if you find any security vulnerabilities. If they find, goes back to the dev team, “We found this security issue. This is a very high risk for our business, and you have to fix it”. Again, goes through the PR, has to go through testing again, rinse, repeat, until you have a clean report or no high-risk vulnerabilities, and you finally can go to production. This has a few caveats.

First, it can cause production delays, because if you have to go through this testing and rinse, repeat, all this cycle until you get a clean report so you can go to production, that can take a while. Or, it’s even worse, companies are not even doing security testing through the software development lifecycle, and they’re doing this once a year, or not doing at all. There’s a very interesting research that was done by the Ponemon Institute, and they say that fixing software defects, or worse, fixing security risks once the product is in production, costs way more than if you are handling that during the development. That’s why in the industry, we say there is a shift left happening, because many years ago, we went through all that cultural change of having unit testing done as part of our development cycle, and now we’re going through this again.

However, we’re talking about security this time. It is much cheaper and much cost effective for the team, for the company as well, for you to handle all those security vulnerabilities, and make sure that your software is secure when you’re doing development. Security has to be from day one. It’s not a technical debt. It’s not something that we’re going to add in the next sprint. It has to be part of your user story. It has to be part of your acceptance criteria. It has to be part of your deliverable. I would like to show you a few things that I’ve learned throughout the years.

My name is Loiane. This talk is from a developer to other developers and leads, so we can go through this cultural change and make sure that security is indeed part of our development phase.

What is API Security?

First of all, whenever we say API security, if you decide to Google this, search this, go into YouTube, try to find a tutorial, you’re going to find a lot of tutorials talking about authentication and authorization, especially if you’re working with Spring Boot. All my examples here are going to be with Java, because this is the technical stack that I’m most familiar with. All the examples you can easily translate them into a different programming language, framework, or platform. Going back to my question, if you go to YouTube and you search by Java security or Spring security, you’ll find a lot of tutorials about authentication and authorization. Security is not only about that. If we take a look at the OWASP Top 10 vulnerabilities that are found each year, and this list is going to change year after year, you find a lot of the same things happening over and again. What I’m going to show you here, at least with all the tips and all these best practices, we can at least make sure that half of this list are not going to happen within our software.

Better Authorization

Let’s go through it first. Let’s suppose that everybody is doing authentication, so at least user password, or you’re using an OAuth service. You’re doing that in your software. We still need to handle authorization, which is why you have to make sure that the user that is trying to access your application or trying to perform a certain action within your application, is indeed able to perform that action. How do we make authorization better within our applications? Let’s start with the first example, with a bad practice. We’re checking here if we can update a course. This is a RESTful API, so we’re doing a Post here. We have the ID. We also have the object, the data that we’re trying to update. I get the user that’s authenticated. I’m checking if this user has a role student. If the user has a role student, they cannot update the course. If I am somebody that I don’t know anything about this application, and I’m reviewing this code, I don’t know who exactly can actually update this record. It’s not clear just reading the code.

A better way of doing that is deny by default. I’m going to write my code, I’m going to write my business logic, and by default, nobody is going to have access to it. What I’m going to do is I’m going to list whoever can actually update it, and everybody else, I’m just going to not allow it. When I read this code now, at least I can see that only admins and only teachers can actually update this record, so it’s a little bit better. The other thing is, the majority of the frameworks that we work with, they do have some support to role-based access control. In Java, for example, we handle a lot of things through annotations. When you’re working with Spring Boot, you do have some annotation that you can easily add all the roles that are actually allowed to do this. We are working with the deny by default approach. You’re free to write your own business logic and leave that part of the authorization, the security check, outside the main business logic. This is great. However, this works perfectly for small systems or systems that you don’t have a lot of roles.

I really wish that my applications were the same, that I watch those YouTube tutorials and I have user or admin. That’s it. That will be a wonderful world. Unfortunately, it’s not like that. What can happen here is role explosion. I’m going to start with the user and admin. Maybe I have a teacher as well, but now it will be really good to have teaching assistants as well. I’m going to grant access to them to my system so they can do a few things on behalf of the teacher as well. Or, maybe we’re working with an eLearning platform, I have an account manager. The account manager will also be able to do those things in my system. We start adding more roles to the system. Now my business logic is only one line of code, and I have more code, just doing the pre-authorization part. It can be, when you’re reading this, not so good, and we can do better. If you’re working with something like this, which actually looks like the projects that I work with, sometimes the authorization level goes to the button that I see on the screen or the link that I can click on the screen. RESTful here, it’s really going to depend on what’s the role that I have, if I’m able to perform that particular action or not. It really depends on the role and all the actions that I’m able to perform.

When we handle situations like this, it is much easier if we have something that is a little bit more dynamic. There are many different ways that you can do this. If we’re using Spring security and Java, of course, you can use a custom security expression. You can design this according to your needs, according to the size of your project and your business. You can maybe have all the mapping, all the authorization within a database or another storage, and you load that, and you have a method or a function that’s going to calculate if the user really has access or not. Of course, annotations for the win. We can actually use the annotation and have our method here with the privilege. Now it’s a little bit more clear for me that the only users that are able to actually perform this action are the ones that have the course update privilege, that’s mapped somewhere else. When we go into those more complex cases, this can be a little bit easier for us. There is no more hardcoding with all those roles within the system.

The other issue that we might face is, I’m logged in. I’m checking if I am authorized or not. Should I have access to update that particular record? If I am a teacher, let’s say we are in university, and there are many classes, should anyone update it? If I know the ID, should I be able to update it, just because I know the ID? I know that some of you are using the incremental identity that is generated automatically by the database. We have to be very careful with that. Again, we still can bypass even if we are authorized to use the system. Be very careful with that, and remember to always deny by default.

How exactly do we make that better? One thing that you can do is, once you have the information, again, you go through the authorization, you have to find a way to check if that particular record can be updated by that particular user. Maybe there is some kind of ID, the course teacher ID, you’re going to match that against the user ID that’s trying to update that record. That way you can make sure that only that that certain user is able to actually perform that action. However, one thing that I see happening a lot, is we’re getting that object, the course object, directly from the request. I still have my ID from the path variable that I’m parsing through my request, but the course that I’m actually checking my logic came from the request. It can be something as simple as using Postman or any other similar tool.

You can manipulate it, or if you’re a little bit more smart, you can use another tool to intercept the request, change the JSON that is being sent to the request, and the ID here might be something. Again, you can bypass any authorization logic that you have and still update that record in the database, and something that should not happen. Never trust the request. When you have to do something like this, always go back to the database or to the true source of the data, the data source. Check for the true data to make sure that that is actually able. There is a tradeoff here. This is going to be a little bit more slower, because we have to go to the data source. There is a request, milliseconds, but again, it is a small tradeoff that we are willing to pay here just to have our APIs more secure.

Property Level Issues

When we are working with objects, there are still a few other issues that we can run into. This has a very fancy name. Just to give you an example, if I have a user and I’m trying to get the data from the user that is logged in, I have a user and password. I’ve done this multiple times myself, exposing the entity directly. Because why am I going to create another class or another object that’s just a copy of my entity, and then I’m going to expose it. This can lead to some issues. In this particular case, if I’m only trying to expose the username, and I have some common sense, and I know that I’m not going to expose the password in the JSON, so using annotations, I can simply annotate my Get method and have a JsonIgnore. What happens if tomorrow we receive another requirement and we have to capture another field, for example, sensitive data such as social security number or something else.

The developer that is working on this unintentionally forgets to annotate the method to get the social security number, and when we’re sending back that information through the request, you are exposing something that you’re not supposed to. This can go through pull request reviews, code reviews, and we’re not going to notice. That can happen. A way that we have to avoid this is creating the data transfer objects or DTOs. You can use records if you’re using a more modern version of Java, or you can just create a class. You have to explicitly tell what are the properties that you want to expose in this case. It’s a much better way of doing that. If tomorrow we get, again, the requirement to add sensitive data to our object, we’re not going to expose because the public contract, I don’t have that information here, and that social security number or whatever other sensitive data that we have to capture, it’s going to stay internally within the system.

Then we can enter into another very good discussion here. Should I create a DTO for a request and have another DTO for a response? Again, this is conflict territory. Each one of us will have their own point of view on this. If you are reusing the same DTO for both requests and responses, just be careful. For example, for the request, do not use the ID, if you have the ID or whatever primary key or whatever unique property that you’re using to identify that object from your DTO when you’re handling requests. This can also slip through the cracks, and then, again, something might happen. It’s always best to have one for request, another one for response. In case you have something again, so you have a metric against duplicated lines of code that you have within a project, be very careful with that.

Password/Key Exposure

Now we’re able to handle authorization a little bit better. The second part will be the password and key exposure. This seems a little bit of common sense. Who here is going to expose the password and commit that to your GitHub repo and have the database password? There are a few different ways that this can still happen. Many companies, you have your URL, and then you have your resource name, something to help to identify the project. Then you are creating a developer database. Again, I really wish my project was the same thing as those tutorials, that I can simply have a MySQL database and a Docker image with two tables, and that’s it. That would be wonderful as well. Especially when you’re working with legacy systems and you have that huge database with maybe hundreds of tables as well, it’s a little bit more complicated, with lots of data as well.

Some of the companies, they will have their own database in a server or in a cloud that everybody is going to access that database. I don’t know about you, but me, personally, I’m not so good with names. That’s the hardest thing to do. How am I going to name a class, a variable? What name do I give to my database? I’m just going to give the company name, or maybe the project name, and dev, to indicate that this is a development environment. I’m going to use prod to indicate that this is production. This can be a little bit dangerous. Then for the password, again, I’m not going to remember all the 30 passwords that I have to do for all the services that we use. I’m just going to use something as well as my learningPlatform@Dev. Then for production, I just change that to production. If something like this gets committed into a repository and somebody sees that information, I wonder what happens if I change this from dev to prod or to another upper environment? Be really careful with that. Never leave passwords or any secrets within your properties YAML file or even hardcoded even for lower environments.

Another issue here, is this last line right here. If you’re working with JPA, if you’re working with Hibernate, there is a way for you that the framework is going to be responsible for checking all the entities that you have in a source code, and it’s going to create all the tables for you. It can create, drop, update. There are many different options. This is a big issue. Never use a user ID that is able to make schema changes in your database. Again, deny by default. You’ll start with, I need read access to my database, because I have my user ID, so you grant that read access. If your application is also writing to the database, then you grant the write access. If you need any access to execute any store procedures, then you add that access as well, but never grant more access than is actually needed. Be very careful with that. This only works for tutorials. This does not work for real applications.

Input Validation

The third part that I would like to bring to your attention is input validation. This seems to be also common sense. This seems to be something that is very basic as well. We are failing in this, for lots of code that we review. We are just not even adding any kind of validation, and we need to start changing that as well. We have our frontend. It’s beautiful, fully validated. I have all the error messages, user experience, chef’s kiss. Then if you take a look at the API that’s feeding that frontend, that’s just this. I have my create method. I have my DTO. There’s nothing. It’s just simple code. This is a big red flag. How can we improve that? Never trust the input. Again, if you have your frontend fully validated, the user is entering all the data, hits the submit or save button. Sends the request to the API. It passes the data it saved perfectly. Then, again, if you try Postman, or if you try any of the other approaches to actually evoke your API without a frontend, then you start to run into issues. There are no validations.

Always remember that if you’re working with an API that is being used by a frontend, the API exists independently from the frontend. We really have to start validating the API. First step, the same validations that you are applying in your frontend, you have to apply in your API as well. That’s the minimum that we have to do. I know it’s a lot of work, because there’s a lot of validations that can go through, especially when we’re working with forms, and we do have a lot of forms in some of the applications, but again, always add the validations to your API, at least the same. Remember that your API has to have more validations than your frontend. It is the one that has to be bulletproof and has to hold the fort when we’re talking about security.

Make sure that you’re validating type, length, formatting, range, and enforcing limits. Java is a beautiful language, because we have something that I like to call annotation driven development. We just start adding all the annotations, and magically, it’s going to do all the work behind the scenes for us. When you are annotating your entities, you have the @Column, for example, just to map this particular property from your class to the column in the database, or to the property in the document. Make sure that you’re adding the length as well, if it’s nullable or not, if it is unique. Try to map your database mapping into your code as well, because, again, that’s going to be at least one layer that we can add a security.

In Java, we have a really nice project that’s called the Jakarta Beans Validation. If you’re a little bit old school, the Java EE Beans Validation as well. Hibernate also has one of the implementations that’s called the Hibernate Validators, that you can use to enhance all your entities or all your documents as well. Do not forget to validate strings, when we have a name. Even if you look at this code right here, I see you have some validations, but that’s not enough. I don’t have all the validations. There is too much damage that I can do if I only have validations for the size, but I’m not validating the string itself. If you try to do a request, can I do !##$, and something, I’m just going to look at my keyboard and add some special characters or weird characters. Is that a valid name? Should it be allowed? Validate strings.

One thing that we usually tend to do, I just go to my keyboard. Let me look and I’m going to type and I’m going to create my regex from my keyboard. If you go to the ASCII table, or if you take a look at the Unicode table, you have hundreds of characters. Characters that I don’t even know that exist, or I don’t even know the name. Be very careful with that. Always prefer to work with an allowed list. What does it exactly mean? A name. If I need to have or I’m only allowed to have alphanumeric characters with maybe a space, parenthesis, underscore, so that is my name, anything else is going to be deny by default. One other thing that you can do is maybe sanitize as well. It really depends on the project. You can use the approach that, if the user tries, I’m going to not allow it, just going to throw an error. Or you can try to automatically remove those characters, or you sanitize those characters as well. Different approaches for different projects. Just make sure that you are choosing the one that is a better fit for you.

Always remember to secure all the layers. For example, we’re working here with three layers. We have our blob controller, and validate all the parameters that your methods have. Do not be shy to use those annotations. It only takes seconds to actually add those annotations over there. One thing that is very important, especially if you’re working with pagination, never forget to add an upper limit to your page size. My frontend only allows 100 records per page. That’s fine. Here, what if I parse a million, 5 million? What if I try to do a DDoS attack and send multiple requests with 5 million? Is your server able to handle that many requests? You can bring down your service, and that can cause business loss, financial loss to the company as well.

Always make sure that you’re adding validation to each and every parameter that your API is receiving. Again, in the service, you’re going to repeat that. The good thing is, you’ve done that in the controller, so Control C, Control V in the service, or maybe you’re doing the other way around, the service and then the controller. Make sure that you are propagating all those validations across all the layers. Because, what can happen, depending on the application that you are working with, you can have a service that is being consumed by only one controller, but again, maybe next week, next month, or next year, you have another controller also using that same service. What’s going to happen?

If the developer that is now coding the controller, that developer does not do any validation in the controller, at least the service is going to be able to handle any kind of validation and reject any kind of requests. Again, the entity or documents as well, don’t be shy to use and add all those annotations. The beautiful thing about this is, if you are handling a request, and sometimes if you have a column or a property that is only able to handle 10 characters, and let’s say that you are sending 50 characters through the request, you don’t get that truncate message, that exception, and it’s going to fail to write into the database. The other beautiful thing about this as well is if you are on the cloud and the service that you’re using is charging you per request when you have all these validations in place, you are saving a failed request to the database so that can actually bring some cost saving benefits to the organization when you have all these validations in place.

SQL injection. It’s 2024, we still have to talk about SQL injection. That is still happening. Make sure that you are validating, sanitizing your inputs, escaping those special characters that can be used for SQL injection. I know sometimes we don’t want to use some kind of Hibernate thing. When you have something a little bit more complex, you want to write your own native queries. Make sure you’re not using concatenation. Please, at least a prepare statement. Be a lazy developer, use what the framework has to offer you. Don’t try to do things on your own. Many developers have gone through the same issues before, and that’s why we have frameworks to try to abstract a few of these things for us. I’m still seeing code during code reviews with concatenations in place. Sad, but it’s life.

File Upload

Still talking about input. We’re only talking about validating the request. What about files? I work for an industry where we handle a lot of files. I’m not talking about images. I’m talking about Excel files, Word files, PDFs, things like that where you have to read those files, parse it, and then do something with the data that’s within the file. Then you go through with the business logic. First rule of thumb, always make sure that you are adding limits to the file size. If the file is too big, ask the user. Again, really depends on what’s the business use case here. Try to find, what’s your limit, something that is acceptable. Make sure that you are setting that into your application. Again, if you’re using Java Spring, two lines of code. Easy. Five seconds and you’re done. Make sure to also check for extension and type validations. These can be very deceiving. If you remember a few slides back, never trust the input, because here you can go to the content header, and you can manually change it and deceive the code, if you’re checking for the extension in the content header. What do we do?

The issue that we can run into this with the extension is, if your library is expecting one extension, and it’s actually something else, you can run into all sorts of issues. Also with the file name, there is one very famous vulnerability that’s called the path traversal vulnerability, where the file name, again, we don’t know what’s the file name. You can use those tools to intercept the request and change it, and have something that is malicious. You can completely wipe out directories of files. I don’t know if you’re using a NAS, if you’re using an S3 bucket, or any kind of storage, but there is a lot of damage that you can do only with a malicious file name. Make sure to also validate that. Be a lazy developer. Use tools that are already available, if you are able to add these dependencies to your project. If you need something that is very simple, very quick, you can use Apache Commons IO. There is a file, you choose file because we love a you choose class. There is a you choose file that you can use to normalize the file name.

If you need something a little bit more robust, you can use Apache Tika that you can actually read the metadata of the file, get the real file extension, sanitize the name of the file. I cannot tell you how many times this library has helped me to close a few vulnerability issues for the applications that I have worked with. Whenever I’m working with file upload, the first thing that I do, do I have Apache Tika in my pom.xml? If I have, then uncover, and then just can copy paste the boilerplate code, or you can create a static method just to run those validations for you and have some reusability as well. Again, if you are indeed saving the file somewhere, be sure that you are running the file through a virus scan. If you’re working with spreadsheets or CSVs or documents, again, deny by default. Do my Excel file need to have macros or formulas? My Word document, do I need to allow embedded objects? Does it make sense for my application? Do I have a valid business justification? Make sure that we have all those validations in place. Then you can safely store your file and live happily ever after.

Exception Handling and Logging

Exception handling and logging, this is where we have to be a little bit careful as well. We as developers, and I find this really funny, whenever I’m using a service on the internet and an error occurs, and I see, they’re using this tech stack. That’s really cool. For me, it is, but for somebody that doesn’t have good intentions, might not be. Never expose the stack trace. Log the stack trace, because we as developers, we’re going to rely on logs to do some debugging and try to fix some of the production issues. Log it, but do not expose it. Return a friendly and helpful message. Please do not return something like an error occurred, please get in touch with the administrator. What does it mean? Something that is helpful to whoever is seeing the message, but you’re not exposing anything.

You’re not exposing the technology stack that you are using. Because what happens is, if you expose the technology stack, the person that does not have good intentions might see, let me see if there is any vulnerabilities. You’re using Spring. Does Spring have any vulnerabilities that I can try to exploit? That is one of the reasons. Again, if you’re using Spring, one line of code that you can add to your properties file or YAML file to not expose the stack trace. Also, be careful with what you are logging. We’ve watched some talks during this conference here that we as developers, we are responsible. We have to be accountable for the code that we are writing. The beautiful thing of being a developer is that you can work within any industry. With power comes responsibilities. Different industries will have different regulations, so make sure that you’re not logging the password, even for debugging purposes.

If you work with personal identifiable information, like first name, last name, email, phone, address, something that can help to identify a person, do not log those in. We have several regulations, GDPR, California has the California Privacy Act. Other states are passing their own regulations. We have to study our programming language, and at the same time, we have to keep ourselves up to date with all these regulations that can impact our jobs as well, to make sure that we’re being ethical, and we are writing code that is not infringing any of those laws: financial information, health care data, any kind of confidential business information. Log something that is still helpful to you, to help you to debug those production issues, but do not log something that is sensitive.

One of the things that you can use to remove those sensitive data, especially if you’re using the toString to log something, again, remove any sensitive data for your toString. There are annotations that you can do this. I personally prefer to not use annotations on this, because, again, you can forget to annotate in case you’re adding a new property. I like to explicitly tell what’s my toString here, so I can actually safely log that information if I have to. In case you do have to log user IDs or credit card numbers or any sensitive confidential data, you can mask that data and still be presented in a helpful way to you, or you can use vault tokens as well.

There are many different ways that you can do this, in case you absolutely have to log it. Be very careful with that. Last but not least here, apply rate limits to your APIs. There are many flavors in the industry. It all depends on the size of your application. If you need something that is very quick and easy, you can use Spring AOP. There’s also a great library, Bucket4j. If you need a more robust enterprise solution, Redis for the win, among other solutions out there as well. Do apply because, in case your API does have any kind of vulnerability, at least here, you’re going to prevent some data mining. At least if you have some rate limit, you can control the damage that’s done here. At least have one of the things. If you cannot have it all, at least try to apply a few validations, rate limit so you can decrease the size of the damage.

Testing

Testing. After all we’ve talked about, of course, we have to test all of this. It’s not only our business logic. For testing, make sure that you are adding those exception edge cases as well to your testing. If you only care about percentage of code coverage, this is not going to add any code coverage to your reports, but at least you are testing if you have your validations in place. You know if your security checks are in place.

One of the things that really helps me, especially when I have to write this kind of data, you can use other data sources for this. You can have your invalid data into some sort of file, and load it. There are many ways of doing this. In case you’re writing the data yourself, use AI to help you with this. You write two, three, and then the AI is going to pick it up and bind the test, all the rest of the data for you. This is a way to also improve that.

The AI Era

Again, we are in our AI era here, so make sure that you are taking advantage of that. If you are starting to work with projects with AI, because, of course, now it’s AI, our companies are going to ask, can you just put an AI on that? Just make sure that we have an AI. In case you are working in one of those projects and you are handling prompt engineering, make sure to validate and sanitize that as well. This is a really cool comic. Make sure that you are validating and sanitizing your input. It doesn’t matter the project, always validate and sanitize. Use AI as an ally here. It’s a great IntelliSense tool. I really like to use as my best friend coding with me.

You’re not sure how to write a unit test for a validation, just ask Copilot, CodeWhisperer, whatever tool that you are using, it can help you with that. In case you’re using GitHub now, they’re coming out with a lot of services. I really think that this is adding the security within the pipeline itself. Keep your dependencies up to date, that also helps a lot. Add some code scanning. For any security vulnerability, make sure that you’re not exposing those passwords. It can also help a lot with that if you do have access to services like this. Of course, there are a lot of other services within the industry as well. It really depends what your company is using. There are great services out there that you can achieve a very similar result.

Education and Training

Of course, you’re not going to go back tomorrow and say, team, I think we need to start incorporating a little bit more security within our code. This change does not happen from night to day or from day to night. It is a slow process. We need to mentor junior developers on this and the rest of our team as well. This is a work in progress, through many months. One of the things that I like to do with the folks that I work with is, whenever we’re having demos of the product, I’ll start asking questions. This is a really nice, cool feature, you’re handling a file upload? Are you checking for the file name? Are you validating that? Or, if we have some RESTful API, what are you using for validation? Start asking questions.

Next time you’re having those sessions again, ask the same questions again. Next time she’s going to ask about that, let’s just add it so when she asks, we’ve already done it. That’s a different way of doing that. Provide feedback. Make sure that the requirements are part of your user stories, it is part of the requirements, so we can start to incorporate it as part of the development. One thing that I like to use as well is some security checklist whenever I’m doing code reviews. This is only a suggestion. These are some of the things that I find mostly in the code reviews that I do. Always be kind with the code reviews that you are doing. These are some of the things that I usually check whenever I’m doing code reviews. You can evolve from this. Adapt to something that works better for your team. Again, many flavors available out there.

Questions and Answers

Participant 1: Do you have any recommendations for libraries for file content validation?

Groner: It really depends on what kind of validation you are using. For example, for all the Word documents that I handle, all the spreadsheets that I handle, we usually do not allow macros, formulas, embedded objects. For the content itself, it really depends on the use case that you have. It can be something manual. You can use some OCR tool to help you to do that as well. It’s really going to depend.

Participant 1: Since you mentioned Excel files. We do have a use case where users upload Excel files. I was just wondering if there are any off-the-shell libraries that we can use, or do we have to write custom code?

Groner: Depending on what you need, we usually write our own. We only validate for things that we do not allow. If you have a data table you’re only trying to extract that data table, we’re going to run all the validations on all the types that we have all over again, and validate all the business logic to make sure that that data is what we are expecting. That level of detail, it’s usually that we usually write something. Depending on the use case, Google has services for that, and there are a few services out there that you can try to use to help you to go through that.

Participant 2: You’re logging what we should expose, what we should return. In our team, we are having this double-edged sword in the sense that we don’t want to return sensitive information, like expose our business logic, how we do our profile management. When we have issues escalated to our helpdesk or service centers, we can’t find the exact errors by looking at Splunk, because our APIs don’t return those important crumbs for us. How can we approach this better? Our architects suggested, for instance, maybe we should use error codes. Like, this is the error code 2 or 3. Have you encountered this issue before? What should we do?

Groner: There are a few different ways that you can approach this. One is you can definitely have your own dictionary of the error codes, as you mentioned, just to help you a little bit with the debugging process. The other way around it is, you can try to mask the data. It’ll still be something that is meaningful and it’s easier for you to consume, but not something that’s going to be exposing any sensitive data. Because often when we’re running into production issues, it can be something like a software defect where we have to fix, but it can also be data consistency issues as well.

Those cases are a little bit more difficult to do the debugging. If you have some masking that you still have, like the nature of the data itself, you can still go through that without actually having access to the database, or something like that. It will be one of the approaches that I would try to use. This is very specific. It really depends on the business case, but it helps a little bit. The other thing that you can do as well is some kind of vault. If you have that data, you have some token, and you can log the token that can help you to retrieve the data. That will be another approach as well.

Participant 3: Do you have any suggestions for any tool in the CI/CD pipeline to scan the code quality and check for security inside the code?

Groner: There are a few, like Snyk. There’s Sonar. Depending on how you configure Sonar, you can try to catch those as well. Personally, we use a lot of checkmarks to do that, like checkmarks for code. There is still a team of InfoSec that is reviewing the checkmark, what it’s flagging to review if it’s a real issue or not. There is Black Duck for any kind of CVEs that we have out there for dependencies. There are other tools on the market, but these are some that we use internally, that’s global to the organization. If you’re using GitHub, GitHub now, they’re rolling out a lot of features, and they have the code scanning. A lot of them are free to use if you’re actually using GitHub, but a few of them, you still have to have the license of the product in order to be able to use.

Participant 4: You mentioned having validation at all levels. One of the things we’ve done is pulled out, like we don’t have authentication at every level, we just handle that, not even in the service authorization, pull it up to the top level. For something like validation, we also have that at the top level, and not have underlying services or something that we handle just at the base level, like in a controller, so that we don’t have to keep adding that in. Is there a difference, in your opinion, on like authorization and authentication versus validation, and why you do validation at every single level, like why that’s different?

Groner: I think it’s really going to depend on the team itself. You definitely can do a validation only on the controller level, if you want to keep your service layer a little bit more clean. I would definitely add that to the entity as well, because sometimes, we make mistakes. You’re going to forget something in the controller level, so at least you have another layer protecting you. If your team has the discipline to always add those validations into the controller, and if that’s working for you, that’s great. You can continue doing that. It also really depends on the nature of the project. If you have your controller, and then you’re calling your service, and maybe you’re using microservices architecture, and you don’t have multiple controllers, that works really well. If you’re working in a monolithic application where you have thousands of controllers and then you have thousands of services.

Then, in one controller, you’re making reference to 10 different other services, that becomes a little bit more complex, and you can actually make a mistake when you are trying to reuse that service in a different file. Then if you forget something, that is one of the reasons that I would say, to add in to all layers. It depends on the project. If that’s working for you, that is great. For the validation and authorization itself, usually, this is only done on the highest layer, usually in the controller, if we’re talking about Java, Spring, or something like that. That’s usually where we handle. You don’t necessarily need to handle the services in the service layer, unless you have a service that’s calling another service. Then you need to have some kind of authorization and authentication, some mechanism in there as well, in case you are interfacing with a different service, like connectivity to a different web service, or what have you.

See more presentations with transcripts

Transcript

Groner: I would like to share with you a little bit about a career as a staff engineer in non-technology companies. My name is Loiane Groner. I’m very passionate about working in technology. I’ve been on this road for over 15 years.

Whenever we talk about being a staff engineer, and we search online for job postings, we see a lot of articles and information about being staff engineers in big tech companies, especially the famous FAANG companies. However, this survey that was conducted by Oracle and Burning Glass Technologies from 2019 shows that approximately 90% of the jobs in the tech industry are in non-big tech companies, meaning they are not from the technology industry sector. If we take a look at this other chart, also published by Oracle and the Burning Glass Technologies from 2019, we see all these different industry sectors with a percentage of the job posts that we see they are tech related. This is just to emphasize that although we’re working in technology, the company that we work for might not necessarily be a technology company, meaning the vast majority of the IT jobs are outside of the tech industry. However, we almost don’t see a lot of material out there, talking about what comes after you’re a senior in these companies that are not related to technology. However, when it comes to career in non-tech companies, I’m sure that you have seen some diagram or career path that looks like the one that we have on the screen right now. This traditional career ladder, it’s what we call the manager’s path. After you become a senior engineer, you’re probably going to work as a tech lead for some time. If you want to take the next step into your career, the next step will be a manager level. If someone would like to continue working as a technical person beyond the senior level, especially the tech related companies, they created another path, which is what we call the staff engineering path. These individuals can continue climbing up the ladder, and progressing within their careers.

What Exactly is a Staff Engineer?

What exactly is a staff engineer that we talked about so much? We have the manager’s path, that we have agreed that this is pretty traditional. After the senior/tech lead level, instead of becoming a manager and having people reporting directly to you, there is what we call the staff-plus engineering path. In this path, you have the staff engineer, you have the senior staff engineer, principal engineer. Then if you want to continue, there’s also distinguished engineer, fellow engineer, as we have seen in some of the FAANG companies. This staff-plus track was designed to be able to retain talent, to be able to retain those people that want to continue working on a technical level. It doesn’t mean that because now you are in a technical track, and because you’re also not going to have any person reporting directly to you as you being their manager, it doesn’t mean that you won’t have to deal with people. This is still a leadership position, and you’re still going to help other people to grow within their careers as well. You’re just not going to be their manager. What exactly does it mean to be a staff engineer?

Three Pillars of Staff Engineer Roles

If you are on this path, or if you like to continue your career growing as a staff engineer in this staff-plus path, I highly recommend that you read these two books. They are amazing. The first one is the, “Staff Engineer.” There’s also a website that you can go into and read several articles, interviews, and guides. There’s also this other book published by Tanya Reilly, that is an individual guide, and has a lot of great insights as well. Tanya mentions in the book that there are three pillars that are related to being a staff engineer. The first one is big-picture thinking. The second one is project execution. The third one is leveling up. All these three pillars are based on the knowledge that you have acquired throughout your career, and also the experience. They have a big impact within the organization as well.

1. Big-Picture Thinking

Let’s dive into each one of them, and see also how these three pillars can be applied to non-technology companies. Let’s start with the big-picture thinking. This pillar means that you’re able to take a step back and see the whole picture, and to be able to make decisions that are going to be better for the organization. This means that you have to think beyond your current time. Whether that means if it’s a project that’s going to take years in order to be able to complete. If a system that you’re working on can be easily decommissioned. What exactly does your company need in the medium term, say, two, three years? Let’s see an example. This example is also from Tanya’s book, and I think it’s fantastic. When you are a senior engineer, or when you are a tech lead, and you work within a team, you have a problem to solve. You have different solutions, different approaches, or different tools that you might need to use, or implement to make your project successful. When you have all these options, usually when you’re a senior or a tech lead, you’re going to review all these options, and you’re going to make the decision based on what’s best for your team. This can be, for example, solution or approach A. This means that this is going to be easier for the team to implement, or it’s going to be quicker for the team to implement, meaning this has a higher value to the team. However, when we take a step back, and we review all the options and take the entire organization into consideration, this might not be the best solution. If we take a look, again, at both approaches, or both solutions that we have to consider, solution A and solution B, solution A although it might be better for the team might not be what’s best for the organization. For the organization, for example, in the long run, solution B might be better. For the organization, that’s going to have a higher value. It might take even more time for the team to implement that solution. Meaning it’s not what’s best for the team, but it’s what’s best for the organization. This is one of the reasons why it’s important to have staff engineers within your organization, so you can have people that are going to take a step back and analyze all the options available, and they’re going to make the best decision that’s possible.

Why is this important and how this correlates with the world that we’re living in today? If we take a look at this article that was published by Tech Brew back in November 2022, it says that many technology companies are cutting costs, and are aiming to do more with less. The Wall Street Journal also published an interview with Microsoft’s CEO. The CEO also mentioned that both their clients, Microsoft’s clients, and also Microsoft itself will have to do more with less. What does it mean to do more with less? Being a staff engineer in a non-technology company means that besides the technical expertise that you’re bringing to the table, you also learn about the business, the problems that the clients are facing. It’s also part of your job to help your clients to scale their business, especially in times of economic uncertainty. The staff engineers can play a key role within every organization. For example, my personal goal as an engineer is to make sure that the applications that I’m working on today will still be up and running in 10 years, and also, the decisions that I’m making will also help to scale the business. How the business can do more with less, and how can I help them using technology as a tool? That is the perspective that we need to have in order to make that happen.

When we say doing more with less, automation definitely plays a part in it. When we automate certain tasks, especially if those tasks are manual, this means that we are freeing up personnel so these personnel can work in different projects, or even take on new clients. Again, this goes for both technology and also for the business. Automation is one of the ways that we can help the business to do more with less. As a staff engineer, one of your jobs is to help to make decisions of tools that are going to be adopted at an organization level. Again, going back to the vision of the single team, if you leave it up for each individual team to make decisions on the tools and processes they’re going to adopt, each team might adopt a different tool or technology. Taking a step back, as a staff engineer, can we decide on a global tool that’s going to be used by everyone and even save some cost with it? Being a staff engineer, you might have an input on your department, within your organization, or even a global level or even in the tech industry. One of the other roles that a staff engineer also plays is to be able to define processes and best practices that are going to be adopted throughout your organization. This is another reason why non-tech companies should have a staff engineer path as well. Why? Because title matters. Having a title, the seniority means that you have a seat at the table. This means a lot. It is important to have the ability of providing an input so it can be taken into consideration. It doesn’t mean that whatever you say is what’s going to be adopted. It’s important to have the opportunity to have a say in it. This is one of the ways that non-tech companies can make sure that your voice is heard. This is really important.

When you work for companies that are not tech related, besides the technical knowledge that you bring to the table, you also have the knowledge about the business. You develop relationship with the clients, the business, the product owners, and other stakeholders as well. As we all know, it takes time to develop relationships. It’s not like, “You want to be friends,” and that’s a done deal. It really takes time until you can develop that partnership. In case someone decides to leave, training a new person can take time. It’s not like you’re starting from scratch. After all, you’re a senior. However, whenever you’re starting at a new company, there are all the internal processes of the company that you don’t know, and you have to learn. There are also all the relationships that you will have to develop. Having this knowledge about the business, knowing how companies work, this is also one of the reasons that for certain sectors in the industry, when you see a job description, you’ll probably see at the end of the job description, something like previous experience in the industry is a plus. Meaning that although some training is going to be required, because you’ll have to get used to the processes and the stakeholders, at least you’re not starting from scratch.

One of the takeaways is if you’re working for a non-technology company, and your company doesn’t have a staff-plus track yet, you start a discussion so this can be considered in the near future. For companies that still don’t have this path, depending on the size of the company, this is not something that you’re going to be doing in a week or even in a month. You’ll have to train managers on how to properly help staff engineers as well. It also requires some reorganizations within the company. If you have been through some reorgs within the company that you work for, you know that sometimes it might take a while until things go back to the normal track again. Having this path, it’s very important because this can help your company to retain talent as well.

2. Project Execution

If we go into the second pillar that Tanya describes in the book, we know that at a staff level, projects are going to be more complex. They’re going to involve more people. They’re going to involve more political capital. They’re going to require more influence and sometimes even culture change as well. As a staff engineer, what exactly is your role when it comes to project execution? I believe we can agree that we do have a pretty solid job description for junior engineers, mid-level senior engineers, tech leads, managers, and so on. What exactly is in a job description for a staff-plus engineer? Sometimes I like to joke with my colleagues at work that I don’t know what my title is anymore. One day, I’m helping with architecture, the other day, I’m helping define the strategy for long term. The next day, I’m helping the team to resolve a big production issue. The next day, I have a completely different task. Some days, I have no idea what my job title will be based on the tasks that I’m performing that day. I like to say that, today I’m going to be whatever you need me to be. Meaning that as a staff engineer, we do wear a lot of different hats, we play different roles depending on what the company needs. At the end of the day, your job as a staff engineer is to make sure that the organization is successful. You can be a technologist, but your job is to help the organization to achieve its goals. This means that this can be your team, your technology team, or it can be the business, the clients. As senior professionals, we do a lot of things that are not in our job description all the time. It might not be ideal, but if that’s what it takes, consider doing it.

Your work as a staff engineer needs to be important for the company. It doesn’t mean that your work should be playing with the latest and most modern technologies out there, meaning only doing fun stuff. Sometimes what’s important, it’s going to be the work that nobody sees. These important tasks might involve gathering data that doesn’t exist, going through old documents, combing through code that was written 10 years ago that nobody has touched. This is what we call the projects that nobody wants. Within a company, there are a number of tasks that they just need to get done, and somebody has to do it. This is still meaningful work. Meaningful work comes in many forms, and sometimes includes working in projects that nobody wants to work. Sometimes it also includes working with old boring technologies. There is one side where you need to keep your skills sharp, because this is how you’re going to be helping your organization to make strategic decisions. On the other hand, there is also all these other projects that they need to get done, and might involve old technologies. However, I’ve learned from experience that there is a lot to learn, even with old technologies. Sometimes I get surprised with the things that I didn’t know that I didn’t know. At the end of the day, it’s all about balance. There are still those projects that need to get done, and on the other hand we still want to play with the fun stuff.

3. Leveling Up

The third and last pillar is leveling up. With the increase in seniority, this means that you also have more responsibility with your hard skills, but also with the people around you. These responsibilities can include teaching, mentoring, and also the accidental influence of being a role model. After all, you are in a senior leadership position, and people will look up to you. As a staff engineer, it means that you don’t have direct reports. However, you’re also responsible for leveling up other people that work with you, within your team or within your organization. I like to include the teaching and mentoring into the community category. This can be done within your company or outside of your job responsibilities. I’m personally very passionate about this topic. This is a great way of teaching, giving back, leveling yourself up, as well as others. In my humble opinion, teaching is the best way to learn. This can be achieved in different ways. You can mentor other people. You can also contribute, writing blog posts, or that documentation in the Confluence, or wiki page of the company that you work for. You can record videos, publish them internally, or as well on the internet. One of the things that you can do, for example, is write a summary of everything that you have learned throughout this conference, or even put together a presentation of all the things that you have enjoyed and learned. Writing blog posts, presenting at conferences, can be a great way of building influence within the industry. Personally, sharing what I have learned with the community has opened many doors.

Just to give an example of how this works, and how this can be beneficial for both you and the company that you work for. Back in 2010, I was working with Ext JS, which is a frontend framework. I was writing blog posts back then of things that I was learning so I can deliver my tasks. I decided to put together a presentation and present it internally for the company that I used to work for. I continue writing blog posts. Eventually, to my surprise, I got an invite to write a book. That was a dream come true for me. I continued writing, and I did write more books because I really liked the process. This opened so many doors for me within the community. I started to get invited to present at conferences. I had the huge honor of presenting at QCon in Sao Paulo as well in Brazil. One year later, the company that I worked for, was deciding on a framework that we could start creating screens using HTML5. Again, to my surprise, and also a dream come true, I was invited to spend two weeks in New York, just training my peers. For me, this is amazing. I have no words to describe how thankful I am. I personally thought that I was living a dream or it was a movie, because it was really awesome.

For over 10 years, I’m still sharing a little bit of what I’m learning throughout my career. Because of this passion and commitment to the community, I was honored by some of these big tech companies that I’m a big fan of with some awards. I’m very thankful for it. Because of that, that continued to open more doors for me and I was able to present, for example, about Angular at QCon again. Later, the company that I worked for, was migrating from Ext JS to a different frontend framework, and I was able to provide my input on why we should adopt Angular and why it was good for us to do that. This is just an example of how contributions that you do outside your working hours can be beneficial for you, and also for the company that you work for. It’s a win-win situation. You might ask yourself that, especially in times like this, should we be sharing everything that we know, or should I just keep to myself so I can keep my job? My personal take on this is to just keep sharing. Great things can happen when you share what you know. You’re leveling up yourself and you’re also leveling up the people around you. This can be an opportunity to get more important projects, or even get a different role. Sharing will help you to keep growing within your career. Eder has presented an amazing talk about this subject as well, on how community contributions can help within your career. I highly recommend that you watch it.

One of the takeaways when you go back to work, just put together a summary of things that you have learned throughout this conference, or things that you have enjoyed, and present it back to your peers. This can be a great way to help you build your influence. Speaking about influence. Influence is one of the soft skills that you need to succeed in this career, however, it’s not the only one that you need. Every increase in seniority demands more soft skills. You’re going to be dealing with more people. When we make the shift from a senior or tech lead into the manager’s path, we’re going to shift our learning and we’re going to focus on learning more soft skills, because we’re going to be dealing a lot more with people. When you go into the staff path, besides your hard skills because you need to keep that technical knowledge growing, you also need to develop the soft skills. After all, you’re still a senior and you’re still in a leadership position as well. Meaning that soft skills are equally important as hard skills. Besides influence, communication is a must. The message that you’re trying to convey needs to be clear and precise. One of the challenges that some of us face, I know that I face this challenge every day, is how to do that in a second language. Do you know how smart I am in Portuguese? There you go. Influence is another one that is very important as well, not only to be able to provide your input, but also in case you need to get a sponsorship for a project, or a process that you’re trying to implement within your organization. Even though you are an individual contributor, your job is to do what’s best for the organization. This means that you’ll have to work with different teams. Teamwork is also essential.

Time management and delegating are also two important skills. Everybody’s time is valuable. As a staff engineer, you won’t have time to do every single thing that you would like to. You’ll have to choose what tasks are more important. How do you spend your time? Priorities might change, and you might not code as much as you’d like to. That’s very important to be able to know how to delegate and also how to manage your time. Again, even if you’re not a direct manager, but you can also help your peers with mentoring and coaching as well. Besides problem solving, critical thinking, that are also pretty basic soft skills. There is another soft skill that I think that is imperative, and that is learning how to disagree. Even though you are a staff engineer, it doesn’t mean that you can just simply pull rank. Sometimes your input, or the approach that you have suggested, are the ones that are going to be implemented. It’s very important to learn how to disagree, and disagree respectfully. Of course, technology is full of up and downs. It’s very important as well, to develop your emotional intelligence, so you are able to handle pressure and difficult situations that might come your way. Keep calm and carry on.

I also would like to leave this link for you because Charity presented an amazing talk as well at QCon, where she describes the importance of the manager and the staff engineer path pendulum. Right now, you might be a staff engineer, maybe your next role, you can be a manager, and vice versa. There are many things that you can learn by making the switch. The manager path is important because you want to continue developing the soft skills and learning how to deal with people. At the same time, you don’t want to be too long away from a technical path, so you can continue developing your hard skills as well. Again, at the end of the day, it’s all about balance. There are a lot of benefits of doing exactly this with your career. Of course, you also have the option to step back and not want to be a staff engineer anymore. You want to go back and be a tech lead, or being a senior engineer. That’s completely fine as well. Just remember that it’s your career, and you have to do what’s best for you as well.

Conclusion

Although we are very unique human beings, it’s almost impossible not to compare yourself with other people. However, don’t do that. It’s very bad for you. Instead, if you want to make sure that you’re making progress towards your career, compare yourself with you from yesterday. This way, you’re going to be able to see if you are indeed in the right path, or if there is anything that you have to change. Last but not least, although we are technologists, we’re attending a tech conference, we’re learning about new languages, architectures, and tools. Although we work with technology, it’s all about people. It’s about your clients, your team, your peers. It’s about being respectful and cultivating good relationships.

See more presentations with transcripts