Article originally posted on InfoQ. Visit InfoQ
Transcript
Mohnani: My name is Urvashi Mohnani. I’m a Principal Software Engineer on the OpenShift container tools team at Red Hat. I have been in the container space for about 7 years now. I’m here to talk to you about a developer’s journey towards Kubernetes.Let’s do a quick refresher on what containers are. Containers are software packages that bundle up code and all of its dependencies together so that the application can run in any computing environment. They’re lightweight and portable, making them easy to scale and share across the various environments. When run, containers are just normal Linux processes with an additional layer of isolation and security, as well as resource management from the kernel.
Security comes in the form of configuring which and how many permissions your container has access to. Resources such as CPU and RAM can be constrained using cgroups. The isolation environment can be set up by tweaking which namespaces the process is added to. The different categories of namespaces you have, user namespaces, network namespaces, PID namespaces, and so forth. It really just depends on how isolated you want your container environment to be. How do we create a container? The first thing we need is a containerfile, or a Dockerfile. You can think of this as the recipe of what exactly goes inside your container.
In this file, you will define the dependencies and any content that your application need to run. We can then build this containerfile to create a container image. The container image is a snapshot of everything that was in the recipe. Each line in the recipe is added as a new layer on top of the previous layers. At the end of the day, we compress all these layers together to create a tarball. When we run this container image, that’s when we get a container.Since containers are just Linux processes, they have always existed. You just had to be a Linux guru to be able to set up all the security, isolation, and cgroups around it. Docker was the first container tool to make this more accessible to the end user by creating a command line interface that does all the nitty-gritty setup for you, and all you have to do is give it a simple command to create your container.
Since then, many more container tools have been created in the open-source world, and they target different areas of the container space. We have a few listed on the slide here. We have Buildah that focuses on building your container images. Skopeo focusing on managing your container images. Podman, that is a tool for not only running your containers, but also develop and creating pods. There is CRI-O, which is a lightweight daemon that is optimized for running your workloads with Kubernetes. Kubernetes itself, which is a container orchestration platform that allows you to manage your thousands of containers in production. Together, all these various container tools give you a holistic solution, depending on what area you really need to focus on in the container space. For this talk, I’m going to use Podman, which is an open-source project, to highlight how we can make a developer’s journey from local to prod, seamless. A few things I would like to mention is that, Podman is open source and completely free to use. It is daemonless, focuses on security first, and is compatible with all OCI compliant container images and registries.
Towards Kubernetes
You’ve been running your containers locally, how’d you get to production? There are a few key challenges in going there. Some of them are paranoid sysadmins, different technologies and environment, and a different skill set as well. We call this the wall of discrepancies. Security doesn’t match up. You have low or no security in your local dev environment while production has highly tightened security. Container processes have different permissions available to them. Locally, you have root privileges available, while in production rootless is required. In fact, even the way you define your container is different between the two environments. All of this just adds a lot of overhead for the developer and can definitely be avoided. Let’s take a look at how we can target some of these. When you run a container locally with a tool like Podman, you can use a bunch of commands and flags to set up your container. I have an example here where I’m running a simple Python frontend container and I want to expose the port that’s inside it.
To do that, I have used a publish flag so that when I go to localhost, port 8088, I’m able to access the server that’s running inside that. Another way that you can define or you can run containers locally is using a Docker Compose file. This is a form of YAML that the Docker Compose tool understands. Here’s an example of how you would define that. Let’s say you have your container running locally. You want to now test it out in a Kubernetes environment. Wouldn’t it be great if you could just copy paste either the CLI command that you have there, or the Docker Compose and just paste in the cluster? Unfortunately, you cannot do that. For those of us here who have run in Kubernetes before, know that Kubernetes has its own YAML format to define your container and your workloads. As you can see, there are three formats going on around here, so when you want to translate from your local dev to a Kubernetes environment, you have to translate between these formats. That can be tedious and can also be error prone, as some configurations could be lost in the translation, just because fields that are named one way in the Kubernetes YAML format are not exactly the same in the flags that are used in the CLI.
You really have to keep referring back to documentation to figure how they map around. This is where the podman kube command can help. In an effort to make the transition from Podman to Kubernetes and vice versa, easy on the developer, Podman can automatically generate a Kube YAML for you when you pass it a container ID or a pod ID that’s running inside Podman. At this point, you can literally do a copy and paste of that Kube YAML file, put it in your cluster and get running. Of course, users can further tweak that generated Kube YAML for any specific Kubernetes use cases or anything that they want to update later on.
I mentioned vice versa, so you can go from Podman to Kubernetes, but you can also go from Kubernetes to Podman with one command. Let’s say you have an application running in your Kubernetes production cluster. Something goes wrong with it, and you really want to debug it. You have some issues getting the right permissions, or access to try and figure it out on the production cluster itself, and you wish you could just replicate that workload locally here. Good news for you is that you can do that with the podman kube play command. You just have to go into your cluster, grab the Kube YAML of that workload, pass it to podman kube play, and Podman will go through all the container definitions, pod definitions, any resources defined in that, and create it on your local system, and start it for you. Now you have the same workload running locally, and you have all the permissions and access you need to be able to debug and test it, just with two commands, podman kube generate and podman kube play.
Outside of Kubernetes, Podman also works really well with systemd. You can use Podman and systemd to manage your workload using systemd unit files. This is especially useful in Edge environments where running a Kubernetes cluster is just not possible due to resource constraints. Edge devices are also a form of production environment, where you’re running your applications there. As we can see here, when you want to do that with systemd, systemd has its own different format. In addition to the three that we just spoke about, there’s a fourth one that you probably have to translate your workloads to if you want to move them to Edge environments. In the effort of standardizing all of this and making it easy for the developer, Quadlet was added to Podman. What Quadlet does is that it’s able to take a Kube YAML file, convert it to a systemd unit file under the hood, and start those containers with Podman and systemd for you, so the user doesn’t have to do anything. All you need is that one Kube YAML file that defines your container workload, and you can plug it into Podman, into a Kubernetes cluster, and into an Edge device using systemd.
Rootless First
That was on the container definition. Remember I mentioned that Podman focuses on security first. This can actually be seen in its architecture. Podman uses a fork-exec model. What this means is that when a container is created using Podman, it is a child of itself. This means that root privileges are not required to run. If someone is trying to do something weird on the machine, when you take a look at the audit logs, you can actually trace it back to exactly which user was trying to do what. When you compare that to the Docker daemon, root access is required, although now you can set up rootless context. If someone is trying to do something weird, when you take a look at the audit logs, it points to this random UID, which essentially is the Docker daemon, but it doesn’t tell you which user was trying to do what there. In the rootless first scenario, there are two things to keep in mind. When you run your container, you want to run it as rootless on the host, which is default when you run with Podman.
You also want your container application that’s running inside the container to be run as a rootless user. This is something that is often overlooked, because just running rootless on the host is considered enough, and container engines, by default, give you root privileges inside the container when you just start it up. This is something that developers usually don’t keep in mind. When you are running in a production Kubernetes based cluster, that is focused on security, so something like OpenShift, running rootless inside the container is a hard requirement. Keeping this in mind and practicing it while you’re doing your development will save you a lot of headaches when you then eventually translate from your local development to a production cluster. In the rootless first scenario, you want to run rootless inside and outside of the container.
Security
Continuing with the security theme, when you use Kubernetes out of the box, it provides you with three pod security standards. These are privileged. Here, your container process has basically all the permissions and privileges possible. You definitely do not want to be using this when you’re running in production. Second one is baseline. Here, your process has some restrictions, but not enough restrictions where you’re banging your head on the wall trying to get your container working. It’s secure, but it’s also broad enough to give you the ability to run your containers without issues. The third one is restricted. This is the one that’s heavily restricted. You basically have zero or very little permissions. This is probably the one you want to use when you’re running in production, but it’s often the most difficult to get started with. We always advise that you start with baseline, like middle line, get there somewhat, and then continue on tightening the security. Let’s take a deeper dive into security. There are two key aspects to it. The first one is SELinux. SELinux protects the host file system by using a labeling process to allow or deny processes from accessing any resources on the system. In fact, any file system CVEs that have happened in the past have been mitigated if you had SELinux enabled on your host.
To take advantage of this, you need to have SELinux enabled both on the host and in the container engine. Podman and CRI-O are SELinux enabled by default, while other container engines are not. If you’re running a Kubernetes cluster using CRI-O, you will have SELinux enabled by default. If your host also has SELinux enabled, then your file system is protected by the SELinux security policies. Always setenforce 1. The second one is capabilities. You can think of capabilities as small chunks of permissions that you give your container process. The more capabilities your container has, the more privileges it has. On the right, this is the list of capabilities that Podman enables by default. It’s a pretty small list. It has been tightened down enough that you are secure, and also, you’re able to still run your containers without running into any security-based issues. When we compare this with the list allowed by the baseline pod security standard given by Kubernetes, they have the same list and actually have a few more capabilities that you can enable as well. When you run in production, you probably want to have even fewer capabilities enabled so that you can shrink your attack surface even further.
To reiterate on the two themes over here, one is that Podman makes it easy for you to transition between your local environment and your pod environment by giving you the ability to translate your CLI commands to Kube YAMLs, or by just being able to understand a Kube YAML and being able to plug that into Podman, Kubernetes, and systemd. The second one is, Podman’s focus on security first helps you replicate an environment that is secure, or quite secure to match what you would expect in a production environment.
Obviously, it’s not going to get you 100% there, but it can at least get you 50% there, so when you do eventually transition over, you run into fewer frictions and have already targeted some of the main security aspects that come when you move to production. With Podman, you can run your containers. You can also run pods, which gives you an idea of what it’s like to run in Kubernetes, because Kubernetes is all pod based. You can run your Docker Compose YAML file with one command. You can convert it to Kube YAML, and deploy those to Kubernetes clusters like kind, minikube, OpenShift, vanilla Kubernetes itself. All of these capabilities and tools are actually neatly put together in a desktop application called Podman Desktop that runs on any operating system. It works on Mac. It works on Linux. It works on Windows. In fact, I’m using a Mac, and I will show you that.
Demo
This is what the desktop app looks like. I’m running on a Mac. I’m on an M2 Mac right now. It gives you information on what Podman version we are running right now, and just some other resources. On the left, we have tabs to take a look at any containers that we have, any pods, images. I’ve already pulled down a bunch of images. You can see the volumes. You can see any extensions that are available. Podman has a lot of extensions available to help you interact with different tools. You can set up a kind cluster, or you can set up a minikube cluster. You can talk to the Docker daemon, if you would like to do that. You can set up Lima as well. There’s a bunch of extensions that you can enable to take advantage of these different tools. For the demo, I am going to start a simple Python application that’s running a web server. This is just the code for it. I have already built and pre-pulled my images down because that takes a while.
If you would like to build an image, you can do that by clicking this button over here, build, and you can browse to the file where your containerfile is stored. In Podman, you can select the architecture you want to build for, and it will build it up for you. Since I already have it built, I’m just going to go ahead and run this container. I have my Python application as a web server that also has a Redis database that I need for the application. You’ll see why once I start it. First, I’m just going to click on this to start it up, give it a name, let’s call it Redis. I’m going to configure its network so that my Python frontend can actually talk to it once I start that. My container is up and running. When it starts, there are all these different tabs that you can take a look at. The logs obviously show you the logs of the container. We can inspect the container, so this gives you all the data about the container that’s running, any information you may or may not need.
By default, we create the Kube YAML for you as well. If you want to just directly run this in a Kubernetes cluster, you can just copy paste this, and deploy it there. With the terminal, you can also get dropped into a shell in the container and play around with it there. Now when I go back to my containers view, I can see that I have the Redis container running. Now let’s go ahead and start the frontend. Let’s give it the name, python frontend. I need to expose a port in this one so I can access it. I’m going to expose it to port 8088. I’m going to go back here and add it to the same network that I had added the Redis database to. Let’s start that. That’s up and running. Similar thing here, you can see the logs. You can inspect the container. You can see the Kube YAML. It can also be dropped into a terminal over here. Same thing. When I go back to my containers, now I see I have two containers running. This is running locally on my Mac right now. Since I’ve exposed the port 8088, let’s go to a browser window and try and talk to that port. There you go. That’s the application that I was running. Every time someone visits this page, the counter will go up by 1, and that is what the Redis database is needed for to store this information. That was me running my container locally.
Let’s say that I want to put this in a pod to replicate how it would run when I run it in Kubernetes, but I still want to run it locally on my machine using Podman. Very simple to do. Go ahead and select this. You can put one, or probably as many containers as you would like in a pod. I’ve not tested the limit on that, but if you do find it out, you can do that. Then I’ll click on that create pod button that showed up there. Click on create pod here. What it will do is now it will create my pod with these two containers inside it. You can update the name of the pod to whatever you would like it to be. I have just left it as my pod. Here we can see I have three containers running inside it, one is the infra container. Then I have the Redis and the Python frontend containers.
Yes, when I click on that, I can actually see the containers running inside it. Same thing with the pod here, when you go you can see the logs in there. I can see the logs for both the containers. You can inspect the container, and you can also get the Kube YAML for the whole pod with both the containers inside. When I go back to containers here, we can see that the first two containers that I had started have been stopped in favor of this new pod with these containers inside it. It’s still exposed at port 8088, so let’s go ahead and refresh. As you can see, the counter started back by 1 because a new container was created, but every time I refresh, it’s going to go up. I successfully run my container and I podified it. That’s what we call it. This is all local on Podman.
Now I have this pod. Let’s say that I want to now actually deploy it in the Kubernetes cluster, but I’m not ready to deploy it in a prod remote Kubernetes cluster yet. I want to test it out still locally using something like kind or minikube. As I mentioned earlier, Podman has those extensions. If you go to resources, you can actually set those up with the click of a button. I have already set up minikube on my laptop right now. We can, in fact, see the minikube container running inside Podman over here. If I go to my terminal and I do minikube status, you can see that my miniKube cluster is up and running. Podman also has this tree icon over here where you can see the status of Podman machine and get to the dashboard. It has this Kubernetes context thing as well. In the kubeconfig file that’s on your laptop, you can sign into multiple different Kubernetes clusters, as long as you have the credentials for it. It can see the context of those different clusters available to you, and you can switch between them.
You can decide which one you want to deploy to, which one you want to access, which one you want to see which pods are running in. Right now, I want to do it on minikube, which is running locally on my computer. That’s what I have selected. Now all I do is I go to this icon over here, I click on deploy to Kubernetes. It will generate the Kube YAML file for me. You can tell it which namespace you want to deploy it into. I just wanted a default namespace, and I’ll click on deploy. When we do that, we can see that pod was created successfully and the containers are running successfully. When we go to my terminal and we do kubectl get pods, we can see my pod is up and running over there. We can also actually see this on the Podman Desktop interface, when we go to pods.
Podman Desktop is able to see which pods and deployments are running in the Kubernetes cluster you’re currently pointing at, and it will tell you that this is the pod. You can see that the environment is set to Kubernetes, so you know it’s the Kubernetes cluster and not your local Podman. Now, same thing here. Let’s get the service so we can see my-pod-8088 services there. I want to expose this so I can actually access the web server running inside it. I’m just going to do minikube service on that, and run that. There you go. It opened a browser for me with that new container and minikube cluster. Every time I refresh, the counter goes up by 1. I was able to, with a click of a button, deploy my container that I had running locally on Podman into a minikube cluster.
What’s the next step? Pretty obvious. You want to deploy it remotely into a cluster that’s probably a production cluster, or a cluster that you test on right before you send it out to production. The really easy way of doing that is basically the same steps again. I’m going to go over here and switch out my context to point to a remote OpenShift cluster that I have running on AWS right now. I’m going to click that. When we do that, we’ll see that you no longer see the pod that’s running in minikube, because now it’s pointing to my OpenShift cluster. I can just go ahead here and do the same thing, deploy to Kubernetes. It would have been deployed on the OpenShift cluster. You would have just switched the context and it would have done the same thing it did with minikube, and launched it over there. It was pretty cool, since we would expose the port that we had the application running on, it was running in an AWS environment.
This was just demoing, moving from local to prod. I did all of this using the Podman Desktop UI. If you’re someone like me who really prefers to use the terminal and type instead of clicking on a bunch of buttons, all of this can be done using the Podman command line interface as well. You can do podman images, it will show you a list of all your images. You can do podman ps, it will show you a list of all your containers running. You can do podman pod ps, and it will show you your pods running. I mentioned that you can also go from prod back to local or to Podman. You can also do that by going back to the Podman Desktop app and clicking on this play Kubernetes YAML button over here. You can browse and point it to the Kube YAML that you wanted to play. You can select whether you wanted to run with Podman or run in the Kubernetes cluster that you’re currently pointing at. That’s something you can do. I’m not going to do that from here. I want to show you how it works with the command line, so I’ll do it from there. This is basically the Kube YAML that I’m going to play.
Very simple. It’s there. It’s a very simple nginx server that I have defined over here. I’m going to go back into my terminal, and let’s do podman kube play. I’m going to set the publish-all flag, just because I want to expose the port that I have defined in there, and pass it the kube.yaml. There you go, the pod was created. When we do podman pod ps, we can see the nginx pod was created. When we do podman ps, we can see the nginx container was also created over here. We can see that it’s exposed at localhost port 80. We can go back to our browser and we can go to localhost 80, and nginx server is up and running. With the Kube YAML file, I was able to just do podman kube play, and get that running locally with Podman. That is basically the demo I had for you, that highlighted that path of moving from Podman to Kubernetes, Kubernetes back, and all the different stuff that you can do with the different ways you can test, play, and deploy eventually to your production cluster.
Podman Desktop
Podman, you can use to run, build container images. You can run your containers and pods. It integrates really well with Kubernetes. As we saw, it has all those features to be able to easily deploy to Kubernetes and pull it back from there. It has a concept of pods to help you replicate what a Kubernetes environment would look like when you do run your workloads in Kubernetes after containerizing them. You can do image builds, set up the registries you would like to pull images from, load images for testing, and all of that. With the click of a few buttons, you can set up a kind cluster locally with Podman, a minikube cluster locally, and can connect to various Kubernetes contexts. One thing I’d like to highlight again is the extensions that Podman supports. We have support for kind, minikube, Docker daemon, OpenShift local, Lima, and many more. It’s just a way of giving all of these tools to the developers so that they can play around with it and have access to everything and anything they might need when developing their containerized workloads.
K8s and Beyond
I know this talk focuses on Kubernetes, but there’s a lot more the developer might need, and there are a bunch of cool features that have been added recently to Podman and Podman Desktop. One thing is AI Lab. AI is really big right now. We’re all super excited about it, and so is Podman Desktop. They added a new extension called AI Lab, where you can run your AI models locally, so that you can then create your container applications, using that as an inference point, basically. The next one is Bootc, where you can create and build bootable container images. The idea here is that, in future, your operating systems will be installed and upgraded using container images. I think it’s pretty cool. It’s still pretty much under development, but you have the ability to start playing around with that right now.
The final one is farm build, which is actually a feature I worked on personally, where you can build multi-arch images from one system. Given that Silicon Macs are so popular nowadays, having the ability to have different architecture images is very important now. In fact, I actually used this command when I was creating the demo for this talk, because my Mac is an M1 architecture, so I was doing all of that with Podman Desktop on my Mac. If OpenShift AWS had worked, that was on an x86 architecture, so I would have needed that architecture image for that part of the demo. If you’re excited by all of this, one of my colleagues has put together a bunch of demos and info on all of this. You can find that at this link.
AI Lab
I can show you the AI Lab extension in Podman Desktop, just because I think it’s very cool. Back to the Podman Desktop, I’ve already enabled it. I just click on AI Lab over here, and it gives me this view. I can take a look at the recipes catalog. These are some things that it gives you out of the box. You can set up a chatbot, or summarize code generation. I’m going to set up a quick chatbot. I’ll click on Start AI app. What it does, it checks out the repository, it pulls down the model. I chose to pull down the granite model, but there are different bunch of models you can pull down from InstructLab. It sets up the llamacpp-server and Streamlit chat app. When I go to this running tab, I can see that app is up and running, and I can just go to the interface that they provide me by default. Let’s ask it a question.
Let’s ask it, what is InfoQ Dev Summit? It’s going to do its thinking and give us an answer. I’m just using the interface that they gave me, but while you’re developing your applications, you can also just connect to it for your use case. I haven’t really given it much resources right now to run. That’s why it’s pretty slow. The more powerful machine you have, the better your performance would be. I think it gave us a pretty accurate answer on what InfoQ Dev Summit is. With the click of a few buttons, I have a personal chatbot running on my machine with Podman Desktop. Then there’s also the Bootc extension over here. This helps you create those bootable OS container images. You click on this, it gives you the ability to be able to switch between disk images and all of that. That’s something you can also play around with.
Get Started with Podman and Podman Desktop
Podman is open source, completely free to use. Same for Podman Desktop. There’s a pretty big community around it, discussions, PRs, issues, contributions, everything are welcome. You can check out our podman.io document page to get started.
See more presentations with transcripts
