I’m bored and I want to learn this Kubernetes thing.
Why?
A colleague told me “I’m bored and I want to learn this Kubernetes thing. Any hints?“. I’ll use this page as notes to myself. They are based on my personal preferences, might be completely wrong and should not be seen as a decent advice or even an advice. You’ll probably be better if you do the opposite.
Background
Kubernetes is massively-massive on its own. But if one adds the world around it, then it is not just massive. It is scary. There are a lot of options and a lot of choices. They all look awesome. And they are all very-very-very nice and obviously they should be used. Which then might result in what’s called “analysis paralysis”. Talking about it, I’ve just spent half an hour looking for Markdown editor for Windows. And then couple of minutes trying to figure out how to enable spell check. Just so you know - Zettlr and expand settings pop-up to see labels. Also, I might have to update this blog to Astro 4.0 because apparently it is out. And may be try another engine while I’m at it. Anyway, back to the topic of this post.
Plan
I have to come-up with something which touches as much of the world of k8s, but in an accessible way. It should allow trying various things, but should not require two years of digging into details (if one wants, of course). My idea is:
- something small - an API which sends messages using Azure Service Bus (but really, anything that has free tier). Then separate component which processes these messages.
- Both should be source-controlled
- Compiled and deployed as automatically as possible
- Lives in containers (duh)
- Should use some sort of configuration and secrets
- A database outside the cluster
- Should require some sort of policy in k8s
- Monitoring, observability and logging are nice to have, but not required (I’m hearing the boo-ing, but will ignore it)
- Should use fancy database
- Should use messaging
In short - should be very impractical, but with trying to touch as much things as possible. Some about k8s, some not.
Sources
What is it that we are building
That was a real yak shave, if I’ve ever seen one. I started writing this on my Windows laptop, had to go out, come back and decided to finish it on my PC which runs Linux. Turns out, there is Zettlr for Linux. I’m installing things anyway, let’s upgrade Astro. Did that. Build script was giving warnings about old version of Node.js. Updated Node. In the meantime Ubuntu installed some updates and asked for restart. Restarted. By that time it was too late to do anything else. Typical.
Back to the post. Task at hand is to build two components:
- API - simple, one endpoint accepting some sort of payments. It should publish them to a messaging platform
- Event listener component which processes events from the API and stores them in the fancy database
This makes no sense, right? It is overly and pointlessly complex. Still, bear with me, it will make sense eventually. All sources should live private repository in GitHub, GitLab or similar. Of course, repository can be public, up to you. Both now offer private repositories for free. I would go for GitHub.
Dapr
IMHO, Dapr is an interesting way to solve problems we face over and over and over again. It saves me from trying to remember for 1000th time how to connect to Azure Service Bus or Kafka. Let’s use it.
Messaging
Because Dapr, messaging choices are limited to what’s supported by Dapr. In other words, there is no really-real limit. Kafka is always a good start, but I’m not aware of free hosted option. Azure Service Bus is a handy alternative - cost of operations is so low that for test purposes it is free. It is possible to run a Kafka-compatible option locally - Redpanda. RabbitMQ can also be an option.
Messages sent by the API should use CloudEvents spec.
Database
There is absolutely nothing wrong with using SQL database. But for extra fun, try something like fauna.
Build
All build outputs should end-up in containers because, well, we need them in k8s. Apart from manually writing Dockerfiles and such, try Buildpacks.io. If manual is preferred way, try Podman. For automated build, use GitHub Actions. Or whatever is in GitLab. All containers should be stored somewhere. DigitalOcean offers private registry with very nice free option. Why DO and not public Docker Hub repository - to figure out how to authenticate with private registry. If you want to try something different, there is Dagger. And Tilt. And Earthly.
Which Kubernetes
Any cloud offers managed service, but they also ask for money. What if we want to run this locally and preferably for free? There are options of course - k3s, k0s, microk8s. I’m excluding minikube and kind from the list as I’ve got a requirement to offer a “proper, real k8s running on multiple nodes, but easy to install”. What - k3s. Why - When I first looked into it, it was the the only “small” option and I could run it on Raspberry Pi. It made all the hard choices for me and it takes about 5 minutes to install on three nodes. If one wants to really understand what’s going on and make all the choices, there is “Bootstrap Kubernetes the hard way. No scripts.”. Why is this Windows Copilot so insistent on sending URLs to it?
Deploy
Manually write files for all necessary k8s objects - deployment, service, configuration, etc. Manually deploy these in your brand new k8s. This might sound simple, but in fact it is not:
- You’ll need Dapr installed in k8s.
- Configure secrets for access to your private registry.
- Figure out how to deal with secrets - there is Vault. But there is a simpler way - Sealed Secrets. Let’s go for the simple option. Also check Pulumi if you don’t like yaml. Fun fact - all k8s policies can be written using json. Yaml is not mandatory. But no one is doing json. Obviously we all love to spend hours trying to figure out where did we put that extra space which now breaks the whole policy.
Deploy harder
Now, I know that everyone deploys and tests in production and DEV/QA/UAT environments are for people who obviously have low self-esteem. Still, let’s simulate QA, UAT and PROD environments using k8s namespaces. Which then leads us to how to deal with environment specific configurations and secrets. There is Kustomize. There is also Helm. Choosing one or the other is just personal preference. I prefer Helm. No matter which, Monokle Desktop can help with edit and preview.
Deploy even harder
Deploy either manually or from a pipeline is basically push - we tell k8s something. But there is also pull model (aka GitOps). Try Flux or Argo CD. Flux has less dependencies (Argo needs Redis). Argo has UI and Flux does not. I would go for Flux (because less dependencies).
Everything else
This is long enough, but just barely scratches the surface of what and how in k8s.
Mesh networking
Pods can talk to each other without any extra help. But with mesh networking we get insights, extras and TLS. Forget about everything else, but free TLS inside network is pretty much mandatory these days. Dapr has TLS, so for basic needs, it is enough. Also check Linkerd. Specifically Linkerd policies.
Check security issues
Trivy will tell you how many CVEs are in your container. Chances are that these are in your base container. Frankly, there isn’t much you can do. You can, of course, build your own base container. But are you really that good? And you will be that good in keeping it up to date? I don’t think so. Still, try Chainguard.
Policy
Like “Do not allow containers from random registries”. Kyverno can do this and more. Why is this important - because predictability, security and fanciness. We really don’t want random things in our cluster. Some can be harmless, some might be trying lateral attacks.