Kubernetes development is not one-size-fits-all. Maybe you’re learning Kubernetes with Minikube on your local machine; maybe you’re part of a large organization with many clusters; maybe your cluster is an on-prem lab, or lives in the cloud.
But whether you’re a cluster operator managing policies, an app developer test-driving a new service, or a data scientist running Kubeflow, chances are you are doing some (or all) of: connecting to a cluster, inspecting its state, creating resources, and debugging those resources.
As a developer relations engineer for Kubernetes, I work a lot with demo code, samples, and sandbox clusters. This can get interesting to keep track of (read: total chaos). So in this post I’ll show some of the tools that make my Kubernetes life a lot better.
This environment can work no matter what flavor of Kubernetes you’re running, and all these tools are made possible by the amazing open source community.
~/.zshrc file has an essential line:
source <(kubectl completion zsh)
This enables tab completion for kubectl commands. No more copy-pasting pod names!
On any given day, I might switch between three clusters. This might describe you
too! Do you get annoyed anytime you have to open your
Luckily, there is kubectx for exactly this
kubectx lets you navigate between cluster
easily. My favorite thing is running
kubectx -, which takes you to the cluster
you were using last.
Now that we’ve got a cluster to work with, let’s do stuff.
Maybe you’ve felt that kubectl commands can get very long, with lots of command-line flags. I have found that kubectl tab completion, plus a comprehensive set of aliases (command shortcuts), helps a lot.
Here is a great list of kubectl aliases, that lets you run things like:
krew lets you browse, install, and use kubectl plugins, so that you can run custom commands.
Now that we have a cluster ready to go, let’s deploy something.
Developing on Kubernetes means writing, managing, updating, and deploying lots of YAML files. I keep all my YAML files in Git. Adopting GitOps early (rather than keeping files locally) lets me see revision history as I’m doing early debugging, and sets me up for success later if/when I start formalizing a pipeline for the application I’m working on. (example: a Github webhook for a CI/CD pipeline.)
I use VSCode as my text editor, plus the Moonlight theme. And while VSCode has a ton of great features on its own, Red Hat’s YAML Support plugin is very handy for validation, autocompletion, and formatting.
Right now my process of writing Kubernetes YAML is fairly manual. But usually for every new project, I’m writing the same Kubernetes specs: ConfigMap, Secret, Volume, Deployment, Service.
I’m actively looking into ways to streamline this process, whether through text editor extensions, templating, or other tools. If you have tools that you use to help write and manage YAML, comment down below! ⬇️
We have our YAML files. Now we can deploy the resources! Given my souped-up
kubectl environment, it’s tempting to do this manually.
But this can be a struggle road, dragging you into the same
kubectl apply, and
kubectl delete pod commands. No fun.
A tool called skaffold can automate away much of this pain. skaffold is magical: it watches your codebase for changes. When you save changes locally, skaffold will automatically docker build, push a new image tag, and redeploy into your cluster.
One really cool thing skaffold does is auto-generate image tags. So in your actual YAML, you just list the image repo, not the tag, and skaffold will populate the new tag(s) on deploy.
spec: containers: - name: helloworld image: gcr.io/megangcp/helloworld imagePullPolicy: Always ports: - containerPort: 8080
All that skaffold needs is a (yes, YAML) configuration file:
apiVersion: skaffold/v1beta3 kind: Config build: artifacts: - image: gcr.io/megangcp/helloworld deploy: kubectl: manifests: - kubernetes/*
This is a minimal config where I specify my image repo (in this case, in Google Container Registry, but any image registry like DockerHub also works). I also specify directory where my manifests live.
skaffold is highly customizable, and can work with deploy tools like Helm in addition to kubectl.
skaffold abstracts the docker build process, but sometimes I want to look into my newly-built images to inspect things like: what is the image size compared to previous versions? what are the contents of each of my image layers?
dive is an amazing tool for inspecting Docker images.
With dive, I can examine filesystem changes between different layers. This is super helpful if something in my Docker build has gone wrong.
Now we’ve got pods running Kubernetes. What next?
Every so often (read: all the time) something goes wrong — with my spec, or with my application code.
My kubernetes debugging workflow is usually:
- Describe the pod (
kdpoalias). Is it my spec’s fault? (example: is the Deployment trying to mount a Secret I accidentally put in a different Namespace?) If not…
- Get the pod logs. the skaffold
command will combine all the logs for every container deployed and stream all of
it to stdout. But I’ve found that when I have two or more pods running, that
format gets noisy. At the same time, the usual
kubectl logscommand can result in an endless cycle of copy-pasting new pod names.
stern is a great alternative for tailing logs in a more customizable way. stern uses regular expressions to select on pods — and given that all pods start with their deployment name, you can follow logs for all the pods in a deployment, without having to know the exact pod name. Super helpful:
If the logs aren’t giving me clues for what’s gone wrong, usually I’ll…
- Exec into the pod (
kexalias with tab completion):
Kubernetes is a big, complex piece of software with a large configuration model. I hope sharing some of these tools might help you, wherever you’re at in your k8s journey.
Here’s the full list of tools and plugins mentioned in this post: