In a previous article, we worked through a common scenario for building a custom docker image for our k3s cluster. In this article, we are going look at creating a custom image from scratch. Well, it's not really from scratch. You'll always have a base image to start from. Most of the bases are really just stripped down Linux images. We have bases for alpine, debian, ubuntu, etc. From these bases we can do whatever we want! Let's try that out!

Materials needed

To follow along with this article, you will need:

• Your running k3s cluster
• A working docker registry
• docker installed on your development PC

The files and source code used in this project are available as downloadable resources.

• bride-api project

Example: Custom backend API service

For the sake of example, let's build a backend service using Node JS. Again, I've pre-made an image for us to use in this example. Let's go get it.

cd ..
git clone https://gitlab.com/carpie/bride-api
cd bride-api

In case you are wondering, this is an incredibly useful service that, upon request, serves up quotes from a now quite old movie.

We know that we are going to be building an ARM image so let's go ahead and copy over that magically ARM emulator for use in our docker file.

cp /usr/bin/qemu-arm-static .

Crafting the Dockerfile

I've provided two docker files in this repository. Node actually has its own base image that we really should use if we are going to do something like this. You can see how that would work by reading Dockerfile.recommended. However, in order to demonstrate the process of really creating something from scratch, we are going to use a more manual method. That method is provided in Dockerfile.manual, but let's go ahead and build it from scratch here to better show the thought process that goes in to it.

If we think about what steps we would do to build a custom Linux box to run whatever it is we want to run, those are the same steps we do in a docker file. Once the docker file is written, we build it in to an image and we can reuse that image anytime we need that software. We could also use that image as a base for new images.

Let's stub out Dockefile with some comments

# Start with a base
# Install node manually
# Copy in our custom code (our service)
# Build our service code (in the image itself)
# Specify the command to run when the image is started

Let's fill these in one by one...

# Start with a base
FROM arm32v7/debian:buster-slim
COPY qemu-arm-static /usr/bin/

We start with a base. In this case we've used debian:buster-slim as it is a bit smaller than standard debian. On the second line, we've copied in our emulator so that the following RUN commands will work while we are building the image on our amd64 PC. Also, it will allow us to test the image locally on our PC before deploying as well.

...
# Install node manually
SHELL ["/bin/bash", "--login", "-c"]
RUN apt-get update && apt-get install -y wget libatomic1 && rm -rf /var/lib/apt/lists/*
RUN wget -qO- https://raw.githubusercontent.com/nvm-sh/nvm/v0.35.3/install.sh | bash
RUN nvm install lts/erbium

In this block, we install node using the node version manager, nvm. The steps required to install node are specific to node and nvm. Obviously, if you were installing something else, you would have your own set of steps for installation.

In nvm's case, we need a bash login shell because nvm stores some environment variables it needs to run in .bashrc. The above SHELL command lets us set that up.

Next, we install some utilities we need using apt just like we would on a real debian box. In the case of node, we need wget and libatomic1. The rm command just cleans up the lists that apt-get update creates since we don't need those just hanging around taking up space in our image.

Next we use wget to download and install nvm. This is taken directly from nvms installation instructions. And finally, we install node 12 (erbium with nvm.

...
# Copy in our custom code (our service)
COPY ./ /app
WORKDIR /app

In this block we use the COPY command to copy our source into the directory /app in the image. This directory is arbitrary. I made it up. Then we set WORKDIR to /app. This means the rest of the RUN commands and the CMD command will be invoked in that directory.

...
# Build our service code (in the image itself)
RUN npm ci

Here we simply invoke npm ci to install the node packages our service needs.

...
# Specify the command to run when the image is started
CMD npm start

With the CMD command, we specify the command that will be executed when we invoke the image.

That's it! All done!

Building the image

Now we can build an image.

docker build -t docker.carpie.net/bride-api:v1.0.0 .

This may take a bit. It's going to download our base debian image, execute all those RUN commands to install stuff, copy our application code, and install its needed node packages.

Because we used the emulator image, we can test this out locally.

docker run -it --rm -p 1987:1987 docker.carpie.net/bride-api:v1.0.0

We can see the message PB quotes being served on port 1987. Now if we invoke curl in another terminal we should get a quote back.

$ curl http://localhost:1987
You keep using that word. I do not think it means what you think it means.

And there's our quote. Great! Now kill the container by pressing Ctrl-C in its terminal.

Things are now working locally. Let's push that image up to our docker registry.

docker push docker.carpie.net/bride-api:v1.0.0

Now the image is in our registry, we just need to deploy it. I've pre-created the yaml file, bride.yaml. Let's take a look at it.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: bride-node
  labels:
    app: bride-node
spec:
  replicas: 1
  selector:
    matchLabels:
      app: bride-node
  template:
    metadata:
      labels:
        app: bride-node
    spec:
      containers:
      - name: bride-node
        image: docker.carpie.net/bride-api:v1.0.0
        ports:
        - containerPort: 1987
      imagePullSecrets:
      - name: regcred
---
apiVersion: v1
kind: Service
metadata:
  name: bride-node-service
spec:
  selector:
    app: bride-node
  type: NodePort
  ports:
    - protocol: TCP
      port: 1987

Looking at the important parts, we see the image is set to what we just pushed, docker.carpie.net/bride-api:v1.0.0. And the container port is 1987. Also, we have our imagePullSecrets entry set so that Kubernetes can authenticate with our registry.

You may notice there is no ingress section. Typically you won't expose backend services to the outside world. Instead they will be used by other services inside your cluster. Kubernetes makes this easy to do and we'll explore that in just a moment. Because we are going try this out from our development PC, we make a concession for convenience and specify a service type of NodePort.

Let's do our standard deploy.

kubectl apply -f bride.yaml

Give it some time to spin up and check to see if it's running.

$ kubectl get pods
NAME                               READY   STATUS    RESTARTS   AGE
docker-registry-6748fd66d6-mskkt   1/1     Running   7          142d
mysite-nginx-679b7ff485-6xpc9      1/1     Running   8          147d
four-nginx-5b649489bd-t82jm        1/1     Running   0          25m
bride-node-5dc6f4f665-fhkk6        1/1     Running   0          2m41s

Looks good. Let's take a look at the services to better see what NodePort does.

$ kubectl get services
NAME                    TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)
...
bride-node-service      NodePort    10.43.176.241   <none>        1987:32419/TCP

We can see our service is of type Nodeport and has its port 1987 mapped to a node port of 32419. So what this does is open up that port on all our nodes. So we can connect any node on that port and it will route to our service. Just so it's clear, this routing is done by Kubernetes. Traefik has no knowledge of this service since we didn't ingress it.

To see this in action, we can use curl to hit all our nodes.

$ curl kmaster:32419
Alright, now let\'s see, where were we? Oh, yes. In the pit of despair.
$ curl knode1:32419
Have fun storming the castle!
$ curl knode2:32419
Inconceivable!

No matter which node we connect to, we get routed to our service and we get back a quote.

So that's cool for testing and some manual things we may do from our PC. But what if we had another pod that wanted to use our backend service? It's incredibly simple. Kubernetes maintains an internal DNS, so we just reference our service by it's name and service port!

Let's try that by spinning up a quick busybox pod running on the cluster that we can interact with.

$ kubectl run -it --rm --generator=run-pod/v1 busybox --image=busybox
If you don't see a command prompt, try pressing enter.
/ # wget -qO- bride-node-service:1987
Rodents of Unusual Size? I don't think they exist.
/ # wget -qO- bride-node-service:1987
Let me explain... No, there is too much. Let me sum up.

This is a pod running inside the cluster that we shell'd into. From inside the pod, we can use wget to grab quotes from our service just using it's name and service port. We can communicate between services just by knowing their names and ports, no matter where they are running or how many replicas they have! Easy!

To exit this pod, hit Ctrl-D or type exit and the pod will be destroyed.

There we go! We have a custom backend service, built from scratch, running on our cluster! We can build any image we wish using the same general procedure.

Cleanup

If we want to clean these samples off our cluster, we can just use the delete command that we've used before on the same yaml file.

kubectl delete -f bride.yaml

In the next article

In the next article, we are going to finally tackle adding persistent storage to our cluster!

Thanks for reading!