In this article, we are going to install our own docker registry on our k3s cluster. This will provide storage for custom docker images that we are going to build in the future. This will also provide a place on our cluster for k3s to pull custom images from when deploying.

2021-09-07: This article has been updated for k3s versions v1.21.0+k3s1 and above. This version of k3s uses Traefik v2 (for new installations) which has different configuration syntax than was used in the original article. If you need a reference to the old configurations, they can be found here.

Materials needed

To follow along with this article, you will need a running k3s cluster. You will also need a working cert-manager install on the cluster with a Let's Encrypt issuer set up on it. If you don't have that setup, check out the previous articles linked above to learn how to do that.

What are we building?

Let's run through a quick set of requirements for our registry.

• We want the registry to be a private registry for our own images.
• We want the registry to be internet accessible so we can push to it from anywhere, not just in the cluster.
• As a internet-accessible registry, we want to make sure it is secured with TLS. We'll have traefik automatically procure a TLS certificate for us.
• And we want to configure basic auth on it so that it requires a username and password to access to allow us to control who accesses it.

Creating basic auth credentials

First, we'll create our basic auth credentials file. You will need the htpasswd utility for this. On Ubuntu-based systems, this utility is packaged in the apache2-utils package. We can install that with a standard apt install command.

sudo apt install -y apache2-utils

If you're not using Ubuntu you can search your package manager for the utility or you can search the web for htpasswd generator to find online generators. I can't vouch for the security of those, but they are available if you can't find the tool or don't want to install it.

Once the tool is installed, we can invoke it, giving it the name of the output file, htpasswd, and a username, in this case we'll use registry as our username. It will prompts us for a password, which we'll make up. We will also need to confirm the password.

htpasswd -Bc htpasswd registry

This will create the file htpasswd on our local file system.

We need to store this file in Kubernetes so that our docker registry can access it. We will do that by adding it as a Kubernetes secret. To do that, we use kubectl to create a secret named docker-registry-htpasswd from the local htpasswd file.

kubectl create secret generic docker-registry-htpasswd --from-file ./htpasswd

We can confirm that that worked

$ kubectl describe secret docker-registry-htpasswd
Name:         docker-registry-htpasswd
Namespace:    default
Labels:       <none>
Annotations:  <none>

Type:  Opaque

Data
====
htpasswd:  70 bytes

Creating the registry

Deployment configuration

Now we're ready to create the registry YAML configuration file. We will call the file registry.yaml. We're going to look at it in parts, but all the configuration can be stored in the same file. If you don't want to create the file yourself, you can download it (and all the files used in this article). Let's look at the Deployment configuration first

apiVersion: apps/v1
kind: Deployment
metadata:
  name: docker-registry
  labels:
    app: docker-registry
spec:
  replicas: 1
  selector:
    matchLabels:
      app: docker-registry
  template:
    metadata:
      labels:
        app: docker-registry
    spec:
      containers:
      - name: docker-registry
        image: registry
        ports:
        - containerPort: 5000
        volumeMounts:
        - name: storage
          mountPath: /var/lib/registry
        - name: htpasswd
          mountPath: /auth
          readOnly: true
        env:
        - name: REGISTRY_AUTH
          value: htpasswd
        - name: REGISTRY_AUTH_HTPASSWD_REALM
          value: Docker Registry
        - name: REGISTRY_AUTH_HTPASSWD_PATH
          value: /auth/htpasswd
        - name: REGISTRY_STORAGE_DELETE_ENABLED
          value: "true"
      volumes:
      - name: storage
        emptyDir: {} # FIXME -make this more permanent later
      - name: htpasswd
        secret:
          secretName: docker-registry-htpasswd

Everything up to the spec section is pretty much boilerplate. The important things we did there were setting all the names and labels. In this case, we made up the name docker-registry for our deployment/app.

We made our container name docker-registry as well and we specified that we want it to use the registry image because the official docker hub name for a docker registry is just registry. The registry image supports ARM targets automatically, so we don't have to specify that here. When we pull the image from our Pi, it will detect the ARM architecture and pull down the correct one.

We set the containerPort to the default registry port 5000.

We also set up a couple of volumes. One for storage of our images and the other for our authentication secret (e.g. the htpasswd file).

Fort the storage, we specified a name of storage and a type of emptyDir with a note to make this more permanent later.

Just a quick note about emptyDir. This kind of volume is created when the pod is created on a node. It uses the storage mechanism of the node. In our case that will be the SD card on the Pi. Also, it only exists as long as the pod is running on that node. It will survive crashes of the pod, but pod deletes or switches to another node will cause the storage to be deleted. This is obviously not ideal, but we have not yet learned about better Kubernetes storage options. We will soon though, so stay tuned!

Next, we used volumeMounts to mount the volume named storage at a path, in the container, of /var/lib/registry. which is where docker registyr looks to store pushed images by default.

For authentication, we created another volume arbitrarily named htpasswd and set its type to secret. We then specified the secretName to use, the secret we created earlier, docker-registry-htpasswd.

With that specified, we added another entry in volumeMounts that mounts the volume named htpasswd at a path of /auth. Just to make sure some bug doesn't have the ability to overwrite our authentication file, we specified that mount as readOnly.

That gets the file in the container, but now we have to tell the registry about it. The registry image is set up to allow configuration with environment variables. Fortunately, Kubenetes makes that easy!

To do the configuration, we created another section in the container called env. We then added the following variables

Variable	Value	Description
REGISTRY_AUTH	htpasswd	Tells the registry we're using htpasswd-based basic authentication.
REGISTRY_AUTH_HTPASSWD_REALM	Docker Registy	The basic auth "Realm" name that shows up in browser when you are prompted for username/password.
REGISTRY_AUTH_HTPASSWD_PATH	/auth/passwd	The path to the credientials file in our container.
REGISTRY_STORAGE_DELETE_ENABLED	true	Unrelated to auth, allows use to delete images from the registry.

Just a note on that last one, the registry doesn't allow deletes by default because images going away in public registries are generally bad things. However, we have a private registry, so we may want to do that in the future. So, we just went ahead an added that here.

That does it for configuring the pod! Let's do a quick recap...

We've setup a volume for storage. We've set up a volume for authentication pointing to a secret with our username and pasword in it, and we told the registry to use it via environment variables.

Service configuration

Adding a service configuration will make it so that we can have traffic routed to our pod, so let's do that.

---
apiVersion: v1
kind: Service
metadata:
  name: docker-registry-service
spec:
  selector:
    app: docker-registry
  ports:
    - protocol: TCP
      port: 5000

This one is super simple. We have specified an arbitrary name docker-registry-service and used the selector field to direct this service to the app we created above docker-registry. (It's important that the app selector matches the name we used for the app label in the Deployment). We specified that we are using TCP as our protocol and we want to receive and send traffic on/to port 5000. Service configured!

Ingress configuration

Now we want to set up the ingress record.

---
apiVersion: cert-manager.io/v1
kind: Certificate
metadata:
  name: docker-carpie-net
  namespace: default
spec:
  secretName: docker-carpie-net-tls
  issuerRef:
    name: letsencrypt-prod
    kind: ClusterIssuer
  dnsNames:
  - docker.carpie.net
---
apiVersion: traefik.containo.us/v1alpha1
kind: IngressRoute
metadata:
  name: docker-registry-ingress-secure
spec:
  entryPoints:
    - websecure
  routes:
  - match: Host(`docker.carpie.net`)
    kind: Rule
    services:
    - name: docker-registry-service
      port: 5000
  tls:
    secretName: docker-carpie-net-tls

Here we have requested a TLS certificate named docker-carpie-net that will store automatically retrieved certificate files for host docker.carpie.net, in a secret named docker-carpie-net-tls.

We have also specified an ingress route record named docker-registry-ingress-secure. We've told Traefik to consider traffic coming in on the websecure entry point (e.g. HTTPS/port 443).

In the routes section, we specified that we want to match on the host name docker.carpie.net (in my case at least, yours will be a domain you own)and that we want to route matching traffic to the services docker-registry-service on port 5000.

In the tls section, we tell Traefik that we are using the TLS secret docker-carpie-net-tls. This also tells the ingress controller that we want to use TLS and it should provide TLS termination for us.

We save the file and the configuration is complete!

Double checking the domain name

In order for the TLS certificate request to work, remember, the domain name we're using need to resolve to our k3s cluster. This is done by specifying an "A" record for it with your DNS manager. If you need a refresher on how to do that, reread this section in my cert-manager article. Once that's set, let's proceed...

Deploying the registry

If you've made it this far in the k3s series, the deploy command should be routine to you...

kubectl apply -f registry.yaml

Give that bit of time to spin up and then check it...

$ kubectl get pods
NAME                               READY   STATUS    RESTARTS  AGE
mysite-nginx-679b7ff485-6xpc9      1/1     Running   1         2d14h
docker-registry-6748fd66d6-mskkt   1/1     Running   0         21s

It looks good. Let's check our certificate...

$ kubectl get certificates
NAME                    READY   SECRET                  AGE
k3s-carpie-net-tls      True    k3s-carpie-net-tls      2d14h
docker-carpie-net-tls   True    docker-carpie-net-tls   29s

There's our new certificate and READY is True so we're good to go!

Testing pushes to the registry

Ok, let try this out. The first we need is to have docker on our development machine. If you are running on Ubuntu or a derivative, you can install docker.io with apt which is what we'll do here.

sudo apt install -y docker.io

If you are not on Ubuntu, or you want the latest and greatest, you can just follow the docker installation instructions.

On linux, you need to be in the docker group to run docker as a regular user. So, we'll add our (my, in this case) user

sudo adduser carpie docker

This only needs to be done once, and obviously, you should use your own username. Sadly, getting added to a group requires a logout and back in to take effect. So please do that if you need to.

With docker installed, let's pull the arm32v7 version of our old friend nginx from docker hub to our local docker cache. I'm pulling the arm32v7 specifically because that's what the Pi needs and ultimately this is going to run on the Pi. If I just pulled nginx, I'd get the amd64 version since that's what my PC is.

We can check our local cache with docker images to see what we have

$ docker images
REPOSITORY          TAG            IMAGE ID            CREATED           SIZE
arm32v7/nginx       latest         333dc4d94c11        10 days ago       97.8MB

And there it is. So far so good...

Now when pushing images to a registry, docker uses the leading part of the image name as the destination. Since my registry is docker.carpie.net I need to name image with that name as a prefix to get it to go where I want when I push it.

So let's do that with docker tag

docker tag arm32v7/nginx:latest docker.carpie.net/arm32v7/nginx:latest

We've tagged the latest version of our local arm32v7/nginx as docker.carpie.net/arm32v7/nginx Now when we push that tagged image, docker knows to deliver it to the docker.carpie.net registry.

Let's try a push. We really expect it to fail right now because we set up authentication, but we haven't done any sort of login. Let's make sure that is working...

$ docker push docker.carpie.net/arm32v7/nginx:latest
The push refers to repository [docker.carpie.net/arm32v7/nginx]
c0f6d0a8c6b4: Preparing
d2abb7e3ea3a: Preparing
e01ca37ce623: Preparing
no basic auth credentials

There we go, no basic auth credentials. Good! Our basic auth is preventing anoymous pushes.

Now let's login with docker login

docker login https://docker.carpie.net

Remeber we specified registry as our username, and we'll type the password we chose when we set up our credentials.

Now that we are logged in, Let's try the push again...

$ docker push docker.carpie.net/arm32v7/nginx:latest
The push refers to repository [docker.carpie.net/arm32v7/nginx]
c0f6d0a8c6b4: Preparing
d2abb7e3ea3a: Preparing
e01ca37ce623: Preparing
latest: digest:
sha256:585c1ec805ab799d7a8e5082d94aace5c3f1455b75f103ca5ca2b45fdbee75fc size: 948

That's better!

Let see if we can pull it. Right now, we have both images in our local docker cache. So let's delete them both

docker rmi arm32v7/nginx docker.carpie.net/arm32v7/nginx

A quick check

$ docker images
REPOSITORY          TAG            IMAGE ID            CREATED           SIZE

Ok the cache is clear. Let's pull that image back from our registry.

docker pull docker.carpie.net/arm32v7/nginx

Now check our local cache

$ docker images
REPOSITORY                       TAG       IMAGE ID            CREATED           SIZE
docker.carpie.net/arm32v7/nginx  latest    333dc4d94c11        10 days ago       97.8MB

And there it is! It's working!

Seeing what's in our registry

At some point, if you're like me, you'll forget what is in your registry. Sadly, there is no docker command line way to query a registry to determine it's contents.

We can use the docker API however. Here's how that would look using curl...

curl -X GET --basic -u registry https://docker.carpie.net/v2/_catalog | python -m json.tool

We use --basic to tell curl we're using basic auth. -u registry is our use name. The URL is the API url for our registry's catalog and | python -m json.tool is just a pipe to a python tool that will pretty-print the JSON result of this API.

This will return something like

{
    "repositories": [
        "arm32v7/nginx"
    ]
}

That shows us that we have a single repository in our registry, arm32v7/nginx.

Congratulations on a working docker registry!

In the next article

In the next article, we are going to learn how to populate the registry with our own custom images. We'll have some hoops to jump through to create those custom ARM images on our amd64 PC, but nothing we can't handle!

Thanks for reading!