author-pic

Amila Senadheera

Tech enthusiast

Automate certificate management in Raspberry Pi Kubernetes cluster


Published on November 07, 2022

Under Exposing Web Apps running in our Raspberry Pi cluster post, I have explained how to expose a service running in the cluster. But I didn't mention how to secure the network communication. What I have done was using proxied DNS records (provided by cloudlflare) which make secured connections from clients to the proxy server and do unencrypted communication between proxy to my ingress controller. So users see the HTTPS lock icon in the browser but they never knew Cloudflare to actual server communication in the public network was unencrypted. It was fine since I was serving a blog page, not a payment portal.

Now I think it would be better to avoid unencrypted communication between Cloudflare to the raspberry pi cluster aka use DNS only in Cloudflare and support TLS at the cluster level. There are two ways we can do this:

  • Terminate TLS at ingress controller - This is totally fine as all the infrastructure runs in a private network.
  • TLS passthrough at ingress controller and terminate TLS at Pod level - This is even better but I may post on doing this in a later post.

I will discuss the first scenario in this post.

Securing NGINX-ingress using cert-manager

What is cert-manager?

cert-manager is X.509 certificate controller for Kubernetes. It will obtain certificates from a variety of Issuers, both popular public Issuers as well as private Issuers, and ensure the certificates are valid and up-to-date and will attempt to renew certificates at a configured time before expiry.

cert-manager is a k8s operator which does all the above operations on our behalf. Otherwise, someone else has to be on alert about certificate expiry dates and do it manually.

cert-manager Installation

Visit cert-manager releases and download the latest version of cert-manager.yaml and keep it under your cluster configs directory. And apply it to your cluster:

kubectl apply -f ./cert-manager.yaml

Configure Let's Encrypt Issuer

There are multiple issuers you can choose as the Issuer. I will use Let's Encrypt as our Certificate Authority.

cert-manger has a CRD called Issuer, and we need to configure it like below. We need two Issuers for staging (to test the functionality without being affected by rate limiting restrictions in production ACME Server URL) and production. Make sure to provide an actual email address.

Staging Issuer

apiVersion: cert-manager.io/v1
kind: Issuer
metadata:
  name: letsencrypt-staging
spec:
  acme:
    # The ACME server URL
    server: https://acme-staging-v02.api.letsencrypt.org/directory
    # Email address used for ACME registration
    email: user@developer-diary.com
    # Name of a secret used to store the ACME account private key
    privateKeySecretRef:
      name: letsencrypt-staging
    # Enable the HTTP-01 challenge provider
    solvers:
      - http01:
          ingress:
            class: nginx
kubectl apply -f ./staging-issuer.yaml

Production Issuer

apiVersion: cert-manager.io/v1
kind: Issuer
metadata:
  name: letsencrypt-prod
spec:
  acme:
    # The ACME server URL
    server: https://acme-staging-v02.api.letsencrypt.org/directory
    # Email address used for ACME registration
    email: user@developer-diary.com
    # Name of a secret used to store the ACME account private key
    privateKeySecretRef:
      name: letsencrypt-prod
    # Enable the HTTP-01 challenge provider
    solvers:
      - http01:
          ingress:
            class: nginx
kubectl apply -f ./production-issuer.yaml

Here we are using HTTP01 challenge provider.

Check the status of staging issuer:

kubectl describe issuer letsencrypt-staging
Name:         letsencrypt-staging
Namespace:    default
Labels:       <none>
Annotations:  <none>
API Version:  cert-manager.io/v1
Kind:         Issuer
Metadata:
  ...
  ...
Status:
  Acme:
    Last Registered Email:  user@developer-diary.com
    Uri:                    https://acme-staging-v02.api.letsencrypt.org/acme/acct/75130654
  Conditions:
    Last Transition Time:  2022-11-06T16:39:32Z
    Message:               The ACME account was registered with the ACME server
    Observed Generation:   1
    Reason:                ACMEAccountRegistered
    Status:                True
    Type:                  Ready
Events:                    <none>

You can see that the status with ACMEAccountRegistered reason if all went well.

Modify the Ingress resource to have TLS

Now we need to inform the cert-manager to issue certificates for the domain names we have configured in the Ingress resource.

Go to your Ingress resource YAML, if you followed every post then it's host-ingress.yaml. Update it with the following annotations and a new section called tls under spec:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: host-ingress
  annotations:
    ...
    cert-manager.io/issuer: "letsencrypt-staging"
spec:
  ingressClassName: nginx
  tls:
    - hosts:
        - developer-diary.com
      secretName: dev-diary-tls
  rules:
    - host: developer-diary.com
      http:
        paths:
          - path: /*
            pathType: Prefix
            backend:
              service:
                name: dev-diary-svc
                port:
                  number: 9000
kubectl apply -f ./host-ingress.yaml

Cert-manager will read these annotations and use them to create a certificate, which you can request and see:

kubectl describe certificate dev-diary-tls

This will show something similar to the following:

Name:         dev-diary-tls
Namespace:    default
Labels:       <none>
Annotations:  <none>
API Version:  cert-manager.io/v1
Kind:         Certificate
  ...
  ...
Events:
  Type    Reason     Age   From                                       Message
  ----    ------     ----  ----                                       -------
  Normal  Issuing    49s   cert-manager-certificates-trigger          Issuing certificate as Secret does not exist
  Normal  Generated  48s   cert-manager-certificates-key-manager      Stored new private key in temporary Secret resource "dev-diary-tls-vzzrs"
  Normal  Requested  48s   cert-manager-certificates-request-manager  Created new CertificateRequest resource "dev-diary-tls-r9t4b"
  Normal  Issuing    8s    cert-manager-certificates-issuing          The certificate has been successfully issued

Now we can observe that the certificate has been issued successfully within seconds. This will create a secret with public and private key pair using the name used as secretName in the Ingress resource. Run the following command and check:

kubectl describe secret dev-diary-tls
Name:         dev-diary-tls
Namespace:    default
Labels:       <none>
Annotations:  cert-manager.io/alt-names: developer-diary.com
              cert-manager.io/certificate-name: dev-diary-tls
              cert-manager.io/common-name: developer-diary.com
              cert-manager.io/ip-sans: 
              cert-manager.io/issuer-group: cert-manager.io
              cert-manager.io/issuer-kind: Issuer
              cert-manager.io/issuer-name: letsencrypt-staging
              cert-manager.io/uri-sans: 

Type:  kubernetes.io/tls

Data
====
tls.crt:  5599 bytes
tls.key:  1675 bytes

You can see alternative name and common name and other certificate-related properties have been added as annotations.

Now, if you visit the browser with developer-diary.com browser will still show certificate is not valid. That's totally fine as we are using the staging issuer of Let's Encrypt. But if you inspect the certificate it should be Let's Encrypt-issued certificate and not one issued by Cloudflare. Make sure to turn off the proxy in DNS A record if you are using Cloudflare DNS.

Now we are good to move to production issuer. Just change the cert-manager.io/issuer to letsencrypt-prod apply:

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: host-ingress
  annotations:
    ...
    cert-manager.io/issuer: "letsencrypt-prod"
...
...
kubectl apply -f ./host-ingress.yaml

And, delete the existing secret issued by staging issuer which is dev-diary-tls:

kubectl delete secret dev-diary-tls

This will make cause the updated issuer to issue a new secret with the same name. Now you can do the same Certificate and Secret inspection we did to verify that all went well. Now if you visit your website in the web browser it will show the secured lock icon and you can inspect the new certificate as well. The cert will expire within three months by default and will get renewed automatically before the due date as I hope :)

There are many things to learn still you can refer to the resources in cert-manager and Let's Encrypt.

That's it. Thanks for reading!

If you like it, share it!


© 2022, All Rights Reserved.

Made withusing Gatsby, served to your browser from a home grown Raspberry Pi cluster