Automate certificate management in Raspberry Pi Kubernetes cluster
Published on November 07, 2022
Under Exposing Web Apps running in our Raspberry Pi cluster post, I have explained how to expose a service running in the cluster. But I didn't mention how to secure the network communication. What I have done was using proxied DNS records (provided by cloudlflare) which make secured connections from clients to the proxy server and do unencrypted communication between proxy to my ingress controller. So users see the HTTPS lock icon in the browser but they never knew Cloudflare to actual server communication in the public network was unencrypted. It was fine since I was serving a blog page, not a payment portal.
Now I think it would be better to avoid unencrypted communication between Cloudflare to the raspberry pi cluster aka use DNS only in Cloudflare and support TLS at the cluster level. There are two ways we can do this:
- Terminate TLS at ingress controller - This is totally fine as all the infrastructure runs in a private network.
- TLS passthrough at ingress controller and terminate TLS at Pod level - This is even better but I may post on doing this in a later post.
I will discuss the first scenario in this post.
Securing NGINX-ingress using cert-manager
What is cert-manager?
cert-manager is X.509 certificate controller for Kubernetes. It will obtain certificates from a variety of Issuers, both popular public Issuers as well as private Issuers, and ensure the certificates are valid and up-to-date and will attempt to renew certificates at a configured time before expiry.
cert-manager is a k8s operator which does all the above operations on our behalf. Otherwise, someone else has to be on alert about certificate expiry dates and do it manually.
cert-manager Installation
Visit cert-manager releases and download the latest version of cert-manager.yaml and keep it under your cluster configs directory. And apply it to your cluster:
kubectl apply -f ./cert-manager.yaml
Configure Let's Encrypt Issuer
There are multiple issuers you can choose as the Issuer. I will use Let's Encrypt as our Certificate Authority.
cert-manger has a CRD called Issuer, and we need to configure it like below. We need two Issuers for staging (to test the functionality without being affected by rate limiting restrictions in production ACME Server URL) and production. Make sure to provide an actual email address.
Staging Issuer
apiVersion: cert-manager.io/v1
kind: Issuer
metadata:
name: letsencrypt-staging
spec:
acme:
# The ACME server URL
server: https://acme-staging-v02.api.letsencrypt.org/directory
# Email address used for ACME registration
email: user@developer-diary.com
# Name of a secret used to store the ACME account private key
privateKeySecretRef:
name: letsencrypt-staging
# Enable the HTTP-01 challenge provider
solvers:
- http01:
ingress:
class: nginx
kubectl apply -f ./staging-issuer.yaml
Production Issuer
apiVersion: cert-manager.io/v1
kind: Issuer
metadata:
name: letsencrypt-prod
spec:
acme:
# The ACME server URL
server: https://acme-staging-v02.api.letsencrypt.org/directory
# Email address used for ACME registration
email: user@developer-diary.com
# Name of a secret used to store the ACME account private key
privateKeySecretRef:
name: letsencrypt-prod
# Enable the HTTP-01 challenge provider
solvers:
- http01:
ingress:
class: nginx
kubectl apply -f ./production-issuer.yaml
Here we are using HTTP01 challenge provider.
Check the status of staging issuer:
kubectl describe issuer letsencrypt-staging
Name: letsencrypt-staging
Namespace: default
Labels: <none>
Annotations: <none>
API Version: cert-manager.io/v1
Kind: Issuer
Metadata:
...
...
Status:
Acme:
Last Registered Email: user@developer-diary.com
Uri: https://acme-staging-v02.api.letsencrypt.org/acme/acct/75130654
Conditions:
Last Transition Time: 2022-11-06T16:39:32Z
Message: The ACME account was registered with the ACME server
Observed Generation: 1
Reason: ACMEAccountRegistered
Status: True
Type: Ready
Events: <none>
You can see that the status with ACMEAccountRegistered
reason if all went well.
Modify the Ingress resource to have TLS
Now we need to inform the cert-manager to issue certificates for the domain names we have configured in the Ingress resource.
Go to your Ingress resource YAML, if you followed every post then it's host-ingress.yaml. Update it with the following annotations and a new section called tls
under spec
:
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: host-ingress
annotations:
...
cert-manager.io/issuer: "letsencrypt-staging"
spec:
ingressClassName: nginx
tls:
- hosts:
- developer-diary.com
secretName: dev-diary-tls
rules:
- host: developer-diary.com
http:
paths:
- path: /*
pathType: Prefix
backend:
service:
name: dev-diary-svc
port:
number: 9000
kubectl apply -f ./host-ingress.yaml
Cert-manager will read these annotations and use them to create a certificate, which you can request and see:
kubectl describe certificate dev-diary-tls
This will show something similar to the following:
Name: dev-diary-tls
Namespace: default
Labels: <none>
Annotations: <none>
API Version: cert-manager.io/v1
Kind: Certificate
...
...
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Issuing 49s cert-manager-certificates-trigger Issuing certificate as Secret does not exist
Normal Generated 48s cert-manager-certificates-key-manager Stored new private key in temporary Secret resource "dev-diary-tls-vzzrs"
Normal Requested 48s cert-manager-certificates-request-manager Created new CertificateRequest resource "dev-diary-tls-r9t4b"
Normal Issuing 8s cert-manager-certificates-issuing The certificate has been successfully issued
Now we can observe that the certificate has been issued successfully within seconds. This will create a secret with public and private key pair using the name used as secretName
in the Ingress resource. Run the following command and check:
kubectl describe secret dev-diary-tls
Name: dev-diary-tls
Namespace: default
Labels: <none>
Annotations: cert-manager.io/alt-names: developer-diary.com
cert-manager.io/certificate-name: dev-diary-tls
cert-manager.io/common-name: developer-diary.com
cert-manager.io/ip-sans:
cert-manager.io/issuer-group: cert-manager.io
cert-manager.io/issuer-kind: Issuer
cert-manager.io/issuer-name: letsencrypt-staging
cert-manager.io/uri-sans:
Type: kubernetes.io/tls
Data
====
tls.crt: 5599 bytes
tls.key: 1675 bytes
You can see alternative name and common name and other certificate-related properties have been added as annotations.
Now, if you visit the browser with developer-diary.com browser will still show certificate is not valid. That's totally fine as we are using the staging issuer of Let's Encrypt. But if you inspect the certificate it should be Let's Encrypt-issued certificate and not one issued by Cloudflare. Make sure to turn off the proxy in DNS A record if you are using Cloudflare DNS.
Now we are good to move to production issuer. Just change the cert-manager.io/issuer
to letsencrypt-prod
apply:
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: host-ingress
annotations:
...
cert-manager.io/issuer: "letsencrypt-prod"
...
...
kubectl apply -f ./host-ingress.yaml
And, delete the existing secret issued by staging issuer which is dev-diary-tls
:
kubectl delete secret dev-diary-tls
This will make cause the updated issuer to issue a new secret with the same name. Now you can do the same Certificate and Secret inspection we did to verify that all went well. Now if you visit your website in the web browser it will show the secured lock icon and you can inspect the new certificate as well. The cert will expire within three months by default and will get renewed automatically before the due date as I hope :)
There are many things to learn still you can refer to the resources in cert-manager and Let's Encrypt.
That's it. Thanks for reading!
If you like it, share it!