Automate certificate management in Raspberry Pi Kubernetes cluster
Published on November 07, 2022
Under Exposing Web Apps running in our Raspberry Pi cluster post, I have explained how to expose a service running in the cluster. But I didn't mention how to secure the network communication. What I have done was using proxied DNS records (provided by cloudlflare) which make secured connections from clients to the proxy server and do unencrypted communication between proxy to my ingress controller. So users see the HTTPS lock icon in the browser but they never knew Cloudflare to actual server communication in the public network was unencrypted. It was fine since I was serving a blog page, not a payment portal.
Now I think it would be better to avoid unencrypted communication between Cloudflare to the raspberry pi cluster aka use DNS only in Cloudflare and support TLS at the cluster level. There are two ways we can do this:
- Terminate TLS at ingress controller - This is totally fine as all the infrastructure runs in a private network.
- TLS passthrough at ingress controller and terminate TLS at Pod level - This is even better but I may post on doing this in a later post.
I will discuss the first scenario in this post.
Securing NGINX-ingress using cert-manager
What is cert-manager?
cert-manager is X.509 certificate controller for Kubernetes. It will obtain certificates from a variety of Issuers, both popular public Issuers as well as private Issuers, and ensure the certificates are valid and up-to-date and will attempt to renew certificates at a configured time before expiry.
cert-manager is a k8s operator which does all the above operations on our behalf. Otherwise, someone else has to be on alert about certificate expiry dates and do it manually.
Visit cert-manager releases and download the latest version of cert-manager.yaml and keep it under your cluster configs directory. And apply it to your cluster:
kubectl apply -f ./cert-manager.yaml
Configure Let's Encrypt Issuer
There are multiple issuers you can choose as the Issuer. I will use Let's Encrypt as our Certificate Authority.
cert-manger has a CRD called Issuer, and we need to configure it like below. We need two Issuers for staging (to test the functionality without being affected by rate limiting restrictions in production ACME Server URL) and production. Make sure to provide an actual email address.
apiVersion: cert-manager.io/v1 kind: Issuer metadata: name: letsencrypt-staging spec: acme: # The ACME server URL server: https://acme-staging-v02.api.letsencrypt.org/directory # Email address used for ACME registration email: firstname.lastname@example.org # Name of a secret used to store the ACME account private key privateKeySecretRef: name: letsencrypt-staging # Enable the HTTP-01 challenge provider solvers: - http01: ingress: class: nginx
kubectl apply -f ./staging-issuer.yaml
apiVersion: cert-manager.io/v1 kind: Issuer metadata: name: letsencrypt-prod spec: acme: # The ACME server URL server: https://acme-staging-v02.api.letsencrypt.org/directory # Email address used for ACME registration email: email@example.com # Name of a secret used to store the ACME account private key privateKeySecretRef: name: letsencrypt-prod # Enable the HTTP-01 challenge provider solvers: - http01: ingress: class: nginx
kubectl apply -f ./production-issuer.yaml
Here we are using HTTP01 challenge provider.
Check the status of staging issuer:
kubectl describe issuer letsencrypt-staging
Name: letsencrypt-staging Namespace: default Labels: <none> Annotations: <none> API Version: cert-manager.io/v1 Kind: Issuer Metadata: ... ... Status: Acme: Last Registered Email: firstname.lastname@example.org Uri: https://acme-staging-v02.api.letsencrypt.org/acme/acct/75130654 Conditions: Last Transition Time: 2022-11-06T16:39:32Z Message: The ACME account was registered with the ACME server Observed Generation: 1 Reason: ACMEAccountRegistered Status: True Type: Ready Events: <none>
You can see that the status with
ACMEAccountRegistered reason if all went well.
Modify the Ingress resource to have TLS
Now we need to inform the cert-manager to issue certificates for the domain names we have configured in the Ingress resource.
Go to your Ingress resource YAML, if you followed every post then it's host-ingress.yaml. Update it with the following annotations and a new section called
apiVersion: networking.k8s.io/v1 kind: Ingress metadata: name: host-ingress annotations: ... cert-manager.io/issuer: "letsencrypt-staging" spec: ingressClassName: nginx tls: - hosts: - developer-diary.com secretName: dev-diary-tls rules: - host: developer-diary.com http: paths: - path: /* pathType: Prefix backend: service: name: dev-diary-svc port: number: 9000
kubectl apply -f ./host-ingress.yaml
Cert-manager will read these annotations and use them to create a certificate, which you can request and see:
kubectl describe certificate dev-diary-tls
This will show something similar to the following:
Name: dev-diary-tls Namespace: default Labels: <none> Annotations: <none> API Version: cert-manager.io/v1 Kind: Certificate ... ... Events: Type Reason Age From Message ---- ------ ---- ---- ------- Normal Issuing 49s cert-manager-certificates-trigger Issuing certificate as Secret does not exist Normal Generated 48s cert-manager-certificates-key-manager Stored new private key in temporary Secret resource "dev-diary-tls-vzzrs" Normal Requested 48s cert-manager-certificates-request-manager Created new CertificateRequest resource "dev-diary-tls-r9t4b" Normal Issuing 8s cert-manager-certificates-issuing The certificate has been successfully issued
Now we can observe that the certificate has been issued successfully within seconds. This will create a secret with public and private key pair using the name used as
secretName in the Ingress resource. Run the following command and check:
kubectl describe secret dev-diary-tls
Name: dev-diary-tls Namespace: default Labels: <none> Annotations: cert-manager.io/alt-names: developer-diary.com cert-manager.io/certificate-name: dev-diary-tls cert-manager.io/common-name: developer-diary.com cert-manager.io/ip-sans: cert-manager.io/issuer-group: cert-manager.io cert-manager.io/issuer-kind: Issuer cert-manager.io/issuer-name: letsencrypt-staging cert-manager.io/uri-sans: Type: kubernetes.io/tls Data ==== tls.crt: 5599 bytes tls.key: 1675 bytes
You can see alternative name and common name and other certificate-related properties have been added as annotations.
Now, if you visit the browser with developer-diary.com browser will still show certificate is not valid. That's totally fine as we are using the staging issuer of Let's Encrypt. But if you inspect the certificate it should be Let's Encrypt-issued certificate and not one issued by Cloudflare. Make sure to turn off the proxy in DNS A record if you are using Cloudflare DNS.
Now we are good to move to production issuer. Just change the
apiVersion: networking.k8s.io/v1 kind: Ingress metadata: name: host-ingress annotations: ... cert-manager.io/issuer: "letsencrypt-prod" ... ...
kubectl apply -f ./host-ingress.yaml
And, delete the existing secret issued by staging issuer which is
kubectl delete secret dev-diary-tls
This will make cause the updated issuer to issue a new secret with the same name. Now you can do the same Certificate and Secret inspection we did to verify that all went well. Now if you visit your website in the web browser it will show the secured lock icon and you can inspect the new certificate as well. The cert will expire within three months by default and will get renewed automatically before the due date as I hope :)
That's it. Thanks for reading!
If you like it, share it!