Venturing into the world of deploying an MQTT broker in Kubernetes can lead you to some intriguing challenges, especially when compared to the more traditional tasks of serving webpages or APIs

1. Unlike HTTP-based services, MQTT is served over TCP. Consequently, some common ingress solutions, such as ingress-nginx, do not natively support serving TCP traffic. We'll explore how we could bypass this limitation.
   
2. Ensuring secure communication between the MQTT broker and clients is crucial. However, manually creating and managing SSL certificates can be a tedious process. We'll demonstrate an automated method for managing SSL certificates, making your MQTT communication secure and worry-free

Certificate

First things first, let's generate the certificate we want to use. The "de facto standard" for this task is cert-manager, and we'll use a cert-manager-generated certificate in the same way we do with other ingress-generated certificates. This ensures the certificate stays up-to-date.

apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: mosquitto
  annotations:
    kubernetes.io/ingress.class: "nginx"
    cert-manager.io/cluster-issuer: "letsencrypt"
spec:
  tls:
  - hosts:
    - mqtt.example.com
    secretName: mosquitto-certs
  rules:
  - host: mqtt.example.com
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: mosquitto-mqtts
            port: 
              number: 8883

As long as your DNS points to the Ingress controller's load balancer for the specified host making it possible for cert-manager to validate your host, you should have a secret containing the certificate in your cluster.

Broker

Now, it's time to deploy an MQTT broker. Although we often prefer cloud-native brokers like EMQX, we've opted for Mosquitto in this example due to its simplicity.

We'll refer to the generated secret when specifying the certificate location in the config.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: mosquitto
spec:
  replicas: 1
  selector:
    matchLabels:
      app: mosquitto
  template:
    metadata:
      labels:
        app: mosquitto
    spec:
      containers:
      - name: mosquitto
        image: eclipse-mosquitto
        ports:
        - containerPort: 8883
        - containerPort: 9001
        volumeMounts:
        - mountPath: /mosquitto/config/mosquitto.conf
          subPath: mosquitto.conf
          name: config
        - mountPath: /mosquitto/certs/
          name: certs
      volumes:
      - name: config
        configMap:
          name: mosquitto-config
      - name: certs
        secret:
          secretName: mosquitto-certs
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: mosquitto-config
data:
  mosquitto.conf: |
    # DO NOT USE IN PRODUCTION
    allow_anonymous true

    # MQTT with TLS (MQTTS)
    listener 8883
    protocol mqtt

    # Fetch the generated certificates
    cafile /etc/ssl/certs/ca-certificates.crt
    keyfile /mosquitto/certs/tls.key
    certfile /mosquitto/certs/tls.crt
---
apiVersion: v1
kind: Service
metadata:
  name: mosquitto-mqtts
spec:
  type: ClusterIP
  selector:
    app: mosquitto  
  ports:
  - port: 8883

Keep in mind that the example above allows anonymous access to the broker. In a production environment, you'll need to secure your endpoint with an ACL, password list, or similar method and disable anonymous access.

After deploying the components mentioned above, you should have a working MQTT broker with a certificate securing it. However, it's not yet accessible from outside since TCP traffic isn't routed by ingress-nginx.

Magic

To overcome this, we can leverage a lesser-known feature in ingress-nginx that allows routing specific ports to TCP services within the cluster. Note that we're not using any host headers here, so any request to any host pointing to the ingress service will be routed on the specified port.

https://kubernetes.github.io/ingress-nginx/user-guide/exposing-tcp-udp-services/

To do this we will use a configmap called tcp-services to keep track of routed ports.

apiVersion: v1
kind: ConfigMap
metadata:
  name: tcp-services
  namespace: ingress-nginx
data:
  8883: "mosquitto/mosquitto:8883"

The last part is to tell ingess-nginx to read the configmap and allow port 8883 into to load balancer.

Add this arg to the ingress controller

  containers:
  - args:
    - /nginx-ingress-controller
    - --publish-service=$(POD_NAMESPACE)/ingress-nginx-controller
    - --election-id=ingress-nginx-leader
    - --controller-class=k8s.io/ingress-nginx
    - --ingress-class=nginx
    - --configmap=$(POD_NAMESPACE)/ingress-nginx-controller
    - --validating-webhook=:8443
    - --validating-webhook-certificate=/usr/local/certificates/cert
    - --validating-webhook-key=/usr/local/certificates/key
    # Add this
    - --tcp-services-configmap=$(POD_NAMESPACE)/tcp-services

Add port 8883 to the ingress load balancer ports.

  ports:
  - name: mqtts
    port: 8883
    protocol: TCP
    targetPort: 8883
  - appProtocol: http
    name: http
    port: 80
    protocol: TCP
    targetPort: http
  - appProtocol: https
    name: https
    port: 443
    protocol: TCP
    targetPort: https

Assuming you have a service named "mosquitto-mqtts" in the "mosquitto" namespace, this should route the port to the broker. Now, you're all set with a fully functioning Mosquitto MQTT broker with encrypted communication in Kubernetes. Enjoy!