NATS with JetStream on Kubernetes

Setting up NATS as a messaging layer with JetStream for persistence and streaming.

Why NATS?

NATS is a lightweight, high-performance messaging system. JetStream adds persistence and streaming capabilities - essentially giving you durable queues and replay.

Installation

Add the Helm repo:

helm repo add nats https://nats-io.github.io/k8s/helm/charts/
helm repo update

Install with the values file:

helm install nats nats/nats -n data -f k8s/nats-values.yaml

This deploys:

• 3 NATS pods clustered for HA
• JetStream enabled with 10Gi persistent storage per pod
• Prometheus metrics exporter on port 7777
• nats-box utility pod for debugging

Verify:

kubectl get pods -n data

nats-box

The Helm chart includes nats-box, a utility pod with the nats CLI for testing:

# Connect to nats-box
kubectl exec -it deployment/nats-box -n data -- sh

# Then inside:
nats server info
nats stream ls
nats pub test "hello"
nats sub test

Optional - can disable in values if not needed. Useful for quick debugging without installing the CLI locally.

JetStream

JetStream is enabled in the values file. Streams are created by your app as needed - no Kubernetes CRDs required.

Key settings in k8s/nats-values.yaml:

config:
  jetstream:
    enabled: true
    fileStore:
      enabled: true
      pvc:
        enabled: true
        size: 10Gi
        storageClassName: local-path

Metrics to HyperDX

NATS exposes Prometheus metrics via a sidecar exporter on port 7777. I scrape these with the OTel cluster collector:

- job_name: 'nats'
  scrape_interval: 30s
  kubernetes_sd_configs:
    - role: pod
      namespaces:
        names:
          - data
  relabel_configs:
    - source_labels: [__meta_kubernetes_pod_label_app_kubernetes_io_name]
      action: keep
      regex: nats
    - source_labels: [__address__]
      action: replace
      regex: ([^:]+)(?::\d+)?
      replacement: $$1:7777
      target_label: __address__
    - source_labels: [__meta_kubernetes_pod_name]
      target_label: pod

Verify scraping:

kubectl logs -n observability deployment/otel-cluster-collector --tail=20 | grep -i nats

Dashboard

Created a NATS dashboard with 5 charts:

• Connections - nats_connz_num_connections (sum, grouped by pod)
• Memory - nats_account_memory_used (sum, grouped by pod)
• Storage - nats_account_storage_used (sum, grouped by pod)
• Subscriptions - nats_connz_subscriptions (sum, grouped by pod)
• Collector Health - up metric (min, 5-min granularity, filtered to nats pods)

Dashboard JSON: dashboards/hyperdx-nats.json

Files