Kubernetes Data Layer: Postgres, NATS, and Namespace Strategy

Setting up shared stateful services properly - with the right namespace strategy from the start.

The problem with default namespace

I initially put Postgres in the default namespace. It worked, but it's not best practice. The default namespace should be left empty or used only for quick testing. Production workloads deserve proper organization.

Namespace strategy

After some back and forth, I settled on:

data/               <- shared stateful services
  postgres
  nats
  redis (later)

observability/      <- monitoring stack
  clickhouse
  hyperdx
  otel-collectors

app-name/           <- one namespace per app
  app-a/
  app-b/

Why this matters:

• Shared infrastructure in data - accessible by all apps
• Each app isolated in its own namespace
• NetworkPolicies can enforce: apps reach data, apps don't reach each other
• Clear separation of concerns

Moving Postgres to data namespace

First, delete from default:

kubectl delete cluster echo-db -n default
kubectl delete pvc -l cnpg.io/cluster=echo-db -n default

Create the new namespace:

kubectl create namespace data

Update k8s/postgres-cluster.yaml to use namespace: data and enable monitoring.

Apply:

kubectl apply -f k8s/postgres-cluster.yaml

Verify:

kubectl get pods -n data

Postgres metrics to HyperDX

CloudNativePG exposes Prometheus metrics on port 9187. To get them into the ClickHouse/HyperDX stack, I add a prometheus receiver to the OTel cluster collector.

The collector uses kubernetes service discovery to find pods with the cnpg.io/cluster=echo-db label and scrapes their metrics endpoint:

prometheus:
  config:
    scrape_configs:
      - job_name: 'postgres'
        scrape_interval: 30s
        kubernetes_sd_configs:
          - role: pod
            namespaces:
              names:
                - data
        relabel_configs:
          - source_labels: [__meta_kubernetes_pod_label_cnpg_io_cluster]
            action: keep
            regex: echo-db
          - source_labels: [__address__]
            action: replace
            regex: ([^:]+)(?::\d+)?
            replacement: $$1:9187
            target_label: __address__
          - source_labels: [__meta_kubernetes_pod_name]
            target_label: pod

HyperDX Dashboard

I created a simple Postgres dashboard with 4 charts:

• Connections - cnpg_backends_total (avg, grouped by pod)
• Database Size - cnpg_pg_database_size_bytes (sum, grouped by pod)
• Replication Lag - cnpg_pg_replication_lag (max, grouped by pod)
• Collector Health - cnpg_collector_up (min, grouped by pod)

Dashboard JSON: dashboards/hyperdx-postgres.json

To import: HyperDX doesn't support Grafana dashboard import. Create manually or use their Dashboard API.

NATS with JetStream

See dedicated article: NATS with JetStream on Kubernetes

NetworkPolicies

TODO: Document default-deny policies and whitelist rules.

Files