Kubernetes

The deploy/ directory ships a Kustomize base with environment overlays for dev, staging, and production. All manifests are production-ready: non-root containers, read-only filesystems, network isolation, and graceful shutdown.

Base Manifests

deploy/base/ contains five manifests bundled by kustomization.yaml:

• Deployment — single replica, resource requests/limits, security context, health probes, downward API for POD_NAME and POD_NAMESPACE
• Service — ClusterIP on port 2575 with sessionAffinity: ClientIP (1-hour timeout) so long-lived MLLP connections are not broken by load balancer churn
• ConfigMap — all runtime tunables (MLLP_ADDR, timeouts, MLLP_MAX_CONNECTIONS, PRE_SHUTDOWN_DELAY, LOG_LEVEL)
• PodDisruptionBudget — minAvailable: 1 prevents simultaneous eviction of all replicas during node drains
• NetworkPolicy — restricts ingress to port 2575 from explicitly labeled namespaces only

Security

The pod and container security contexts enforce a minimal attack surface:

# Pod-level
securityContext:
  runAsNonRoot: true
  runAsUser: 65534
  runAsGroup: 65534
  fsGroup: 65534
  seccompProfile:
    type: RuntimeDefault

# Container-level
securityContext:
  allowPrivilegeEscalation: false
  readOnlyRootFilesystem: true
  capabilities:
    drop:
      - ALL

Each control matters specifically in healthcare environments:

• runAsNonRoot / runAsUser: 65534 — prevents the process from running as root, limiting blast radius if the container is compromised. UID 65534 (nobody) has no filesystem privileges.
• readOnlyRootFilesystem — any exploit that attempts to write a payload to the container filesystem will fail immediately. This also satisfies HIPAA technical safeguard requirements around unauthorized modification.
• drop: ALL capabilities — removes all Linux capabilities including NET_BIND_SERVICE, CHOWN, and SETUID. The MLLP server binds on port 2575 (above 1024), so no capabilities are needed.
• seccompProfile: RuntimeDefault — applies the container runtime’s default seccomp filter, blocking uncommon syscalls (e.g. ptrace, kexec_load) that are not needed by the server and are frequently exploited.

Health Probes

Both liveness and readiness probes use tcpSocket on the MLLP port. MLLP does not have an HTTP layer, so TCP is the correct probe type — a successful TCP handshake means the server is accepting connections.

livenessProbe:
  tcpSocket:
    port: mllp
  initialDelaySeconds: 5
  periodSeconds: 10
  timeoutSeconds: 3
  failureThreshold: 3

readinessProbe:
  tcpSocket:
    port: mllp
  initialDelaySeconds: 5
  periodSeconds: 5
  timeoutSeconds: 3
  failureThreshold: 2

The readiness probe runs twice as frequently (periodSeconds: 5) and has a lower failure threshold (2 vs 3). This removes a pod from the Service endpoint slice quickly when it is not ready, before the liveness probe would restart it. The liveness probe’s higher threshold avoids unnecessary restarts under transient load.

Overlays

Dev reduces resource requests to minimize cluster footprint. Staging uses base defaults to mirror production behavior under realistic conditions. Production doubles the connection limit, increases resource limits, and adds topology spread constraints to distribute pods across availability zones.

Horizontal Pod Autoscaler

The production overlay includes an HPA that scales on CPU utilization:

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: mllp-server
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: mllp-server
  minReplicas: 2
  maxReplicas: 10
  metrics:
    - type: Resource
      resource:
        name: cpu
        target:
          type: Utilization
          averageUtilization: 70

The minimum of 2 replicas ensures the PDB (minAvailable: 1) can be satisfied during a rolling update or node drain. The HPA scales up when average CPU across all pods exceeds 70%, adding replicas up to the maximum of 10.

Note: when the HPA is active, do not set replicas in the Deployment manifest — the HPA owns that field. The patches/replicas.yaml in the prod overlay sets an initial value; after the HPA takes control it manages replicas independently.

Persistent Storage

The deploy/overlays/persistent/ overlay adds a PersistentVolumeClaim and mounts it for outbox.db:

# pvc.yaml
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: mllp-data
spec:
  accessModes:
    - ReadWriteOnce
  storageClassName: local-path
  resources:
    requests:
      storage: 10Gi

The deployment patch sets MLLP_DB_PATH=/data/outbox.db, mounts the PVC at /data, and switches the update strategy to Recreate:

spec:
  replicas: 1
  strategy:
    type: Recreate

Recreate is required because ReadWriteOnce volumes can only be attached to one node at a time. A RollingUpdate would leave the old pod holding the volume mount while the new pod tries to attach it, causing the rollout to stall. Recreate terminates the old pod first, releases the volume, then starts the new pod.

The persistent overlay also disables readOnlyRootFilesystem on the container (readOnlyRootFilesystem: false) because the database requires write access to the mounted path.

Apply persistent storage on top of an environment overlay:

# Prod with persistent storage
kustomize build deploy/overlays/persistent | kubectl apply -f -

Pre-shutdown Delay

The base ConfigMap sets:

PRE_SHUTDOWN_DELAY: "5s"
SHUTDOWN_TIMEOUT: "30s"

When Kubernetes terminates a pod it sends SIGTERM and simultaneously removes the pod from the Service endpoint slice. However, the endpoint update propagates through kube-proxy or the CNI with a small delay — typically 1–5 seconds. Without a pre-shutdown delay, the server would stop accepting connections before all load balancers have updated their routing tables, causing in-flight connections to be refused.

PRE_SHUTDOWN_DELAY=5s tells the server to wait 5 seconds after receiving SIGTERM before it stops accepting new connections. Existing connections continue to be processed during this window.

The base Deployment sets terminationGracePeriodSeconds: 45. This value must be greater than PRE_SHUTDOWN_DELAY + SHUTDOWN_TIMEOUT (5s + 30s = 35s). The 10-second margin gives Kubernetes time to forcibly kill the container if the process does not exit cleanly within the grace period.

If you increase SHUTDOWN_TIMEOUT in an overlay, increase terminationGracePeriodSeconds proportionally.

Network Policy

The NetworkPolicy restricts who can reach the MLLP port:

spec:
  podSelector:
    matchLabels:
      app: mllp-server
  policyTypes:
    - Ingress
  ingress:
    - from:
        - namespaceSelector:
            matchLabels:
              network-policy/mllp-client: "true"
      ports:
        - protocol: TCP
          port: 2575

Only pods in namespaces labeled network-policy/mllp-client: "true" can open TCP connections to port 2575. All other ingress — including from the same namespace — is dropped. There is no egress policy; outbound connections to upstream systems are unrestricted.

To allow a client namespace:

kubectl label namespace <client-namespace> network-policy/mllp-client="true"

Deploy Commands

# Preview rendered manifests
kustomize build deploy/overlays/dev
kustomize build deploy/overlays/staging
kustomize build deploy/overlays/prod

# Apply to cluster
kustomize build deploy/overlays/dev     | kubectl apply -f -
kustomize build deploy/overlays/staging | kubectl apply -f -
kustomize build deploy/overlays/prod    | kubectl apply -f -

# Or use kubectl's built-in kustomize support
kubectl apply -k deploy/overlays/prod

# Diff before applying
kubectl diff -k deploy/overlays/prod

TLS with Secrets

Store TLS credentials as a Kubernetes Secret and mount them into the container. Never put certificate data in a ConfigMap.

Create the secret from local files:

kubectl create secret generic mllp-tls \
  --from-file=tls.crt=/path/to/server.crt \
  --from-file=tls.key=/path/to/server.key \
  --namespace healthcare-prod

Add the volume and environment variables in a Kustomize patch (e.g. deploy/overlays/prod/patches/tls.yaml):

apiVersion: apps/v1
kind: Deployment
metadata:
  name: mllp-server
spec:
  template:
    spec:
      containers:
        - name: mllp-server
          env:
            - name: TLS_CERT_FILE
              value: /etc/mllp/tls/tls.crt
            - name: TLS_KEY_FILE
              value: /etc/mllp/tls/tls.key
          volumeMounts:
            - name: tls
              mountPath: /etc/mllp/tls
              readOnly: true
      volumes:
        - name: tls
          secret:
            secretName: mllp-tls

Register the patch in your overlay’s kustomization.yaml:

patches:
  - path: patches/tls.yaml

The mounted path is read-only, which is compatible with readOnlyRootFilesystem: true — only the volume mount path is writable, not the container root.

For certificate rotation, update the Secret and perform a rolling restart:

kubectl rollout restart deployment/mllp-server -n healthcare-prod