Part 2 ended with a right sized platform that could rebuild itself. That also included its open endpoints, broad permissions, badly placed replicas and unverified pull requests.
Right at the beginning we found that STACKIT limited which services could be private, the network gate depended on settings in two systems, Helm charts brought somebody else’s defaults, and Kubernetes treated a placement rule that looked strict as optional.
So we compiled a list of the most pressing issues and addressed them one after another. I tried not to get too technical in here, but it gets harder the deeper we dive into real platform engineering.
How Private Can You Get?
We wanted the data tier on private endpoints, with operators entering through one hardened jump host. A private endpoint gives a database no public address, so a stolen password is useless without a foothold inside the network. This is similar to the VPC and PrivateLink approach on AWS.
STACKIT’s catalogue ruled that out. PostgresFlex, OpenSearch and Secrets Manager use public endpoints protected by authentication and an ACL, an access control list of accepted source addresses. Object Storage has no address filter. Private SKE was behind a feature flag and its routing tables were in public preview. Basing the foundation on those features would add churn without making the managed data tier private. A jump host would add a single point of failure and another machine to patch.
We accepted public endpoints behind a short ACL. The team connects through a VPN with a fixed exit address, while CI uses the runner’s floating IP from Part 2; the cluster API and secret store admit both addresses, but the datastores admit cluster egress only, which keeps direct operator traffic out. I would choose private endpoints in a landing zone that offers them. STACKIT’s managed data tier did not, but that might change.
A Gate Split Across Two Systems
Prod has two active access policies: the Camunda UIs are VPN-only, while Zeebe gRPC also admits our fixed-address benchmark runner, which Part 5 introduces. Both enter through the same load balancer.
The external load balancer (STACKIT uses yawol, “Yet Another Working OpenStack Loadbalancer”) filters connections at L4, where it sees a source address and port. Its loadBalancerSourceRanges setting applies one list to the whole load balancer. To let the benchmark runner reach Zeebe, that list has to contain both the VPN and runner ranges. At L4, the runner then has a network path to every hostname behind the load balancer, including the UIs. OIDC would still protect the login, but the UIs would no longer be VPN-only.
The SKE internal Loadbalancer (we use nginx) sees the requested hostname at L7. Its default rule admits only the VPN, while the Zeebe ingress overrides that rule with the VPN and runner ranges. That keeps the UIs behind the VPN without blocking the benchmark runner from Zeebe.
We initially thought the L4 field was ignored. Before publishing that claim about a named provider, we retested with a throwaway load balancer: the excluded range blocked ten out of ten probes, and the caller’s address restored access. The current service enforces the field, so we use it as an outer gate.
The L7 rules depend on nginx seeing the original client address. The external load balancer therefore prepends it with PROXY protocol, while the internal load balancer is configured to parse that information and trust it only from the load-balancer network. We verified the chain in the live ingress logs, where nginx recorded the VPN exit address instead of the load balancer’s internal address.
# Illustrative: one outer list, then a narrower default at L7
service:
loadBalancerSourceRanges:
- <vpn-cidr>
- <benchmark-runner-cidr>
annotations:
yawol.stackit.cloud/tcpProxyProtocol: "true" # LB: prepend the real client IP
config:
use-proxy-protocol: "true" # nginx: parse and trust the header
proxy-real-ip-cidr: <proxy-cidr> # nginx: only from this hop
whitelist-source-range: <vpn-cidr> # default: UIs stay VPN-only
---
# Annotation on the Zeebe gRPC Ingress only
metadata:
annotations:
nginx.ingress.kubernetes.io/whitelist-source-range: >-
<vpn-cidr>,<benchmark-runner-cidr>The coupling fails in opposite directions. Remove a PROXY-protocol setting while the allowlist stays narrow and every client gets blocked with a HTTP 403 Forbidden; lose the shared value file that carries the L7 allowlist and every UI becomes reachable from the internet while ArgoCD stays green, the pods stay healthy and login continues to work.
Policy tests cover both layers. They assert that the L4 list remains a superset of every active L7 rule, each hostname stays narrow, and all three PROXY-protocol settings are present. The VPN range lives in one file used by both environments, so it changes once.
For example, this shortened Conftest rule checks every per-host L7 allowlist against the L4 list on the rendered load-balancer Service:
deny contains msg if {
service := lib.ingress_nginx_service
l4_cidrs := {
trim_space(cidr) |
some cidr in service.spec.loadBalancerSourceRanges
}
some ingress in lib.docs
ingress.kind == "Ingress"
annotation := object.get(
ingress.metadata.annotations,
"nginx.ingress.kubernetes.io/whitelist-source-range",
"",
)
some l7_cidr in {
trim_space(cidr) |
some cidr in split(annotation, ",")
}
l7_cidr != ""
not l7_cidr in l4_cidrs
msg := sprintf(
"L7 admits %s on %s, but the L4 gate blocks it",
[l7_cidr, ingress.metadata.name],
)
}The VPN range gates almost every human path into the platform. If its exit address changes, the UIs, kubectl, operator access to Zeebe gRPC and SSH to the runner reject at once. Direct operator access cannot reopen the ACL. Recovery follows a separate path: ArgoCD pulls from Git, so a commit restores UI access, and CI runs the applies that reopen the provider ACLs.
Connector Direction Decides Your Public Surface
Adding a public /inbound/* route would’ve made the setup more complicated so we postponed inbound support for now. A source filter would not have been enough. HMAC verification happens inside the connector runtime, after the request has crossed the load balancer and ingress and reached the pod, so unauthenticated traffic already consumes platform capacity. If inbound access returns, we will use polling where the external API supports it, place a relay in front of the cluster, or add an edge that enforces limits before traffic reaches the pod.
Outbound connectors remain in use. They leave through cluster NAT and call the HTTPS destination named in a process model, using credentials from the managed secret store. Because the Camunda namespace permits port 443 to 0.0.0.0/0, permission to deploy a model also carries permission to make the platform send payloads and connector credentials to an external host.
Restricting that egress is harder than writing SaaS names into a NetworkPolicy. SKE uses Calico OSS, which matches IP ranges and cannot express DNS names. A forward proxy can restrict hosts, but for HTTPS it sees the hostname in CONNECT, not the request path. Path filtering requires TLS interception, which puts every connector credential and payload through one decryption point and breaks certificate pinning. We recorded broad HTTPS egress as an open risk and deferred that design. A process team cannot choose either connector direction without changing the platform’s exposure.
Check The Defaults Of Every Chart You Install
Installing our own platform charts also meant owning their defaults. The monitoring stack and ingress controller arrived with cluster-wide read on secrets, and application images used floating tags rather than digests, which means a version’s content can change without you noticing. Those charts are packaged for many cluster shapes, so we had to review and narrow the defaults ourselves.
Grafana and Loki could read every Secret in the cluster, but they only need the monitoring namespace. We replaced their ClusterRoles with namespace-scoped Roles. We also limited the Prometheus Operator to monitoring and camunda.
ingress-nginx is different. It handles TLS for apps in several namespaces, so it must read their certificate Secrets. We kept this permission and restricted its network access.
Also, our Terraform modules now reject unsafe or invalid input at plan time. A caller range that is empty or wide open fails validation. The variable naming an environment accepts only the environments that exist (test or prod).
variable "allowed_caller_cidrs" {
type = list(string)
validation {
condition = length(var.allowed_caller_cidrs) > 0 && !contains(var.allowed_caller_cidrs, "0.0.0.0/0")
error_message = "Name the VPN and CI egress ranges. Empty or 0.0.0.0/0 is not a valid answer."
}
}Every stateful resource carries destroy protection in Terraform: the databases, the search cluster, Object Storage, the secret store. No apply takes the data with it, whatever a refactor does to the resource addresses. The guard cannot take a variable and the same modules serve both environments, so tearing down test is a deliberate two-step. Destroy protection later blocked Terraform from replacing a managed service after it had been renamed outside our configuration; Part 6 covers that incident.
We test monitoring separately because it cannot report its own failure. A heartbeat fires constantly into the same alert channel as everything else, and silence is the failure signal. Moving the stack out of the cluster with STACKIT Observability would remove it from the cluster’s failure domain and reduce our maintenance work, at the price of a managed-service bill; we have not done it yet.
Terraform State And Camunda Backups Must Survive
Two kinds of Object Storage buckets hold what the platform cannot recreate. The state buckets store what Terraform knows about every environment, including cluster credentials. The backup buckets receive the nightly Camunda and OpenSearch backups, our recovery path when anything else is lost. The destroy protection from the last section guards this data against Terraform, but Terraform is not the only writer. CI uploads state and the clusters upload backups with their own S3 keys, and a leaked key talks straight to the storage API, with no plan and no review in between. The buckets have to hold against their own credentials.
Every bucket therefore got Object Lock and a default retention when it was created. Creation is the only moment STACKIT can set the lock, so either you build this control in from the start or you recreate the bucket later. Versioning comes with the lock and cannot be switched off. Locked versions stay restorable for their retention window, even against the credential that wrote them. State keeps seven days in GOVERNANCE mode, which an operator can still clear early: state rewrites on every apply, and undeletable versions would stall every migration. Prod backups keep thirty days in COMPLIANCE.
The difference between the two modes is who can bypass them. GOVERNANCE allows a credential with one specific bypass permission to delete locked versions or shorten their retention early; our bucket policy denies the writer exactly that permission. COMPLIANCE has no bypass: nobody deletes a locked version until the window ends, the project owner included, and retention can only be extended. The mode is stamped onto each version at write time, so changing a bucket’s default only affects future writes.
resource "stackit_objectstorage_compliance_lock" "project" {
project_id = var.project_id
}
resource "stackit_objectstorage_bucket" "backups" {
project_id = var.project_id
name = "prod-backups"
object_lock = true
lifecycle { prevent_destroy = true }
}
resource "stackit_objectstorage_default_retention" "backups" {
project_id = var.project_id
bucket_name = stackit_objectstorage_bucket.backups.name
mode = "COMPLIANCE"
days = 30
}An Object Lock alone would not survive long, because the settings that protect a bucket are changed over the same S3 API that writes objects. Any credential the bucket policy admits could downgrade the retention or rewrite the policy itself. So each bucket splits its permissions into three credential groups.
The writer, the only key CI and the workloads hold, can read and write objects, and the policy denies it everything destructive: deleting versions, changing the policy, the lock, the lifecycle or the encryption.
A monitor credential that reads the five configuration values our checks compare and cannot even list the objects. It’s for verification only.
Admin rights belong to nobody permanently; when a bucket needs administrative work, we mint a key into the admin group, use it and delete it again, so there is no standing credential to steal.
Bucket policies, lifecycle rules and default encryption have no provider resource, so an idempotent script applies those three (manually). A weekly GitHub Actions job compares every bucket against a checked-in declaration: versioning, lock mode and days, policy, lifecycle, SSE-S3 encryption.
Replicas Need A Hard Placement Rule
The pod placement test produced the same kind of surprise. Prod runs three nodes, one per availability zone, and our two-replica components had weight-100 preferred pod anti-affinity to keep each pair apart. Connectors, Console and the Web Modeler REST API still landed both replicas on one node.
It happened during a reschedule when two nodes became Ready one second apart. The first node ended up carrying 78% of the cluster’s CPU requests, along with both replicas of the three affected deployments; losing it would remove authentication, every UI, Connectors, Web Modeler, GitOps control and secret sync in one go.
Kubernetes was doing what we had asked. preferred contributes a score alongside image locality and resource fit, and it never removes a node from consideration. Other scores won during the scheduling burst.
The ingress controller behaved differently because its rule filters nodes. It held a replica Pending until a second node was ready. Zeebe’s required hostname and zone anti-affinity kept one broker Pending for twenty minutes until an unused zone had a ready node.
A disruption budget (PDB) caps how many pods of a workload may go down at once during a planned action. It cannot protect a co-located pair from losing its node because Kubernetes is not evicting either pod. Nothing reports the mismatch.
Policy tests now reject every multi-replica prod workload without hard hostname anti-affinity. For a Helm-managed component, the relevant part of the values file looks like this:
connectors:
replicas: 2
affinity:
podAntiAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
- labelSelector:
matchExpressions:
- key: app.kubernetes.io/component
operator: In
values: [connectors]
topologyKey: kubernetes.io/hostnameThe Camunda chart exposes topologySpreadConstraints only for brokers. Other components silently ignore the key, so required anti-affinity is the common lever.
After the GitOps change, the loaded node dropped from 78% to ~50% of CPU requests. No pair protected by a disruption budget shared a node. CPU requests still differed across the three nodes.
A fourth node would not have changed the scheduling race. Kubernetes placed most pods on the first node that became ready, while the node that arrived twenty minutes later received only pods with volumes pinned to its zone; one node pool across three zones has no symmetric fourth member anyway, because four means a 2/1/1 split. Placement needed enforcement. Capacity was a separate question.
Terraform-owned Helm releases need their placement values in Terraform. ArgoCD combines hard per-component anti-affinity with a soft spread across the release:
locals {
argocd_ha_values = {
global = {
affinity = { podAntiAffinity = "hard" }
topologySpreadConstraints = [{
maxSkew = 1
topologyKey = "kubernetes.io/hostname"
whenUnsatisfiable = "ScheduleAnyway"
labelSelector = {
matchLabels = { "app.kubernetes.io/part-of" = "argocd" }
}
}]
}
}
}hard separates matching replicas. With ScheduleAnyway, Kubernetes tries to spread the whole release while still permitting every ArgoCD pod to run during a one-node window.
Enforced placement costs something: a replica can wait Pending while only one node is up, and a rolling update now needs a third schedulable node or it stalls with the old pods serving. Test keeps none of this: one node, one zone and one replica of each. Singletons use preferred mutual scatter instead. This covers Camunda’s Identity, Optimize and Web Modeler WebSockets, GitOps-owned cert-manager and Terraform-owned External Secrets. A hard rule would add no availability and could strand a pod.
The scheduler evaluates these constraints when it places a pod. It does not move existing pods later. A descheduler could address the remaining concentration, but it would evict running pods, so we deferred it.
The Checks A Change Has To Survive
The allowlist and the placement rule had both looked correct in configuration. We changed the pull-request pipeline to test what the rendered configuration would do. Part 2 explains how a change ships; these are the checks it passes first.
Eight jobs run in parallel for every infrastructure pull request:
terraform fmt checks canonical formatting across every Terraform file.
terraform validate checks syntax and provider schemas without state access.
TFLint flags Terraform-specific errors and recommended-rule violations.
tfsec scans Terraform for security misconfigurations.
Checkov applies policy-as-code checks to Terraform.
terraform test runs module input-validation tests with mocked providers.
terraform-docs generates README files for each module.
Gitleaks scans the repository on every pull request.
The GitOps repo:
Renders Helm charts with helm template
Builds the overlays with kubectl kustomize
Checks the combined test and production output with Conftest policies written in Rego.
Gitleaks scans the repository on every pull request.
For example, this Conftest rule fails when either default-deny policy is missing from a guarded namespace:
default_deny_namespaces := {
"camunda", "argocd", "external-secrets-system", "secrets-manager", "ingress-nginx", "cert-manager"
}
deny contains msg if {
some ns in default_deny_namespaces
some np in {"default-deny-ingress", "default-deny-egress"}
not lib.has_ns("NetworkPolicy", ns, np)
msg := sprintf("[%s][network-policies] %s NetworkPolicy %q missing", [lib.release, ns, np])
}Neither tfsec nor Checkov has native STACKIT rules. The S3-compatible backend triggers AWS-shaped checks that do not apply or cannot be configured through the provider. We document the suppressions. STACKIT constraints belong in Terraform tests or custom policies. Versioning runs separately on the data plane.
None of these jobs receives cloud credentials. A scanner that runs on every pull request has no reason to sit inside the apply identity’s blast radius.
Runtime secrets live in the managed secret store, CI credentials stay in the pipeline’s store, and Git holds non-secret configuration. Gitleaks checks both repositories without a path filter because a credential can land anywhere; local rules cover signed webhook URLs that the defaults miss.
What’s Next
This part showed how we applied hardening over the whole platform. Restricting access to all of our components, adding pipelines that verify every change, deliberate pod placement to even the load, and chart defaults that make sense for us.
Part 4 explains how we use AI as support to build this platform, where its strengths are, and especially its weaknesses.