Part 1 explained how we sized separate environments for test and prod, integrated Azure Entra as our identity provider, and successfully deployed them with Terraform. Users could log in, model processes and complete instances. Nobody could rebuild or recover them, though, and that set the goal for the next phase: building a real platform.
What that actually means is that Camunda config changes shouldn’t touch infrastructure, Terraform shouldn’t be executed from my local machine, backup mechanisms need to be in place and verified, and disks shouldn’t fill up without anyone noticing. I’ve already built similar groundwork on an AWS platform project for a logistics company, which was a great plus.
Terraform For Infrastructure, GitOps For Apps
We decided to split the code. A database resize should not share a release rhythm with a Camunda values change, so Terraform owns the STACKIT projects, SKE, PostgreSQL, OpenSearch, Object Storage and the secret store; ArgoCD owns ingress, certificates, monitoring and Camunda from the GitOps repository. Terraform pushes. ArgoCD pulls.
Infrastructure changes less frequent and behind a heavy review gate; application config changes all the time. Terraform’s identity never edits a running workload like Camunda, and ArgoCD’s Git token never touches Terraform state. Terraform hands ArgoCD one repository credential and a root Application, then leaves the application layer to reconcile itself.
Each environment runs its own ArgoCD syncing an app-of-apps: one root Application that owns the child Applications for ingress, cert-manager, monitoring, Camunda, and the rest, ordered by sync waves so the secret operator is up before anything that needs a secret.
The two environments deliberately track different things. Test follows the main branch, so a merge lands there and you find out quickly whether it works. Prod follows a promotion tag. Pointing both clusters at one branch would be a tempting simplification, but it means every merge would be a prod deploy.
During a necessary manual operation, we learned how literal ArgoCD ownership is. I scaled a StatefulSet to zero, and it returned within seconds. I disabled self-heal on the child Application and tried again, but the parent restored the child’s sync policy from Git. For a manual operation, we now pause reconciliation along the entire ownership chain, including the parent.
# Illustrative app-of-apps child: ordered by sync wave, self-healed
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
name: cert-manager
annotations:
argocd.argoproj.io/sync-wave: "1" # the secret operator is wave 0
spec:
syncPolicy:
automated:
selfHeal: trueA second ArgoCD problem was easier to overlook. Kubernetes filled fields inside the StatefulSet volume claim templates that Git did not contain, so the Application stayed out of sync in ArgoCD while self-heal kept trying to resolve the same difference. We added a narrow ignoreDifferences rule to the ArgoCD Application:
spec:
ignoreDifferences:
- group: apps
kind: StatefulSet
jqPathExpressions:
- .spec.volumeClaimTemplates[]?.status
- .spec.volumeClaimTemplates[]?.spec.volumeMode
- .spec.volumeClaimTemplates[]?.metadata.annotationsWe ignored only the fields Kubernetes generated. Ignoring the complete volumeClaimTemplates list would also hide changes we want ArgoCD to report, including a new PVC size. We used the same approach for strategy fields defaulted by External Secrets.
One Apply, Two Clouds, No Stored Keys
One deploy identity across infra, test and prod would have undermined the repository split. STACKIT has no AssumeRole-style impersonation like e.g. in AWS. We started with local Terraform applies, where STACKIT authenticated through service-account key files.
When we moved the applies into GitHub Actions, carrying that approach forward would have meant copying a STACKIT key into the infra, test and prod GitHub environments. We chose federated identities for CI instead. Each STACKIT service account and Entra CI application now trusts only its matching GitHub environment and reaches only its assigned resources.
This split does not mean one apply stays inside one cloud. The shared-services stack creates STACKIT infrastructure and Entra app registrations, while the environment stacks provision STACKIT and manage Camunda’s Entra credentials. Both providers authenticate during the same run.
The job may request OIDC tokens. Each provider requests a separate GitHub token because the audiences differ: Azure expects api://AzureADTokenExchange, while STACKIT expects sts.accounts.stackit.cloud. Entra and STACKIT verify the issuer and a subject containing the repository and GitHub environment, then issue short-lived access tokens.
We considered using Azure Entra as the single broker to remove one trust chain. Entra cannot mint a token with the arbitrary audience STACKIT requires, and obtaining a suitable Entra application token another way puts a stored Azure credential back into the job. That buys an extra exchange and merely moves the long-lived secret. Two direct trust chains keep provider login keyless, and their subject checks mean a test job cannot satisfy the prod trust even when someone triggers it manually.
We use access control lists (ACLs) to give access to STACKIT resources. I’ll get into the details in Part 3. Those ACLs added a requirement: a fixed source address. Hosted GitHub runners use changing address ranges, so Terraform now runs on a small self-hosted VM in the shared project. A floating public IP survives VM replacement and the allowlist contains the runner address for automation.
That runner is privileged. terraform plan executes provider code from the branch, so an untrusted pull request could run code from an allowlisted host with live OIDC access. Pull requests receive static analysis on GitHub-hosted runners. Plans and applies use the self-hosted runner only from trusted refs, and applies follow a merge to main.
State lives in a third, shared project. Its buckets sit outside the stacks CI applies, have versioning enabled and are checked for access-policy drift. If a pipeline can alter prod state, it can steer the next prod apply. State therefore needs its own boundary.
Blue bootstrap creates yellow runner, identity and state resources. That runner provisions green infrastructure, where ArgoCD reconciles red applications, volume claims and policies. Purple marks differences between test and prod.
You Build It, You Run It
A platform isn’t done after it’s deployed. Someone has to operate it and make sure it runs. A platform team e.g. needs insights into the health of the cluster, and ways to analyze and act on errors.
For that we added monitoring to prod with the kube-prometheus-stack. It contains Kubernetes manifests, default Grafana dashboards, and Prometheus rules combined with documentation and scripts to provide easy to operate end-to-end Kubernetes cluster monitoring with Prometheus using the Prometheus Operator. We also added some dashboards from Camunda, showing live metrics for Zeebe & Co.
We’re aware that operating our monitoring stack inside the same cluster it monitors is a risk. This is a low priority open issue right now while building the platform blueprint. We might switch to managed STACKIT Observability later on.
Camunda’s secondary storage, the searchable copy behind Operate, Tasklist, and Optimize, runs on managed OpenSearch in prod and on the relational database in test. STACKIT’s managed OpenSearch tops out at version 2.x, and the application credential lacks index-lifecycle permissions. Native retention is therefore off, with a cronjob pruning old indices instead. This prevents the disk from filling up in the middle of the night and paging people out of bed. We opened a service request with STACKIT to address this issue in the future.
Backups made the difference between rebuilding software and recovering a platform. A nightly job writes Camunda’s state to object storage, and the return path lives in a runbook. The same happens for OpenSearch, where we write regular backups to a snapshot repository in the object storage. Both the state and backup buckets are created with Object Lock, versioning and default encryption, and a weekly job verifies those settings; Part 3 covers that as well.
Why “As Code” Is Harder Than It Looks
Almost everything was now “as code.” A few manual boundaries remained.
ArgoCD can reconcile the platform only after ArgoCD exists. It needs a cluster, and Terraform needs an identity before it can create one. One hand-executed step remains at the bottom of the stack: an operator uses a short-lived key for the initial apply, revokes it and lets the keyless path take over.
We revoked the bootstrap key and initially left its identity holding owner rights on all three projects. The identity now has no active credential between uses. Removing a credential does not remove the permissions attached to its identity.
Some values don’t exist when the code is written. Terraform first has to create the managed database, password and search cluster before it can write their coordinates into the secret store; External Secrets then syncs them into Kubernetes for the application. That secret-store entry is the handoff between infrastructure and applications.
Timing crosses the repository boundary as well. Adding a database starts in Terraform, continues in the secret store and finishes when GitOps adds the sync and application configuration.
Review is the last boundary. Git reproduces a bad configuration as reliably as a good one, and ArgoCD keeps either version synchronized, so our policies inspect the rendered result and assert the details whose absence leaves everything green: the access gate, sync policies and chart versions.
What’s Next
Our first platform iteration is now rolled out and rebuilt by pipelines. Infrastructure and apps are structurally separated and short-lived tokens are minted for each run. The endpoints were open and the network was flat. Part 3 deals with that.