I tripped over back-pressure while load-testing our Camunda cluster on STACKIT. In the end we tuned the exporter and left the broker’s flow-control settings at their defaults. If you want the measurements, they’re in Part 5 of the series. What stayed with me was the mechanism. While we were building there was never time to look at it properly, so this article does.

The question underneath: when a client sends one more command, who decides whether the system takes it, and what happens when the answer is no? Camunda 7 and Camunda 8 answer that in very different places. Camunda 7 lets the infrastructure around the engine answer: the web container, the connection pool, the database, none of which was built to say no on purpose, so the engine accepts the call, writes the work down and slows until something times out. Camunda 8 answers inside the engine. It rejects early and lets the client retry. It has to, because a log-based engine that buffered without limit would turn overload into unbounded latency and, eventually, a full disk. One gap remains with Self-Managed defaults, where the export backlog is bounded only by the disk. More on that under circuit 2.

I checked the code and config defaults against 8.10.4, the latest release as I write this; older versions may differ. The Camunda 7 side comes from its docs and how it’s built. I didn’t load-test it for this article.

Where You Come From; An Async Call To Camunda 7

To keep the comparison fair I’ll stick to one Camunda 7 setup: a remote engine in Camunda Run, called over REST, with an async continuation on the start event, so a call only persists state and returns. External task workers fetch and lock their work. That is the closest thing Camunda 7 has to Camunda 8’s command plus job worker model, so whatever differs comes from the engines.

The call goes through the web container into the engine, which writes the instance and its job to the database, commits and responds. From there the job executor moves the instance forward, and workers fetch, lock and complete their tasks against the same database. The call itself is cheap, so the REST endpoint should stay responsive while the job table fills up behind it, and nothing in the engine looks at that backlog before taking the next call. When the database or the connection pool can’t keep up, calls get slower and then fail with timeouts or server errors. The client never gets a signal it can act on. Camunda’s best-practices guidance describes no admission control in the engine; where it mentions throttling, it means protecting the services the engine calls.

The Path Of A Command In Camunda 8

A client sends a command to the gateway, which routes it to the leader of one partition. The leader appends the command to the partition's log, replicates it to the followers and waits until it's committed. Then the stream processor reads the command back, executes it, writes the follow-up records and answers the client. Exporters read the same log afterwards and copy it into secondary storage, where Operate, Tasklist and the search APIs find it, and the log can only be compacted behind the slowest of those exporters and the latest snapshot.

There is no shared database. State is split into partitions, each with a single writer, and work sits in order in an append-only log instead of in rows that compete for locks.

Two things matter from here on. The command is written before it’s executed, so the broker always holds work it has accepted and not answered yet. And processing and export share the log, so a slow reader at the end holds disk at the start.

Three Circuits And A Pause

Four independent mechanisms decide when the broker says no. Three of them are circuits that watch load, each at a different point: the commands in flight, the export behind them and the disk under both. The fourth is a pause, new in 8.10 and unrelated to load, so it comes last.

Circuit 1; The In-Flight Limit

Circuit 1 is about latency. Per partition, the leader counts the user commands it has accepted and not answered yet. A command is in flight from the moment it’s written until it has been processed, committed and answered. Once the count reaches the limit, new commands are rejected immediately.

The client sees RESOURCE_EXHAUSTED over gRPC, or a 503 over REST. Not a 429. The gateway has no limiter of its own, and for commands that can go to any partition it tries the next partition before giving up, while a command pinned to one partition, through a correlation key for example, goes straight back to the client.

Commands that finish or cancel work are exempt. Completing or failing a job still goes through above the limit, because work already inside has to be able to leave. Throwing a BPMN error from a job and updating job retries are not exempt, so workers can see rejections on those two calls. Unlike the next circuit, this one is on by default.

Camunda 7 caps concurrency through the size of the web container and the connection pool. Whoever sized the pools set that cap without meaning to. It works by making callers wait and knows nothing about how far behind the engine is. Starts, worker fetches and completions all queue for the same connections.

Circuit 2; The Write Rate And Exporter Lag

Circuit 2 is a limit on records written per second, with an optional throttle that shrinks that rate as the unexported backlog grows. Past an acceptable backlog, by default 100k records, the allowed rate drops in proportion. It rejects client commands, and it also slows the broker’s own scheduled writes and processing results, so the whole partition runs at the exporter’s pace.

It’s off by default in Self-Managed and on in SaaS. With it off, the first circuit only sees processing and commit, so it can stay quiet while the export backlog grows behind it. The log keeps growing and clients get no pushback until the disk runs low.

If you run Self-Managed, export lag needs its own alert.

I’d turn it on when export is your constraint and you can’t make export cheaper. If export keeps up with everything the engine accepts, the throttle caps a rate nothing ever reaches. We never turned it on: we were exporting too much data, and changing what got exported fixed it. From 8.9 the switch lives under one prefix:

camunda:
  processing:
    flow-control:
      write:
        enabled: true
        limit: 1000 # example rate, not a default
        throttle:
          enabled: true

In 8.6 to 8.8 the same settings sit under zeebe.broker.flowControl.write.*. The rate is yours to set, and I wouldn’t trust a default number, because the docs disagree on it.

Camunda 7 has nothing like circuit 2. Runtime and history live in the same database, so there’s no separate export path to fall behind.

Circuit 3; Free Disk Space

Circuit 3 watches free disk, and the broker checks it every second. By default, below 2 GB free it rejects every client command and pauses processing, and below 1 GB it stops accepting replicated events too: the log refuses to grow and compacts what it can. Both recover on their own once space is back.

The disk fills because processing and export share one log, and a stuck exporter keeps it growing however healthy processing looks. Circuit 2 is the earlier, gentler version of circuit 3. Alert on how fast the volume fills as well as how full it is, because under load the log can reach the watermark quickly.

In Camunda 7 the database itself is the buffer, with no watermark the engine respects, so I’d expect a full database to end in failed transactions.

The Pause; Leadership Rebalancing

The pause isn’t a circuit, because it doesn’t watch load at all. In 8.10, when a partition hands its leadership to another broker during a coordinated rebalance, it freezes its log. It rejects every write, internal ones included, so the new leader has a fixed log position to catch up to. Clients get the same RESOURCE_EXHAUSTED or 503, with a message saying the partition is paused for a leadership transfer.

The pause ends when the transfer succeeds or fails, and if it runs too long a watchdog makes the old leader step down. Only one partition pauses at a time, and only when someone triggers a rebalance; a crash, a normal leader election or a rolling update leaves the log alone. So if you see a short burst of rejections on one partition and nothing else looks loaded, check whether somebody just rebalanced the cluster.

Camunda 7 has no leadership to hand over, so there’s nothing to pause.

How The In-Flight Limit Moves

There is no fixed number of in-flight commands. The limit moves with what the partition observes, and an algorithm decides how.

AIMD is the default and a good one, and most setups never need to change it. It grows the limit by one while the partition is busy and answers in time, and cuts it by 10% when a command takes longer than the default request timeout of 200 ms. It only grows while the partition is at least half full. By default it starts at 100 and moves between 1 and 1,000. As far as I can tell, it’s also the only one you can point at a latency target. If you do touch something, change that request timeout before you change the algorithm. Both sit in the snippet below.

AIMD and Vegas are TCP congestion-control algorithms. Gradient and Gradient2 are Netflix’s own, from their concurrency-limits library, which builds on the same TCP ideas. A partition limiting unanswered commands has the same problem as a TCP connection limiting unacknowledged packets.

Camunda’s docs give no guide for choosing between them, so the last column is my reading, not Camunda’s. Fixed makes sense in a benchmark, where you want the limit out of the equation. The docs also suggest it when the node’s clock is unreliable. I’d look at Gradient2 if your baseline latency drifts, on shared storage or a busy network for example. Legacy Vegas mimics the old Zeebe defaults, for keeping old behavior on an upgraded cluster. Switching is one line, and AIMD’s timeout sits right next to it:

camunda:
  processing:
    flow-control:
      request:
        algorithm: aimd # fixed, vegas, gradient, gradient2, legacy-vegas
        aimd:
          request-timeout: 200ms # the default

In 8.8 and earlier the setting lives under zeebe.broker.flowControl.request.*, or zeebe.broker.backpressure.* on older clusters. The nearest Camunda 7 equivalent is a thread pool size you set once and hope still fits under load.

Two Systems, Two Contracts

In this setup an accepted request means the same thing in both products: the work is persisted.

Camunda 8 builds a backlog too: jobs waiting for workers. None of the circuits watches your workers. Where the two differ is command admission, and what that makes the client responsible for.

If you’re coming from Camunda 7, this is where old habits bite. Against a Camunda 8 broker, a rejected request is normal operation. A retry loop, timeout or alert built for a system that waits does the wrong thing against one that says no. A client that treats RESOURCE_EXHAUSTED as a failure gives up on work the broker would have taken shortly after, and an alert on every 503 fires whenever the broker does its job.

Summary

The client owns the retry now, and the Java client leaves most of it to you. Its job workers back off on their own when polling gets rejected. Commands don’t: their retries are off by default, and the retry policy you can switch on with useDefaultRetryPolicy covers gRPC only, not REST. The Spring Boot starter can retry completing, failing and throwing errors from jobs after a rejection, but camunda.client.worker.defaults.max-retries is 0 by default. A client that backs off lets the in-flight count drain, and once commands are answered inside the timeout again the limit recovers.

• Retry with backoff and jitter in every client that sends commands.

• Make retries idempotent. A timeout doesn’t prove the command failed, in either product.

• Watch one signal per circuit.

  • Circuit 1: rejected requests (zeebe_dropped_request_count_total, which counts rejections from every circuit, the pause included) and the current limit (zeebe_backpressure_requests_limit).

  • Circuit 2: export lag, the committed position (zeebe_log_appender_last_committed_position) minus the exported one (zeebe_exporter_last_exported_position), as in Camunda’s Grafana dashboard.

  • Circuit 3: free disk and its fill rate from your volume metrics, since the broker exports no disk metric.

• Watch the job backlog as well.

• Decide consciously about circuit 2 in Self-Managed.

On STACKIT circuits 1 and 3 stayed at their defaults, and our work went into the exporter. Having gone through the mechanism now, I’d make the same call again: they regulate themselves.

I recently built a custom connector that manages DNS entries. The customer runs critical infrastructure, and DNS is not something you change from just anywhere. So the connector runtime had to run in a secure environment, on a machine that can only be accessed in person. If you want to deploy something there, somebody builds it and walks over with a USB stick, including several flights of stairs.

I wrote the connector as a Spring bean with constructor injection. In the environment where we built and tested it, the runtime had automatic connector discovery disabled (via CONNECTOR_OUTBOUND_DISCOVERY_DISABLED) and everything worked. When the connector moved to the secure machine, nobody carried that environment variable over. The runtime now found the connector twice: once as a Spring bean, and once through SPI. The bean registration was fine. The SPI path tried to build the class on its own, found no no-args constructor, and the runtime didn’t start. The error said the class couldn’t be built. It didn’t say why the runtime was trying to build it in the first place. Every failure meant another walk to the desk, fix and back to the machine, and it took more of those walks than I like to admit before it was clear that discovery wasn’t disabled anymore.

I mixed up the two mechanisms myself. What kept them apart lived in the environment, not in the artifact, and environment variables don’t travel with jars. This article sorts every way to get a custom connector into a Camunda connector runtime into one of two families, so you can pick one on purpose.

Two Families, One Question

The connector runtime collects connectors from several sources and puts them into one factory. After that, it doesn’t matter how a connector got there. Secrets, input validation, retries and error handling all run through the same job handler. The only real difference is who builds the instance:

• SPI family: the runtime builds your class itself, reflectively, without Spring.

• Spring family: your Spring context builds your class, and the runtime picks up the bean.

Each family has its own rules. A few switches cut across both, and those are the ones that cause confusion.

Two things apply to both families. First, your connector class implements OutboundConnectorFunction, or OutboundConnectorProvider if you use the Operation API. Both families look for exactly these two types. Second, it carries @OutboundConnector. A class without the annotation is skipped silently, by SPI and by bean discovery alike. The runtime doesn’t log an error for it. Jobs of that type just sit there, because no worker picks them up.

Note: I use outbound connectors in the examples, but inbound connectors follow the same rules. Only the names change. The class implements InboundConnectorExecutable and carries @InboundConnector, and the services file is named after that interface. Env var registration uses CONNECTOR_{NAME}_EXECUTABLE, and the discovery switch is CONNECTOR_INBOUND_DISCOVERY_DISABLED. The Spring family has one extra rule for inbound: the bean must be @Scope("prototype"), or the runtime refuses to start.

The SPI Family; The Runtime Builds Your Class

The runtime uses Java’s ServiceLoader. It reads META-INF/services/io.camunda.connector.api.outbound.OutboundConnectorFunction from every jar on the classpath. Each line in that file is the fully qualified name of a connector class. Operation API connectors go into a file named after OutboundConnectorProvider instead.

That file comes into existence in one of two ways. You write it by hand, or, what I didn’t know, the element-template-generator-maven-plugin writes it for you. The plugin does that by default, as long as the file doesn’t exist yet. Both paths end up in the same mechanism, so pick whichever you like. ServiceLoader calls a public no-args constructor, and nothing else. That rules out constructor injection and everything else Spring would normally do for you. The connector builds what it needs itself:

@OutboundConnector(
    name = "Invoice Lookup",
    type = "com.acme:invoice-lookup:1",
    inputVariables = {"invoiceId"})
public class InvoiceLookupConnector implements OutboundConnectorFunction {

  private final InvoiceClient invoiceClient = InvoiceClient.fromEnvironment();

  @Override
  public Object execute(OutboundConnectorContext context) {
    var request = context.bindVariables(InvoiceRequest.class);
    return invoiceClient.findById(request.invoiceId());
  }
}

The services file contains a single line:

com.acme.invoice.InvoiceLookupConnector

There is a sibling in this family: registration by environment variable. You set CONNECTOR_INVOICELOOKUP_FUNCTION=com.acme.invoice.InvoiceLookupConnector, plus _TYPE, _INPUT_VARIABLES and _TIMEOUT if you need them. The runtime loads the class by name and instantiates it reflectively, again without Spring. As soon as one of these variables exists, the runtime skips outbound SPI scanning completely, so env var registration replaces the services files instead of adding to them.

The SPI family is the only one that works with the official connectors Docker image. You don’t write any Spring Boot application. You put your fat jar on the image’s classpath, as described in the Camunda docs on hosting custom connectors. One thing not to count on with this image: Spring picking up beans from your jar. The image only scans its own packages, so a @Component in your package is just a class to it.

The Spring Family; Your Context Builds Your Class

If your connector needs beans, you need your own runtime. That’s a Spring Boot application with io.camunda.connector:spring-boot-starter-camunda-connectors as a dependency. Every bean of type OutboundConnectorFunction or OutboundConnectorProvider in that context gets registered as a connector. The starter doesn’t replace the SPI family, though. It still runs SPI and env var discovery and adds the beans on top, all into the same factory. Now constructor injection works, and the invoice connector looks like this:

@Component
@OutboundConnector(
    name = "Invoice Lookup",
    type = "com.acme:invoice-lookup:1",
    inputVariables = {"invoiceId"})
public class InvoiceLookupConnector implements OutboundConnectorFunction {

  private final InvoiceClient invoiceClient;

  public InvoiceLookupConnector(InvoiceClient invoiceClient) {
    this.invoiceClient = invoiceClient;
  }

  @Override
  public Object execute(OutboundConnectorContext context) {
    var request = context.bindVariables(InvoiceRequest.class);
    return invoiceClient.findById(request.invoiceId());
  }
}

Look at the constructor. There is no no-args variant, and that’s exactly why SPI cannot load this class. For the Spring family, the services file must not exist for this connector. The clean way is to tell the plugin not to write it:

<plugin>
  <groupId>io.camunda.connector</groupId>
  <artifactId>element-template-generator-maven-plugin</artifactId>
  <version>${version.connectors}</version>
  <executions>
    <execution>
      <goals>
        <goal>generate-templates</goal>
      </goals>
    </execution>
  </executions>
  <configuration>
    <connectors>
      <connector>
        <connectorClass>com.acme.invoice.InvoiceLookupConnector</connectorClass>
        <writeMetaInfFileGeneration>false</writeMetaInfFileGeneration>
      </connector>
    </connectors>
  </configuration>
</plugin>

The flag sits on each <connector> entry, not on the plugin. Older examples in the plugin README use a <connectorClasses> element, but the real one is <connectors><connector>. Camunda does the same thing for its own AI Agent connector, whose components are Spring beans.

There is a catch. The plugin writes the file into src/main/resources, your source tree, and it only writes it if it doesn’t exist yet. Switching the flag to false stops new files. It does not remove the one from your last build, and that one is probably in Git by now. Delete it by hand. The same goes for renames: the file keeps the old class name. A services file that names a class that doesn’t exist, or one that doesn’t implement the interface, stops the runtime at startup. Only a listed class without @OutboundConnector gets skipped silently.

The other way is to switch off discovery for the whole runtime and keep the plugin defaults:

env:
  - name: CONNECTOR_OUTBOUND_DISCOVERY_DISABLED
    value: "true"

This disables SPI and environment variable registration, but never Spring beans. It also disables SPI for every other connector jar on that runtime’s classpath. And it’s the variable that got lost in my story. I’d use it as a second line of defence only. It is an environment variable and nothing else: there is no Spring property for it, so application.yaml won’t help you.

One more advanced gotcha, new in 8.10: the runtime doesn’t use the instance Spring already created for your bean. When it opens the job worker, it builds its own instance from the class, with constructor injection, and reuses that one instance for every job (one per tenant, if you run several). A @Bean factory method is skipped in that step. If you declare the connector itself through a @Bean method, mark it @Scope("prototype"), and the runtime asks Spring for the bean instead. In 8.9, the runtime still used the bean from the context.

Why Blending Breaks

With both families active for the same connector, you can end up in several places:

• Constructor injection, SPI file present: SPI tries to build the class and fails, because there is no no-args constructor. The runtime doesn’t start. That was my case.

• No-args constructor, SPI file present, also a bean: both registrations succeed, and the runtime refuses to start. Registering the same connector type twice throws. There is no “last one wins”, no matter in which order the runtime reads its sources.

• Field injection, SPI only: if the class has @Autowired fields and a default constructor, ServiceLoader builds it happily. The fields stay null until the first job hits them.

• Discovery “re-enabled” with false: the runtime only checks whether CONNECTOR_OUTBOUND_DISCOVERY_DISABLED exists. Its value is ignored. Set it to false and discovery stays off, and every SPI-only connector on that runtime disappears without a word.

• Type overrides on a mixed runtime: CONNECTOR_{NAME}_TYPE is looked up by the connector’s name. SPI and bean registrations of the same class share that name, so they get the same type and you’re back at the duplicate error. Env var registration is different, because you pick the name yourself. Register a bean and the same class by env var under another name with its own _TYPE, and both start. Now two workers handle job types you never meant to have.

Two switches really do apply to both families: CONNECTOR_OUTBOUND_DISABLED and, since 8.10, CONNECTOR_OUTBOUND_ENABLED. Each takes a comma-separated list of connector types, meaning the job types such as com.acme:invoice-lookup:1, not connector names or classes. _DISABLED is a block list, _ENABLED an allow list, and setting both stops the runtime at startup. They filter every source, and they match the type after any _TYPE override. A filtered connector stays registered but gets no job worker. That’s also why they can’t fix a duplicate: the duplicate check runs first, so blocking one of two registrations of the same type still ends in the startup error. Like the discovery switch, they’re environment variables only.

Which Family To Pick

Start with one question: does the connector need beans from a Spring context? That might be a client with connection pooling, or a component another team already maintains as a bean. If yes, you’re in the Spring family. There is no trick that gets injection into SPI. If no, both families work, and the question becomes who runs the runtime.

The SPI family is cheaper to operate. You ship a jar, and Camunda ships the runtime. A Camunda upgrade means a new base image tag and a rebuild. You pay for that in code: the connector builds its own clients and reads its own configuration, and testing that wiring is on you.

The Spring family turns the runtime into your application. You pick the starter version, keep it in line with your Camunda version, build the image and patch its dependencies. For a team that already runs Spring Boot services, that’s routine. For a team that only wanted to add one connector, it’s one more application to maintain.

It gets harder when one runtime should host both. Say three connectors are plain SPI jars and one needs beans. You can put all four into your own Spring runtime, and SPI keeps working there as long as discovery stays on. That rules out the discovery switch, so writeMetaInfFileGeneration=false on the bean connector is your only guard. The same applies if your runtime pulls in Camunda’s out-of-the-box connectors as dependencies, because those register through SPI too. The alternative is two runtimes: the stock image for the SPI connectors, and your own runtime for the bean connector, with discovery disabled as a second line of defence. That’s one more deployment. In return, each runtime follows one family, and the stock one has no variable to lose. If you can afford the extra deployment, I’d split.

Summary

Pick the family per connector, and per runtime where you can. Then make the build enforce it, so the choice doesn’t depend on a variable someone has to remember.

• If you need Spring beans, use the Spring family: your own Spring Boot runtime with the starter, writeMetaInfFileGeneration=false on the connector, no services file in src/main/resources. Disabling discovery as well is a backup, and it shouldn’t be your only guard.

• If you want to drop a jar into the stock image, use the SPI family: public no-args constructor, no injection, plugin defaults or a handwritten services file, jar on the image’s classpath.

• If one runtime hosts both families, the build flag is your only guard, because disabling discovery would drop the SPI connectors too. Two runtimes cost a deployment and keep each family on its own.

• CONNECTOR_OUTBOUND_DISCOVERY_DISABLED turns off SPI and environment variable registration, never beans. The presence of the variable is enough, the value doesn’t matter (incl. false).

• Same type registered twice means the runtime doesn’t start. Nothing overrides anything.

• Inbound connectors follow the same rules under their own names. Inbound beans must be prototype-scoped.

If I had put writeMetaInfFileGeneration=false into the DNS connector’s build from day one, the USB stick would’ve gone down those stairs once.

At a talk about agentic orchestration last year, someone in the audience asked me: “Why should we use Camunda for agent orchestration when we could just use n8n?” Fair question. Both draw boxes and arrows, both call APIs, and n8n gets you a working automation in an afternoon. My answer was one word: Enterprise. What I meant is everything that only starts to matter once a process runs at volume, crosses teams and has to pass an audit.

Two weeks ago Forrester published The Forrester Wave: Adaptive Process Orchestration Software, Q3 2026. It evaluates 13 providers and names four of them Leaders, Camunda among them. Forrester’s take says Camunda “is best suited for large enterprises requiring scalable, reliable, and observable process orchestration that includes agentic AI at scale.” That sentence is pretty much the answer I tried to give in that room, only with an analyst’s name on it.

Full disclosure: I work at Consid, a Camunda partner, and I’m a Camunda Champion, so read what follows as an opinion from someone who is invested in the topic.

Two things had to come together. Camunda 8 supplied the architectural foundation, but it was already on the market when Forrester rated Camunda a Strong Performer twice. What changed is the evaluation: the new category put the properties of that foundation at the center. In my view, without the painful jump from Camunda 7 to Camunda 8, there would have been little for the new criteria to reward.

From Consultancy to Leader

Camunda started in 2008 in Berlin as a BPM consultancy, founded by Jakob Freund and Bernd Rücker. In 2013 it forked the Activiti engine and became a product company, and the fork became Camunda 7: a Java engine most teams embedded in their application, backed by a relational database. Developers loved it because the engine wasn’t hidden away: it was developer-centric and open source. Business teams loved it because BPMN was the real execution model and not a drawing next to the code.

Camunda has been in Forrester’s Waves since 2023, always as a Strong Performer until this one. The first two were Digital Process Automation (DPA) Waves and the latest belongs to a different category, so read the placements as a direction across changing tests.

• Q4 2023, DPA: first entry, as a Strong Performer, with top scores in six criteria including end-to-end orchestration. According to Camunda, the report already singled out Zeebe for automation at scale.

• Q3 2025, DPA: Strong Performer again. Forrester wrote that Camunda suits enterprises that “require best-in-class orchestration.”

• Q3 2026, Adaptive Process Orchestration (APO), a category Forrester announced in September 2025: Leader, with the highest possible scores in the architecture criteria, adoption and vision.

Camunda scored top marks on orchestration and vision in both DPA Waves and still stayed out of the Leader circle. My reading is that a DPA evaluation also weighs things Camunda never wanted to be good at, like low-code app building for citizen developers. Forrester itself wrote in the 2025 DPA Wave that the low-code promise for citizen developers “is largely overstated, with most DPA projects deployed by experienced professionals.”

Gartner started its own category in the same period. Its Magic Quadrant for Business Orchestration and Automation Technologies (BOAT), which also scores use cases like platform consolidation, placed Camunda among the Visionaries in October 2025 and again on September 14, 2026. According to Camunda, Gartner calls process orchestration its strongest capability. The methods differ too much to line up as one ranking, and I can’t say exactly why each firm landed where it did, but the pair shows that the lens matters: an evaluation built around orchestration puts Camunda in a different place than a wider one.

Why a New Category Changes the Picture

In September 2025, Bernhard Schaffrik wrote that deterministic workflow engines, RPA bots and DPA tools are not powerful enough for autonomous operations. Forrester calls autonomous operations the North Star for many enterprises, and the DPA evaluations were built around deterministic automation.

At CamundaCon Amsterdam in May 2025, Schaffrik had called for solving “the trust gap with a blend of deterministic and non-deterministic approaches.” Four months later that blend was the definition of a new market. Forrester defines APO as “an automation platform that uses AI agents and nondeterministic control flows, in addition to traditional deterministic control flows, to meet business goals, perform complex tasks, and make autonomous decisions.”

Read that definition with a BPMN model in mind. Deterministic flow around the parts that must be controlled, an ad-hoc sub-process with an AI agent inside for the parts that can’t be scripted in advance. That is the agentic orchestration Camunda shipped with 8.8, and it is what Camunda calls outer and inner orchestration: BPMN coordinates agents, people and systems, and inside the agent it governs which tools the agent may call.

So the category followed the market, and the market moved toward the kind of platform Camunda had been building since Zeebe work started in 2017. The APO report opens on trust: companies hold back on agents because they don’t trust AI output to be reliable, and Forrester’s answer is the right architecture. The criteria that carry weight in this Wave are architecture criteria, and read as questions, they are the ones a bank’s architecture board asks before it lets an agent decide anything: how much volume the platform takes, whether it stays up while you upgrade it, and whether someone can reconstruct afterwards what the agent did and which rule allowed it.

Camunda landed right where those questions sit. Forrester rates it above par for running complex processes at scale, with top marks for reliability and adaptivity, and above par for runtime observability and optimization. It also credits the platform with strong auditability, live versioning and updates without downtime, which answers the architecture board’s three questions almost one by one. On strategy, vision and adoption get top marks, the user community stands out, and customers praise the support and a product team that is easy to reach. The weak spot is the design environment, which Forrester rates below par and sees as built for technical users, but the Wave puts most of its weight on architecture, not on how processes are drawn.

The switch reshuffled the whole field. Of the five Leaders in the 2025 DPA Wave, only Appian is a Leader in the APO Wave. Pegasystems, Salesforce and ServiceNow are Strong Performers under the new test, and IBM isn’t part of the APO Wave. Camunda is the only vendor that went from Strong Performer in DPA to Leader in APO.

Why This Needed Camunda 8

I don’t think Camunda would be a Leader in this Wave with Camunda 7. Camunda 7 runs demanding workloads and will be supported for years. The question is where its limit sits.

Camunda 7 keeps process state in a relational database. You could embed the engine or run it remotely and let workers fetch external tasks over REST, which took your code out of the engine’s JVM. Either way, every state change, every fetch-and-lock and every completion is a database transaction, and the history goes into the same database. You scale by adding engine nodes and tuning the job executor and history level, until that shared database is the limit and the one component every node depends on.

Zeebe, the engine behind Camunda 8, was started in 2017 to move that limit. It has no central database. State lives in a replicated, partitioned log, workers pull jobs over gRPC or REST, and you scale by adding partitions and brokers. That doesn’t remove limits, it moves them. Camunda launched Camunda 8 on Zeebe in April 2022, runs its SaaS on it, and in 8.8 built the Orchestration Cluster around it.

Agent workloads make that capacity matter more. In Camunda 8’s native agent pattern, the agent loop runs in the engine and each tool call activates a BPMN activity in the agent’s ad-hoc sub-process, so a busy agent adds many state changes. An agent that runs elsewhere and only calls Camunda puts less of that on the engine.

For an executive, the short version: on Camunda 7, growth eventually means a bigger, more carefully tuned central database. On Camunda 8, it means adding capacity to a distributed cluster.

The move had a price, and customers carry a large part of it. Code that ran inside the Camunda 7 engine, such as delegates and listeners, has to be reworked for Camunda 8, mostly into job workers, and the tests with it. Camunda’s migration tooling cuts that work down without removing it. Camunda 7 Community Edition got its final release on October 14, 2025. Enterprise support runs until April 2030, with paid extended support until a few years later. For teams with years of Camunda 7 code in production, that is a hard message, and I understand the frustration. Nobody has to move everything at once, though: new processes can start on Camunda 8 while the existing Camunda 7 instances run out, which is the drain-out approach in Camunda’s migration guide. As I see it, Camunda had two options: keep refining a well-loved engine whose ceiling it knew, or break with it and build for the market it expected a few years later.

Camunda’s commercial numbers point the same way. Camunda crossed $100 million in ARR in September 2024 after almost ten years, and according to its press release from August, ARR is nearing $200 million. Camunda credits enterprise customers spending more than $1 million a year and its native agentic orchestration, a Camunda 8 feature. The growth happened in the Camunda 8 era, but it doesn’t show how much of it came from the engine rather than from the agent wave, sales or a growing market, or what Camunda 7 would have earned in the same years.

That is why the bet paid off for Camunda. Whether it pays off for you is a separate question. The architecture is worth evaluating when your processes run at high volume, cross many systems and teams, have to stay up through upgrades, have to be explained to auditors, and increasingly include agents that decide steps on their own. If a few of your core processes look like that, the shift deserves a serious look. If your automation is small, local and stable, a Leader badge changes nothing, and a migration from Camunda 7 costs what it costs.

Where Camunda Is Still Building

Camunda doesn’t score top marks everywhere and it makes sense to take a closer look. Of the criteria Forrester rates as on par, three carry the most weight:

1. content and context intelligence

2. the AI agent library and discovery

3. orchestration testing

In my view none of them points at a weakness in the architecture. They’re the questions the whole market is still working on, because agents haven’t been in production long enough for anyone to have finished answers. Forrester used material vendors provided by July 17, 2026. Camunda 8.10 ships on October 13, and a good part of it lands right on these three points.

Testing is the hardest. A deterministic BPMN path is easy to test: you drive the process and assert on the path it took. An agent can take a different path on every run, so the test has to check whether the decision stayed inside the rules. Camunda started on this early. Since 8.9, Camunda Process Test has judge assertions: a model scores how well a process variable matches an expectation written in plain language, and the test passes above a threshold. 8.10 brings them into JSON test cases, and Camunda Hub gets a test mode that runs test cases with assertions from the modeler. What I want to see next is tests that catch the failures that matter, the wrong approval or the skipped check, reliably enough to block a release. The building blocks for that are already in the product.

Context is about what the agent knew when it decided. Here Camunda has an advantage that is easy to miss: a native agent runs inside a process instance, so the steps before it, the variables it got and the human decisions around it are already recorded by the engine. With 8.10, Operate adds the agent instance on top, with its system prompt, resolved tools, every prompt and tool call, and the model’s reasoning where the provider returns it. For an auditor that splits cleanly: the process trail, which steps ran, which tools the agent called with which inputs, and who approved what, is recorded by the engine and can be checked. The model’s displayed reasoning can’t, because it is not a complete account of why the model decided. Having both next to the process they belong to is still more than most teams can show today. What I’d like Camunda to build next is context as a managed asset: system prompts, policies and knowledge sources defined once, versioned and approved like a process model, and referenced by every agent that needs them. Camunda already handles process models that way, so the step is shorter than it sounds. The test is whether a team can show, for any agent decision, which inputs, which policy version and which tool calls led to the outcome.

Discovery is where Camunda Hub comes in. In Camunda an agent is a connector, an element template with a version, and with 8.10 the engine knows agents as definitions with their own API. At CamundaCon in May I saw Hub flag models still on old connector template versions. Through the Agent Instance API, agents built with LangGraph or CrewAI report into the same view, so Camunda doesn’t have to run every agent to see it. What I’d want next is one inventory across real deployments: every agent, its version, its owner and every process that calls it. Today that view stops at the cluster. Until an organization can answer “which processes use this agent, and who owns it” across its clusters, the gap is open, but it is an inventory on top of data Camunda already collects.

The engine work that made this placement possible is behind Camunda, and that was the hard part. What’s left is making agent governance operationally dependable, release after release. Given what 8.9 and 8.10 already brought, I’m betting Camunda gets there.

What’s Next

Camunda’s next move is ProcessOS, announced at CamundaCon in May. It leaves the closed beta with 8.10 and becomes available to all paying customers. It brings AI into how processes are discovered, designed, built and optimized, the same area where Forrester rates Camunda’s autonomous process design as on par. At CamundaCon, Camunda showed it on its own Quote to Cash process: a discovery agent queried the real systems and documented the process in under an hour, and a build agent generated the BPMN and DMN from the redesign, with approvals still enforced in BPMN. I wrote about it in May, and I’ll look at it hands-on once 8.10 is out.

• Camunda 8 supplied the architecture. The APO Wave made it decisive.

• The bet paid off for Camunda. Your migration case is your own assessment, and April 2030 is closer than it sounds.

• Agent testing, context and discovery are where Camunda goes next, and 8.9 and 8.10 already moved on all three.

When someone asks me the n8n question again, my answer will still be one word, and now boards and architects can read an independent evaluation of what it means for agentic processes in the report Camunda offers. It won’t tell anyone whether Camunda fits their processes, and it doesn’t make n8n the wrong tool for the work it does well. The next step is showing that agents on Camunda can be tested, explained and governed as reliably as the processes around them, and with 8.9 and 8.10 that work has started.

Camunda 8.10 ships on October 13, 2026. Camunda 8.7 reaches end of standard maintenance that same day, which leaves anyone still on 8.7 with no overlap between the release going out of support and the one replacing it. The release also removes everything 8.8 deprecated.

The headline features are Camunda Hub and an engine API for agents. For existing users, most of the actual work sits a bit deeper: removed APIs, changed defaults, Helm 4 for self-managed, and the Web Modeler migration. Camunda’s own what’s-new page starts with unified authentication, Optimize data filters, and that migration.

Camunda Hub Replaces Console And Web Modeler

Camunda merged the two products into one. Center of excellence teams manage infrastructure, access, and workspaces at Organization level; delivery teams model and deploy inside a Workspace. Almost every name changed.

Workspaces have their own members, roles, projects, and clusters, and the Catalog holds element templates synced from one Git repository. Credentials live as organization-level cluster variables; connectors reference them as camunda.secrets.KEY.

At CamundaCon in May, I saw Camunda Hub show which workers and connectors belong to each cluster. It also flagged models using old connector template versions, and it kept tenants, secrets, and connections in one place. Today I open models one by one to find those templates, and I cannot see the worker-to-cluster connection at all. If the released Hub matches that demo, I stop doing both.

The Hub API v2 arrives with the release, and Web Modeler API v1 is deprecated in 8.10 and removed in 8.12 (Oct 2027), with the Console self-managed API on the same schedule, which leaves anything built against either of them two releases to move across before it stops working. Camunda migrated SaaS to the new workspace structure in August. Self-managed gets the same structure at GA.

In our SaaS organization the migration arrived in the modeler. Folders came back as process applications, created by Camunda Migration, with an info panel saying only the location changed, not names, files or version history.

Self-managed users have to replace two chart settings:

# 8.9
console:
  enabled: true
webModeler:
  enabled: true

# 8.10
camundaHub:
  enabled: true

The images become camunda/hub and camunda/hub-websockets, and any leftover camunda.console.* configuration fails the application on startup. The chart change is the easy part: there is no rolling upgrade from Web Modeler to Hub, so every Web Modeler instance has to stop and its database has to be backed up before Hub starts, and because the database migration cannot be reversed, that backup is the only way back to 8.9. The component upgrade guide lists every removed setting. Management Identity is still separate and still required. Console Admin and Web Modeler Admin both gain access to Hub cluster pages, so check who holds those roles before the upgrade.

What The Engine Now Knows About Agents

The orchestration cluster models agents as first-class entities now. An agent definition comes from a zeebe:agentDefinition extension on the model, and an agent instance carries state, token usage, tool inventory, and conversation history grouped by loop iteration. Eight new endpoints cover definitions, instances, updates, and history search.

None of it shows up for an agent modeled before 8.10. Operate fills the panel once the element template moves to latest v1 or v2 and the model is redeployed; until then it stays empty.

Select the agent element in Operate and you get the instance key, model, system prompt, resolved tools, and usage against the configured token and call limits. The state moves through Initializing, Tool discovery, Thinking, Tool calling, Idle, and Completed. Below the diagram, the conversation history groups every prompt, message, tool call, input, and output by loop iteration, inside the same process instance, and shows the model’s reasoning inline as a Thinking entry where the provider returns it.

The 8.9 version of that documentation page had you switching on execution counts and hunting tool inputs in a parent scope. The 8.10 page says you rarely need raw process variables. Operate also renders a LangGraph or CrewAI runtime beside the process instance when it registers through the Agent Instance API.

The engine resolves the tools from the agent’s ad-hoc sub-process. That list is the BPMN boundary on what the agent can call, and the engine enforces it.

The Agentic Control Plane is an Optimize dashboard over completed Camunda AI agent instances. It is scoped per cluster, keeps 30 days of data, and does not manage or govern agents. The Camunda-provided LLM shipped in May and is documented identically for 8.8 and 8.9, so an 8.9 cluster already has it.

On May 20, after Camunda’s ProcessOS keynote at CamundaCon, I published Camunda Announces ProcessOS and said I would follow up when 8.10 shipped. Two weeks before GA, the docs cover the Control Plane dashboard, live agent state and conversation history in Operate, and processes as MCP tools. ProcessOS, Test Studio, and the Business Value Dashboard are not in them. ProcessOS itself is still in beta behind a waiting list, and Camunda gives priority to companies that bring a use case and a forward deployed engineer with it. Fingers crossed I get my hands on it soon.

EDIT: I wrote this against what was checked into the docs two weeks before GA, and Camunda of course works double shifts before a go-live. Two updates came in from Camunda after I published. ProcessOS, Test Studio, and the Business Value Dashboard will ship with 8.10 after all, and ProcessOS leaves the closed beta and becomes available to all paying customers; the release notes are being finalized now.

What Changed Outside The Agent Work

The orchestration cluster API gained 44 endpoints in 8.10 and removed none. Every active element instance can report what it is waiting for: a timer with its due date, a message with its name and correlation key, a signal, a conditional expression, a user task, or a job. Operate shows it under a Waiting label and details tab by default.

Business ID is now searchable across process instances, decision instances, user tasks, and message subscriptions. It also narrows message correlation as an extra filter, and FEEL reads it through camunda.processInstance.businessId.

Camunda added four endpoints for suspending and resuming process instances, plus a SUSPENDED state and a new permission, and Operate can suspend a single instance or a batch. Suspension halts the instance where it stands: no jobs activate or complete, no messages or signals correlate, and no timers fire. Variables stay readable and writable, so you can fix bad data before resuming. Timers that came due during the pause fire the moment the instance resumes, so a long suspension ends with every overdue timer firing at once.

Physical tenants sit between logical tenants and separate clusters as a third isolation option. Each tenant gets dedicated Raft groups, isolated secondary storage, and its own document store, while brokers and gateways stay shared, and in 8.10 the whole thing is self-managed only and configured statically with a rolling restart, with no runtime create, rename, or delete.

What Breaks In 8.10

Camunda removes everything it deprecated in 8.8, and the 8.10 migration guide maps each removal to its replacement. The Operate API, Tasklist API v1, Tasklist v1 mode, public start forms, and user task access restrictions are all gone. Java projects have to swap ZeebeClient for CamundaClient and Zeebe Process Test for CamundaProcessTest. Both show up as compile errors. The Camunda Spring Boot Starter also moves to Spring Boot 4.1.x, which pulls projects on 4.0.x into a migration of their own. Anything calling the Operate API or Tasklist v1 stops working until those integrations are rewritten. On self-managed, Optimize also rejects its static API token with a 401 straight after the upgrade, so any client that sends OPTIMIZE_API_ACCESS_TOKEN has to move to OIDC bearer tokens first.

Four behavior changes may not fail loudly:

• Java client OAuth credentials are cached in memory by default, with the file cache now opt-in

• JobIntent.COMPLETED no longer carries variables by default, which breaks custom exporters reading them

• Message subscription search now returns start event subscriptions, with a new type field to filter them out

• Resource APIs became eventually consistent

A custom exporter can keep running while it drops fields from a report.

Element templates can declare operations, so the modeling menu lists send email or upload object as searchable entries. Nineteen single-operation connectors were renamed, REST Outbound Connector to Send REST Request among them. Template IDs and runtime behavior did not change, so deployed models keep running and only the documentation and training material need updating. AWS connectors moved to AWS SDK for Java v2. STRICT became the default connector secret filter back in Connectors 8.9.10 and stays the default here, which the 8.10 announcement still lists as a breaking change.

Camunda 7 Gaps That Close

Several 8.10 items restore something Camunda 7 had and Camunda 8 did not. Camunda does not present them that way, and the release notes never mention Camunda 7.

Business ID is the big one. Camunda’s own data migrator maps the Camunda 7 businessKey onto it, and in 8.10 it stops being a label you carry around. Camunda 7 correlated messages by business key directly.

Execution listeners now take headers, which covers the field injection a Camunda 7 listener class relied on for its configuration, and a cancel listener runs when an instance is terminated, a hook Camunda 7 fired by default and Camunda 8 never had. Suspend and resume come back as API endpoints.

The SUSPEND authorizations have also been added, which closes the permission gap from the Camunda 7 section, and the docs follow shortly.

SaaS And Self-Managed Upgrade Notes

On SaaS

Optimize data filters are new in cluster settings, on by default for new clusters and opt-in for existing ones. Saving a filter restarts the cluster, and the records it filters out are gone for good.

Camunda’s official SaaS multi-tenancy support applies to generation 8.8 and later clusters. The same per-cluster toggle already appears in the 8.8 and 8.9 docs, and the 8.10 release note says no upgrade is required. Tenancy checks remain off by default, and turning them on locks out any principal without a tenant assignment.

On Self-Managed

Helm v4 is required and chart 15.x fails fast on Helm v3. Do the CLI switch while you are still on 8.9, because chart 14.x is the last one that accepts both. Nothing has to be migrated with it: Helm is client-side, both CLIs read the same release secrets, and there is no helm 3to4 step for Camunda charts. Server-side apply is on by default in Helm 4, which turns duplicate env entries into an install error. Bitnami subcharts are gone, another migration that belongs on 8.9. Minimum versions rise across ElasticSearch and OpenSearch (v2.19+), and PostgreSQL 14, SQL Server 2019, and H2 2.3 are dropped.

Four defaults then move underneath the install, and none of them stops the upgrade. RocksDB memory allocation stops being 512 MB per partition and becomes 10% of broker memory shared across all of them, which puts break-even at 5 GB per partition and most clusters below it. The smaller block cache sends reads that used to come from memory to disk, so throughput falls under load and the first sign of it is a latency graph. OpenSearch index replicas go from 0 to 1, which on a single OpenSearch node plan leaves a replica with nowhere to go, so cluster health sits yellow and the alert fires for a redundancy you are not getting, and on a real cluster it roughly doubles the disk every new index costs. The exporter switches to Optimize mode and drops every record value type Optimize does not consume, job records included: anything reading the Zeebe record indices directly quietly thins out. Optimize stops flattening object variables, so reports built on their properties come back empty, and the reimport that fixes them only works while the records are still inside the Zeebe retention window.

We made the Helm 4 switch on our own self-managed 8.9 platform on STACKIT in mid-September, a month before GA, and it was a non-event. Chart 14.x takes both CLIs, so it came down to a version pin in CI and a new test rule that refuses a Helm 3 binary. Test and prod rendered byte for byte identical across the Helm 4 patches we tried, every policy assertion passed, and nothing in the cluster moved. The reason to do it early is ArgoCD, which renders every Application with Helm 4 from 3.5 on: Helm 4 keeps explicit null fields that Helm 3 dropped, so a chart can render differently with no version change at all.

We turned that exporter filter on by hand in early August, on the same platform. Export was the binding constraint in our load tests then, and switching it on doubled export to 851.000 records/minute with the exporter thread still 40 percent idle, which moved the bottleneck off export and onto the engine’s accept ceiling. In 8.10 it arrives as a default, so the throughput comes for free, and so does the record loss that pays for it.

Summary

• Still on 8.7: standard maintenance ends on October 13, the day 8.10 ships. Nothing else on this list matters until that hop is scheduled.

• Custom integrations: inventory every Operate API, Tasklist API v1, ZeebeClient and ZeebeProcessTest dependency now. The library removals surface as compile errors, the API removals as failing calls.

• Self-managed Kubernetes: switch to the Helm v4 CLI while chart 14.x still accepts both, then re-render your values against chart 15.x and check custom exporters for the changed completed-job records.

• Agent models: move the element templates to latest v1 or v2 and redeploy, or the new Operate panel stays empty.

• Everyone: the Hub names reach runbooks, training material and support conversations long before anyone touches a cluster.

The full detail is on Camunda's what's-new page for 8.10: https://docs.camunda.io/docs/next/reference/announcements-release-notes/8100/whats-new-in-810/.

In the last article in this series, I want to give you an overview of our learnings and what we eventually paid during development. We can’t answer every question yet, but we are very close to a production-ready platform that holds the load a customer expects.

To make it short: One day of STACKIT billing across our three projects came to 70 €. Prod accounted for 51 €, test for 16 € and shared infrastructure for 3 €.

That is only the provider bill. The people who build and run the platform cost more than the infrastructure does. We did not do any of this to save money.

What Held

I would use Camunda’s AWS single-region reference again. STACKIT publishes no equivalent blueprint, so we mapped the published architecture component by component, and that mapping is where every major prod choice came from, including the three brokers spread across three availability zones.

The two-repository split and keyless pipeline from Part 2 also held up. Every change that reached a cluster passed a review with a person’s name on it. Infrastructure and platform changes kept separate permissions and separate failure boundaries.

Above the infrastructure layer we wrote exactly one STACKIT-specific line: a load-balancer annotation that turns on PROXY protocol. Everything else is standard charts, manifests and operators. Nothing in the Camunda layer knows which cloud it sits on.

Decision records, a findings log and runbooks kept earlier work available to the next person, and to the next agent session. A hand-maintained set of numbered Markdown files stops working once the platform is live and several people are filing findings at once, so we froze both registers and moved new work into tracked GitHub issues.

Part 5 put a number on the sizing: roughly 72 to 80 accepted process instances per second on the current hardware.

What Hurt

STACKIT’s managed Kubernetes gives you a cluster and stops there. Ingress, certificate automation and secret synchronization were ours to install, and then ours to keep running, including their upgrades and their alerts. On Camunda’s reference cloud two of those three are managed services and the third is an add-on. That work carries on after go-live, so price it in. It is the warning I would give first.

Managed OpenSearch came with a narrower limit. Its application credential has no index-lifecycle permissions, and with Camunda’s retention enabled the engine never finished starting up. We turned native retention off and scheduled a job to prune dated indices instead.

In August, prod’s search cluster was renamed outside Terraform. The provider treats that name as immutable, so the next plan proposed destroying the secondary store and building a new one. Destroy protection stopped the apply. I shipped a temporary name override so the rest of the prod changes could go out, opened a ticket, and removed the override once STACKIT put the original name back.

Ingress ate several days. spec.loadBalancerSourceRanges does enforce the L4 restriction, so the source gate holds at layer 4. Separating the UI audience from the Zeebe audience needs L7 rules on top of that, and those rules only work when the PROXY protocol chain in front of them is correct. Part 3 has the detail, together with our decision to postpone inbound connectors and the broad HTTPS egress that outbound connectors still need.

Entra was worse. One audience value in the wrong field produced a login loop, and a group-mapping rule then used equality against a claim that arrives as a list. We only found them by walking the whole authentication path, from directory membership through to Camunda authorization.

There are few Camunda-on-STACKIT examples to copy, and some provider differences only showed up under load. We upgraded the search plan, and then found an exporter setting that doubled capacity on the hardware we already had. Part 5 tells that story. The bigger plan is independently justified now for its heap and shard budget, and we still bought it in the wrong order: capacity problems come with a bill attached, and the order you investigate them in decides how big it is.

What It Costs

The prod bill shows where the money went.

Test came to 15,80 € for the same platform shape at smaller sizes. Shared infrastructure cost 3,20 €, and almost all of that is one CI runner on a fixed public address. The managed services are gated by address, so a changed IP would lock that runner out of every one of them. Three euros a day is the floor you cannot easily design away.

We hibernate the test and prod clusters outside working hours. On the measured day that put 39 node-hours on prod’s bill instead of 72, and 13 on test’s instead of 24. The infra project has no cluster to hibernate, so its 3,20 € lands every day of the month, weekends included.

August came to 1.240 € for prod, and that is what we actually paid. It is not a rate. We swapped resources in the first week and woke the clusters when there was work rather than to a timetable, so the nodes billed 426 hours across the three of them over the whole month. Take the month as evidence that hibernation works, and take the numbers that follow from one regular day instead.

So take one regular day, the one in the table above, where both clusters woke at 07:00 and slept at 20:00. Both projects run the same machine type at 0,41 € per node-hour, and compute is the only line the schedule moves: around the clock, prod’s three nodes take it from 15,96 € to 29,46 € a day and test’s single node from 5,32 € to 9,82 €. Everything else on the bill, the replicated stores and the storage and the cluster management, holds at its measured level either way, which is why a weekend with both clusters asleep still costs 49,05 € across the three projects. Scaling that one day two ways gives the months below, both of them arithmetic rather than a billed month: the left column keeps the hibernation schedule, the right one never lets the nodes sleep. Put customer processes on this platform and the right column is your bill.

Three services account for 75 % of the prod bill: the nodes, the search cluster and the database. Add cluster management and the share reaches 85 %. All the block storage together is 13 %. The load balancer, the public IPs, DNS, the secret store and object storage come to 1,4 % between them, which is noise. The two replicated data stores cost more than all the compute put together. That is what it costs to let prod survive the loss of an availability zone, and it is why test’s day costs less than a third of prod’s.

A full month sharpens it. In August the search cluster cost 376,53 €, the database 280,81 € and the nodes 174,30 €, so the two replicated stores came to nearly four times the compute rather than the 1,4 times the single day showed. Hibernation is why: the stores bill every hour and the nodes do not.

The larger cost is people. I built this platform with an agent, which is Part 4’s story. Running it takes two, because nobody is on call every day of the year.

Camunda charges two things: a fixed amount per root process instance, and the infrastructure it runs on. Self-managing changes only the second one. You still owe the per-instance amount, and in place of Camunda’s infrastructure component, which is billed at roughly cost recovery and lands below what we pay for prod, you carry your own bill and the people who run it. There is no list price to check that against, because agreements get negotiated around volume and need. Sovereign self-managed prod costs more either way, on the bill and on the team. For an organization that needs EU legal sovereignty, SaaS on US infrastructure is not an option, whatever its price.

Running It

Someone still has to run it. Our runbooks cover credential rotation per credential, an identity-provider outage, a failed apply against a cluster that is not awake, restoring either environment, an unstuck GitOps sync, rebalancing partition leaders, and the day the office address changes and every gate rejects at once. You read them after the build team has moved on, or while something is already broken.

We run the monitoring stack ourselves, the kube-prometheus-stack with Loki for logs, so the alerts and dashboards are ours to keep working. Buying STACKIT’s managed observability instead would have added another 200 € to the monthly bill.

Keeping the platform current is the piece we haven’t automated yet. An agent pins the version it learned, so a chart or provider release that has already been superseded can arrive in a pull request looking current, and a reviewer is a poor detector for that. We plan to put Renovate on both repositories and let a bot watch upstream releases.

Backups run nightly into object storage. Both environments have been restored end to end, and the data came back.

The Verdict

The repositories rebuild both environments, policy checks guard the exposed paths, and Part 5 found the current ceiling. A team of two can run this platform today. That is the bar I would set before calling something a platform.

STACKIT carries Camunda and its process data with no US operator underneath. Two dependencies sit outside that data path: Entra for identity and GitHub for code hosting. The capacity number belongs to our synthetic process shape, so your own process mix still has to be measured.

My verdict: it makes sense when legal sovereignty and less dependence on American hyper-scalers are real requirements. You control where the platform and the process data live, and you carry the higher infrastructure bill and the team that comes with it. Take the requirement away and I would struggle to justify either.

For this build, sovereignty means about 2.650 € a month of projected infrastructure across the three projects, the people who keep it running, and the per-instance fee you would owe Camunda either way.

We had sized prod from Camunda’s reference architecture and our own arithmetic. No real load had passed through it. We spent the next week finding a capacity number we could support with measurements.

The prod setup under test had three g2i.8 nodes, 8 vCPU and 32 GiB each, one per availability zone, with Zeebe across them as three brokers, three partitions, replication factor 3 and a 64 Gi volume per broker. Secondary storage is a managed OpenSearch service, three nodes in HA, which started the week at 2 vCPU and 8 GiB per node. Two exporters run on every partition: the newer one feeds Operate, Tasklist and the search APIs, and the legacy one is there because Optimize needs it.

The first graph looked good. For a few minutes we thought the platform had exceeded our expectations, until the job-completion count showed that the generator was producing far less load than configured. Later runs repeated that pattern: one metric supported a capacity estimate, then another showed which limit we had measured. Across six runs, we produced four estimates and discarded three.

Note: The bench cluster is specific to our setup. The measurement errors apply to any Camunda 8 benchmark.

How Camunda Absorbs Load

Camunda absorbs load by measuring itself. Nothing here caps the request rate. Each broker derives its own ceiling from how fast it is answering, and three loops move that ceiling.

The first loop: A command is not executed when it arrives. The broker appends it to its partition’s log, and the stream processor reads it back later. So a broker always holds requests it has accepted and not yet answered. It counts those, in flight, and measures the round trip from accepting a request to answering it. Above a limit on the in-flight count, requests are rejected instead of queued. That rejection is backpressure, and limit and count are both per partition.

The limit moves. An adaptive algorithm recalculates it, AIMD by default: a round trip faster than the broker expects raises the limit by one, a slower one cuts it by a tenth. Ours sits at an initial 100 in-flight / partition, with a floor of 1 so sustained load cannot walk it down to nothing.

Exporters close the second loop. Each walks the same log from its own position into secondary storage, the copy that Operate, Tasklist, Optimize and the search APIs read. Flow control limits the export write rate too, and can throttle it as the unexported backlog grows. Slower writes leave the stream processor waiting on its results, the round trip lengthens, the limit shrinks. A slow search cluster reaches clients as engine backpressure.

Disk closes the third loop. The log compacts only up to the earliest exporter position, so an exporter that stops advancing keeps it growing. Below 2 GB free the broker rejects every client command and pauses processing, and below 1 GB it stops accepting replicated events too. Both limits can be configured.

Whichever loop closed, the client gets RESOURCE_EXHAUSTED over gRPC, or an HTTP 503 saying the same, and should retry with backoff and jitter. A flood of start requests and a slow OpenSearch look identical from out there. Completing and failing a job are exempt, accepted above the limit, because work already inside has to be able to leave.

The First Run Measured The Client

We use the camunda-8-benchmark project from Camunda Community Hub to put load on prod. It starts process instances at a configured rate and completes the service tasks in the synthetic model.

The pilot started from a laptop, with one generator driving prod over the public gRPC path at 150 process instances per second. The first dashboard looked almost too good: backpressure stayed under 1%, with no engine ceiling in sight. We wanted to believe the sizing had headroom.

The job-completion count showed the problem. The model has ten sequential service tasks, so 150 PI/s needs roughly 1.500 completions per second. The generator plateaued between 585 and 606. We traced the gap to the ingress recycling HTTP/2 connections after about a thousand requests. Each recycle killed the job-push streams, leaving pushed jobs on dead streams until timeout. Prod had looked comfortable because it never received the load we thought we were sending.

So we built a permanent cluster whose only job is generating load: three nodes, one generator per node so they cannot contend for a CPU or a network queue. Generators inside prod would have been cheaper and wrong twice, because their CPU competes with the brokers and they skip the public ingress path under investigation. The bench cluster comes in over the internet instead, admitted by a narrow allowlist on that one route.

Every run since reports the generators’ own CPU against their limit. Peak draw went from 868 millicores on the first cluster run to 98 of a 3.000 limit on the last. A load test that cannot show its generator had headroom has not established what it measured.

The Bottleneck Was Downstream

With three generators, we offered the same 150 PI/s aggregate and watched the client path stay clean: zero GOAWAY, zero closed streams, zero UNAVAILABLE. Prod now received the intended load. Every error was engine-side, and backpressure reached 42% of start attempts against a 10% target.

The engine was only one path. Export lag climbed on all three partitions to 2,85 million records after five and a half minutes of load, and draining it afterwards took fourteen minutes. The original sizing exported about 280.000 records per minute, so every minute of load cost roughly 2,5 minutes of export tail. Export capacity, not the engine, bounded how long a run could last. Both sides of that ratio changed again later in the week.

We answered with three changes at once. That got the next run moving and made attribution impossible. Each broker went from a 2 Gi memory request and a 4 Gi limit to 4 Gi and 8 Gi, with its CPU limit left at 3 cores. The managed search cluster moved from three nodes at 2 vCPU and 8 GiB to three at 4 vCPU and 16 GiB. We also set the exporter’s index refresh interval to 5 seconds. Together they raised export capacity about 36%, to 380.000 records per minute.

The broker resize added memory headroom. Peak working set stayed at 3,73 GiB, which was 91% of the old limit and 47% of the new one. Because the image exits the JVM on out-of-memory, a longer run at the old limit could have cost us quorum.

Capacity Arithmetic Uses The Accepted Rate

Export remained the bottleneck, so we wanted the rate at which its backlog stopped growing. Two reports estimated that rate instead of measuring it, and both came out about 2,4 times too high.

The method was reasonable. Export capacity and record generation are both rates, so dividing one by the other tells you what share of the current load the exporter can keep up with. Apply that share to the load and you have a sustainable rate.

The load we applied it to was the wrong number. We’re offering 150 process instances per second, but backpressure was rejecting 40 - 47% of those starts, so the engine only ever accepted around 80. Nearly half the load in the calculation never reached the engine.

Capacity arithmetic has to use accepted starts. Nothing in the tooling tells you which of the two rates you are looking at, so it is worth checking before either one goes into a calculation. Once we saw the mistake, another spreadsheet felt less useful than a measured crossing point, so we went back to the cluster.

Backpressure Is Not A Capacity Signal

Sustainable means export lag neither grows nor shrinks over a long window, measured as the slope of the lag curve. We kept a standing backlog between steps instead of draining, so the exporter stayed saturated and the slope was generation minus capacity. Draining between steps would have made every slope depend on the exporter warming up again.

# Export lag: positions appended to the log, minus positions the exporter
# has walked. Take the slope over a long window; the level tells you nothing.
sum(
    max by (partition) (zeebe_log_appender_last_appended_position)
  - max by (partition) (zeebe_exporter_last_exported_position)
)

We planned to increase the rate in fixed steps from 75 PI/s. The first step produced +415,000 records per minute of lag, so we reduced the rate instead. Four steps at 75, 45, 36 and 30 PI/s gave lag slopes of +415.000, +118.000, +2.000 and −85.000 records per minute. The line between them crossed break-even at 35,9 PI/s.

Backpressure was useless as the instrument. At 30, 36 and 45 PI/s it sat at zero while export fell behind by up to 118.000 records per minute. The benchmark tooling’s ramp strategies climb until the engine pushes back, so they would have reported double the truth. Export capacity itself was a hard constant near 427.000 records per minute, within 0,4 % whether the brokers were under full load or idle. Nothing was bought between that and the 380,000: same brokers, same plan, one drain reading in an earlier run against a saturated exporter here.

The Cheapest Capacity Was Free

The largest improvement was already available in the exporter configuration. The legacy exporter serialized everything the engine produced even though Optimize, its only consumer, imports five record types.

Cutting the job records alone, 18% of that exporter’s write volume, had bought no throughput at all, because the metric that bounds run length counts log positions walked rather than documents written. What worked was dropping records before serialization instead of after, with the exporter’s own filter for what Optimize imports, plus skipping a class of variable writes on the newer exporter. Same hardware, same load profile, one configuration change:

• export capacity from 427.000 to at least 851.000 records per minute, a measured 2x, with 40% of the exporter thread still idle

• the post-run drain from 2,68 million records and nearly nine minutes to 26.000 records and about 90 seconds

We had bought hardware before finding the configuration limit. The larger plan raised capacity 1.36x and added a permanent monthly charge; the filter doubled it on the machines already running. The upgrade was still useful because it doubled the search cluster’s heap, and shard count has since outgrown what the old plan could hold. We should have finished reviewing the exporter configuration before upgrading the search plan. Part 6 shows the cost.

What It Holds

Doubling export capacity also doubled the projected break-even rate. The 35,9 PI/s result belonged to roughly 427,000 records per minute of export; at 851.000, the calculated balance is near 72 PI/s. At that rate, engine acceptance became the limiting factor. On the final run we increased the offered rate again, with export tracking generation at every step.

On the final run, accepted throughput peaked around 80 process instances per second and fell to 72.6 when we offered nearly twice that, because rejecting a start still costs the engine work. We trust the range of 72 to 80 accepted process instances per second at this broker CPU limit, with no connectors. Prod completed 963,902 jobs across 25 minutes of load. One instance of this model produces ten jobs and about 186 log records, and that second number is the conversion to redo before comparing any of this to a different process.

That ceiling is self-imposed. Each broker runs with a CPU limit of 3 cores, sat at 2,7 of them under load, and spent 27 to 48% of its scheduler periods throttled against that limit. The nodes underneath have 8 vCPU, and the busiest of the three peaked at 2,7 cores of the 7,9 it has to give. We set the limit to cover the broker’s three processing threads and left the request at 1 core, because that busy node also carries the ingress controllers and every non-broker Camunda component and is already at 79% of its CPU requests. The measured ceiling belongs to those resource limits; raising them remains untested.

The relevant alert changed with the bottleneck. Export backlog identified the limit in the middle runs while backpressure stayed silent. After the exporter change, backpressure identified the engine limit and export lag stayed flat.

Operate and the v2 search and statistics APIs read the secondary store, which in the early runs still reported ten thousand active instances more than an hour after everything had finished. Ground truth is the broker’s exported position. Never diagnose engine state from the search APIs under load.

The process shape is synthetic, ten sequential service tasks and two timer catch events, not a customer’s mix. The longest window we held flat was 22 minutes and the final run carried 25 minutes of load, so nothing here says what a day of it does. We also measured throughput and not client response time, so how long a start takes to return at 70 PI/s is still an open question.

What You Turn If You Need More

If you need more throughput, start with the broker CPU limit because the nodes already have unused capacity. Whether raising it moves the accept ceiling is untested: append rate held flat near 13.500 records per second across all three steps of the final run, which does not look CPU-bound, so change one variable and measure. The next option is more cores per broker or bigger nodes, both of which add a permanent cost. More partitions provide horizontal scale, but raising the partition count is a one-way door.

Whichever lever you pull, export has to move with it. Its capacity is a hard constant for a given configuration, and break-even scales with that constant. Raising the accept ceiling alone relocates the bottleneck.

What’s Next

The number that survived six runs is 72 to 80 accepted process instances per second at this sizing. The method is what carries beyond our setup: measure the accepted rate and never the offered one, read the export backlog slope while export is the bottleneck, and find out what the exporter thread is doing before buying hardware.

Part 6 closes the series with the bill: what the platform costs to run, what held, what hurt, and whether the sovereignty trade was worth it.

I was lucky to be in assignments where I’ve learned a lot about building and running platforms in the cloud. I also learned that you need many resources to build and run a platform. Since we cannot allocate all of our time to this platform blueprint, we decided early to use agents for implementation and documentation.

Many of the Terraform and Kubernetes manifests behind Parts 2 and 3 came through pull requests opened by our coding agent. It mapped the reference architecture onto an unfamiliar provider, read live state and pushed branches.

The agent could prepare a branch, but it could not merge the pull request or run a terraform apply, and neither could one of us without going through the same review path. A reviewed merge started the only apply workflow, using an identity scoped to one environment.

My work shifted. I typed fewer resource blocks and spent more time comparing proposals with live state, reading plans and checking whether the recorded reason matched the change. But that only works when the reviewer knows enough to stop a plausible answer.

The Same Guardrails You Would Give A Person

Before we started handing work over to an agent, we ensured that an AI harness was in place. Each repository has an operating guide and skills that defines its job, what belongs elsewhere and the rules we learned while building. Stability, blast radius, least privilege and recoverability come before convenience. Providers and backends stay in roots. Reading live state is routine while changing it directly is strictly forbidden.

The guidance gets more specific near the work. Our runbooks folder has its own rules because an incident procedure needs different checks from a Terraform module, while Terraform and Helm conventions load only for sessions that use those tools. The agent starts each task with a small, relevant piece of the platform’s history instead of one huge prompt.

Every check from Part 3 runs on all pull requests, and the per-environment identities from Part 2 limit the eventual apply. We already needed those controls for ourselves. Everyone has a bad afternoon, copies a value from the wrong tab or points a command at the wrong environment.

For larger changes, Claude writes and ChatGPT reviews before I do. The second model has none of the discussion that produced the diff and must judge the evidence on the page, which is useful for finding unstated assumptions or a solution that has grown beyond the problem. I make the final decision. Two models can agree on the same bad idea with impressive fluency.

Everything Goes In The Repo

A human who debugged the managed OpenSearch will probably remember its provider-assigned high port number. A new agent session starts without that memory and uses the default port number.

Before closing a session, we put what it learned into the repository. Decision records capture the reason for an architecture, the alternatives we rejected and the condition that would make us revisit it; the next session reads that before proposing a change, and a reviewer can inspect the reasoning without reconstructing it from a diff. Review became the bottleneck quickly, so that context saved real time.

A findings log records smaller provider constraints, such as a bucket that cannot be deleted while it still contains object versions. Without that entry, a later session has to investigate the same failure again. That short feedback loop produced more decision records, findings and runbooks than we would have written by hand. We also keep track of open issues in GitHub, if there is no time or urgency to fix it right away.

They still go stale. At one point, decision records referred to Terraform variables that no longer existed. We treat stale documentation as a bug because the agent will follow it just as confidently as current instructions.

Where Our Agent Was Confidently Wrong

The load-balancer work from part 3 showed what happens when the agent is uncertain. An early test appeared to show that the allowed source ranges did nothing, so it treated the observation as a provider limitation and moved filtering into the ingress controller. A bad L7 configuration then returned 403 for every request in test. Its next proposal concluded that this cloud could not gate the surface reliably.

After two days, we had three architectures around a mechanism we had never isolated. I stopped the sequence, removed the workarounds and repeated the L4 check with a disposable load balancer. An excluded caller was blocked in ten out of ten attempts, and access returned when we restored the admitted range. spec.loadBalancerSourceRanges was enforced. We never found what had invalidated the early observation.

L7 filtering still has a job. Our UI and Zeebe routes have different audiences, and the ingress controller needs the real client address from the PROXY protocol chain. The final design kept both layers for distinct reasons. The agent had sounded equally confident through all three conclusions.

Version drift was quieter. The agent used chart versions it knew from training data instead of checking the registry, and both the GitOps controller and log shipper deployed cleanly several releases behind current. Nothing failed to draw our attention to their age.

The generated Terraform also pinned the STACKIT provider to 0.80, where service-account key files looked unavoidable. Workload identity federation had arrived in 0.82, and Terraform could manage the trust from 0.95. Both versions predated our build. We found the gap when local applies moved to GitHub Actions, upgraded the provider and gave infra, test and prod federated identities. The old dependency had excluded a safer design from consideration.

It also invented plausible details. We saw variable names that sounded native to the repository and did not exist. Once, it drafted a commit message describing a step that had never run, turning an intention into an apparent fact.

During a prod outage, the agent added and removed a dummy range on a managed service’s access list in the hope that the provider gateway would reconfigure itself. Nothing changed. We eventually traced the failure to an egress policy pinned to the conventional port for a service the provider had placed on a high range. The guess had changed the control plane before we understood the traffic path.

Runbooks An Agent Can Run

Somebody follows a runbook under time pressure and without the author’s context. Any missing precision has to be supplied during the incident.

Our standard grew out of procedures we would use while tired. Every command block is self-contained and names its environment and stack. Commands, paths, metrics and thresholds must come from the repository or a read-only query. Verification includes the expected output. A destructive step identifies its point of no return and the data at risk.

# Illustrative verification step: self-contained, with a named healthy value
export KUBECONFIG=~/.kube/config/k8-test.yml
kubectl -n camunda get externalsecret app-connection-config
# READY is True within ~30s. If it stays False, the credential did not sync,
# and you are in step 4, not step 5.

An experienced operator may be able to rescue a step that says “check it worked” by inventing the test from memory. An agent may instead report success against a condition it never evaluated. We write the check down, including the healthy value, which makes the same runbook usable by a tired person and an agent under supervision.

We have not handed a runbook to an agent and walked away. The flat confidence in the last section is why.

What’s Next

The agent made us faster at producing changes. Review, written context and executable checks became a larger share of the work.

Supervision stays in the operating model for one reason I have not found a technical control for: an engineer will often hesitate when they are out of their depth, while the agent can sound exactly as confident when it is guessing. That is how the load-balancer investigation lasted two days.

Part 5 measures what the platform holds under load. We ran six tests and got three wrong answers before the numbers meant anything.

Part 2 ended with a right sized platform that could rebuild itself. That also included its open endpoints, broad permissions, badly placed replicas and unverified pull requests.

Right at the beginning we found that STACKIT limited which services could be private, the network gate depended on settings in two systems, Helm charts brought somebody else’s defaults, and Kubernetes treated a placement rule that looked strict as optional.

So we compiled a list of the most pressing issues and addressed them one after another. I tried not to get too technical in here, but it gets harder the deeper we dive into real platform engineering.

How Private Can You Get?

We wanted the data tier on private endpoints, with operators entering through one hardened jump host. A private endpoint gives a database no public address, so a stolen password is useless without a foothold inside the network. This is similar to the VPC and PrivateLink approach on AWS.

STACKIT’s catalogue ruled that out. PostgresFlex, OpenSearch and Secrets Manager use public endpoints protected by authentication and an ACL, an access control list of accepted source addresses. Object Storage has no address filter. Private SKE was behind a feature flag and its routing tables were in public preview. Basing the foundation on those features would add churn without making the managed data tier private. A jump host would add a single point of failure and another machine to patch.

We accepted public endpoints behind a short ACL. The team connects through a VPN with a fixed exit address, while CI uses the runner’s floating IP from Part 2; the cluster API and secret store admit both addresses, but the datastores admit cluster egress only, which keeps direct operator traffic out. I would choose private endpoints in a landing zone that offers them. STACKIT’s managed data tier did not, but that might change.

A Gate Split Across Two Systems

Prod has two active access policies: the Camunda UIs are VPN-only, while Zeebe gRPC also admits our fixed-address benchmark runner, which Part 5 introduces. Both enter through the same load balancer.

The external load balancer (STACKIT uses yawol, “Yet Another Working OpenStack Loadbalancer”) filters connections at L4, where it sees a source address and port. Its loadBalancerSourceRanges setting applies one list to the whole load balancer. To let the benchmark runner reach Zeebe, that list has to contain both the VPN and runner ranges. At L4, the runner then has a network path to every hostname behind the load balancer, including the UIs. OIDC would still protect the login, but the UIs would no longer be VPN-only.

The SKE internal Loadbalancer (we use nginx) sees the requested hostname at L7. Its default rule admits only the VPN, while the Zeebe ingress overrides that rule with the VPN and runner ranges. That keeps the UIs behind the VPN without blocking the benchmark runner from Zeebe.

We initially thought the L4 field was ignored. Before publishing that claim about a named provider, we retested with a throwaway load balancer: the excluded range blocked ten out of ten probes, and the caller’s address restored access. The current service enforces the field, so we use it as an outer gate.

The L7 rules depend on nginx seeing the original client address. The external load balancer therefore prepends it with PROXY protocol, while the internal load balancer is configured to parse that information and trust it only from the load-balancer network. We verified the chain in the live ingress logs, where nginx recorded the VPN exit address instead of the load balancer’s internal address.

# Illustrative: one outer list, then a narrower default at L7
service:
  loadBalancerSourceRanges:
    - <vpn-cidr>
    - <benchmark-runner-cidr>
  annotations:
    yawol.stackit.cloud/tcpProxyProtocol: "true"  # LB: prepend the real client IP
config:
  use-proxy-protocol: "true"                      # nginx: parse and trust the header
  proxy-real-ip-cidr: <proxy-cidr>                # nginx: only from this hop
  whitelist-source-range: <vpn-cidr>              # default: UIs stay VPN-only
---
# Annotation on the Zeebe gRPC Ingress only
metadata:
  annotations:
    nginx.ingress.kubernetes.io/whitelist-source-range: >-
      <vpn-cidr>,<benchmark-runner-cidr>

The coupling fails in opposite directions. Remove a PROXY-protocol setting while the allowlist stays narrow and every client gets blocked with a HTTP 403 Forbidden; lose the shared value file that carries the L7 allowlist and every UI becomes reachable from the internet while ArgoCD stays green, the pods stay healthy and login continues to work.

Policy tests cover both layers. They assert that the L4 list remains a superset of every active L7 rule, each hostname stays narrow, and all three PROXY-protocol settings are present. The VPN range lives in one file used by both environments, so it changes once.

For example, this shortened Conftest rule checks every per-host L7 allowlist against the L4 list on the rendered load-balancer Service:

deny contains msg if {
  service := lib.ingress_nginx_service
  l4_cidrs := {
    trim_space(cidr) |
    some cidr in service.spec.loadBalancerSourceRanges
  }

  some ingress in lib.docs
  ingress.kind == "Ingress"
  annotation := object.get(
    ingress.metadata.annotations,
    "nginx.ingress.kubernetes.io/whitelist-source-range",
    "",
  )
  some l7_cidr in {
    trim_space(cidr) |
    some cidr in split(annotation, ",")
  }
  l7_cidr != ""
  not l7_cidr in l4_cidrs

  msg := sprintf(
    "L7 admits %s on %s, but the L4 gate blocks it",
    [l7_cidr, ingress.metadata.name],
  )
}

The VPN range gates almost every human path into the platform. If its exit address changes, the UIs, kubectl, operator access to Zeebe gRPC and SSH to the runner reject at once. Direct operator access cannot reopen the ACL. Recovery follows a separate path: ArgoCD pulls from Git, so a commit restores UI access, and CI runs the applies that reopen the provider ACLs.

Connector Direction Decides Your Public Surface

Adding a public /inbound/* route would’ve made the setup more complicated so we postponed inbound support for now. A source filter would not have been enough. HMAC verification happens inside the connector runtime, after the request has crossed the load balancer and ingress and reached the pod, so unauthenticated traffic already consumes platform capacity. If inbound access returns, we will use polling where the external API supports it, place a relay in front of the cluster, or add an edge that enforces limits before traffic reaches the pod.

Outbound connectors remain in use. They leave through cluster NAT and call the HTTPS destination named in a process model, using credentials from the managed secret store. Because the Camunda namespace permits port 443 to 0.0.0.0/0, permission to deploy a model also carries permission to make the platform send payloads and connector credentials to an external host.

Restricting that egress is harder than writing SaaS names into a NetworkPolicy. SKE uses Calico OSS, which matches IP ranges and cannot express DNS names. A forward proxy can restrict hosts, but for HTTPS it sees the hostname in CONNECT, not the request path. Path filtering requires TLS interception, which puts every connector credential and payload through one decryption point and breaks certificate pinning. We recorded broad HTTPS egress as an open risk and deferred that design. A process team cannot choose either connector direction without changing the platform’s exposure.

Check The Defaults Of Every Chart You Install

Installing our own platform charts also meant owning their defaults. The monitoring stack and ingress controller arrived with cluster-wide read on secrets, and application images used floating tags rather than digests, which means a version’s content can change without you noticing. Those charts are packaged for many cluster shapes, so we had to review and narrow the defaults ourselves.

Grafana and Loki could read every Secret in the cluster, but they only need the monitoring namespace. We replaced their ClusterRoles with namespace-scoped Roles. We also limited the Prometheus Operator to monitoring and camunda.

ingress-nginx is different. It handles TLS for apps in several namespaces, so it must read their certificate Secrets. We kept this permission and restricted its network access.

Also, our Terraform modules now reject unsafe or invalid input at plan time. A caller range that is empty or wide open fails validation. The variable naming an environment accepts only the environments that exist (test or prod).

variable "allowed_caller_cidrs" {
  type = list(string)

  validation {
    condition     = length(var.allowed_caller_cidrs) > 0 && !contains(var.allowed_caller_cidrs, "0.0.0.0/0")
    error_message = "Name the VPN and CI egress ranges. Empty or 0.0.0.0/0 is not a valid answer."
  }
}

Every stateful resource carries destroy protection in Terraform: the databases, the search cluster, Object Storage, the secret store. No apply takes the data with it, whatever a refactor does to the resource addresses. The guard cannot take a variable and the same modules serve both environments, so tearing down test is a deliberate two-step. Destroy protection later blocked Terraform from replacing a managed service after it had been renamed outside our configuration; Part 6 covers that incident.

We test monitoring separately because it cannot report its own failure. A heartbeat fires constantly into the same alert channel as everything else, and silence is the failure signal. Moving the stack out of the cluster with STACKIT Observability would remove it from the cluster’s failure domain and reduce our maintenance work, at the price of a managed-service bill; we have not done it yet.

Terraform State And Camunda Backups Must Survive

Two kinds of Object Storage buckets hold what the platform cannot recreate. The state buckets store what Terraform knows about every environment, including cluster credentials. The backup buckets receive the nightly Camunda and OpenSearch backups, our recovery path when anything else is lost. The destroy protection from the last section guards this data against Terraform, but Terraform is not the only writer. CI uploads state and the clusters upload backups with their own S3 keys, and a leaked key talks straight to the storage API, with no plan and no review in between. The buckets have to hold against their own credentials.

Every bucket therefore got Object Lock and a default retention when it was created. Creation is the only moment STACKIT can set the lock, so either you build this control in from the start or you recreate the bucket later. Versioning comes with the lock and cannot be switched off. Locked versions stay restorable for their retention window, even against the credential that wrote them. State keeps seven days in GOVERNANCE mode, which an operator can still clear early: state rewrites on every apply, and undeletable versions would stall every migration. Prod backups keep thirty days in COMPLIANCE.

The difference between the two modes is who can bypass them. GOVERNANCE allows a credential with one specific bypass permission to delete locked versions or shorten their retention early; our bucket policy denies the writer exactly that permission. COMPLIANCE has no bypass: nobody deletes a locked version until the window ends, the project owner included, and retention can only be extended. The mode is stamped onto each version at write time, so changing a bucket’s default only affects future writes.

resource "stackit_objectstorage_compliance_lock" "project" {
  project_id = var.project_id
}

resource "stackit_objectstorage_bucket" "backups" {
  project_id  = var.project_id
  name        = "prod-backups"
  object_lock = true
  lifecycle { prevent_destroy = true }
}

resource "stackit_objectstorage_default_retention" "backups" {
  project_id  = var.project_id
  bucket_name = stackit_objectstorage_bucket.backups.name
  mode        = "COMPLIANCE"
  days        = 30
}

An Object Lock alone would not survive long, because the settings that protect a bucket are changed over the same S3 API that writes objects. Any credential the bucket policy admits could downgrade the retention or rewrite the policy itself. So each bucket splits its permissions into three credential groups.

1. The writer, the only key CI and the workloads hold, can read and write objects, and the policy denies it everything destructive: deleting versions, changing the policy, the lock, the lifecycle or the encryption.

2. A monitor credential that reads the five configuration values our checks compare and cannot even list the objects. It’s for verification only.

3. Admin rights belong to nobody permanently; when a bucket needs administrative work, we mint a key into the admin group, use it and delete it again, so there is no standing credential to steal.

Bucket policies, lifecycle rules and default encryption have no provider resource, so an idempotent script applies those three (manually). A weekly GitHub Actions job compares every bucket against a checked-in declaration: versioning, lock mode and days, policy, lifecycle, SSE-S3 encryption.

Replicas Need A Hard Placement Rule

The pod placement test produced the same kind of surprise. Prod runs three nodes, one per availability zone, and our two-replica components had weight-100 preferred pod anti-affinity to keep each pair apart. Connectors, Console and the Web Modeler REST API still landed both replicas on one node.

It happened during a reschedule when two nodes became Ready one second apart. The first node ended up carrying 78% of the cluster’s CPU requests, along with both replicas of the three affected deployments; losing it would remove authentication, every UI, Connectors, Web Modeler, GitOps control and secret sync in one go.

Kubernetes was doing what we had asked. preferred contributes a score alongside image locality and resource fit, and it never removes a node from consideration. Other scores won during the scheduling burst.

The ingress controller behaved differently because its rule filters nodes. It held a replica Pending until a second node was ready. Zeebe’s required hostname and zone anti-affinity kept one broker Pending for twenty minutes until an unused zone had a ready node.

A disruption budget (PDB) caps how many pods of a workload may go down at once during a planned action. It cannot protect a co-located pair from losing its node because Kubernetes is not evicting either pod. Nothing reports the mismatch.

Policy tests now reject every multi-replica prod workload without hard hostname anti-affinity. For a Helm-managed component, the relevant part of the values file looks like this:

connectors:
  replicas: 2
  affinity:
    podAntiAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
        - labelSelector:
            matchExpressions:
              - key: app.kubernetes.io/component
                operator: In
                values: [connectors]
          topologyKey: kubernetes.io/hostname

The Camunda chart exposes topologySpreadConstraints only for brokers. Other components silently ignore the key, so required anti-affinity is the common lever.

After the GitOps change, the loaded node dropped from 78% to ~50% of CPU requests. No pair protected by a disruption budget shared a node. CPU requests still differed across the three nodes.

A fourth node would not have changed the scheduling race. Kubernetes placed most pods on the first node that became ready, while the node that arrived twenty minutes later received only pods with volumes pinned to its zone; one node pool across three zones has no symmetric fourth member anyway, because four means a 2/1/1 split. Placement needed enforcement. Capacity was a separate question.

Terraform-owned Helm releases need their placement values in Terraform. ArgoCD combines hard per-component anti-affinity with a soft spread across the release:

locals {
  argocd_ha_values = {
    global = {
      affinity = { podAntiAffinity = "hard" }
      topologySpreadConstraints = [{
        maxSkew           = 1
        topologyKey       = "kubernetes.io/hostname"
        whenUnsatisfiable = "ScheduleAnyway"
        labelSelector = {
          matchLabels = { "app.kubernetes.io/part-of" = "argocd" }
        }
      }]
    }
  }
}

hard separates matching replicas. With ScheduleAnyway, Kubernetes tries to spread the whole release while still permitting every ArgoCD pod to run during a one-node window.

Enforced placement costs something: a replica can wait Pending while only one node is up, and a rolling update now needs a third schedulable node or it stalls with the old pods serving. Test keeps none of this: one node, one zone and one replica of each. Singletons use preferred mutual scatter instead. This covers Camunda’s Identity, Optimize and Web Modeler WebSockets, GitOps-owned cert-manager and Terraform-owned External Secrets. A hard rule would add no availability and could strand a pod.

The scheduler evaluates these constraints when it places a pod. It does not move existing pods later. A descheduler could address the remaining concentration, but it would evict running pods, so we deferred it.

The Checks A Change Has To Survive

The allowlist and the placement rule had both looked correct in configuration. We changed the pull-request pipeline to test what the rendered configuration would do. Part 2 explains how a change ships; these are the checks it passes first.

Eight jobs run in parallel for every infrastructure pull request:

• terraform fmt checks canonical formatting across every Terraform file.

• terraform validate checks syntax and provider schemas without state access.

• TFLint flags Terraform-specific errors and recommended-rule violations.

• tfsec scans Terraform for security misconfigurations.

• Checkov applies policy-as-code checks to Terraform.

• terraform test runs module input-validation tests with mocked providers.

• terraform-docs generates README files for each module.

• Gitleaks scans the repository on every pull request.

The GitOps repo:

• Renders Helm charts with helm template

• Builds the overlays with kubectl kustomize

• Checks the combined test and production output with Conftest policies written in Rego.

• Gitleaks scans the repository on every pull request.

For example, this Conftest rule fails when either default-deny policy is missing from a guarded namespace:

default_deny_namespaces := {
  "camunda", "argocd", "external-secrets-system", "secrets-manager", "ingress-nginx", "cert-manager"
}

deny contains msg if {
	some ns in default_deny_namespaces
	some np in {"default-deny-ingress", "default-deny-egress"}
	not lib.has_ns("NetworkPolicy", ns, np)
	msg := sprintf("[%s][network-policies] %s NetworkPolicy %q missing", [lib.release, ns, np])
}

Neither tfsec nor Checkov has native STACKIT rules. The S3-compatible backend triggers AWS-shaped checks that do not apply or cannot be configured through the provider. We document the suppressions. STACKIT constraints belong in Terraform tests or custom policies. Versioning runs separately on the data plane.

None of these jobs receives cloud credentials. A scanner that runs on every pull request has no reason to sit inside the apply identity’s blast radius.

Runtime secrets live in the managed secret store, CI credentials stay in the pipeline’s store, and Git holds non-secret configuration. Gitleaks checks both repositories without a path filter because a credential can land anywhere; local rules cover signed webhook URLs that the defaults miss.

What’s Next

This part showed how we applied hardening over the whole platform. Restricting access to all of our components, adding pipelines that verify every change, deliberate pod placement to even the load, and chart defaults that make sense for us.

Part 4 explains how we use AI as support to build this platform, where its strengths are, and especially its weaknesses.

Part 1 explained how we sized separate environments for test and prod, integrated Azure Entra as our identity provider, and successfully deployed them with Terraform. Users could log in, model processes and complete instances. Nobody could rebuild or recover them, though, and that set the goal for the next phase: building a real platform.

What that actually means is that Camunda config changes shouldn’t touch infrastructure, Terraform shouldn’t be executed from my local machine, backup mechanisms need to be in place and verified, and disks shouldn’t fill up without anyone noticing. I’ve already built similar groundwork on an AWS platform project for a logistics company, which was a great plus.

Terraform For Infrastructure, GitOps For Apps

We decided to split the code. A database resize should not share a release rhythm with a Camunda values change, so Terraform owns the STACKIT projects, SKE, PostgreSQL, OpenSearch, Object Storage and the secret store; ArgoCD owns ingress, certificates, monitoring and Camunda from the GitOps repository. Terraform pushes. ArgoCD pulls.

Infrastructure changes less frequent and behind a heavy review gate; application config changes all the time. Terraform’s identity never edits a running workload like Camunda, and ArgoCD’s Git token never touches Terraform state. Terraform hands ArgoCD one repository credential and a root Application, then leaves the application layer to reconcile itself.

Each environment runs its own ArgoCD syncing an app-of-apps: one root Application that owns the child Applications for ingress, cert-manager, monitoring, Camunda, and the rest, ordered by sync waves so the secret operator is up before anything that needs a secret.

The two environments deliberately track different things. Test follows the main branch, so a merge lands there and you find out quickly whether it works. Prod follows a promotion tag. Pointing both clusters at one branch would be a tempting simplification, but it means every merge would be a prod deploy.

During a necessary manual operation, we learned how literal ArgoCD ownership is. I scaled a StatefulSet to zero, and it returned within seconds. I disabled self-heal on the child Application and tried again, but the parent restored the child’s sync policy from Git. For a manual operation, we now pause reconciliation along the entire ownership chain, including the parent.

# Illustrative app-of-apps child: ordered by sync wave, self-healed
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: cert-manager
  annotations:
    argocd.argoproj.io/sync-wave: "1"   # the secret operator is wave 0
spec:
  syncPolicy:
    automated:
      selfHeal: true

A second ArgoCD problem was easier to overlook. Kubernetes filled fields inside the StatefulSet volume claim templates that Git did not contain, so the Application stayed out of sync in ArgoCD while self-heal kept trying to resolve the same difference. We added a narrow ignoreDifferences rule to the ArgoCD Application:

spec:
  ignoreDifferences:
    - group: apps
      kind: StatefulSet
      jqPathExpressions:
        - .spec.volumeClaimTemplates[]?.status
        - .spec.volumeClaimTemplates[]?.spec.volumeMode
        - .spec.volumeClaimTemplates[]?.metadata.annotations

We ignored only the fields Kubernetes generated. Ignoring the complete volumeClaimTemplates list would also hide changes we want ArgoCD to report, including a new PVC size. We used the same approach for strategy fields defaulted by External Secrets.

One Apply, Two Clouds, No Stored Keys

One deploy identity across infra, test and prod would have undermined the repository split. STACKIT has no AssumeRole-style impersonation like e.g. in AWS. We started with local Terraform applies, where STACKIT authenticated through service-account key files.

When we moved the applies into GitHub Actions, carrying that approach forward would have meant copying a STACKIT key into the infra, test and prod GitHub environments. We chose federated identities for CI instead. Each STACKIT service account and Entra CI application now trusts only its matching GitHub environment and reaches only its assigned resources.

This split does not mean one apply stays inside one cloud. The shared-services stack creates STACKIT infrastructure and Entra app registrations, while the environment stacks provision STACKIT and manage Camunda’s Entra credentials. Both providers authenticate during the same run.

The job may request OIDC tokens. Each provider requests a separate GitHub token because the audiences differ: Azure expects api://AzureADTokenExchange, while STACKIT expects sts.accounts.stackit.cloud. Entra and STACKIT verify the issuer and a subject containing the repository and GitHub environment, then issue short-lived access tokens.

We considered using Azure Entra as the single broker to remove one trust chain. Entra cannot mint a token with the arbitrary audience STACKIT requires, and obtaining a suitable Entra application token another way puts a stored Azure credential back into the job. That buys an extra exchange and merely moves the long-lived secret. Two direct trust chains keep provider login keyless, and their subject checks mean a test job cannot satisfy the prod trust even when someone triggers it manually.

We use access control lists (ACLs) to give access to STACKIT resources. I’ll get into the details in Part 3. Those ACLs added a requirement: a fixed source address. Hosted GitHub runners use changing address ranges, so Terraform now runs on a small self-hosted VM in the shared project. A floating public IP survives VM replacement and the allowlist contains the runner address for automation.

That runner is privileged. terraform plan executes provider code from the branch, so an untrusted pull request could run code from an allowlisted host with live OIDC access. Pull requests receive static analysis on GitHub-hosted runners. Plans and applies use the self-hosted runner only from trusted refs, and applies follow a merge to main.

State lives in a third, shared project. Its buckets sit outside the stacks CI applies, have versioning enabled and are checked for access-policy drift. If a pipeline can alter prod state, it can steer the next prod apply. State therefore needs its own boundary.

Blue bootstrap creates yellow runner, identity and state resources. That runner provisions green infrastructure, where ArgoCD reconciles red applications, volume claims and policies. Purple marks differences between test and prod.

You Build It, You Run It

A platform isn’t done after it’s deployed. Someone has to operate it and make sure it runs. A platform team e.g. needs insights into the health of the cluster, and ways to analyze and act on errors.

For that we added monitoring to prod with the kube-prometheus-stack. It contains Kubernetes manifests, default Grafana dashboards, and Prometheus rules combined with documentation and scripts to provide easy to operate end-to-end Kubernetes cluster monitoring with Prometheus using the Prometheus Operator. We also added some dashboards from Camunda, showing live metrics for Zeebe & Co.

We’re aware that operating our monitoring stack inside the same cluster it monitors is a risk. This is a low priority open issue right now while building the platform blueprint. We might switch to managed STACKIT Observability later on.

Camunda’s secondary storage, the searchable copy behind Operate, Tasklist, and Optimize, runs on managed OpenSearch in prod and on the relational database in test. STACKIT’s managed OpenSearch tops out at version 2.x, and the application credential lacks index-lifecycle permissions. Native retention is therefore off, with a cronjob pruning old indices instead. This prevents the disk from filling up in the middle of the night and paging people out of bed. We opened a service request with STACKIT to address this issue in the future.

Backups made the difference between rebuilding software and recovering a platform. A nightly job writes Camunda’s state to object storage, and the return path lives in a runbook. The same happens for OpenSearch, where we write regular backups to a snapshot repository in the object storage. Both the state and backup buckets are created with Object Lock, versioning and default encryption, and a weekly job verifies those settings; Part 3 covers that as well.

Why “As Code” Is Harder Than It Looks

Almost everything was now “as code.” A few manual boundaries remained.

ArgoCD can reconcile the platform only after ArgoCD exists. It needs a cluster, and Terraform needs an identity before it can create one. One hand-executed step remains at the bottom of the stack: an operator uses a short-lived key for the initial apply, revokes it and lets the keyless path take over.

We revoked the bootstrap key and initially left its identity holding owner rights on all three projects. The identity now has no active credential between uses. Removing a credential does not remove the permissions attached to its identity.

Some values don’t exist when the code is written. Terraform first has to create the managed database, password and search cluster before it can write their coordinates into the secret store; External Secrets then syncs them into Kubernetes for the application. That secret-store entry is the handoff between infrastructure and applications.

Timing crosses the repository boundary as well. Adding a database starts in Terraform, continues in the secret store and finishes when GitOps adds the sync and application configuration.

Review is the last boundary. Git reproduces a bad configuration as reliably as a good one, and ArgoCD keeps either version synchronized, so our policies inspect the rendered result and assert the details whose absence leaves everything green: the access gate, sync policies and chart versions.

What’s Next

Our first platform iteration is now rolled out and rebuilt by pipelines. Infrastructure and apps are structurally separated and short-lived tokens are minted for each run. The endpoints were open and the network was flat. Part 3 deals with that.

In the opening article I explained the reasons why I put Camunda on a European cloud. We still had to figure out what running it on STACKIT means.

Luc Weinbrecht and Sebastian Roseneck of envite had already given us a good starting point at CamundaCon 2026 in Amsterdam. Their deployment brought up Camunda 8.8 in a STACKIT environment and proved that the pieces fit together.

We used their component map as base and moved on to the decisions a platform team would need to make: an existing identity provider, separate environments, and how to operate it.

Where The POC Left Off

STACKIT is not one of Camunda's supported cloud providers, so there was no reference architecture where changing some variables produced a deployment. Sebastian and Luc used Camunda's AWS single-region architecture, preserved its shape and implemented each component with STACKIT modules.

Some mappings were direct. Object Storage is based on S3, PostgresFlex is Postgres, and a Gardener extension handles DNS. SKE hands you a bare cluster, which meant ingress, secret sync and TLS certificate automation were missing out of the box.

Identity Without Keycloak

Consid’s Microsoft Entra tenant was already our authority for users and groups, and because Camunda 8.9 lets each component connect via OIDC (OpenID Connect) to Entra, we could remove the Keycloak broker and database that the POC used. This already spared us some moving parts in the middle to operate.

Camunda’s OIDC setup also supports providers like Okta or Auth0. An existing Keycloak remains a good option when it already brokers SAML or Active Directory, or when other applications depend on its claim and group mappings.

# Camunda global identity: straight to Entra
global:
  identity:
    auth:
      type: MICROSOFT
      issuerBackendUrl: https://login.microsoftonline.com/<tenant-id>/v2.0
      # each web component gets its own Entra app registration

Direct OIDC still needs careful wiring. Each web component needs its own app registration and one helm value pointing to the wring registration sent Web Modeler into a login loop. After fixing that, Camunda Admin denied everybody because our rule compared the group claim for equality even though Entra sends a list. The match had to use contains.

We tested the whole path after that, starting with membership in an Entra group and ending at the authorization Camunda receives. Only members of the mapped group could enter.

Two Environments Sized For The Job

Once login worked, we could have cloned the environment and renamed it prod. Instead, we wrote down the job of each one. Test supports developers and validates processes at a lower cost. Prod carries the full product surface and must survive a zone failure, so copying the infrastructure back 1:1 would pay for availability that test doesn’t need.

On STACKIT, each environment is a project. The project is the main boundary for resource ownership, quotas and access control. For the first local applies, I created a project-scoped service account per environment and authenticated Terraform with a service-account key on my laptop.

STACKIT has no AWS AssumeRole-style impersonation. When we later moved the applies to GitHub Actions, Identity Federation let each job exchange its GitHub OIDC token for a short-lived STACKIT access token. Part 2 explains that transition.

Camunda’s Architecture

Camunda distributes process instances across so called partitions. Each instance stays on one partition for its entire lifetime. Every partition has one leader that processes commands and followers that copy its event log. Similar to Kafka. With a replication factor of three, the partition exists on three brokers: one leader and two followers. If the leader fails, a follower can take over. If the partition loses its majority of brokers, processing stops until quorum is restored. When setting up Camunda, one has to decide how many brokers, partitions and replication the platform needs. They affect parallelism and fail safety.

Cloud providers normally offer availability zones (AZs) in each region. Each AZ represents a physical different server, rack or even data center. In a Kubernetes cluster we can place nodes of the same cluster in different AZs. This prevents a full outage in a cluster if one AZ goes down. We also make use of the AZs to ensure high availability.

Our Final Environments

Test is deliberately lean: A single broker setup (1 broker on 1 node, in 1 AZ running 1 partition). Camunda 8.9 uses PostgreSQL for secondary storage there, which removes managed OpenSearch, Optimize and the monitoring stack from that environment. It runs in one zone without data-store replicas. Developers still get a shared Camunda installation, while avoiding paying for a miniature copy of every prod service.

Prod carries the full load: A triple broker setup (3 brokers on 3 nodes, in 3 AZs running 3 partitions, replicated over all 3 brokers), OpenSearch for secondary storage, Optimize for process analytics, and monitoring with Prometheus. Three availability zones and replicated data stores keep the platform running when one zone fails.

Both environments run the same node type, a g2i.8 (general purpose, 8 vCPU Intel Icelake, 32 GiB RAM). Everything else differs by design: secondary storage, zones, node count, Optimize and monitoring. STACKIT adds new options from time to time, as of writing this, there is already a newer generation g3i.8 (Intel Emerald Rapid) available which is slightly more expensive, but we resisted the temptation to upgrade.

Storage was the less visible difference. STACKIT defaults block volumes to 500 IOPS (input/output operations per second); the AWS reference assumes a 3.000-IOPS gp3 disk, while Zeebe expects fast SSD storage with 1.000 IOPS as the floor. Reusing the reference volume sizes without checking the storage class would have left the write-ahead log below that minimum, and Kubernetes would have reported nothing wrong.

We assigned IOPS based on workload: 2.000 for Zeebe’s log and snapshots, 1.000 for PostgreSQL, and 500 for node root disks that carry the OS and logs. Test and prod use the same tiers so their behavior stays comparable; only the volume sizes change.

The Helm edit took only minutes. Deciding what each value had to achieve took most of the time. When we finished, Entra logins worked and process instances completed in test and prod. That was enough to call it a real deployment.

We added a small landing page so nobody had to keep a list of application URLs and client endpoints. It brings Operate, Tasklist, Web Modeler, Console, Admin, Management Identity, Optimize (prod), Grafana (prod) and ArgoCD together in one place. Entra and the authorization rules behind each application still decide what a user can open.

What We Needed To Address Next

My laptop still held credentials. Neither environment had a recovery path, the network trusted far too much, and the sizing rested on arithmetic rather than a load test.

We tackle the operational gaps first. Part 2 turns our setup into a real platform, then Part 3 narrows the network trust. Part 4 looks at what the coding agent contributed and where it failed. In Part 5 we put the platform under load, and Part 6 closes with the operating model, the bill and the verdict.

I don’t follow American politics regularly, the few important things are reported in German news, and my Instagram or LinkedIn algorithm does the rest. The topic of cloud sovereignty prevails in the IT world for several years now and one thing caught my attention, even if it might not sound significant on the first read.

On June 29th, 2026, the US Supreme Court ruled 6 to 3 in Trump v. Slaughter that the president can remove commissioners of the FTC (Federal Trade Commission) at will. The decision overturned a restraint on presidential power that had held since 1935.

The European Union has relied on the FTC when assessing whether US data protection is adequate. NOYB (None of your Business, an Austrian data-privacy advocacy group) has since asked the European Commission to withdraw the adequacy decision; its letter points out that the EU-US Data Privacy Framework refers to the FTC 259 times and treats it as an independent supervisory authority. The framework is still valid. Whether it stays that way is a decision European companies do not control.

There was already another concern regarding the sovereignty of data. Under the US CLOUD Act, American authorities can force a US company to handover data it controls no matter where the data is stored. Frankfurt tells you where the disks are. Company ownership tells you which court can reach them.

Architects, developers and transformation leads usually don’t decide on long-term cloud strategy, but we are still asked whether the resulting architecture is safe to depend on. Our part is to translate that legal uncertainty into operational consequences so leadership can make an explicit decision.

What Sovereignty Actually Means

Many arguments regarding sovereignty end at the datacenter: the infrastructure is in Frankfurt (a.k.a. eu-central-1), the ops team of the managed services sits in Europe. Case closed. That answer covers location and operations. It says nothing about the law that can reach the company behind the service.

The discussion should be split into three questions:

An EU region and an EU operating company answer the first two. But if the parent entity is American, the CLOUD Act can still reach it. European law still applies, but it doesn’t exclude the jurisdiction of the ownership.

For an ops team, this means two concrete things: a government policy can change access to a service, and US authorities can force the provider to hand over data.

Process Automation Is The Wrong Place To Accept That Risk

The type of work changes the risk level of that provider dependency. If Camunda coordinates your most critical business processes like orders, claims, or payments, losing control over the platform means losing the assets that generate revenue. If you’re dealing with sensitive data, you need to ensure privacy and integrity. Who owns the provider belongs in the business-continuity plan, even though it sits outside the architecture diagram.

Bernd Rücker (Camunda co-founder) has always told developers to only keep data in the engine that drive a process forward, and I totally agree. Customer records should stay in the systems that own them. This has changed over the last years, when Camunda has grown into a platform that can also handle documents, so there is more at stake than a set of process variables. A CLOUD Act request can reach that data through the provider, with no European court in a position to stop it.

Camunda on STACKIT

Camunda 8 can be self-managed since from the beginning. You can place your platform and its process data with a provider that meets all three levels of sovereignty from above. In our case, identity and code hosting remain separate choices (Consid currently uses GitHub and Azure Entra, both of which are owned by Microsoft), and each organization has to decide how far the requirement should go.

We checked four things:

☑️ An entity under EU law

☑️ No foreign parent

☑️ European-owned infrastructure

☑️ No hidden US dependency underneath

STACKIT met all of them. Schwarz Digits (part of the Schwarz Group) first built the cloud for their own retail companies, including Kaufland and Lidl. They then offered it outside the group. That operating history mattered more to me than a long feature catalogue. STACKIT’s catalogue is smaller than AWS’s and already included the services required for this platform.

For transparency reasons: I work for Consid, which offers the Camunda-on-STACKIT architecture described here. Consid partners with Camunda and Schwarz Digits, and I’m a Camunda Champion. Nobody pays me or provides anything in return for this series. I’m sharing our work on my blog, because I want the lessons to be available for everyone.

Next Up

This is the beginning of a seven part series. The following articles show what we did: the first Camunda deployment, the platform work around it, hardening, our use of a coding agent, putting load on the platform and the resulting costs of STACKIT.

Cloud sovereignty is risk management. The events of 2026 made the dependency concrete enough to require an explicit decision, including the cost and operating consequences on either side.

The Day One keynote announced ProcessOS. The Day Two keynote actually ran it. Bernd Rücker, Co-founder and Chief Technologist, and Bastian Körber, VP Product Management, took a process from a one-paragraph brief to a deployed and running instance in under thirty minutes. They were upfront that this was lab conditions, not a production rollout. I think that honesty mattered, because the thirty-minute number is not the real point of the session.

The more hands-on half of the keynote was what sits underneath the demo. ProcessOS is not just “AI writes a BPMN file”. It is a pipeline that goes from intent to executable artifacts, plus a stack of capabilities that Camunda is building so the output actually holds up in production. The framing was direct. Generation is the easy half. Trust is the half most teams underestimate.

This article walks both halves. The demo first, because the phase model is the easiest way to understand how ProcessOS thinks. Then the Four Pillars of Trust and the roadmap to Camunda 8.10 in October 2026, where the trust half lives. If you want the announcement framing, read the Day One write-up first. This one assumes you have it.

From Brief To Running Process

The interesting part of the demo was not the domain. It was the sequence of phases ProcessOS runs through.

There are several phases, and each one has a clear input, an agent step, and a user review:

• Brief and goals. A plain-English brief, picked KPIs (cycle time, manual effort, quality, compliance, cost, throughput), free-text constraints, and a single sentence describing the desired outcome. The brief on stage was short, a few lines pointing at the company wiki for policy and naming the backend services the process should wire up to.

• Discovery. The Discovery agent reads the wiki, OpenAPI specs, policies, and runbooks that are in scope. It builds a Markdown decomposition of the as-is process and a draft BPMN. A live event stream showed the discovery phase running through to phase complete.

• Calibrate as-is. ProcessOS does not pretend its findings are ground truth. It surfaces open questions, risks, and confirmations against specific BPMN elements, and asks you to answer per element to lock down the as-is.

• Design and review to-be. The agent proposes the redesign with concrete before-and-after items per step, each tied back to a KPI you picked. One step in the demo replaced a gut-feel decision with a DMN business rule producing a routing class. The BPMN diff that followed showed 47 additions, 28 removals, and 3 changes. Feedback can be left on individual steps and goes back to the agent.

• Generate artifacts. The Build agent produces real Camunda 8 artifacts, not pseudocode. Four artifact types: BPMN, DMN, forms, and Java workers, sitting as files in the project repository, ready to compile.

The last phase was deploy and test. ProcessOS lints the artifacts, deploys all the BPMN, DMN, and form files to a Camunda 8 cluster, generates synthetic test variables, and starts one process instance via the Zeebe REST API. About thirty seconds. Bernd then switched to Operate and walked a live instance through its happy path. If you skip the deploy step, the artifacts stay in the project folder and you can run the generated worker scaffold yourself.

To make the openness point, Bernd also opened a coding agent in the same project folder and asked it to extend the generated Java application with a small web form. Same repo, same artifacts, different agent. The point is that ProcessOS is not a closed surface. The output is a normal Camunda 8 project that any coding agent or human can pick up from.

Why Speed Alone Is Not The Story

Thirty minutes from brief to running instance is a strong headline. It is also the kind of headline that gets teams in trouble. I have seen enough demos that looked good in the room and stalled in production to be careful here, and the speakers were careful too. They put it directly: “Trust is structural, not aspirational. Governance lives in the engine, the diagram, and the deploy path. Not in a policy doc.”

The strategic framing on the slide stacked three goals:

• Build solutions quicker. That is what ProcessOS is for.

• Build solutions that hold. That is what agentic orchestration plus the Four Pillars of Trust is for.

• Scale adoption. That only works if the first two are real.

The middle goal is where most of the engineering investment is going. Generating BPMN was never going to be the hard part. Operating, evolving, and trusting agent-driven processes in production is.

The Four Pillars Of Trust

This is the framework Bernd walked through on stage. The pillars sit on top of agentic orchestration as the foundation. That foundation gives you durable state, governance, auditability, and operational control. The pillars cover what comes after a process is in production, when you have to operate it, debug it, and evolve it. Each pillar in his slide was split into what is shipping today and what is coming with 8.10.

Testing. Non-determinism breaks the usual test pyramid. You cannot assert on exact strings when the LLM is in the loop.

• Today: mock the LLM with Camunda Process Test, when(agent calls tool X).then(complete with Y). Assert output quality in natural language using an LLM as judge.

• 8.10: Test Studio for no-code acceptance tests written by business analysts.

Analytics. Problems at scale are invisible without dedicated signals. Aggregate process metrics are not enough once an agent is making decisions inside the flow.

• Today: bottleneck detection across the fleet in Optimize, plus process monitoring and incident tracking in Operate.

• 8.10: full agent visibility per instance, with conversation history, system prompt, tool calls, and token usage. An Agentic Control Plane for fleet-wide averages and outliers across tokens, durations, failure rates, and tool call distribution.

Instance Operations. Production problems need surgical fixes, not restarts. If your only remediation is to redeploy, you will keep redeploying.

• Today: see the incident and fix it in place. Retry, modify, migrate. Every action goes to the audit trail. Drill into individual agent runs to see the complete memory, including every tool call and reasoning step.

• 8.10: waiting state visibility, variable trace, and instance priority. Know what the process is waiting for without guessing.

Evolution. Everything changes. Models, prompts, business rules. The platform needs to make change cheap and traceable.

• Today: every change versioned, every instance traceable, running cases stay on their version, and you know which model ran when. Promote a proven agent pattern to a deterministic step without rewriting the process.

• 8.10: swap model or credentials without touching the process via environment handling and a connection manager. Optimize from production data, with token trends, tool frequency, and prompt versioning.

Read across the pillars and the pattern is consistent. The “today” rows already give you a working production setup. The “8.10” rows close the visibility and operational gaps that agentic systems specifically introduce.

What To Expect In The Next Months

The roadmap puts the trust half on a calendar. The slide split it into three layers, each on a different clock:

• The Foundation. Camunda 8 as it is today. The event-streaming engine, the SDKs, BPMN as an open standard, Process Test for self-validating agents, the Docs MCP server. Most readers are running on it already.

• The Connective Tissue. Arrives with 8.10 in October 2026. One Camunda Hub workspace, AI Coding Assistant reaches full release, Test Studio, Agentic Control Plane, and processes exposed as MCP tools. This is where the Four Pillars deliverables show up.

• The Compounding Loop. Beyond 8.10. Platform-generated Solution Packs, agents that tune from production signals, and overnight experiments that surface improvements with a diff. The long bet and the one with the most execution risk.

There is also a people side. ProcessOS is being positioned as a transformation offering, with forward-deployed engineers running discovery engagements at the customer site. Design partners are open now. If you are in a real re-engineering program, that is the program to ask about.

Looking Ahead

The Day Two keynote did three things well. It showed the pipeline end to end on something that looked like a real process. It admitted the demo was lab conditions and pointed at what is harder than generation. And it tied the roadmap to a clear date, October 2026 with 8.10, instead of a vague future.

Key takeaways

• The demo phase model is the right mental shortcut for ProcessOS. Brief, Discovery, Calibrate, Design, Artifacts, Deploy and test. Each phase is reviewable.

• The artifacts are normal Camunda 8 artifacts. BPMN, DMN, forms, and Java workers in a project repo. Coding agents and humans can pick it up from there.

• Bernd’s Four Pillars of Trust frame where the production story lives. Roughly half of each pillar is available today. The other half ships with 8.10.

• The roadmap names three paces. Foundation today, Connective Tissue in October 2026, Compounding Loop beyond. Plan for the right one for your team.

• ProcessOS is also a transformation offering. The design partner program is the way in if you have a real re-engineering program lined up.

What I will be watching over the next months: the closed beta opening up, the 8.10 release notes for the Four Pillars deliverables, and the first platform-generated Solution Pack reaching full release. If you saw the keynote and want to compare notes, reach out. I will follow up once 8.10 is out.

Related Articles

Today at CamundaCon in Amsterdam, Camunda announced ProcessOS. The framing is bold: an operating system for business processes. It will be released with Camunda 8.10. If you have followed Camunda over the last year, this is the next big step after the push into agentic orchestration.

The short version is that ProcessOS is an intelligence layer that sits inside the Camunda Hub. It uses skills and agents to help you discover, design, build, and optimize processes. It works on your company data and gets better over time. The current AI coding agent is Claude Code, but the design is open and not locked to one model or vendor.

Camunda’s framing for this is direct: Every process in your enterprise is legacy, because it was designed for a world where AI did not exist. They call the work of redesigning those old processes into AI-native ones The Great Re-engineering. ProcessOS is meant to be the layer where that work happens. That framing is the main thread of this article, not the full feature list.

What ProcessOS Actually Is

ProcessOS is positioned as the intelligence layer of the Camunda Hub. The Hub also covers Web Modeler, Workspaces, Catalog, Console, the Business Value Dashboard, and the Agentic Control Plane. ProcessOS pulls these surfaces together with a loop of four activities: Discover, Design, Build, Optimize:

The inputs are intent, expressed as desired process improvements. The outputs are running, re-engineered, AI-native processes. In between, ProcessOS draws on two things. The first is organizational memory: knowledge, the catalog of connectors and building blocks, and policies. The second is signals: production data, live systems, and intelligence. That mix is what allows it to learn over time and stay grounded in the real company context, not just in a generic model.

For the people doing the work, there are three ways to use it. You can let agents act directly in an autonomous, agentic way. You can work with agent skills through a coding agent like Claude Code, with a human in the loop. Or you can edit manually in the Web Modeler with Camunda Copilot. Same skills, different levels of automation.

Why It Matters For Companies

Agentic orchestration was the first step. Camunda made it possible to use AI inside a running process, with the platform handling state, retries, and human escalation. ProcessOS is the next step. It uses AI before the process is even doing something, in the work of building the process itself.

For companies, the practical promise is speed. Prototyping and shipping production grade processes can move from months to weeks. That changes the economics of process work. It also changes who can take part. If an analyst can describe intent and get a working draft to review, the bottleneck is no longer only in the modeling tool. The review is still real and the governance is still real, but the first version arrives much faster.

The other point is strategic. Everyone is doing AI right now. What Camunda is offering with ProcessOS is a path where AI is not a feature bolted on the side, but a layer that knows your processes, your data, and your policies. That is a different conversation than just calling a model from a service task. It is the actual work of The Great Re-engineering: turning legacy enterprise processes, designed without AI in mind, into AI-native ones that fit how work gets done now.

From The Keynote

The announcement came from two speakers. Jakob Freund, Camunda’s CEO, opened with a direct question: are you truly ready for AI? His framing was that AI bolted on top of existing processes hits an automation ceiling. The early gains show up, but then coordination breaks down and the costs of chaos rise. The way through is to re-engineer the processes themselves, not to add AI to processes that were never designed for it. He also reminded the audience that agentic orchestration, announced six months ago in New York, is now live and battle proven at scale, with 16 customers already presenting at the event.

Daniel Meyer, Camunda’s CTO, then walked through what ProcessOS does in practice. Camunda used its own Quote to Cash process as the pilot. That process touches about 30 people across 6 divisions, 17 roles, 12 systems, and 14 document artifacts. Every euro of Camunda’s revenue runs through it. The Discovery Agent queried the real systems and produced markdown documentation in under an hour. About 66% of the input came from systems, the rest from short questionnaires for the people involved. A 3 hour stakeholder review followed.

The Design Agent then suggested a radical transformation. Collapse 20 to 79 touchpoints down to 2 in the happy path, with Slack as the single surface for employees and a new Deal Agent coordinating the other agents behind it. Guardrails and approvals stay enforced through BPMN where they are needed. The Build Agent generated the BPMN, DMN and other artifacts from that design, with engineers handling the remaining 20%. Four weeks of calendar time replaced the 9 to 12 months Camunda estimates for the traditional approach. The outcome: cycle time dropped from 115 days to 80, error rate from 10% to about 2%, and around 6000 person-hours freed per year, or about 5 hours saved per deal. The wider pilot, enablement and rollout are still pending, but the demo was meant to make the speed concrete, not just promised.

What’s Next

ProcessOS is in closed beta at the announcement and will arrive with Camunda 8.10. The parts to watch are how the Quote to Cash rollout goes beyond the demo, when the closed beta opens up, and how the skills and tools extend to coding agents other than Claude Code. I will follow up with a closer look once the release is out and the first hands-on examples are available.

If you want the official framing, watch for the keynote recording and the ProcessOS announcement page from Camunda after the event.

The first agentic essentials article was about how agentic orchestration works. This one is about when to use agentic orchestration at all. That is a different question, and I think it matters more. Teams often reach for AI before they have confirmed it fits the problem they actually have.

In this article I compare three problem shapes in the same baggage-support world: a fixed workflow, a rule-based decision, and a messy inbound email that neither of those can handle cleanly. The rule-based part uses DMN, short for Decision Model and Notation, which is a standard way to write business decisions as explicit rule tables. The goal is one honest threshold for when agentic orchestration is the right call and when it is not. I will continue to use the missing-baggage example here.

Three Different Problem Shapes

All three examples below stay in the same baggage-support world. What changes is how much the system already knows when the case arrives, and how hard the next step is to pre-model.

When A Normal Workflow Is Enough

Imagine the passenger uses the portal. They pick “missing baggage”, enter a booking number and bag tag, and confirm their contact details. From there, the process follows the same order every time: query for the bag, contact the last known airport, provide a voucher and then update the passenger.

The steps are known before the case starts. I would not add AI here. Every model call adds latency and cost. On a step with no judgment in it, that overhead buys you nothing.

When DMN Is The Better Fit

Now look at the next action decision. A policy might allow different voucher amounts based on route length or delay. That is a business rule, not an open-ended judgment call.

This is where DMN fits better than AI. You put the policy into a decision table. The engine evaluates the inputs and returns a result. You can also explain later exactly why the outcome was accepted or rejected.

Every rule is visible. Every outcome is traceable. No model call needed. If you want to go deeper on DMN, I covered it in a separate article.

When Agentic Orchestration Makes Sense

Now take the messy version. The passenger sends a plain email. Their bag didn’t arrive, they have moved to a different hotel, and they only have an old photo of the tag.

The first useful step is not obvious. The system might need to check whether a claim already exists, ask only for the missing hotel address, and then continue the search. That is a much better experience than a rigid form that asks for everything upfront or a decision table that refuses to run without clean input. That is where agentic orchestration starts to make sense. The AI works within guardrails and has access to a small set of approved tools, but it decides which tool to use first based on reading the message.

But messy intake is only one case. The same reasoning applies mid-process. A process engine follows the path it was given. A rule table matches conditions to outputs. Neither one reasons. If the passenger’s next message mentions that their medication was in the bag, a fixed flow has no branch for that. A DMN table has no row for it either. An agentic step can read the message, understand what changed, and act on it, for example by sending a care voucher or escalating the priority, without a developer having anticipated every situation the passenger might bring up.

The Cost Of AI

This part often gets skipped in demos. AI adds real overhead:

• Every model call costs money.

• Responses are slower than a rule table.

• The behavior needs evaluation on real, messy inputs.

• Operations teams get one more thing to monitor.

If there are only a few known next steps, that overhead is usually not worth it. A workflow or a decision table will be easier to run and easier to explain.

Start From The Problem, Not The Tool

Do not start from “where can I add AI?” Start from “where does structured modeling stop being enough?” For most processes, a workflow or a rule table is already the right answer.

What’s Next

• Many processes just don’t need AI. If the path is known, use deterministic BPMN. If the decision has clear rules, use DMN.

• Agentic orchestration fits when the input is messy, the next step depends on interpretation, or the situation can change mid-case in ways that were never modelled.

• The fundamental difference is reasoning. A process engine follows its path. A rule table matches its conditions. An LLM reads the situation and decides what to do next.

• AI adds real cost and operational overhead. Start from the problem, not the technology.

Once you decide that agentic orchestration is the right fit, the next question is practical: how does the AI step actually call tools and systems from inside the process? That is what the next article is about.

Camunda 8.9 lands on April 14, 2026. Camunda keeps pushing the direction it made more visible in 8.8: stronger agentic orchestration building blocks, a clearer Orchestration Cluster API story, fewer migration gaps from Camunda 7, and a more mature connector runtime.

The upgrade pressure is not the same for every team. If you mostly run Camunda SaaS with standard clients, the move should feel manageable. If you run your own connector runtime, build custom SDK integrations, or rely on specific API behavior, read the breaking changes carefully. The main watch-outs are the Spring Boot 4 requirement, MCP related connector changes, and broader client and API compatibility cleanup.

The date matters for another reason as well: Camunda 8.6 reaches end of maintenance on the same day. If you are still on 8.6, it is time to move.

Agentic AI And AI Facing Changes

Camunda is clearly putting more weight behind agentic orchestration. In 8.9, the platform pieces around that story get more concrete.

The clearest example is the built-in MCP support in the Orchestration Cluster. Camunda can now expose operational capabilities through an MCP server, so MCP-capable clients can discover and invoke Camunda tools without custom API glue code. That includes tools around process definitions, process instances, user tasks, incidents, and variables. Streamable HTTP support makes that more practical for real MCP clients.

Camunda also adds an MCP server for the Camunda docs. Agents should not only call tools, they also need a trustworthy way to read the product documentation they depend on. In practice, this gives them a cleaner path to Camunda docs instead of relying on copied snippets or ad hoc retrieval.

There is also movement on the A2A side. Camunda adds A2A Client, A2A Client Polling, and A2A Client Webhook connectors in 8.9. That fits the same direction: agents should not only call one model or one tool, but also work across more structured agent-to-agent patterns. This is still earlier-stage than the MCP story, but it shows where the AI integration model is expanding.

For me this is a clear sign that Camunda keeps investing in AI, and 8.9 shows they’re serious about it.

API Direction And Orchestration Cluster

The next theme is platform consolidation around the Orchestration Cluster API. Camunda introduced the Orchestration Cluster and its API with Camunda 8.8 in October 2025. So 8.9 is not the start of that story, but a clear continuation of it.

Operate now uses the Orchestration Cluster REST API as its single interface for accessing and managing process data. That reduces the distance between what the product uses internally and what customers use for their own tooling. On top of that, 8.9 adds more operational capabilities on the same path, for example deleting process and decision definitions or instances consistently through the Orchestration Cluster API and Operate. Even smaller behavior changes, like deletion endpoints now returning a response body, point in the same direction.

The takeaway for me is that the platform is getting more coherent around one API direction.

If you have custom scripts, generated clients, internal wrappers, or direct integrations, this is the release where it makes sense to review your assumptions. If you mostly stay on supported tooling, you mainly need to know where the platform is moving.

Migration Relevant Changes

This part of the release closes a few gaps that Camunda 7 teams have felt for a while. Camunda has pushed Camunda 7 Enterprise Edition end of life to April 9, 2030, but it still stays on the final 7.24 minor release. That gives customers more time, but it also makes the migration path more important. In that window, Camunda keeps trying to smooth out the move to 8 where it can.

The most obvious one is BPMN conditional events. Camunda 8 now supports conditional start, boundary, and intermediate catch events. It makes BPMN models more expressive, removes some workaround patterns, and makes migration conversations from Camunda 7 a bit less awkward.

Business ID is another good example. You can now create process instances with an immutable business identifier, and uniqueness validation can be enabled at cluster level for idempotent starts. That is useful far beyond migration, but migration teams will recognize the value immediately.

Taken together, I think this is probably the clearest “what got easier now” section in the release.

Connector Runtime And SDK Details

First, connectors now use a virtual threads executor by default. That should improve performance and scalability, especially for the I/O-heavy workloads that are typical in connector runtimes. In simple terms, the runtime should handle more concurrent jobs with lower resource consumption.

On the SDK side, the clearest headline is that the Camunda Spring Boot Starter now requires Spring Boot 4.0.x. That alone can decide how much work your upgrade takes, and Camunda ties it directly to the Spring Boot support policy, because support for Spring Boot 3 ends in June 2026. Also, OpenJDK 25 is now supported across Orchestration Cluster, Connectors, Optimize, and related tooling.

Other Practical Changes

After the four bigger themes, the rest of the release is mostly about operational maturity. These are not the headline items, but they still improve how the platform feels in day-to-day work.

Worth Mentioning

One of the nicest operational additions is the new audit log. It is available in Operate, Identity, Tasklist, and the Camunda 8 REST API. That improves traceability, governance, and troubleshooting.

One strong SaaS addition is the Cluster Metrics endpoint for Orchestration Cluster. That matters because it gives teams a cleaner way to get cluster metrics into their own observability stack instead of treating SaaS metrics as something locked inside the platform.

Self-Managed Quick Notes

Self-Managed is not the focus of this article, but there is one change that stands out more than the others: RDBMS support for Orchestration Cluster as an alternative storage path alongside Elasticsearch or OpenSearch.

That is a big deal. For teams that want a more familiar operational model, or already run database-centric platforms, this can lower the barrier to self-managed Camunda noticeably. It does not remove the need for real platform ownership, but it changes the conversation around what self-managed Camunda can look like.

Summary

Taken together, Camunda 8.9 looks like a consolidation release with a purpose. It sharpens the AI and API direction introduced in 8.8, closes a few migration pain points, and improves runtime behavior where it matters.

If you are on 8.6, the maintenance deadline is the forcing function. If you are migrating from Camunda 7, a few long-standing gaps get smaller here. And if you work with connectors, AI integrations, or custom SDK code, 8.9 is the point where a careful upgrade review pays off.

In one of my last articles, I wrote about payload validation for inbound webhooks. The model can reject bad payloads, check values with FEEL, and shape the data before the process continues. That is useful, but only half the story. At some point you need to prove that the webhook really works when the connector runtime is up and the process is deployed.

That is the point of this article. I want to show how to verify the whole inbound path: the webhook is reachable, authentication works, the payload is accepted or rejected as expected, the optional response looks right, and the process really starts or continues.

The setup is simpler than it sounds. Start the connector runtime next to the engine, point an HTTP client at the runtime address, wait until the webhook endpoint exists, then call it like a real consumer would.

What We Need To Verify

If the real entry point is a webhook, starting the instance directly through CamundaClient is not enough. It skips the exact parts that tend to break.

The main things I want to verify are:

• the webhook endpoint exists and is reachable.

• the auth header is required and accepted.

• the payload validation behaves as expected.

• the optional response body contains the data I need.

• the process starts or continues after the inbound connector fired.

Typical failures sit right there:

• the webhook doesn’t exist.

• the auth token is missing or wrong.

• a FEEL check rejects a payload that used to pass.

That is why I like testing the webhook through HTTP. It keeps the test close to the real runtime path, without needing a full external environment.

A Shared Test Setup

For this setup, I want three things in every test:

• the process engine

• the connector runtime

• a tiny HTTP helper that knows how to call the webhook

Camunda Process Test already gives us the first two pieces. With SpringBoot, the shared base class can hold the rest.

@SpringBootTest(
    classes = TestConnectorRuntimeApplication.class,
    properties = {
      "camunda.process-test.connectors-enabled=true",
      "camunda.process-test.connectors-secrets.WEBHOOK_TOKEN=test-token",
    })
@CamundaSpringProcessTest
abstract class WebhookAwareProcessTest {

  protected static final String WEBHOOK_TOKEN = "test-token";

  @Autowired
  protected CamundaClient camundaClient;

  @Autowired
  protected CamundaProcessTestContext processTestContext;

  private RestClient webhookClient(String token) {
    return RestClient.builder()
        .baseUrl(processTestContext.getConnectorsAddress() + "/inbound/")
        .defaultHeaders(headers -> {
          headers.setBearerAuth(token);
          headers.setContentType(MediaType.APPLICATION_JSON);
        })
        .build();
  }

  protected void waitForWebhookReady(String id) {
    Awaitility.await()
        .atMost(Duration.ofSeconds(30))
        .ignoreExceptions()
        .until(() -> webhookStatus(id) != 404);
  }

  protected Optional<Long> startProcessViaWebhook(String id, String body) {
    var response = webhookClient(WEBHOOK_TOKEN)
              .post().uri(id)
              .body(body).retrieve()
              .toEntity(String.class);
    
    assertThat(response.getStatusCode().is2xxSuccessful()).isTrue();
    
    return extractProcessInstanceKey(response.getBody());
  }

  protected void continueProcessViaWebhook(String id, String body) {
    var response = webhookClient(WEBHOOK_TOKEN)
              .post().uri(id)
              .body(body).retrieve()
              .toBodilessEntity();

    assertThat(response.getStatusCode().is2xxSuccessful()).isTrue();
  }

  protected long startProcess(String processId, Map<String, Object> variables) {
    return camundaClient
        .newCreateInstanceCommand()
        .bpmnProcessId(processId)
        .latestVersion()
        .variables(variables)
        .send().join().getProcessInstanceKey();
  }

  protected int triggerWebhookWithToken(String id, String body, String token) {
    return webhookClient(token)
            .post().uri(id)
            .body(body).retrieve()
            .toBodilessEntity()
            .getStatusCode().value();
  }

  private int webhookStatus(String id) {
    return webhookClient(WEBHOOK_TOKEN)
        .get().uri(id)
        .exchange((request, response) -> response.getStatusCode().value());
  }
}

If you prefer file based config, the same connector setup can live in src/test/resources/application.yaml.

camunda:
  process-test:
    connectors-enabled: true
    connectors-secrets:
      WEBHOOK_TOKEN: test-token

And if you already package your connector runtime as a Spring Boot app, use that same app in the test. That keeps the test setup close to the real runtime behavior. The only thing I swap is the secret values.

Trigger, Reject, And Continue Through Webhooks

Once the base class exists, the actual tests get small.

The first test covers the happy path. It waits for the route, starts the process through the webhook, and reads the optional instance key from the response.

class CustomerWebhookStartTest extends WebhookAwareProcessTest {

  @BeforeEach
  void setup() {
    waitForWebhookReady("missing-bag");
  }

  @Test
  void test_process_should_start_as_expected() {
    // given
    var key = startProcessViaWebhook("missing-bag", "{\"bagId\":\"BAG-42\",\"customerId\":\"C-100\",\"lastAirport\":\"BER\"}");

    // then
    assertThat(key).isPresent();
    assertThat(byKey(key.get())).isCompleted();
  }
}

This example assumes your webhook response expression returns the processInstanceKey in correlation. If that field is not part of the response, startProcessViaWebhook(...) returns Optional.empty(). That is why the return type is optional. Once you have the key, you can use your regular process assertions or search API to inspect the instance in more detail.

The next useful test is the negative auth case.

class CustomerWebhookAuthTest extends WebhookAwareProcessTest {

  @Test
  void test_webhook_call_rejected_if_unauthorized() {
    // when
    var status = triggerWebhookWithToken("missing-bag", "{\"bagId\":\"BAG-42\"}", "wrong-token");

    // then
    assertThat(status).isEqualTo(401);
  }
}

This is a cheap test, but it covers a real production risk. Someone changes the auth setup, or the token header is configured differently than expected, and suddenly the webhook is either too open or unusable.

The third test covers the callback case. Sometimes the webhook does not start a new instance. It continues an existing one that is waiting for external input. In that case I still use CamundaClient, but only to move the instance into the waiting state. The continuation itself still happens through the webhook.

class ShipmentWebhookContinuationTest extends WebhookAwareProcessTest {

  @Test
  void test_process_continues_if_webhook_is_triggered() {
    // when
    var key = startProcess("shipment-update-process", Map.of("shipmentId", "S-42", "customerId", "C-100"));

    // check for wait state
    assertThat(byKey(key.get())).hasActiveElements("WaitingForWebhook");

    // when
    continueProcessViaWebhook("shipment-update", "{\"shipmentId\":\"S-42\",\"status\":\"DELIVERED\"}");

    // then
    assertThat(byKey(key.get())).isCompleted();
  }
}

In this case the instance key comes from startProcess(...), because the webhook is only used for the continuation step. After the callback, assert the completed path or variables with the same process assertion style you already use in the rest of your test suite.

That is the split I would keep:

• use HTTP for the real webhook interaction.

• use CamundaClient only when you need to prepare state before the callback.

• return a process instance key where you can.

• optionally, you can query for the instance key by using a ProcessInstanceSearchRequest.

If you already have complex FEEL expressions in your webhook, this is also the place to prove it. Send one payload that should pass and one that should fail. Then check both the response and the process path.

Start Event Or Message Start Event

There is one current detail that is worth calling out. Right now, an inbound webhook can effectively behave in different ways:

• as a message start event

• as a none start event

The difference is whether the inbound webhook connector has a message attached to it or not. In my setup, getting the none start variant meant manually deleting the attached message event definition from the BPMN Xml. There is no visible way in the Web Modeler to do it.

Why does that matter? Because the returned correlation data is different.

With the none start variant, the response can include the processInstanceKey through the correlation object. That is useful when you want to follow the created instance directly from the webhook response.

With the message start variant, the response gives you the message correlation data, not the process instance key. In practice that means you get a messageKey, which is less useful when you want to inspect the created instance right away. You need to look up the started instance yourself.

So if your test needs to assert on the created process instance through the webhook response itself, the plain start variant is currently the better fit. I would treat this as current behavior, not as a permanent rule. Camunda may improve that later.

Summary

If you build validation logic into inbound webhooks, test the webhook itself. Do not bypass the route and then assume the rest will work.

The setup I would start with is:

• start the engine and connector runtime together.

• deploy the process with the real inbound webhook.

• wait until the webhook route exists.

• trigger it through a small HTTP helper.

• inspect the (optional) response body.

The best practices I would keep are:

• keep webhook related calls in a shared base class for others to use.

• reuse production secret names (not values) in tests.

• test both accepted and rejected payloads.

• use the CamundaClient only to reach a waiting state before a webhook callback.

• if you need the created instance key in the response, prefer the plain start variant for now.

That gives you a practical way to verify the webhook behavior you modeled, not just describe it.

You land after a long flight. At baggage claim, your bag never shows up. You report it and ask a simple question: “Can you find it and tell me what happens next?”

Behind the scenes, it is a mix of predictable steps and messy work. Someone checks tracking systems, waits for replies, asks you for missing details, and keeps you updated. I found the missing-baggage example below in an early blog post by Camunda’s CEO Jakob Freund, and it is a good way to picture agentic orchestration.

This article is a non-technical introduction to agentic orchestration with Camunda. I keep it simple on purpose: no code, no deep platform details, and as little jargon as possible. The goal is to explain the basic idea and why BPM and AI fit together.

The Advantages of BPM

Business Process Management (BPM) is running work as a defined process. A process engine executes it step by step and keeps it moving.

Many organizations already use a BPM suite to manage and automate processes. In Camunda, agentic orchestration lets you add AI without replacing that foundation.

Where you put the AI matters. If it sits outside the process, you lose end-to-end visibility. With agentic orchestration, AI work happens inside the process, so the case history stays complete.

BPM is great at the things you want to be linear and reliable:

• Traceability per case (an “instance”).

• Waiting, timers, and escalation (like a 48-hour handoff).

• Built-in human review via user tasks.

The Power of Generative AI

A Large Language Model (LLM) is good at working with unstructured information, like free text or documents. It is also useful when a system needs to interpret a situation and decide what to do next.

In the baggage example, the passenger might write a short description or provide pictures instead of using a form. The LLM can:

• Extract key details.

• Classify information.

• Find the next best action through reasoning.

• Draft a clear status update.

But the LLM should only act inside guardrails.

So the question is not “BPM or GenAI?”. The question is how to combine them without losing the advantages of BPM.

Agentic Orchestration: The Best Of Both Worlds

This is where both parts come together. The BPM engine runs the reliable path, provides guardrails, and keeps the case history. The LLM chooses the next approved task until a solution is reached, or until the process adds a human to the loop.

The AI Toolbox And Loop

Most cases should still run straight through. The AI loop is for exceptions where a human would normally try a few steps to get unstuck.

For missing baggage, the AI loop has a clear goal: reach a stable outcome with safe escalation. The engine exposes a toolbox of approved tasks and blocks everything else:

• Query tracking database

• Ask last known airport for confirmation

• Request more data from passenger

• Issue voucher

• Organize delivery

• Send status update

The loop is simple:

1. The LLM picks the next best task from the toolbox.

2. The engine validates and runs it, stores the result, and pauses if it must wait.

3. The loop ends when the goal is reached, or escalates when it is not safe.

The engine logs each task call and result, so the work stays visible.

Status Messages And Safe Escalation

While the AI loop runs, the passenger might ask for updates. In our example, you can send a status message without interrupting the main work. In BPMN, a process diagram language, this is a non-interrupting message boundary event.

You also define clear “stop and ask a human” rules, like:

• Low-confidence: the LLM cannot match the bag safely, or the data conflicts.

• 24-hour timer: too much time has passed without progress.

In both cases, BPM creates a user task for a human to review and decide. A reviewer sees what the AI tried, decides what to do next, and the case continues or ends.

A Clear End State

When the AI loop finishes, BPM makes the outcome explicit:

• If the bag is found and delivery is arranged, the case ends.

• If the bag is not found, BPM routes into a compensation decision and claim handling, or escalates for approval.

What’s Next

If the basic idea makes sense, do not jump to model choice yet. The first question is whether the problem really needs agentic orchestration at all.

That is where I go next. I stay with the baggage example and look at what should stay a normal workflow, what fits DMN, and where an agentic step is worth the extra effort.

I’m implementing Camunda 8 at a customer, and the hardest part isn’t modeling or coding. It’s keeping acceptance high when changes keep coming.

In part 1, I introduced compatibility as a contract between models, mappings, and runtime code. But rules on paper are not enough.

This part is about enforcing compatibility with automation. Our target setup has three pipelines:

• A connector pipeline that validates connector code and deploys generated templates to Web Modeler.

• A runtime pipeline that runs the latest runtime code against deployed models and deploys and tags code if tests are green.

• A model pipeline that runs the latest models against deployed code and deploys and tags models if tests are green. If it fails, it reruns the tests against the latest runtime code and, if that works, deploys runtime and models together.

We’re currently in the middle of implementation, in the first stage the pipelines check changes against the latest code or model version. The rest will follow in the next weeks. But the theory behind it is still relevant.

Staging Is Where Trust Breaks

Most late surprises are not BPMN errors. They are environment and ownership problems. Most common cases are:

• the runtime code changed and the model didn’t adapt

• the model changed and the runtime code didn’t adapt

• connector secrets are not working

If you only have one environment, every deploy is a gamble. If you have staging but people can still patch things in place, you end up with drift and a false sense of safety.

At the customer, coding and modeling both happen in dev. That keeps one place where new ideas are allowed to break.

Right now we only run two clusters, dev and prod. That works for now, but it keeps the distance short. You can get away with two clusters, but you still need one place to break things before testing starts. We are thinking about a third clusters:

• Dev for fast iteration.

• Test or QA for integration and business testing.

• Prod for stable operations.

And we try hard to prevent changes in test and prod. Modeling happens in dev only. In test and prod, Web Modeler access should be limited to admins for analysis, not editing.

Note: We keep clusters separated by having different stage organizations. Otherwise we’d need to maintain manual RBAC (resource based access control) rules to prevent access to certain models or cluster. This might change with future releases of Camunda 8 and its API.

Handle Models and Templates as Artifacts

The first shift is treating models like artifacts, not like “the thing in Web Modeler”.

Camunda’s Web Modeler can sync models into a Git repo. Once they’re in Git, you can handle them like code:

• Version them via commits.

• Test them in CI.

• Promote the same model version through stages.

This matters because a BPMN deployment is an artifact. If you redeploy from a different source in each stage, you are not promoting. You are rebuilding.

The same applies to connector templates. If you generate them in your connector repo, they are already versioned in Git and can be deployed via a pipeline. If you create them in Web Modeler, sync them to Git.

Using Pipelines That Catch Breakage

Contracts drift unless they’re checked continuously. For us, process tests are the gate: not unit tests, not “it compiled”, but end to end tests that deploy a model, run a realistic path, and assert variables and mappings.

Like I mentioned in part 1: compatibility breaks in both directions, so the process tests must run in both directions too.

Connector Pipeline

Trigger: a code commit in the connector repo.

Goal: prove the connector code is valid and the generated templates are valid and safe to publish.

High Level Steps:

• Check out the latest code.

• Run unit tests.

• Generate connector templates.

• Copy generated connector templates to Web Modeler.

• Publish them to a project or organization.

If you model templates directly in Web Modeler instead of generating them from code, they belong to the model’s lifecycle and should be handled by the model pipeline.

Runtime Pipeline

Trigger: a code commit in the runtime repo (worker updates, connector updates, connector runtime config).

Goal: prove that new code is compatible with what is already deployed in the target stage.

High level steps:

• Check out the deployed model baseline for that stage.

• Deploy those models into your test engine (Camunda offers an in-memory engine for that).

• Start the current runtime with your changes.

• Run process tests.

If this fails, do not deploy the runtime. The new code is not compatible with the models deployed in that stage. You need updated models now, and that handoff goes to the model pipeline.

In the full setup, if this succeeds, deploy the runtime to the next stage and update the runtime stage tag.

Model Pipeline

Trigger: a commit in the model repo, usually from Web Modeler sync.

Goal: prove that new models are compatible with the runtime currently deployed in the target stage.

High level steps:

• Check out the deployed runtime baseline for that stage.

• Deploy the new models into your test engine.

• Keep the runtime version pinned to the deployed baseline.

• Run the same process tests against that setup.

In the full setup, if the first test succeeds, deploy the models to the next stage and update the model stage tag.

If this fails, the model change is breaking against the deployed runtime. Do one additional step:

• Check out the latest runtime baseline for that stage.

• Deploy the new models into your test engine.

• Run the same process tests again, this time against the latest runtime code.

If the second test succeeds, you now have a compatible pair. In the full setup, deploy the runtime and models together and update both stage tags in one combined deployment.

If this still fails, there are real errors left to fix. Either the runtime change needs adjustment, the model change needs adjustment, or a migration step is missing.

What Process Tests Need To Cover

The tests do not need to be huge. They need to be intentional and run against the real runtime.

At the customer, process tests run against the actual workers and connectors. We usually mock downstream services to force edge cases.

For each critical process, we aim for:

• A realistic happy path with stable test data.

• Edge cases that tend to break in production (missing fields, unexpected status codes, timeouts, rate limits).

• Expected failure paths that prove retry and incident behavior.

• Coverage for all relevant paths, not only “the process ended”.

• Assertions on variable shape at key boundaries, not only outcomes.

Testing out of the box connectors is harder. The Camunda testing framework does not cover every scenario yet, so you may need extra harness code.

This is where staging helps. If you run tests only in dev, you miss the things that break in real clusters. If you run them in test or QA, you catch drift earlier.

Promotion And Tagging Strategy

Once the relevant pipeline is green, promotion should be easy. Today that still involves manual glue. The target is to move the same proven artifacts from dev to the next stage.

We are in the middle of building that full promotion flow. Prod is different. We do not deploy to prod continuously. Production deployment is planned and manual.

To keep this honest, you need to know what is running where. That is why stage tags matter, even if the full flow is still being built.

In the target setup, after a successful deployment, the responsible pipeline updates the stage tag. For a combined deployment, both tags move together. Then every compatibility pipeline can resolve refs/tags/latest-test to a specific revision and test against it.

This also helps during incidents. When test fails, you can answer quickly: did we change the model, the runtime, or both.

Of course a detailed branching strategy is also possible, if you’re not into using Git tags. You can use whatever helps pinning deployed versions down.

TL;DR

Staging is not overhead. It’s the price of predictable acceptance.

• Treat models as artifacts and promote the same versions through all stages.

• Use Camunda process tests as the compatibility gate, and run them in both directions, and against different baselines.

• Prevent drift between connector code and element template by generating your templates and publishing them to the Web Modeler for everyone to use.

• Tag deployed versions per stage so everyone knows what is running where.

• Lock down access to test and prod so trust does not decay through drift.

If you already do some of this, pick one gap and close it this week. A single stage tag and one process test can already make acceptance feel less random.

If you build Camunda connectors long enough, you stop thinking about the happy path. You start thinking about rate limits, flaky networks, and APIs that return the same error for five different root causes. That’s where retry logic in a connector becomes part of the connector design.

The tricky part is that retries are not automatic. In Camunda, an outbound connector runs as a job in the connector runtime. You can configure retries (and retry backoff) on the BPMN service task in Camunda Modeler, but the connector still needs to tell the Camunda engine when a failure is retryable. If you throw the wrong exception type from the Camunda Connector SDK, the engine won’t retry. In the worst case, you’ll see retries get set to 0 and your activity level retry settings are ignored for that failure path. The message of this post is simple: know your errors and act accordingly.

I’ll use a small REST API example where the connector gets a rate limit response. Of course, Java technologies like WebFlux WebClient can be wrapped with client side retries, but for this article I want the Camunda engine to retry (including backoff) so the behavior is visible in Operate and consistent across connectors.

Why Retries Matter

Imagine that your Camunda connector calls an external REST API. Most of the time, it works. Sometimes, the API rate limits you and returns something like this:

HTTP/1.1 429 Too Many Requests
Content-Type: application/json; charset=utf-8
Retry-After: 30
X-Request-Id: 049c9564-e6c7-7bc3-8a47-d955de067599

{
  "id": "019c9f14-b575-722b-a0c2-f1ac5834faef",
  "error": "Too Many Requests",
  "message": "Rate limit exceeded.",
  "timestamp": "2026-02-27T12:34:26Z"
}

If you fail fast with a non retry-able error, you create incidents for something that would have resolved itself in 30 seconds.

That is the key difference between a good connector and a noisy connector. A good connector turns transient failures into retries and permanent failures into clear incidents in Camunda Operate.

Force The Camunda Engine To Retry

The Camunda engine can only apply the retry configuration when the connector tells it that a task is retry-able. Otherwise it can end up creating an incident even though you configured retries.

For connectors this makes sense: a low-code modeler can set max retries and backoff on the BPMN task, but they usually don’t know how the connector is implemented. They can define the frame, but the connector decides whether it’s applied. The connector runtime sees the HTTP status, response body, and Java exception types, so it is the only place that can reliably decide between “try again later” and “this will never work”. With job workers, the situation is often different. A technical team usually implements the worker, wires the variables, and owns the activity configuration, so retry behavior is part of the worker contract. That is why connector retries need an explicit error classification policy inside the connector code.

In the Camunda Connector SDK for Java, that signal is a ConnectorRetryException (created via ConnectorRetryExceptionBuilder). If you throw a generic exception or a plain ConnectorException, the connector runtime treats it as non retry-able and the task level retry settings are effectively ignored for that failure path.

As soon as you throw ConnectorRetryException, the retry configuration on the task becomes active. You do not need to encode “retry three times” into the exception. The task already knows how many retries and what backoff it should use.

Classify Errors And Exceptions

Start with a small classification table. Keep it intentionally boring. HTTP status codes are a good starting point, but also exceptions from underlying libraries:

Retryable

• Rate limits (HTTP 429)

• Some server errors (like temporary errors: HTTP 502, 503 or 504)

• Timeouts (SocketTimeoutException)

• Temporary network failures (UnknownHostException, ConnectException)

• etc.

Non retryable

• Authentication and Authorization (HTTP 401, HTTP 403)

• Invalid input (HTTP 400)

• Missing resources (HTTP 404)

• Some server errors (like an unspecific HTTP 500)

• Most other HTTP 4xx responses, unless you have a clear reason to retry

Code Examples

This example shows one Camunda outbound connector function that calls a REST API and throws either ConnectorException (non retryable) or ConnectorRetryException (retryable). These exception types come from the Camunda Connector SDK. The important part is the mapping, not the HTTP client details.

Non Retryable: 401 Unauthorized

If credentials are wrong, retries won’t fix it. Fail fast with a clear error message and keep the root cause.

public CustomerResponse fetchCustomer(String customerId) {
  try {
    return webClient
        .get()
        .uri("/customers/{id}", customerId)
        .retrieve()
        .bodyToMono(CustomerResponse.class)
        .block();
  } catch (WebClientResponseException.Unauthorized e) {
    throw new ConnectorRetryExceptionBuilder()
        .message("API request failed: not authorized (401)")
        .errorCode("UNAUTHORIZED")
        .cause(e)
        .retries(0)
        .build();
  }
}

Retryable: 429 Too Many Requests

If you get rate limited, throw ConnectorRetryException. This activates the retry configuration on the task.

public CustomerResponse fetchCustomer(String customerId) {
  try {
    return webClient
        .get()
        .uri("/customers/{id}", customerId)
        .retrieve()
        .bodyToMono(CustomerResponse.class)
        .block();
  } catch (WebClientResponseException.TooManyRequests e) {
    throw new ConnectorRetryExceptionBuilder()
        .message("API rate limited us (429). Please retry.")
        .errorCode("RATE_LIMIT")
        .build();
  }
}

Advanced: Retry-After And Backoff

Many APIs return a Retry-After header with status 429. If you have it, you can use it as a retry backoff candidate for the next attempt. You still rely on the task retry configuration for the overall retry behavior, but this gives the Camunda engine a better starting point for the next retry.

} catch (WebClientResponseException.TooManyRequests e) {
  Duration retryAfter = parseRetryAfterHeader(e.getHeaders()); // example helper

  ConnectorRetryExceptionBuilder builder = new ConnectorRetryExceptionBuilder()
      .message("API rate limited us (429). Please retry.")
      .errorCode("RATE_LIMIT");

  if (retryAfter != null) {
    builder.backoffDuration(retryAfter);
  }

  throw builder.build();
}

Some Additional Notes

These are a few advanced topics that often come up once a connector runs under real load. They help you prevent new failure modes while you add retries.

• Pick one retry layer. Mixing e.g. WebFlux retries and Camunda engine retries is usually a bad idea because you multiply calls and make backoff unpredictable.

• Think about idempotence. Retrying a “send mail” connector can send the same mail twice if you retry the wrong failure. If the operation has side effects, you either need idempotency keys, deduplication, or a very strict retry policy.

• Always keep the root cause. Whatever you throw should include the original exception as the cause. It will show up in Operate, and someone needs to understand what actually went wrong.

Key Takeaways

• The Camunda engine retries only when you throw ConnectorRetryException.

• If you want to create an incident right away, set retries to 0.

• Model retries and backoff live on the BPMN service task, not in your code.

• Classify exceptions as non retryable and retryable.

• Honor meta information like Retry-After as a backoff hint when you get it.

• Don’t mix client side retries with Camunda engine retries.

• Retry only idempotent operations, or make them idempotent.

• Always attach the root cause so Camunda Operate shows what happened.

Imagine you are waiting for a text message with a one-time code. You cannot log in without it. And it is not just any code. It is your code, for your login attempt.

In Camunda, that is a message. Your process instance waits, and something has to publish the message that wakes it up. The boring part is that you often end up writing the same tiny “publish a message” worker again and again.

This post is about avoiding that work. With the Message Sending Connector, you can use the connector runtime to publish messages for you. You only configure message name, correlation key, and payload mapping. No custom worker code for every new message.

When A Process Waits For A Message

Let’s use a simple example. An order process reserves stock, then waits for a payment confirmation.

In BPMN, that waiting point is e.g. an intermediate message catch event. It means: “wait here until a message arrives.”

Many instances can be running and waiting for the same message name. So we need a way to make sure the right message wakes up the right instance.

Note: Message correlation also matters in other places, like message start or boundary events. The matching idea is the same. I use the intermediate catch example because it makes the “waiting instance” easy to picture.

A Quick Intro to Correlation

The Three Pieces: Subscription, Message, Match

When a process instance reaches an intermediate message catch event, it creates a subscription in the engine. The subscription is like a note pinned on the wall:

• Message name: what the instance is waiting for (for example, PaymentReceived)

• Correlation key: which instance it is meant for (for example, the orderId)

An incoming message has the same two fields, plus optional payload data (variables).

Now the engine correlates:

1. Find subscriptions with the same message name.

2. Compare the correlation key on those subscriptions with the correlation key on the incoming message.

3. If there is a match, continue that instance.

If the message arrives too early, Camunda can buffer it for a limited time and correlate it later when the instance reaches the waiting point.

The correlation key is the address of the message.

A correlation key is usually something you already have in the business:

• Order id

• Claim id

The key should be stable and specific. If two instances could share the same key, it is not specific enough.

Before The Connector: A Custom Worker Every Time

Imagine you want to send a message to another process. This can be an intermediate throw and catch event, or even a message end event.

Typically, that means a small job worker that publishes the message, since the engine only waits for the external trigger. It is rarely hard, but it is repetitive. Both send events have to be implemented.

It sounds small, and it often is. But it comes with real work:

• Credentials and environment configuration

• Mapping and validation (what if the orderId is missing?)

• Retries and operations (logs, monitoring)

And you repeat it for every process that needs message based waiting.

Here is what “publish a message” can look like in code. It can feel too small to deserve its own worker:

client.newPublishMessageCommand()
  .messageName("PaymentReceived")
  .correlationKey(orderId)
  .variables(Map.of("paymentId", paymentId))
  .send().join();

It is straightforward. But “straightforward” still means code to write, deploy, and run.

And if you have more than one process waiting for messages, you repeat the same pattern. Each new message tends to create another “small worker”, another deployment, another thing to operate.

After The Connector: Use The Runtime

The Message Sending Connector is an outbound connector that publishes a message for you. Instead of implementing message publishing as code, you model a step and configure it.

The key idea is simple: you stop rebuilding the “publish a message” runtime. You use the one you already have, if you run connectors.

It is also a nice companion to the inbound Webhook Connector. Webhooks bring events into the engine. This connector helps you send events inside the engine for inter-process communication.

What You Configure

At a high level, you configure three things:

• Message name

• Correlation key

• Variables (payload mapping)

You still need to understand correlation. The connector cannot guess the right key for you. But it removes the repeated implementation work.

What The Runtime Gives You

The connector runtime gives you the operational basics you usually have to rebuild in custom code:

• Auth and connectivity to the engine

• Retries and Idempotence (via message id)

• A standard place to configure and change the mapping

If you are teaching a process to talk to another process, or sending a “ping” back into a waiting instance, this connector removes a lot of friction. Your BPMN model shows clearly where the message gets published, and your runtime stays smaller.

Summary

• Message correlation matches a message name and correlation key to a waiting subscription.

• Before the connector, teams often implement message publishing as a small custom worker, over and over again.

• With the Message Sending Connector, you configure publishing and reuse the connector runtime instead of writing worker code each time.