Publishing events one by one is simple, but it becomes expensive when an upstream application already owns a complete set of records. Every individual call repeats network, authentication, integration-flow, serialisation and acknowledgement overhead. More importantly, a partial failure can leave only part of a logical group visible in Kafka.

Atomic Receiver Batching in the KaTe Kafka Adapter for SAP Integration Suite addresses both concerns. One SAP Integration Suite exchange can publish multiple Kafka records in one Kafka transaction. A consumer configured with read_committed sees either the complete committed batch or none of an aborted batch.

One request, one transaction, many records

The receiver channel first parses and validates the complete batch envelope. It then borrows one producer exclusively, starts one Kafka transaction, serialises and sends every record, waits for all acknowledgements and commits the transaction. If processing fails before a confirmed commit, the transaction is aborted. Producers affected by fencing, transaction timeouts, ambiguous commit results or failed aborts are closed and recreated before reuse.

The feature supports three request styles:

Every published record receives the correlation headers kate.batch.id, kate.batch.index and kate.batch.size. A detailed input can additionally provide a recordId, which becomes kate.event.id. The same batch contract can use the receiver channel’s configured Avro, JSON, JSON Schema or Protobuf value serialisation.

The performance effect of fewer integration calls

The first gain occurs before Kafka: one upstream request and one Cloud Integration exchange carry many records. To illustrate the possible scale, assume a four-partition topic with four concurrent upstream publishing lanes, an average single-record interaction of 50 ms and a 25-ms network share within that time.

If a batch of 100 can be accepted, transformed and committed in approximately the same 50 ms, the corresponding planning ceiling is 8,000 records/s, or 100× the single-record rate. That is the ideal bulk-processing case.

If the network cost is shared but 25 ms of transformation or application work remains for every item, a 100-record batch takes approximately 25 ms + 100 × 25 ms = 2,525 ms. Four lanes then deliver about 158 records/s, or 1.98×. A batch of 10 reaches about 145 records/s, or 1.82×.

This is an illustrative latency model rather than a product benchmark. On the receiver side, four Kafka partitions are distribution lanes, not a strict concurrency ceiling for a producer. Actual results depend on the configured parallel producer count, record and batch size, serialisation, broker acknowledgements, transaction latency, compression, worker-node resources and upstream concurrency.

Interpreting the 50-ms assumption

The 50 ms includes acknowledgement and commit. As with a JDBC batch, sharing fixed call and transaction overhead can make additional records relatively inexpensive, although Kafka timings must be measured separately. Even if 100 records take 100–200 ms, four active publishing lanes would deliver 2,000–4,000 records/s, or 25–50× the single-record baseline. These are illustrative scenarios; compare equivalent durability settings and include batch-collection time when measuring end-to-end latency.

One exchange can also reduce message consumption

Atomic Receiver Batching can reduce SAP Integration Suite message consumption as well as technical overhead. SAP’s current public pricing definition meters Cloud Integration messages in 250 KB blocks — not 256 KB: an outbound message up to 250 KB counts as one message, and each further 250 KB or fraction counts as another. For synchronous receiver calls, request and response sizes are considered together. Contract terms, included volume, SAP-to-SAP exclusions, retries and the structure of the deployed flow determine whether a reduction changes the actual bill. Refer to the current SAP Integration Suite pricing definition and verify the result in the tenant’s Message Usage dashboard.

As a simplified example, publishing 20 small records through 20 separate receiver exchanges normally produces at least 20 metered outbound messages. If one receiver exchange carries the complete atomic batch and its metered outbound size remains at or below 250 KB, it can count as one. A 600-KB batch counts as three 250-KB units rather than 20. Envelope data, replies, retries, splitters and other receiver calls must be included when estimating a real flow.

Atomic visibility, not exactly-once business processing

The delivery contract is atomic and at least once. Transactional visibility prevents a read_committed consumer from seeing a partial batch. It does not prevent an upstream retry from publishing the complete batch again as a new transaction after an uncertain client-side result. Downstream consumers should therefore use the batch ID, batch index, optional event ID or a stable business key for idempotent processing where duplicates matter.

Consumers that must honour the atomic boundary should select Read committed. A read_uncommitted consumer can observe records from open or aborted transactions and is unsuitable when the all-or-nothing contract is required.

A receipt for every publication

The receiver can return no body, a summary or detailed record metadata. A summary confirms the generated batch ID, COMMITTED status and record count. A detailed reply additionally reports the topic, partition, offset and timestamp for every record, providing a precise publication receipt. Replies can be XML, JSON or match the request family.

KaTe Launchpad 1.0.6 or later can generate matching sender, receiver or interoperability batch contracts from the value schema. This avoids hand-maintained envelope definitions and keeps XML WSDL/XSD or JSON OpenAPI contracts aligned with the configured request and reply modes.

When to use atomic receiver batches

The feature is a strong fit for order groups, master-data packages, synchronised state changes or any upstream bulk interface where partial Kafka visibility would be misleading. Begin with batches that comfortably fit the broker and CPI payload limits, configure a transaction timeout that covers transformation, acknowledgement and commit, and increase Number of producers per worker node only when independent batches need to publish concurrently.

The result is more than fewer calls. It is a clear integration contract: the upstream system submits one logical group, Kafka exposes that group atomically, and the caller can receive a record-level publication receipt.

This product update is published by KaTe GmbH. Performance and message-consumption figures are illustrative capacity-planning inputs and must be validated against the customer’s deployed integration flows, payloads and commercial terms.