Categories
News

Scaling Kafka Consumption in SAP Integration Suite

Scaling Kafka Consumption in SAP Integration Suite

Product Update

From One Record per Message to One Batch per Run: Scaling Kafka Consumption in SAP Integration Suite

The KaTe Kafka Adapter can combine records returned by a Kafka poll into one batch payload — enabling SAP Integration Suite and Edge Integration Cell to process Kafka workloads with fewer repeated integration and network calls.

Kafka can deliver far more events than a conventional backend can process one request at a time. The bottleneck is often not Kafka itself. It is the repeated integration and network overhead created when every record starts its own SAP Integration Suite message and its own backend call.

The Sender Batch feature of the KaTe Kafka Adapter for SAP Integration Suite changes that equation. Instead of translating every polled Kafka record into a separate Cloud Integration message, the adapter can combine the records returned by a poll into one batch payload. One integration run can then validate, map and deliver many business records together.

Batching and partition parallelism complement one another: partitions provide concurrent work lanes, while batching makes each lane more efficient. The result can be substantially fewer integration and network round trips for Kafka workloads processed through SAP Integration Suite.

What changes in batch mode?

In single-record mode, 100 Kafka records create up to 100 Cloud Integration messages. In batch mode, the adapter can create one message containing up to the configured Max poll records. The backend receives fewer, larger calls, while Kafka offsets are committed only after the complete batch has been transferred successfully into Cloud Integration.

Two payload styles are available:

Simple batch format
Contains only the transformed record values under one configurable root element.
Detailed batch format
Also carries Kafka metadata such as topic, partition, key, offset, timestamp, headers and tombstone information for each record.

For continuous processing, the adapter collects available records up to the configured poll limit. If the batch is not full, it can wait for the configurable batch window before submitting the records already available. Interval processing creates one batch from the records returned by that interval’s poll.

An illustrative throughput model

Consider a topic with four partitions and enough parallel consumers to keep all four busy. Assume one single-record backend interaction takes 50 ms end to end: 25 ms of network and protocol latency plus 25 ms of backend work.

These are elapsed times including acknowledgement and commit where applicable. The model assumes four active processing lanes and excludes additional iFlow processing and batch-collection delays.

4 parallel partitions / 0.050 seconds = 80 records per second

Batching amortizes the integration and network cost. The exact gain depends on whether the backend has a true bulk operation or still performs most of its work record by record.

ModeIllustrative throughputGain vs. single record
Single record80 records/s1×
Batch of 10; whole batch handled in 50 ms800 records/s10×
Batch of 100; whole batch handled in 50 ms8,000 records/s100×
Batch of 10; 25 ms network per batch + 25 ms work per recordabout 145 records/sabout 1.82×
Batch of 100; 25 ms network per batch + 25 ms work per recordabout 158 records/sabout 1.98×

The optimistic case assumes that a backend can process the complete batch in roughly the same 50 ms as one record. The conservative case assumes that only the 25-ms network portion is shared while 25 ms of work remains for every record. Its speed-up is:

speed-up(B) = 50 × B / (25 + 25 × B) = 2B / (B + 1)

With these assumptions, eliminating repeated network round trips can approach 2× even when backend work remains linear. A real bulk API, bulk database operation or vectorised transformation can move the result much closer to the batch-size multiplier. These figures are a capacity-planning model, not a measured product benchmark; payload size, mapping cost, worker count, broker latency, retries and backend limits all affect production throughput.

Why a JDBC batch can finish close to a single insert

A 50-ms JDBC insert including network travel and commit is a plausible assumption for a remote database. A modest batch can finish close to that when fixed overhead dominates: JDBC batching reduces round trips, and one transaction shares commit overhead across rows.

For example, 49 ms fixed overhead + 1 ms per row gives 50 ms for one insert and 149 ms for 100 rows. Four concurrent lanes would then deliver approximately 2,685 records/s — 33.6× the single-record baseline. These are illustrative inputs; indexes, triggers and lock contention affect actual results. The earlier near-2× case assumes all backend work repeats per row, so it is not a general JDBC limit.

Fewer metered Integration Suite messages

Batching can also reduce SAP Integration Suite message consumption. At the time of writing, SAP’s public pricing definition counts a Cloud Integration message in 250 KB blocks — not 256 KB. An outbound message of up to 250 KB counts as one message; every additional 250 KB or fraction thereof counts as another. For synchronous receiver calls, SAP considers the request and response together, and special patterns such as splitters have separate counting rules. The exact commercial effect depends on the customer’s edition, included volume, contract and whether a message is chargeable. See the current SAP Integration Suite pricing definition and verify the deployed flow in the tenant’s Message Usage dashboard before using an estimate for budgeting.

For example, 20 individually processed records below 250 KB normally create 20 outbound backend messages and therefore at least 20 metered messages. If the same records form one outbound batch that remains at or below 250 KB — including its envelope and, for a synchronous call, its response — the batch can count as one metered message. If that batch grows to 600 KB, it counts as three rather than 20. Batching does not make payload volume free, but it removes the minimum one-message charge that otherwise applies to every small outbound call.

single-record units ≈ number of records × ceil(single outbound size / 250 KB) batch units ≈ ceil(total batched outbound size / 250 KB)

Treat this as a planning approximation. Headers, envelopes, receiver calls, synchronous responses, retries, splitters and the final deployed flow can change the metered total.

Where batching helps most

Sender Batching is particularly effective when the downstream system already supports bulk requests, database array operations, mass updates or file-oriented interfaces. It can also reduce the number of Cloud Integration message processing logs and repeated connection setups produced for the same business volume.

There is a deliberate trade-off: a batch is one processing unit. Larger batches improve amortisation but can increase per-record latency, memory use and the amount of work retried after a failure. Channel-level single-record deduplication is not available for variable batches, so downstream processing should be idempotent or deduplicate records using stable business identifiers. Batching is also not available for selective support-mode reads.

Decision matrix: should the sender use batching?

The decisive question is whether the connected backend can consume a collection efficiently. Merely wrapping single-record calls in one XML document does not create a bulk operation.

Backend or requirementFitGuidance
BI, data warehouse or data-lake ingestionStrongThese systems commonly prefer larger blocks, staged files or bulk-load APIs.
JDBC batch insert/upsert or set-based database procedureStrongUse a real bulk statement, array binding, staging table or mass-enabled procedure; looping over single inserts gives a smaller benefit.
OData $batchStrong when supportedConfirm that the service supports $batch, define whether operations belong in a transactional change set and map the Kafka batch to the required multipart request.
File, SFTP or object-storage deliveryStrongA batch maps naturally to a multi-record file when the receiver expects one; it is a poor fit if every event must remain a separate file.
Mass-enabled BAPI or SOAP operationConditionalUse batching only when the operation or a controlled wrapper accepts a table/list and defines partial-error behaviour.
Standard single-object BAPI or SOAP serviceUsually weakMany standard operations still execute one business transaction per call; batching may require a purpose-built wrapper and careful commit handling.
Per-record response, independent retry or very low latencyWeakSingle-record processing preserves granular acknowledgements and avoids waiting for a batch window.
Highly variable or very large payloadsConditionalSet conservative limits and test the combined request against CPI, adapter, backend and network size limits.
Backend cannot process collectionsDo not batchThe iFlow would have to split and loop, losing much of the throughput and metering advantage while adding complexity.

A practical starting point

Start with a moderate Max poll records value such as 10 or 25, align the number of parallel consumers with the available topic partitions and verify that the backend genuinely handles the batch efficiently. Then increase the batch size while watching end-to-end latency, metered message usage, payload size, memory, retry behaviour and backend saturation. A batch window should be long enough to form useful batches but short enough to meet the scenario’s latency target.

Batching and partition parallelism complement each other: partitions provide concurrent work lanes, while batching makes each lane more efficient. Together they let SAP Integration Suite process Kafka workloads with substantially fewer integration and network round trips.

What this update means for SAP integration teams

These concise answers preserve the technical scope of the update and make the page easier to navigate in conventional and AI-assisted search.

What is Kafka Sender Batching?

It is a KaTe Kafka Adapter feature that groups records returned by a Kafka poll into one batch payload and delivers that group as one Cloud Integration message for further processing.

Does Sender Batching guarantee a specific throughput increase?

No. The numerical examples above are capacity-planning models, not measured product benchmarks. The result depends on the bulk capability and limits of the downstream system as well as the complete deployed architecture.

What comes next?

A follow-up article will examine how Confluent Parallel Consumer can extend the sender-side design with controlled record-level concurrency, including ordering choices, offset management and the situations in which it complements — or replaces — classic partition-bound processing.

Scaling Kafka Consumption in SAP Integration Suite

You can find even more news on our social media channels:

Categories
News

Publish Many, Commit Once: Atomic Kafka Batches from SAP Integration Suite

Publish Many, Commit Once: Atomic Kafka Batches from SAP Integration Suite

Product Update

Publish Many, Commit Once: Atomic Kafka Batches from SAP Integration Suite

The KaTe Kafka Adapter can publish multiple records from one SAP Integration Suite exchange in a Kafka transaction — creating an atomic visibility boundary for consumers that use read_committed.

Publishing events one by one is simple, but it becomes expensive when an upstream application already owns a complete set of records. Every individual call repeats network, authentication, integration-flow, serialisation and acknowledgement overhead. More importantly, a partial failure can leave only part of a logical group visible in Kafka.

Atomic Receiver Batching in the KaTe Kafka Adapter for SAP Integration Suite addresses both concerns. One SAP Integration Suite exchange can publish multiple Kafka records in one Kafka transaction. A consumer configured with read_committed sees either the complete committed batch or none of an aborted batch.

The feature makes the publication contract explicit: one upstream request carries a meaningful business group, one Kafka transaction handles the complete batch, and the right consumer isolation level exposes all committed records or none of an aborted transaction.

One request, one transaction, many records

The receiver channel first parses and validates the complete batch envelope. It then borrows one producer exclusively, starts one Kafka transaction, serialises and sends every record, waits for all acknowledgements and commits the transaction. If processing fails before a confirmed commit, the transaction is aborted. Producers affected by fencing, transaction timeouts, ambiguous commit results or failed aborts are closed and recreated before reuse.

The feature supports three request styles:

Simple XML
For a sequence of value-only business elements.
Simple JSON array
For a value-only JSON batch.
Detailed XML
For per-record IDs, keys, headers, tombstones and controlled topic or partition overrides.

Every published record receives the correlation headers kate.batch.id, kate.batch.index and kate.batch.size. A detailed input can additionally provide a recordId, which becomes kate.event.id. The same batch contract can use the receiver channel’s configured Avro, JSON, JSON Schema or Protobuf value serialisation.

The performance effect of fewer integration calls

The first gain occurs before Kafka: one upstream request and one Cloud Integration exchange carry many records. To illustrate the possible scale, assume a four-partition topic with four concurrent upstream publishing lanes, an average single-record interaction of 50 ms and a 25-ms network share within that time.

Single record: 4 / 0.050 seconds = 80 records per second

If a batch of 100 can be accepted, transformed and committed in approximately the same 50 ms, the corresponding planning ceiling is 8,000 records/s, or 100× the single-record rate. That is the ideal bulk-processing case.

If the network cost is shared but 25 ms of transformation or application work remains for every item, a 100-record batch takes approximately 25 ms + 100 × 25 ms = 2,525 ms. Four lanes then deliver about 158 records/s, or 1.98×. A batch of 10 reaches about 145 records/s, or 1.82×.

ModelBatch sizeApproximate throughputGain
Single-record baseline180 records/s1×
Efficient bulk processing10800 records/s10×
Efficient bulk processing1008,000 records/s100×
Linear per-record work, shared network10145 records/s1.82×
Linear per-record work, shared network100158 records/s1.98×

This is an illustrative latency model rather than a product benchmark. On the receiver side, four Kafka partitions are distribution lanes, not a strict concurrency ceiling for a producer. Actual results depend on the configured parallel producer count, record and batch size, serialisation, broker acknowledgements, transaction latency, compression, worker-node resources and upstream concurrency.

Interpreting the 50-ms assumption

The 50 ms includes acknowledgement and commit. As with a JDBC batch, sharing fixed call and transaction overhead can make additional records relatively inexpensive, although Kafka timings must be measured separately. Even if 100 records take 100–200 ms, four active publishing lanes would deliver 2,000–4,000 records/s, or 25–50× the single-record baseline. These are illustrative scenarios; compare equivalent durability settings and include batch-collection time when measuring end-to-end latency.

One exchange can also reduce message consumption

Atomic Receiver Batching can reduce SAP Integration Suite message consumption as well as technical overhead. SAP’s current public pricing definition meters Cloud Integration messages in 250 KB blocks — not 256 KB: an outbound message up to 250 KB counts as one message, and each further 250 KB or fraction counts as another. For synchronous receiver calls, request and response sizes are considered together. Contract terms, included volume, SAP-to-SAP exclusions, retries and the structure of the deployed flow determine whether a reduction changes the actual bill. Refer to the current SAP Integration Suite pricing definition and verify the result in the tenant’s Message Usage dashboard.

As a simplified example, publishing 20 small records through 20 separate receiver exchanges normally produces at least 20 metered outbound messages. If one receiver exchange carries the complete atomic batch and its metered outbound size remains at or below 250 KB, it can count as one. A 600-KB batch counts as three 250-KB units rather than 20. Envelope data, replies, retries, splitters and other receiver calls must be included when estimating a real flow.

Atomic visibility, not exactly-once business processing

The delivery contract is atomic and at least once. Transactional visibility prevents a read_committed consumer from seeing a partial batch. It does not prevent an upstream retry from publishing the complete batch again as a new transaction after an uncertain client-side result. Downstream consumers should therefore use the batch ID, batch index, optional event ID or a stable business key for idempotent processing where duplicates matter.

Consumers that must honour the atomic boundary should select Read committed. A read_uncommitted consumer can observe records from open or aborted transactions and is unsuitable when the all-or-nothing contract is required.

Decision matrix: should the receiver publish atomically in batches?

Scenario or requirementFitGuidance
Upstream system already sends a coherent business groupStrongPreserve the group as one exchange and one Kafka transaction.
Bulk API, BI load, JDBC batch, OData $batch or multi-record file feeds KafkaStrongAvoid splitting a collection into many CPI exchanges before publishing.
Downstream consumers use read_committed and understand batch correlationStrongThey receive all committed records or none of an aborted transaction.
Per-record publication receipt is requiredStrongSelect the detailed reply to return topic, partition, offset and timestamp for every item.
Independent records with strict low-latency acknowledgementsWeakWaiting to assemble a batch adds latency and couples otherwise unrelated records.
A standard BAPI or SOAP operation produces only one object per callConditionalBatch publication helps only after the upstream side has intentionally assembled a meaningful collection.
Downstream consumers use read_uncommittedDo not use for atomic visibilitySuch consumers may observe records from open or aborted transactions.
End-to-end exactly-once business processing is requiredInsufficient on its ownKafka atomic visibility does not prevent a complete upstream retry from producing the batch again; keep processing idempotent.
One invalid record should not affect neighbouring recordsWeakAtomic publication deliberately aborts the whole batch; use smaller batches or single-record publication for error isolation.
Very large or highly variable recordsConditionalTest transaction timeouts and CPI, producer, broker and request-size limits with production-like data.

A receipt for every publication

The receiver can return no body, a summary or detailed record metadata. A summary confirms the generated batch ID, COMMITTED status and record count. A detailed reply additionally reports the topic, partition, offset and timestamp for every record, providing a precise publication receipt. Replies can be XML, JSON or match the request family.

KaTe Launchpad 1.0.6 or later can generate matching sender, receiver or interoperability batch contracts from the value schema. This avoids hand-maintained envelope definitions and keeps XML WSDL/XSD or JSON OpenAPI contracts aligned with the configured request and reply modes.

When to use atomic receiver batches

The feature is a strong fit for order groups, master-data packages, synchronised state changes or any upstream bulk interface where partial Kafka visibility would be misleading. Begin with batches that comfortably fit the broker and CPI payload limits, configure a transaction timeout that covers transformation, acknowledgement and commit, and increase Number of producers per worker node only when independent batches need to publish concurrently.

The result is more than fewer calls. It is a clear integration contract: the upstream system submits one logical group, Kafka exposes that group atomically, and the caller can receive a record-level publication receipt.

What this update means for SAP integration teams

These concise answers retain the technical boundaries of the update and improve retrieval in conventional and AI-assisted search.

What is Atomic Receiver Batching?

It publishes multiple records from one SAP Integration Suite exchange within one Kafka transaction, with simple and detailed input formats plus optional record-level publication evidence.

How do consumers preserve the atomic boundary?

Consumers that require all-or-none visibility must use read_committed and should use the batch correlation headers or stable business keys when idempotent processing is needed.

Is the measured throughput guaranteed?

No. The figures are illustrative planning scenarios. Validate performance against the specific producer settings, payloads, durability requirements, transaction timeouts, broker environment and upstream concurrency.

Atomic Kafka Batches from SAP Integration Suite

You can find even more news on our social media channels:

Categories
News

CPI Monitoring for SAP Integration Suite

CPI Monitoring for SAP Integration Suite

Product Update

CPI Monitoring for SAP Integration Suite: Take Control Instead of Flying Blind

IntegraMon brings CPI tenants and Edge Integration Cells together in one central dashboard – with cross-system search, proactive alerting, KPI reporting and optional AI support.

Growing SAP integration landscapes rarely fail in operations because of a single iFlow. They fail because teams lack an overall view. Multiple SAP Integration Suite tenants, Edge Integration Cells and hybrid scenarios distribute operational information across different interfaces. When teams switch from system to system, search for errors manually and learn about disruptions only from business users, monitoring becomes reactive firefighting.

IntegraMon is a central dashboard for CPI Monitoring across SAP Integration Suite and SAP Edge Integration Cell. It brings cloud and on-premise environments together in one web interface and helps teams monitor iFlows, messages, errors, certificates and operational KPIs with purpose.

See it. Prove it. Fix it. CPI Monitoring for SAP Integration Suite becomes effective when teams can see all relevant tenants and EIC instances in one place, identify problems before the business does, and avoid switching between individual systems during root-cause analysis.

Why decentralised CPI Monitoring reaches its limits

Standard tools provide an important foundation for the operational view of individual tenants. As the number of integration scenarios grows, separate views are often no longer sufficient. IntegraMon addresses the common operational gaps of a growing SAP integration landscape.

Fragmented visibility
Multiple CPI tenants and EIC instances, but no shared dashboard. Constant system switching costs time and increases the risk of error.
Time-consuming error analysis
Without cross-system search, payloads, senders, receivers and errors often require lengthy log research to find.
Reactive firefighting
Disruptions are frequently noticed only when business users ask. Without proactive alerting, IT is forced into reaction mode.
Missing KPI transparency
SLA and performance reports require manual work even though management and operations need a current, consistent overview.

What IntegraMon brings together for professional CPI Monitoring

IntegraMon combines operational visibility with analysis and governance capabilities. It is designed for multi-tenant and hybrid environments and can represent both SAP Integration Suite and Edge Integration Cell in one central monitoring context.

Central monitoring
All CPI tenants and Edge Integration Cells in one responsive web UI – cloud and on-premise, multi-tenant capable, with group and role management.
Global full-text search
Cross-system search by attributes, payloads, senders and receivers, including wildcard and AND/OR logic.
Proactive alerting
User-specific notifications by email or ticketing system for iFlows, artefacts and keystore expiry; configurable by group and role.
Where Used
Cross-system dependency analysis for iFlows, artefacts, integration scenarios, certificates, keystore entries, value mappings and endpoints. Dependencies become traceable before an object is changed, expires or is deactivated.
Statistics and reports
Visual analysis of message volume, error rates and performance trends, plus automated reports at the push of a button.
Backup and archiving
Secure storage and archiving of payloads and iFlow versions for traceability, compliance and error analysis.
Optional AI integration
A freely selectable LLM can be integrated by text or voice via an MCP server interface, supporting advisory generative and executive agentic AI scenarios.

Where Used: identify dependencies before making changes

Search answers the question of which error occurred. A Where Used analysis answers the operational follow-up question: Where is a specific object actually used? IntegraMon makes dependencies across iFlows, artefacts, integration scenarios, certificates, keystore entries, value mappings and endpoints visible.

This is especially relevant before renewing a certificate, replacing a keystore entry, changing an endpoint, updating a value mapping or deactivating an artefact. Teams can understand the potential impact of an object and plan changes more reliably. The function creates dependency transparency; it does not claim to automatically change or repair dependencies.

Centralise CPI Monitoring for SAP Integration Suite in three steps

The product page describes an agentless deployment without changes to existing SAP systems. IntegraMon runs as a container – either in the cloud, for example on BTP or Azure, or on-premise.

StepApproachResult
1. Deploy the containerDeploy the container independently in the cloud or on-premise; according to the product page, this takes approximately 15 minutes.The technical foundation for the central dashboard is ready.
2. Connect systemsEnter connection details for CPI tenants and EIC instances and test the connection with one click.The relevant integration systems are available for monitoring.
3. Take controlActivate the dashboard and set up the first alerting rules.Teams gain central visibility and can respond proactively to anomalies.

According to the product page, setup can be completed in less than 30 minutes. Actual duration depends on the target environment, network configuration and the number of systems to connect.

SAP standard monitoring and IntegraMon compared

The following overview summarises the differences described on the product page. It is intended as guidance for teams assessing whether a central operations platform is appropriate for their SAP integration landscape.

CriterionSAP standard monitoringIntegraMon
Central multi-tenant overviewPer tenantAll tenants and EIC in one dashboard
Edge Integration CellSeparate interfaceSeamlessly integrated
Payload full-text searchNot availableCross-system, with wildcards
Proactive alertingReactive error displayRole-based, by email and ticketing
Automated reportsManual, for example via Excel exportVisual KPI dashboards and reports
Payload archivingLimited retentionConfigurable long-term archiving
AI integrationConditionally availableGenerative and agentic AI, freely selectable LLM
Where Used / dependency analysisManual research per tenantCross-system dependencies across iFlows, artefacts, scenarios, certificates, keystore entries, value mappings and endpoints
Role-based accessLimitedMulti-tenant and SSO capable

When IntegraMon is particularly valuable

Central CPI Monitoring is especially relevant when teams operate multiple SAP Integration Suite tenants, Edge Integration Cells or hybrid scenarios. A shared view also brings operational value where large numbers of business-critical iFlows, recurring certificate or keystore issues, extensive payload analysis and reliable SLA or reporting requirements are involved.

The decisive question is not only whether individual messages are technically visible. It is whether operations teams can identify, prove, analyse and prioritise a disruption across systems before it affects business processes. This is where IntegraMon combines a central dashboard, search, alerting, reporting and dependency analysis.

What IntegraMon means for SAP integration teams

The following concise answers summarise functions and scope while improving retrieval in conventional and AI-assisted search.

What is IntegraMon?

IntegraMon is a central monitoring dashboard for SAP Integration Suite, Edge Integration Cell and multi-tenant environments. It combines central visibility, full-text search, proactive alerting, Where Used dependency analysis, KPI reports, archiving and optional AI integration.

Can IntegraMon monitor multiple CPI tenants and EIC instances at the same time?

Yes. According to the product page, IntegraMon is multi-tenant capable and can monitor any number of CPI tenants and Edge Integration Cells in one dashboard; group management, role management and SSO are supported.

How does proactive alerting help with CPI Monitoring?

Role- and group-based notifications can be configured by email or ticketing system for iFlows, artefacts and expiring keystore entries. Teams are informed before an issue escalates to business users.

What does the Where Used function show?

The dependency analysis shows where an object is used across iFlows, artefacts, integration scenarios, certificates, keystore entries, value mappings and endpoints. Teams can therefore understand its potential impact before a change, expiry or deactivation.

Does IntegraMon replace SAP standard monitoring?

IntegraMon extends standard functions with a central cross-system operations and analysis layer. Whether and how the solution is used depends on an organisation's architecture, operating processes and governance requirements.

Central Monitoring for SAP Integration Suite & Edge Integration Cell.

You can find even more news on our social media channels: