Categories
SAP Products

Kafka Adapter for ABAP

SAP ABAP

Kafka Adapter for ABAP

Native, bidirectional connectivity between your ABAP applications and Apache Kafka — over TCP/TLS, without a separate Kafka REST proxy.

Integration architecture

ABAP applications
Publish / Consume
KaTe GmbHKaTe Kafka Adapter for ABAPNative TCP/TLS
Kafka records
Apache Kafka
TCP/TLSConnectionAvroSchema RegistryJSONData formats
Technical building blocks
IDocs
Custom ABAP types
JSON
Avro
Schema Registry

Architecture overview

ABAP applications, KaTe Kafka Adapter for ABAP and Apache Kafka

ABAP applications
KaTe GmbHKaTe Kafka Adapter for ABAPNative ABAP integration
Apache Kafka
Publish / ConsumeNative TCP/TLS
Direct connection between ABAP and Kafka

Features

All features at a glance

Bring native Kafka connectivity into your ABAP applications with native producer and consumer APIs, configurable data binding and central administration.

Core functions: publish and consume

Publish application data as Kafka records and consume incoming records directly in standard S/4HANA and ECC functions and ABAP.

  • Flexible publishing: publish Kafka records from IDocs, RAP Events (TBA) or custom ABAP programs, using blocking sends or non-blocking outbox enqueueing with Kafka broker acknowledgements (acks).
  • Controlled consumption: read Kafka records and submit them to SAP integration objects, such as IDocs and ABAP calls, through managed consumer channels with consumer groups. Consumers process partitions in parallel and commit Kafka offsets only after successful SAP receiver processing, with retries for failed records within their partition.
  • Configurable setup: use one or more topics, control offsets and work with Kafka keys, values and headers.

Security and connection management

Secure your Kafka connections with TLS encryption, protected credentials and centrally managed connection profiles.

  • Native TCP/TLS connectivity with SASL/PLAIN authentication.
  • Encrypted credentials for Kafka brokers and Schema Registry.

Advanced features: SAP integration, data formats and error handling

SAP integration, data formats and error handling

SAP integration and data formats

Integrate SAP objects with Kafka

  • IDocs: exchange complete IDocs as Kafka records, with SAP acting as sender or receiver.
  • Custom ABAP data types and tables: bind application data to explicitly configured or generic ABAP types, structures and internal tables.
  • RAP Events (TBA): integration with events and powerful reference data lookup from the ABAP RESTful Application Programming Model.

Supported serializers and deserializers

  • Standard Kafka formats: String, Boolean, Short, Integer, Long, Float, Double, UUID, Void and binary formats (ByteArray, ByteBuffer and Bytes).
  • Confluent Avro: native binary encoding and decoding, Schema Registry integration and supported schema evolution.
  • Confluent JSON Schema: serialization and validation with Schema Registry integration within a documented Draft 7 subset.

Key and Value formats can be freely configured with respective serializers/deserializers and binding to data structures.

Configurable error handling and retries

Keep processing failures visible and control how failed records are retried.

  • Configure retry limits and delays.
  • Review retry status and failure details in monitoring.
  • Retry failed records within their partition while other partitions continue.
  • Review uncertain producer delivery outcomes before deciding on recovery.

Advantages

Why connect to Kafka directly from ABAP?

1 · Connect directly to Kafka brokers

Use Kafka’s binary protocol from ABAP. A separate Kafka REST proxy is unnecessary for broker communication; Schema Registry uses its own HTTPS API.

2 · Work with your application’s data

Integrate IDocs and RAP Events (TBA), or use generic and typed interfaces for custom ABAP data. Select the serialization format independently for keys and values.

3 · Keep configuration and diagnostics together

Manage connection profiles and channels in one user-friendly web console, and investigate operational failures through correlated runtime diagnostics.

Comparison

Choose the integration approach that fits your scenario

Kafka Adapter for ABAP

Direct ABAP APIs and SAP data binding

Application-controlled table extraction

Integration logic within your SAP application

Kafka Adapter for SAP Integration Suite

Integration through Cloud Integration iFlows

Orchestration of data exposed by SAP interfaces

Central transformation and routing across systems

Setup

Set up your Kafka integration in three steps

1

1 · Discuss your scenario

Identify your SAP environment, Kafka topics, data formats and processing requirements with our team.

2

2 · Configure the adapter

Prepare the SAP prerequisites, configure protected connection profiles and define producer or consumer channels with the required data bindings.

3

3 · Validate and start processing

Check connectivity, test your application integration and use the administration console to control and monitor processing.

Compatibility

Find the right setup for your SAP environment

Contact us to discuss your SAP release, Kafka environment and installation prerequisites. Avro, Schema Registry and JSON Schema support follow documented mapping, schema and resource limits.

Administration console

More control during setup and operations

Operations and monitoring

Manage Kafka channels centrally and investigate processing issues with runtime status, application logs and targeted diagnostics.

  • Start, stop, pause and resume managed channels from the administration console, with consumer startup from the beginning or latest position of the stream.
  • Inspect consumer group offsets, processing progress and partition status.
  • Monitor producer outbox entries and review uncertain delivery outcomes.
  • Investigate failures through correlated application logs and temporary Kafka protocol traces.

FAQ

Frequently asked questions

Answers to common questions about the Kafka Adapter for ABAP.

What is the Kafka Adapter for ABAP?

The adapter connects SAP applications directly to Kafka using native ABAP producer and consumer APIs. It brings IDocs and custom ABAP data into Kafka integration, with configurable serialization and a web console for administration. RAP Events integration is TBA.

How are Kafka connections secured?

Broker communication uses TCP/TLS with SASL/PLAIN authentication. Broker and Schema Registry credentials are stored under separate encrypted aliases, keeping connection configuration separate from secrets.

How does the adapter preserve processing order?

Records are processed in order within each assigned partition. A failed record is retried before later records in that partition proceed, while other partitions can continue. Ordering across different partitions is determined by your application design.

Can consumers process records in parallel?

Managed consumer groups distribute partitions across configured consumers. This allows parallel processing across partitions while retaining ordered processing within each partition. Capacity depends on partition count, SAP resources and receiver workload.

Can I choose where consumption starts?

You can configure consumption from the beginning or latest position of the stream. Explicit-offset read APIs also support selected stream positions. Existing consumer-group offsets affect where an established group resumes.

How can I monitor and troubleshoot my integration?

The web console brings channel controls, processing progress, consumer offsets and producer outbox inspection together. Application logs show error and retry details; temporary Kafka protocol traces support targeted troubleshooting. Retry limits and delays are configurable.

When should I use the ABAP adapter instead of a Kafka adapter on middleware?

Use the ABAP adapter when integration logic belongs in SAP — for example, application data integration or custom ABAP extraction of table data to Kafka. Publish and consume Kafka records directly within your ABAP applications, with control over data binding and processing, without routing each exchange through a middleware layer. Use the KaTe Kafka Adapter for SAP Integration Suite when you need central orchestration, transformations or routing across multiple systems. The two adapters complement each other within your integration landscape.

Do I need a Kafka REST proxy?

Unlike other solutions, no separate Kafka REST proxy is needed for broker communication. The adapter speaks Kafka’s binary protocol directly over TCP/TLS from ABAP. Schema Registry integration uses HTTPS separately.

Can I replicate tables to Kafka?

Yes. Publish table data to Kafka using generic or explicitly typed ABAP serialization and a simple custom ABAP program that reads and sends data through a Kafka channel. This enables straightforward data integration tailored to your requirements.

Can I exchange IDocs through Kafka?

Yes. Send and receive complete IDocs as Kafka records using standard SAP IDoc integration. Outbound IDocs are confirmed after Kafka acknowledges the record.

Which Confluent serializers are supported?

The adapter provides serializers and deserializers compatible with Confluent Avro and Confluent JSON Schema, including Schema Registry integration. Avro supports schema evolution; JSON Schema supports validation within a documented Draft 7 subset.

How does the adapter support reliable delivery?

The adapter commits Kafka consumer offsets only after successful receiver processing. Failed records are retried before processing continues within their partition, helping ensure records are not skipped when processing fails.

Contact

Planning Kafka integration from ABAP?

Tell us about your SAP environment, data formats and processing requirements. We can discuss the integration approach and supported configuration with you.

Categories
News

KaTe is now in Americas

KaTe now in Americas

Company News

KaTe Group Inc. Opens in Atlanta: SAP Integration Expertise for the Americas

With KaTe Group Inc. in Atlanta, KaTe is expanding its local presence for customers and partners across North and South America — bringing established SAP integration expertise closer to the region.

KaTe Group Inc. has officially been incorporated in Atlanta, Georgia. The new company presence marks an important step for KaTe: a stronger local connection for organisations, partners and SAP integration teams throughout North and South America.

KaTe has focused on SAP integrations since 2008. From its roots in Germany, the company has developed software solutions first for SAP integration landscapes — including dedicated SAP adapters and IntegraOps for integration operations. Core expertise spans SAP Integration Suite, SAP Edge Integration Cell, SAP PI/PO and SAP ERP/ABAP integration. Consulting and project services complement these products where organisations need architectural, implementation or operational support.

Software solutions for modern SAP integration. KaTe Group Inc. brings dedicated adapter products, IntegraOps for integration operations and practical SAP expertise closer to customers and partners across the Americas. Services are available where they add value — alongside a clear product focus.

A base in Atlanta for North and South America

The KaTe office is located in Atlanta Buckhead at Three Alliance Center. Atlanta is the base for the company’s activities in the Americas and supports closer collaboration across time zones, regional business contexts and individual integration roadmaps.

The goal is not simply a new address. It is to make KaTe’s software solutions, product knowledge and SAP integration experience more accessible to teams working with complex landscapes across North and South America.

Company presence
KaTe Group Inc., incorporated in Atlanta, Georgia, USA.
Regional focus
Customers, partners and the SAP community across North and South America.
Core expertise
SAP Integration Suite, SAP Edge Integration Cell, SAP PI/PO, SAP ERP/ABAP integration and dedicated SAP integration adapters.
KaTe’s solution for SAP integration operations: Monitoring, Transport Management and Disaster Recovery. Learn more about the monitoring capabilities with IntegraMon.

Software solutions for SAP integration and operations

KaTe is primarily a software solutions company. SAP integration landscapes are rarely static: teams run cloud integrations alongside on-premise systems, modernise SAP PI/PO, extend ERP processes through ABAP, connect business-critical systems and build new services on SAP BTP. KaTe develops software that helps organisations address these integration and operations requirements, with expert services available where a project needs additional support.

Solution areaWhat KaTe brings
Dedicated SAP adaptersSoftware products for connecting SAP and non-SAP systems in SAP Integration Suite, SAP PI/PO and hybrid integration scenarios.
SAP Integration SuiteIntegration solutions and adapter capabilities for cloud-based SAP integration architecture.
Edge Integration CellSupport for hybrid integration scenarios that combine cloud capabilities with local requirements.
SAP PI/POProducts and expertise for established middleware landscapes as organisations operate, modernise or evolve their integration architecture.
SAP ERP / ABAP integrationIntegration expertise for connecting SAP ERP processes, ABAP-based extensions and surrounding business systems.
IntegraOpsKaTe’s software solution for SAP integration operations: Monitoring, Transport Management and Disaster Recovery. The monitoring capabilities are available through IntegraMon.

For teams operating SAP Integration Suite and Edge Integration Cell, IntegraOps provides a product-led operations approach across integration environments. It combines Monitoring, Transport Management and Disaster Recovery; IntegraMon provides the central monitoring capabilities. Together with KaTe’s adapter portfolio, this creates a practical software foundation for connecting, operating and evolving SAP integration landscapes.

Building the next chapter together

Bernd Baier, Head of Americas, will represent KaTe from Atlanta and work with organisations across the region. Together with Sebastian Simon and Lars Albrecht, the team is looking forward to building new connections, supporting customers and contributing to the growing SAP integration community in the Americas.

The coming months will bring more conversations around SAP Integration Suite, SAP ERP/ABAP integration, Kafka, MQTT, Edge Integration Cell and the operational challenges that emerge as integration landscapes scale. KaTe looks forward to sharing product knowledge, discussing architectures and helping teams solve the integration problems that matter most to their business.

What the Atlanta presence means for SAP integration teams

For customers and partners in North and South America, KaTe Group Inc. creates a local starting point for conversations about software solutions, integration architecture, dedicated adapters, SAP ERP/ABAP integration, hybrid scenarios and IntegraOps. The experienced KaTe team remains connected across regions, while the Atlanta presence strengthens the company’s ability to engage locally.

Whether an organisation is modernising SAP PI/PO integrations, designing an Integration Suite architecture, evaluating Edge Integration Cell, connecting SAP ERP/ABAP processes or looking for more control in integration operations, the focus remains the same: fit-for-purpose software solutions, reliable operation and practical implementation support.

Frequently asked questions

Find answers to common questions about KaTe Group Inc., its SAP software solutions and its presence in the Americas.

Where is KaTe Group Inc. located in the United States?

KaTe Group Inc. is located in Atlanta Buckhead, Georgia, at Three Alliance Center.

Which regions does KaTe Group Inc. support?

The Atlanta presence supports KaTe’s work with customers, partners and the SAP community across North and South America.

What is KaTe’s primary focus?

KaTe primarily develops software solutions for SAP integration and operations. These include dedicated SAP adapters and IntegraOps. Consulting and project services complement the software portfolio where required.

Which SAP technologies are part of KaTe’s core expertise?

KaTe’s core expertise includes SAP Integration Suite, SAP Edge Integration Cell, SAP PI/PO, SAP ERP/ABAP integration and dedicated SAP integration adapters.

What is IntegraOps?

IntegraOps is KaTe’s solution for SAP integration operations. It combines Monitoring, Transport Management and Disaster Recovery. The monitoring capabilities are available through IntegraMon.

Who represents KaTe in the Americas?

Bernd Baier is Head of Americas. He works together with Sebastian Simon and Lars Albrecht as KaTe develops its presence across the region.

You can find even more news on our social media channels:

Categories
News

Scaling Kafka Consumption in SAP Integration Suite

Scaling Kafka Consumption in SAP Integration Suite

Product Update

From One Record per Message to One Batch per Run: Scaling Kafka Consumption in SAP Integration Suite

The KaTe Kafka Adapter can combine records returned by a Kafka poll into one batch payload — enabling SAP Integration Suite and Edge Integration Cell to process Kafka workloads with fewer repeated integration and network calls.

Kafka can deliver far more events than a conventional backend can process one request at a time. The bottleneck is often not Kafka itself. It is the repeated integration and network overhead created when every record starts its own SAP Integration Suite message and its own backend call.

The Sender Batch feature of the KaTe Kafka Adapter for SAP Integration Suite changes that equation. Instead of translating every polled Kafka record into a separate Cloud Integration message, the adapter can combine the records returned by a poll into one batch payload. One integration run can then validate, map and deliver many business records together.

Batching and partition parallelism complement one another: partitions provide concurrent work lanes, while batching makes each lane more efficient. The result can be substantially fewer integration and network round trips for Kafka workloads processed through SAP Integration Suite.

What changes in batch mode?

In single-record mode, 100 Kafka records create up to 100 Cloud Integration messages. In batch mode, the adapter can create one message containing up to the configured Max poll records. The backend receives fewer, larger calls, while Kafka offsets are committed only after the complete batch has been transferred successfully into Cloud Integration.

Two payload styles are available:

Simple batch format
Contains only the transformed record values under one configurable root element.
Detailed batch format
Also carries Kafka metadata such as topic, partition, key, offset, timestamp, headers and tombstone information for each record.

For continuous processing, the adapter collects available records up to the configured poll limit. If the batch is not full, it can wait for the configurable batch window before submitting the records already available. Interval processing creates one batch from the records returned by that interval’s poll.

An illustrative throughput model

Consider a topic with four partitions and enough parallel consumers to keep all four busy. Assume one single-record backend interaction takes 50 ms end to end: 25 ms of network and protocol latency plus 25 ms of backend work.

These are elapsed times including acknowledgement and commit where applicable. The model assumes four active processing lanes and excludes additional iFlow processing and batch-collection delays.

4 parallel partitions / 0.050 seconds = 80 records per second

Batching amortizes the integration and network cost. The exact gain depends on whether the backend has a true bulk operation or still performs most of its work record by record.

ModeIllustrative throughputGain vs. single record
Single record80 records/s1×
Batch of 10; whole batch handled in 50 ms800 records/s10×
Batch of 100; whole batch handled in 50 ms8,000 records/s100×
Batch of 10; 25 ms network per batch + 25 ms work per recordabout 145 records/sabout 1.82×
Batch of 100; 25 ms network per batch + 25 ms work per recordabout 158 records/sabout 1.98×

The optimistic case assumes that a backend can process the complete batch in roughly the same 50 ms as one record. The conservative case assumes that only the 25-ms network portion is shared while 25 ms of work remains for every record. Its speed-up is:

speed-up(B) = 50 × B / (25 + 25 × B) = 2B / (B + 1)

With these assumptions, eliminating repeated network round trips can approach 2× even when backend work remains linear. A real bulk API, bulk database operation or vectorised transformation can move the result much closer to the batch-size multiplier. These figures are a capacity-planning model, not a measured product benchmark; payload size, mapping cost, worker count, broker latency, retries and backend limits all affect production throughput.

Why a JDBC batch can finish close to a single insert

A 50-ms JDBC insert including network travel and commit is a plausible assumption for a remote database. A modest batch can finish close to that when fixed overhead dominates: JDBC batching reduces round trips, and one transaction shares commit overhead across rows.

For example, 49 ms fixed overhead + 1 ms per row gives 50 ms for one insert and 149 ms for 100 rows. Four concurrent lanes would then deliver approximately 2,685 records/s — 33.6× the single-record baseline. These are illustrative inputs; indexes, triggers and lock contention affect actual results. The earlier near-2× case assumes all backend work repeats per row, so it is not a general JDBC limit.

Fewer metered Integration Suite messages

Batching can also reduce SAP Integration Suite message consumption. At the time of writing, SAP’s public pricing definition counts a Cloud Integration message in 250 KB blocks — not 256 KB. An outbound message of up to 250 KB counts as one message; every additional 250 KB or fraction thereof counts as another. For synchronous receiver calls, SAP considers the request and response together, and special patterns such as splitters have separate counting rules. The exact commercial effect depends on the customer’s edition, included volume, contract and whether a message is chargeable. See the current SAP Integration Suite pricing definition and verify the deployed flow in the tenant’s Message Usage dashboard before using an estimate for budgeting.

For example, 20 individually processed records below 250 KB normally create 20 outbound backend messages and therefore at least 20 metered messages. If the same records form one outbound batch that remains at or below 250 KB — including its envelope and, for a synchronous call, its response — the batch can count as one metered message. If that batch grows to 600 KB, it counts as three rather than 20. Batching does not make payload volume free, but it removes the minimum one-message charge that otherwise applies to every small outbound call.

single-record units ≈ number of records × ceil(single outbound size / 250 KB) batch units ≈ ceil(total batched outbound size / 250 KB)

Treat this as a planning approximation. Headers, envelopes, receiver calls, synchronous responses, retries, splitters and the final deployed flow can change the metered total.

Where batching helps most

Sender Batching is particularly effective when the downstream system already supports bulk requests, database array operations, mass updates or file-oriented interfaces. It can also reduce the number of Cloud Integration message processing logs and repeated connection setups produced for the same business volume.

There is a deliberate trade-off: a batch is one processing unit. Larger batches improve amortisation but can increase per-record latency, memory use and the amount of work retried after a failure. Channel-level single-record deduplication is not available for variable batches, so downstream processing should be idempotent or deduplicate records using stable business identifiers. Batching is also not available for selective support-mode reads.

Decision matrix: should the sender use batching?

The decisive question is whether the connected backend can consume a collection efficiently. Merely wrapping single-record calls in one XML document does not create a bulk operation.

Backend or requirementFitGuidance
BI, data warehouse or data-lake ingestionStrongThese systems commonly prefer larger blocks, staged files or bulk-load APIs.
JDBC batch insert/upsert or set-based database procedureStrongUse a real bulk statement, array binding, staging table or mass-enabled procedure; looping over single inserts gives a smaller benefit.
OData $batchStrong when supportedConfirm that the service supports $batch, define whether operations belong in a transactional change set and map the Kafka batch to the required multipart request.
File, SFTP or object-storage deliveryStrongA batch maps naturally to a multi-record file when the receiver expects one; it is a poor fit if every event must remain a separate file.
Mass-enabled BAPI or SOAP operationConditionalUse batching only when the operation or a controlled wrapper accepts a table/list and defines partial-error behaviour.
Standard single-object BAPI or SOAP serviceUsually weakMany standard operations still execute one business transaction per call; batching may require a purpose-built wrapper and careful commit handling.
Per-record response, independent retry or very low latencyWeakSingle-record processing preserves granular acknowledgements and avoids waiting for a batch window.
Highly variable or very large payloadsConditionalSet conservative limits and test the combined request against CPI, adapter, backend and network size limits.
Backend cannot process collectionsDo not batchThe iFlow would have to split and loop, losing much of the throughput and metering advantage while adding complexity.

A practical starting point

Start with a moderate Max poll records value such as 10 or 25, align the number of parallel consumers with the available topic partitions and verify that the backend genuinely handles the batch efficiently. Then increase the batch size while watching end-to-end latency, metered message usage, payload size, memory, retry behaviour and backend saturation. A batch window should be long enough to form useful batches but short enough to meet the scenario’s latency target.

Batching and partition parallelism complement each other: partitions provide concurrent work lanes, while batching makes each lane more efficient. Together they let SAP Integration Suite process Kafka workloads with substantially fewer integration and network round trips.

What this update means for SAP integration teams

These concise answers preserve the technical scope of the update and make the page easier to navigate in conventional and AI-assisted search.

What is Kafka Sender Batching?

It is a KaTe Kafka Adapter feature that groups records returned by a Kafka poll into one batch payload and delivers that group as one Cloud Integration message for further processing.

Does Sender Batching guarantee a specific throughput increase?

No. The numerical examples above are capacity-planning models, not measured product benchmarks. The result depends on the bulk capability and limits of the downstream system as well as the complete deployed architecture.

What comes next?

A follow-up article will examine how Confluent Parallel Consumer can extend the sender-side design with controlled record-level concurrency, including ordering choices, offset management and the situations in which it complements — or replaces — classic partition-bound processing.

Scaling Kafka Consumption in SAP Integration Suite

You can find even more news on our social media channels:

Categories
News

Publish Many, Commit Once: Atomic Kafka Batches from SAP Integration Suite

Publish Many, Commit Once: Atomic Kafka Batches from SAP Integration Suite

Product Update

Publish Many, Commit Once: Atomic Kafka Batches from SAP Integration Suite

The KaTe Kafka Adapter can publish multiple records from one SAP Integration Suite exchange in a Kafka transaction — creating an atomic visibility boundary for consumers that use read_committed.

Publishing events one by one is simple, but it becomes expensive when an upstream application already owns a complete set of records. Every individual call repeats network, authentication, integration-flow, serialisation and acknowledgement overhead. More importantly, a partial failure can leave only part of a logical group visible in Kafka.

Atomic Receiver Batching in the KaTe Kafka Adapter for SAP Integration Suite addresses both concerns. One SAP Integration Suite exchange can publish multiple Kafka records in one Kafka transaction. A consumer configured with read_committed sees either the complete committed batch or none of an aborted batch.

The feature makes the publication contract explicit: one upstream request carries a meaningful business group, one Kafka transaction handles the complete batch, and the right consumer isolation level exposes all committed records or none of an aborted transaction.

One request, one transaction, many records

The receiver channel first parses and validates the complete batch envelope. It then borrows one producer exclusively, starts one Kafka transaction, serialises and sends every record, waits for all acknowledgements and commits the transaction. If processing fails before a confirmed commit, the transaction is aborted. Producers affected by fencing, transaction timeouts, ambiguous commit results or failed aborts are closed and recreated before reuse.

The feature supports three request styles:

Simple XML
For a sequence of value-only business elements.
Simple JSON array
For a value-only JSON batch.
Detailed XML
For per-record IDs, keys, headers, tombstones and controlled topic or partition overrides.

Every published record receives the correlation headers kate.batch.id, kate.batch.index and kate.batch.size. A detailed input can additionally provide a recordId, which becomes kate.event.id. The same batch contract can use the receiver channel’s configured Avro, JSON, JSON Schema or Protobuf value serialisation.

The performance effect of fewer integration calls

The first gain occurs before Kafka: one upstream request and one Cloud Integration exchange carry many records. To illustrate the possible scale, assume a four-partition topic with four concurrent upstream publishing lanes, an average single-record interaction of 50 ms and a 25-ms network share within that time.

Single record: 4 / 0.050 seconds = 80 records per second

If a batch of 100 can be accepted, transformed and committed in approximately the same 50 ms, the corresponding planning ceiling is 8,000 records/s, or 100× the single-record rate. That is the ideal bulk-processing case.

If the network cost is shared but 25 ms of transformation or application work remains for every item, a 100-record batch takes approximately 25 ms + 100 × 25 ms = 2,525 ms. Four lanes then deliver about 158 records/s, or 1.98×. A batch of 10 reaches about 145 records/s, or 1.82×.

ModelBatch sizeApproximate throughputGain
Single-record baseline180 records/s1×
Efficient bulk processing10800 records/s10×
Efficient bulk processing1008,000 records/s100×
Linear per-record work, shared network10145 records/s1.82×
Linear per-record work, shared network100158 records/s1.98×

This is an illustrative latency model rather than a product benchmark. On the receiver side, four Kafka partitions are distribution lanes, not a strict concurrency ceiling for a producer. Actual results depend on the configured parallel producer count, record and batch size, serialisation, broker acknowledgements, transaction latency, compression, worker-node resources and upstream concurrency.

Interpreting the 50-ms assumption

The 50 ms includes acknowledgement and commit. As with a JDBC batch, sharing fixed call and transaction overhead can make additional records relatively inexpensive, although Kafka timings must be measured separately. Even if 100 records take 100–200 ms, four active publishing lanes would deliver 2,000–4,000 records/s, or 25–50× the single-record baseline. These are illustrative scenarios; compare equivalent durability settings and include batch-collection time when measuring end-to-end latency.

One exchange can also reduce message consumption

Atomic Receiver Batching can reduce SAP Integration Suite message consumption as well as technical overhead. SAP’s current public pricing definition meters Cloud Integration messages in 250 KB blocks — not 256 KB: an outbound message up to 250 KB counts as one message, and each further 250 KB or fraction counts as another. For synchronous receiver calls, request and response sizes are considered together. Contract terms, included volume, SAP-to-SAP exclusions, retries and the structure of the deployed flow determine whether a reduction changes the actual bill. Refer to the current SAP Integration Suite pricing definition and verify the result in the tenant’s Message Usage dashboard.

As a simplified example, publishing 20 small records through 20 separate receiver exchanges normally produces at least 20 metered outbound messages. If one receiver exchange carries the complete atomic batch and its metered outbound size remains at or below 250 KB, it can count as one. A 600-KB batch counts as three 250-KB units rather than 20. Envelope data, replies, retries, splitters and other receiver calls must be included when estimating a real flow.

Atomic visibility, not exactly-once business processing

The delivery contract is atomic and at least once. Transactional visibility prevents a read_committed consumer from seeing a partial batch. It does not prevent an upstream retry from publishing the complete batch again as a new transaction after an uncertain client-side result. Downstream consumers should therefore use the batch ID, batch index, optional event ID or a stable business key for idempotent processing where duplicates matter.

Consumers that must honour the atomic boundary should select Read committed. A read_uncommitted consumer can observe records from open or aborted transactions and is unsuitable when the all-or-nothing contract is required.

Decision matrix: should the receiver publish atomically in batches?

Scenario or requirementFitGuidance
Upstream system already sends a coherent business groupStrongPreserve the group as one exchange and one Kafka transaction.
Bulk API, BI load, JDBC batch, OData $batch or multi-record file feeds KafkaStrongAvoid splitting a collection into many CPI exchanges before publishing.
Downstream consumers use read_committed and understand batch correlationStrongThey receive all committed records or none of an aborted transaction.
Per-record publication receipt is requiredStrongSelect the detailed reply to return topic, partition, offset and timestamp for every item.
Independent records with strict low-latency acknowledgementsWeakWaiting to assemble a batch adds latency and couples otherwise unrelated records.
A standard BAPI or SOAP operation produces only one object per callConditionalBatch publication helps only after the upstream side has intentionally assembled a meaningful collection.
Downstream consumers use read_uncommittedDo not use for atomic visibilitySuch consumers may observe records from open or aborted transactions.
End-to-end exactly-once business processing is requiredInsufficient on its ownKafka atomic visibility does not prevent a complete upstream retry from producing the batch again; keep processing idempotent.
One invalid record should not affect neighbouring recordsWeakAtomic publication deliberately aborts the whole batch; use smaller batches or single-record publication for error isolation.
Very large or highly variable recordsConditionalTest transaction timeouts and CPI, producer, broker and request-size limits with production-like data.

A receipt for every publication

The receiver can return no body, a summary or detailed record metadata. A summary confirms the generated batch ID, COMMITTED status and record count. A detailed reply additionally reports the topic, partition, offset and timestamp for every record, providing a precise publication receipt. Replies can be XML, JSON or match the request family.

KaTe Launchpad 1.0.6 or later can generate matching sender, receiver or interoperability batch contracts from the value schema. This avoids hand-maintained envelope definitions and keeps XML WSDL/XSD or JSON OpenAPI contracts aligned with the configured request and reply modes.

When to use atomic receiver batches

The feature is a strong fit for order groups, master-data packages, synchronised state changes or any upstream bulk interface where partial Kafka visibility would be misleading. Begin with batches that comfortably fit the broker and CPI payload limits, configure a transaction timeout that covers transformation, acknowledgement and commit, and increase Number of producers per worker node only when independent batches need to publish concurrently.

The result is more than fewer calls. It is a clear integration contract: the upstream system submits one logical group, Kafka exposes that group atomically, and the caller can receive a record-level publication receipt.

What this update means for SAP integration teams

These concise answers retain the technical boundaries of the update and improve retrieval in conventional and AI-assisted search.

What is Atomic Receiver Batching?

It publishes multiple records from one SAP Integration Suite exchange within one Kafka transaction, with simple and detailed input formats plus optional record-level publication evidence.

How do consumers preserve the atomic boundary?

Consumers that require all-or-none visibility must use read_committed and should use the batch correlation headers or stable business keys when idempotent processing is needed.

Is the measured throughput guaranteed?

No. The figures are illustrative planning scenarios. Validate performance against the specific producer settings, payloads, durability requirements, transaction timeouts, broker environment and upstream concurrency.

Atomic Kafka Batches from SAP Integration Suite

You can find even more news on our social media channels:

Categories
News

Kafka via SAP Cloud Connector

Kafka via SAP Cloud Connector (SCC)

Product Update

Kafka via SAP Cloud Connector: Secure Hybrid Access Without Compromises

The KaTe Kafka Adapter now connects private Kafka infrastructure through SAP Cloud Connector while preserving full TLS, SNI and hostname verification.

KaTe GmbH has expanded the KaTe Kafka Adapter for SAP Integration Suite with secure private-network connectivity through SAP Cloud Connector. The update enables SAP Cloud Integration and SAP Edge Integration Cell to access private Kafka environments without changing broker advertisements or weakening TLS security.

Kafka brokers, Schema Registry services and OAuth 2.0 identity providers can each use SAP Cloud Connector or the public network independently. Therefore, hybrid Kafka architectures can follow the network and security design that fits each individual service.

Secure hybrid Kafka connectivity for SAP Integration Suite

For organisations that run Apache Kafka in hybrid environments, private Kafka brokers, Schema Registry services and OAuth 2.0 identity providers can now be reached through SAP Cloud Connector where required. At the same time, each endpoint can continue to use the public network independently when that is the right architectural choice.

This matters because Kafka landscapes rarely follow a one-size-fits-all network model. An organisation may operate private Kafka brokers while using a public identity provider, or it may keep its Schema Registry in a separate network zone. The KaTe Kafka Adapter for SAP Integration Suite supports these combinations without forcing every Kafka-related service through the same connection path.

Apache Kafka brokers
SAP Cloud Connector or public network, with full TLS, SNI and hostname verification.
Schema Registry
SAP Cloud Connector or public network, independently configurable from broker access.
OAuth 2.0 identity provider
SAP Cloud Connector or public network, independently configurable from Kafka and Schema Registry access.

No custom broker advertisement rewriting and no TLS compromises

A frequent challenge in hybrid Kafka deployments is handling the broker addresses advertised to clients. The updated adapter avoids custom broker advertisement rewriting for SAP Cloud Connector connectivity. Consequently, teams do not need to introduce an additional address-translation layer simply to establish secure access from SAP Integration Suite to private Kafka services.

Equally important, the connection preserves TLS, Server Name Indication (SNI) and hostname verification. Established certificate validation therefore remains in place instead of being bypassed to make a hybrid connection work.

What this update clarifies for SAP integration teams

The following points are part of this product announcement. They explain the practical impact of the update rather than presenting a separate FAQ.

Can SAP Integration Suite reach a private Kafka broker through SAP Cloud Connector?

Yes. The KaTe Kafka Adapter can connect SAP Cloud Integration to private Kafka brokers through SAP Cloud Connector. The same capability is available for SAP Edge Integration Cell, enabling hybrid integration architectures that combine cloud-based integration with private Kafka services.

Do Kafka, Schema Registry and OAuth 2.0 identity providers have to use the same network route?

No. Each endpoint can be configured independently. Kafka brokers, Schema Registry services and OAuth 2.0 identity providers can use SAP Cloud Connector or the public network according to the organisation’s network architecture and security requirements.

Does private connectivity require weaker TLS validation or custom address rewriting?

No. The integration preserves full TLS, SNI and hostname verification. It also does not require custom broker advertisement rewriting for the SAP Cloud Connector connection path.

A practical option for SAP Cloud Integration and Edge Integration Cell

The enhancement is designed for SAP integration teams that need to connect SAP systems with event-driven Kafka platforms across cloud and private network boundaries. Whether an architecture uses SAP Cloud Integration, SAP Edge Integration Cell or a combination of both, the adapter enables a clear separation between integration logic and network-routing decisions.

The result is true hybrid Kafka connectivity: private where necessary, public where appropriate and consistently secured throughout.

You can find even more news on our social media channels:

Categories
SAP Products

Kafka Adapter for SAP Integration Suite

Kafka Adapter for SAP Integration Suite (CPI) – KaTe GmbH
SAP Integration Suite (CPI)

Kafka Adapter for
SAP Integration Suite

Bidirectional connection between SAP Cloud Integration Iflows and Apache Kafka – the logical successor to the proven KaTe Kafka Adapter for SAP PO.

Verified SAP Partner Free 30-Day Trial
Architecture Overview
SAP Cloud
Integration
Iflow
bidirektional
KaTe
Kafka Adapter
CPI
Publish / Subscribe
Apache Kafka
On-Premise / BTP
Avro Protocol Buffers OAuth2 SASL_SSL Schema-Registry Batch Processing CSFLE / CSPE
Verified SAP Partner
Free 30-Day Trial
SAP Roadmap Compatible

All Features of the Kafka Adapter at a Glance

The KaTe Kafka Adapter for SAP Integration Suite offers much more than the SAP standard adapter – from Avro support to Field-Level Encryption.

Core Functions: Publish & Subscribe
Publish SAP Cloud Integration Iflow messages as Kafka records and subscribe to Kafka records as Iflow messages. Kafka Key, Header, and Data are translated into Iflow messages and vice versa.
  • Topics and partitions flexibly configurable
  • Ack mode and consumer commits customizable
  • Fully integrated into SAP Cloud Integration
  • Observable at channel and message log level
Enterprise-Grade Security
All common Kafka security standards are supported – including advanced encryption at the field and payload level for regulated environments.
  • PLAIN, PLAIN_SASL, SASL_SSL
  • SASL-Plain / SASL-SCRAM
  • OAuth2 & Client Certificate Authentication
  • Field-Level Encryption (CSFLE) & Payload Encryption (CSPE)
  • SOC 2, GDPR, ISO 27001 compliant
High Performance & Flow Control
Full control over message throughput, filtering, and batch processing – for optimal performance with high data volumes.
  • Filtering by headers, keys, and data content
  • Batch processing: n Kafka records as one Iflow message
  • Polling and record count/time unit configurable
  • Processing from the beginning, end, or any stream position

Advanced Features: Error Handling, Formats & Migration

Automated & Manual Error Handling
In "in order" streaming systems, errors can stop the entire processing. To prevent this, our adapter offers both manual and fully automated error handling.
  • Error routing to an error destination
  • Documentation and ignoring of specific cases
  • Error handling even for deserialization errors
  • Deduplication via offset or record header
Avro, Protocol Buffers & Schema Registries
Binary data formats like Avro and Protocol Buffers are standard in Kafka environments. Therefore, our adapter fully supports them – including automatic schema generation via schema registries.
  • Avro and Protocol Buffers ↔ XML conversion
  • JSON ↔ XML transformation
  • Schema Registry integration
  • Integrated schema generation tool
SAP Roadmap Compatible
Because the adapter is available for both SAP Process Orchestration and the SAP Integration Suite, existing SAP PO integrations can be transferred with minimal effort.
  • Available for SAP PO and SAP Integration Suite
  • Import existing interfaces without refactoring
  • Determine your own migration plan and schedule
  • Investments in SAP PO integrations are preserved

Why the KaTe Adapter
Can Do More

The KaTe Kafka Adapter offers crucial features that the SAP standard Kafka adapter in SAP Cloud Integration does not provide.

1
Arbitrary Stream Positions & Group Names
While the SAP standard connector does not support arbitrary positions or group names in the stream, our Kafka connector fully provides this feature, which is crucial in daily operations.
2
All Data Formats incl. Schema Registry
Avro, Protocol Buffers, JSON, and schema registries with automatic schema generation are fully supported – although the SAP standard connector has significant gaps here.
3
Batch Processing in One Iflow Execution
Multiple Kafka records can be bundled as a single Iflow message, significantly increasing throughput. Since the SAP standard connector does not offer this function, this is a decisive advantage.
4
Advanced Error Handling & Encryption
Both manual and automated error handling – including deserialization errors – as well as Field-Level and Payload Encryption (CSFLE/CSPE) are integrated, whereas the SAP standard connector does not provide these features.
5
Private and Public Network Access
The adapter runs equally on SAP Cloud Integration and SAP Edge Integration Cell (EIC). It also supports hybrid scenarios, including access to on-premises Kafka brokers, schema registries, and OAuth 2.0 identity providers through SAP Cloud Connector.

Kafka Adapter Comparison:
KaTe vs. SAP Standard

Feature KaTe Adapter SAP Standard
Arbitrary Stream Positions
✓ Yes
✗ No
Avro / Protocol Buffers
✓ Yes
✗ No
Schema-Registry
✓ Yes
✗ No
Batch Processing
✓ Yes
✗ No
Field-Level Encryption
✓ Yes
✗ No
Advanced Error Handling
✓ Yes
Limited
Filtering & Deduplication
✓ Yes
✗ No
OAuth2 & CSFLE/CSPE
✓ Yes
✗ No
SAP Cloud Connector Support
✓ Yes
✗ No

Set Up Kafka Adapter – Start in 3 Steps

1
Request Free Trial

Contact us – within 48 hours you will receive access to the adapter for your test environment.

2
Install Adapter

Installation in your SAP Integration Suite is completed in just a few steps, as our team guides you through the onboarding process.

3
Go Live

Initial productive scenarios start quickly, allowing you to scale as needed – without upfront investments.

Compatibility: Supported Kafka Platforms & Protocols

Whether on-premise, cloud, or hybrid – the KaTe Kafka Adapter for SAP Integration Suite is compatible with all common Kafka deployments and SAP platforms.

SAP Integration Suite
SAP Cloud Integration (CPI) – fully integrated. SAP Docs ↗
Supported
SAP BTP
SAP Business Technology Platform – Cloud-native
Supported
Apache Kafka
On-premise Kafka clusters of all common versions. Apache Kafka ↗
Supported
Confluent Platform
Confluent Cloud and Confluent Platform on-premise. Confluent ↗
Supported
Azure Event Hubs
Kafka-compatible Azure Event Hubs endpoint
Supported
AWS MSK
Amazon Managed Streaming for Apache Kafka
Supported
SAP Edge Integration Cell
Hybrid on-premise scenarios with EIC
Supported
SAP Cloud Connector
Access on-premises Kafka brokers and schema registries through SAP Cloud Connector (SCC).
Supported

KaTe Launchpad – More Control During Setup and Operations

The KaTe Launchpad extends the Kafka Adapter with an integrated toolbox for developers and operations teams, so that all key functions are accessible directly from the SAP Integration Suite.

Generator

Kafka configurations, topic structures, and SAP Integration Suite Iflow channel definitions are automatically generated, significantly reducing configuration effort and error sources.

Service List Display & Tester

The Service List Display provides an overview of all configured Kafka topics, consumer groups, and broker connections, while the integrated tester enables sending and receiving Kafka messages directly from the SAP Integration Suite.

SAP Technology & Migration Consulting –
Future-Proof Integration

KaTe GmbH is your expert partner for SAP technology and migration projects. Our focus is on the secure and seamless transformation of existing SAP PO landscapes to the modern SAP Integration Suite – including the SAP Edge Integration Cell for hybrid integration scenarios.

  • Migration from SAP PO to SAP Integration Suiteincluding analysis, architecture design, and technical implementation
  • Implementation and Optimization of the SAP Edge Integration Cellfor flexible and secure hybrid integration
  • SAP BTP (Business Technology Platform) ConsultingLeveraging modern services and tools for innovation and process automation
  • Technology-Based Strategy Consultingfrom cloud integration to API management
SAP PO → Integration Suite Migration
Structured Migration Without Downtime

We analyze your existing interfaces in SAP PO and develop a structured migration plan for the SAP Integration Suite, so the transition succeeds without operational disruption.

Learn More
SAP PO Adapter – Free 30-Day Trial
All SAP PO Adapters – No Upfront Payment

Try our SAP PO Adapters for Kafka, MQTT, AMQP, AWS, Azure, GCP, GraphQL, gRPC, and RESTful – free for 30 days.

View All SAP PO Adapters
IntegraMon – SAP Monitoring
Central Dashboard for All SAP CPI Tenants & EIC Instances

IntegraMon unifies all SAP CPI tenants and Edge Integration Cells in a single dashboard – with proactive alerting, AI support, and automated KPI reports.

Discover IntegraMon

Frequently Asked Questions

Answers to the most important questions about the KaTe Kafka Adapter for SAP Integration Suite.

What is the Kafka Adapter for SAP Integration Suite?

As a powerful middleware component, the Kafka Adapter for SAP Integration Suite connects SAP systems bidirectionally with Apache Kafka – both cloud-native and in hybrid scenarios. Since it reliably processes event-driven data streams and enables message exchange between SAP Cloud Integration Iflows and Kafka topics, it is the ideal integration layer for modern SAP landscapes. As the successor to the KaTe Adapter for SAP Process Orchestration (SAP PO), it also offers state-of-the-art features for seamless SAP integration.

What core features does the Kafka Adapter for SAP Integration Suite offer?

With the adapter, SAP Cloud Integration Iflow messages can be published as Kafka records, and Kafka records can be subscribed to as Iflow messages. Kafka keys, headers, and payloads are automatically translated into Iflow messages – and vice versa. In addition, the following core features are available:

  • Full support for Avro, Protocol Buffers, and JSON, including schema registry and automatic schema generation
  • Topics, partitions, and reliability settings (ack mode, consumer commits) are flexibly configurable
  • Batch processing: n Kafka records as one Iflow message
  • Filtering by headers, keys, and data content
  • Fully integrated into SAP Cloud Integration, observable and debuggable
What security standards are supported?

Because regulated environments have special requirements, the Kafka Adapter for SAP Integration Suite meets the highest security standards. Specifically, the following protocols and methods are supported:

  • Kafka Security Standards: PLAIN, PLAIN_SASL, SASL_SSL, SASL-Plain/SASL-SCRAM
  • OAuth2 and client certificate authentication for broker and service registry connections
  • Field-Level Encryption (CSFLE) and Payload Encryption (CSPE) for sensitive data
  • Compliance-ready for SOC 2, GDPR, and ISO 27001

Technical Details & Comparison

How does the adapter support high performance and error handling?

Thanks to integrated batch processing, filtering, and flow control, the adapter delivers high-performance streaming even at high data volumes, so that even large amounts of data are reliably processed. Messages can be filtered by headers, keys, or data content and bundled into optimized batches.

For stable data processing, both automated and manual error handling are available. Serialization errors, data transformation errors, or Iflow errors can be forwarded to an error destination or deduplicated, so that continuous operation is always guaranteed.

How does the KaTe Adapter differ from the SAP standard Kafka adapter?

While the SAP standard Kafka adapter covers only basic functions, the KaTe Kafka Adapter offers significantly more flexibility and depth. Specifically, these include:

  • Support for arbitrary stream positions and freely definable group names
  • Compatibility with Avro, Protocol Buffers, JSON, and schema registries with automatic schema generation
  • Batch processing in a single Iflow execution
  • Advanced error handling – both manual and automated – including for deserialization errors
  • Filtering and deduplication for higher data quality and optimized data throughput
  • Field-Level and Payload Encryption (CSFLE/CSPE)
  • Flexible private network access to Kafka brokers, schema registries, and OAuth 2.0 identity providers through SAP Cloud Connector.
How does SAP Cloud Connector support hybrid Kafka integration?

The KaTe Kafka Adapter supports SAP Cloud Connector for hybrid integration scenarios. It enables SAP Cloud Integration and SAP Edge Integration Cell to reach on-premises Kafka brokers, schema registries, and OAuth 2.0 identity providers without exposing them directly to the public internet.

Is the KaTe Kafka Adapter SAP Roadmap Compatible?

Yes, the KaTe Kafka Adapter is available for both SAP Process Orchestration and the SAP Integration Suite. Existing SAP PO integrations can therefore be transferred to the Integration Suite without significant migration effort, adjustments, or refactoring.

You determine your own timeline and roadmap. Because your investments in SAP PO integrations are fully preserved, neither unforeseen risks nor commercial follow-up questions arise.

Get Started Now

Ready for Kafka in the SAP Integration Suite?

Contact us for a free 30-day trial or an individual consultation. We set up your trial within 48 hours.