https://linktr.ee/eyalestrinhttps://security-24-7.com

An engineer's practical guide to serialization, schema evolution, and reliable event-driven systems using Java, Apache Kafka, and Schema…

Topics, partitions, consumers, and the core ideas interviewers expect you to know.

AWS Lambda event source mappings now support queue consumption mode for Apache Kafka. Using Kafka Share Groups, you can scale consumer parallelism independently of partition count, with per-record acknowledgment, visibility timeouts, and automatic retries on self-managed Kafka 4.2 or later clusters.

While exploring the latest AWS updates, I came across a really interesting improvement for Kafka + Lambda.

AWS Lambda now supports Apache Kafka queues for self-managed Kafka event sources, enabling higher parallelism and throughput beyond partition count limits.

AWS Lambda now supports queue consumption mode for Apache Kafka, enabling parallel processing of records without partition count limitations using Kafka Share Groups.

Topics, partitions, consumers, and the core ideas interviewers expect you to know.

Aiven provides managed open source data technologies on all major clouds.

A detailed guide to Apache Kafka covering topics, partitions, brokers, producers, consumers, replication, delivery guarantees, KRaft, and the ecosystem, with code examples and practical tips.

AWS Lambda now supports queues for Apache Kafka in self-managed Kafka event source mappings (ESM), for Kafka clusters that customers run themselves and for managed offerings such as Confluent Cloud. Customers can now scale processing for Kafka workloads beyond the partition count of a topic, increasing throughput during traffic spikes and improving processing resilience. Customers building event-driven Kafka workloads for use cases such as image and video encoding, fraud detection, notification delivery, and telemetry ingestion treat each event as an independent unit of work, where processing speed matters more than strict ordering. Previously, Kafka ESMs used the Kafka consumer group protocol, which assigns each partition to exactly one consumer within a consumer group. Customers needing higher parallelism had to over-partition topics or move traffic to a separate queuing system, adding infrastructure to operate and producer code to maintain. Queue support for self-managed Kafka ESMs is built on Kafka share groups, which are generally available in Apache Kafka 4.2. This launch enables multiple event pollers to consume concurrently from the same partition, with each event poller supporting up to 10 concurrent invocations. This decouples parallelism from partition count with no changes to producers, topic configuration, or function code. As a result, customers get higher throughput during traffic spikes and more resilient processing, because a slow or failed event no longer holds up other events in the same partition. Queue mode does not preserve partition ordering and is intended for workloads that do not require it. This capability requires a Kafka cluster with share groups enabled, such as Apache Kafka 4.2 or later, and is available in all AWS commercial Regions where self-managed Kafka ESM is available. To get started, create a self-managed Kafka ESM with ConsumptionMode set to Queue through the AWS Management Console, Lambda API, AWS CLI, AWS CloudFormation, AWS SAM, or AWS CDK. Standard Lambda pricing applies. Customers who configure provisioned mode for their self-managed Kafka ESM also pay for Event Poller Unit (EPU) usage. To learn more, see the AWS Lambda Developer Guide and AWS Lambda pricing.    

AWS Lambda now supports https://cwiki.apache.org/confluence/spaces/KAFKA/pages/255070434/KIP-932+Queues+for+Kafka in https://docs.aws.amazon.com/lambda/latest/dg/with-kafka.html (ESM), for Kafka clusters that customers run themselves and for managed offerings such as Confluent Cloud. Customers can now scale processing for Kafka workloads beyond the partition count of a topic, increasing throughput during traffic spikes and improving processing resilience. Customers building event-driven Kafka workloads for use cases such as image and video encoding, fraud detection, notification delivery, and telemetry ingestion treat each event as an independent unit of work, where processing speed matters more than strict ordering. Previously, Kafka ESMs used the Kafka consumer group protocol, which assigns each partition to exactly one consumer within a consumer group. Customers needing higher parallelism had to over-partition topics or move traffic to a separate queuing system, adding infrastructure to operate and producer code to maintain. Queue support for self-managed Kafka ESMs is built on Kafka share groups, which are generally available in Apache Kafka 4.2. This launch enables multiple event pollers to consume concurrently from the same partition, with each event poller supporting up to 10 concurrent invocations. This decouples parallelism from partition count with no changes to producers, topic configuration, or function code. As a result, customers get higher throughput during traffic spikes and more resilient processing, because a slow or failed event no longer holds up other events in the same partition. Queue mode does not preserve partition ordering and is intended for workloads that do not require it. This capability requires a Kafka cluster with share groups enabled, such as Apache Kafka 4.2 or later, and is available in all https://builder.aws.com/build/capabilities where self-managed Kafka ESM is available. To get started, create a self-managed Kafka ESM with ConsumptionMode set to Queue through the AWS Management Console, Lambda API, AWS CLI, AWS CloudFormation, AWS SAM, or AWS CDK. Standard Lambda pricing applies. Customers who configure https://docs.aws.amazon.com/lambda/latest/dg/kafka-scaling-modes.html#kafka-provisioned-mode for their self-managed Kafka ESM also pay for Event Poller Unit (EPU) usage. To learn more, see the https://docs.aws.amazon.com/lambda/latest/dg/polling-kafka-queue.html and https://aws.amazon.com/lambda/pricing/.    

AWS Lambda now supports queues for Apache Kafka in self-managed Kafka event source mappings (ESM), for Kafka clusters that customers run themselves and for managed offerings such as Confluent Cloud. Customers can now scale processing for Kafka workloads beyond the partition count of a topic, increasing throughput during traffic spikes and improving processing resilience. Customers building event-driven Kafka workloads for use cases such as image and video encoding, fraud detection, notification delivery, and telemetry ingestion treat each event as an independent unit of work, where processing speed matters more than strict ordering. Previously, Kafka ESMs used the Kafka consumer group protocol, which assigns each partition to exactly one consumer within a consumer group. Customers needing higher parallelism had to over-partition topics or move traffic to a separate queuing system, adding infrastructure to operate and producer code to maintain. Queue support for self-managed Kafka ESMs is built on Kafka share groups, which are generally available in Apache Kafka 4.2. This launch enables multiple event pollers to consume concurrently from the same partition, with each event poller supporting up to 10 concurrent invocations. This decouples parallelism from partition count with no changes to producers, topic configuration, or function code. As a result, customers get higher throughput during traffic spikes and more resilient processing, because a slow or failed event no longer holds up other events in the same partition. Queue mode does not preserve partition ordering and is intended for workloads that do not require it. This capability requires a Kafka cluster with share groups enabled, such as Apache Kafka 4.2 or later, and is available in all AWS commercial Regions where self-managed Kafka ESM is available. To get started, create a self-managed Kafka ESM with ConsumptionMode set to Queue through the AWS Management Console, Lambda API, AWS CLI, AWS CloudFormation, AWS SAM, or AWS CDK. Standard Lambda pricing applies. Customers who configure provisioned mode for their self-managed Kafka ESM also pay for Event Poller Unit (EPU) usage. To learn more, see the AWS Lambda Developer Guide and AWS Lambda pricing.