Unlocking Kafka Exactly-one Semantic

How to implement “Exactly-one Semantics” at Kafka side. Before that you, you have to check Kafka version:

If its Kafka 0.11 release then you can try enabling Exactly-once semantics feature. Prior to 0.11 Kafka support at least one delivery but that can cause duplicate results

Exactly-once one semantics has been implemented using concept. Idempotence and Transaction Management features

How to enable Exactly-once Semantics

In Producer config, please add/modify below property

enable.idempotence = True

acks=all

retries=Integer.MAX_VALUE

max.inflight.requests.per.connection=1

transactional.id =  < some unique id>

In Consumer config, please add/modify below property

isolation.level =  <read_committed or read_uncommitted>

References:

https://cwiki.apache.org/confluence/display/KAFKA/KIP-98+-+Exactly+Once+Delivery+and+Transactional+Messaging

https://cwiki.apache.org/confluence/display/KAFKA/KIP-129%3A+Streams+Exactly-Once+Semantics

https://www.confluent.io/blog/exactly-once-semantics-are-possible-heres-how-apache-kafka-does-it/