How to implement
“Exactly-one Semantics” at Kafka side. Before that you, you have to check Kafka
version:
If its Kafka 0.11 release then
you can try enabling Exactly-once semantics feature. Prior to 0.11 Kafka
support at least one delivery but that can cause duplicate results
Exactly-once one semantics has
been implemented using concept. Idempotence and Transaction Management features
How to enable Exactly-once
Semantics
In Producer config, please
add/modify below property
enable.idempotence = True
acks=all
retries=Integer.MAX_VALUE
max.inflight.requests.per.connection=1
transactional.id = <
some unique id>
In Consumer config, please
add/modify below property
isolation.level =
<read_committed or read_uncommitted>
References:
https://cwiki.apache.org/confluence/display/KAFKA/KIP-98+-+Exactly+Once+Delivery+and+Transactional+Messaging
https://cwiki.apache.org/confluence/display/KAFKA/KIP-129%3A+Streams+Exactly-Once+Semantics
https://www.confluent.io/blog/exactly-once-semantics-are-possible-heres-how-apache-kafka-does-it/
