Kafka as a Message Queue: can you do it, and should you do it?
|Big Data & Machine Learning||Beginner/novice Level|
|Wednesday from 16:40 til 17:30|
Using Kafka's offset commit mechanism we can implement a message processing system with at-least-once delivery guarantees. However, we can only acknowledge processing of all messages up to a given point (offset). That’s very often enough, but not always.
Sometimes we need selective, out-of-order message acknowledgments, like in a “traditional” message queue. If a message is not acknowledged for a given period of time, it should be re-delivered. Can this be implemented on top of Kafka? Sure! (By the way: this is similar to how Amazon’s SQS works.)
In the talk I’ll describe the architecture and implementation of a message queue built on Kafka: kmq. We’ll go through two crucial components: the queue client and the message redelivery tracker. There will be some live coding, some slides, and a couple of demos.
We’ll also look at the performance (which is surprisingly good) & latency, as well as possible problems that using such an approach can cause, such as “error-flooding”.
Kmq is open-source and available at https://github.com/softwaremill/kmq.messaging Java benchmarks
I am one of the co-founders of SoftwareMill, where I code mainly using Scala and other interesting technologies. I am involved in open-source projects, such as Macwire, Supler, ElasticMQ and others. I have been a speaker at major conferences, such as JavaOne, Devoxx and ScalaDays.
Apart from writing closed- and open-source software, in my free time I try to read the Internet on various (functional) programming-related subjects, any ideas or insights usually end up on my blog: http://www.warski.org/blog