HPP-0456 Kafka as a Message Queue: can you do it, and should you do it? | Devoxx

Kafka as a Message Queue: can you do it, and should you do it?


track.6 Big Data & Machine Learning Beginner/novice Level

Room 6

Wednesday from 16:40 til 17:30

Using Kafka's offset commit mechanism we can implement a message processing system with at-least-once delivery guarantees. However, we can only acknowledge processing of all messages up to a given point (offset). That’s very often enough, but not always.

Sometimes we need selective, out-of-order message acknowledgments, like in a “traditional” message queue. If a message is not acknowledged for a given period of time, it should be re-delivered. Can this be implemented on top of Kafka? Sure! (By the way: this is similar to how Amazon’s SQS works.)

In the talk I’ll describe the architecture and implementation of a message queue built on Kafka: kmq. We’ll go through two crucial components: the queue client and the message redelivery tracker. There will be some live coding, some slides, and a couple of demos.

We’ll also look at the performance (which is surprisingly good) & latency, as well as possible problems that using such an approach can cause, such as “error-flooding”.

Kmq is open-source and available at https://github.com/softwaremill/kmq.

 messaging    Java    benchmarks  
Adam Warski Adam Warski

I am one of the co-founders of SoftwareMill, where I code mainly using Scala and other interesting technologies. I am involved in open-source projects, such as Macwire, Supler, ElasticMQ and others. I have been a speaker at major conferences, such as JavaOne, Devoxx and ScalaDays.

Apart from writing closed- and open-source software, in my free time I try to read the Internet on various (functional) programming-related subjects, any ideas or insights usually end up on my blog: http://www.warski.org/blog

Floor plans