06 May 2017

Redis, Apache Kafka & RabbitMQ - When to use what

I recently had to present a design where the designed consisted of Redis, Apache Kafka and RabbitMQ, among other things of course. At the presentation the obvious question came up, can't we just use one of these?

I understand that for novice users visiting the respective websites it must be difficult to understand the differences. On the front page on the redis webpage they state 

Redis is an open source (BSD licensed), in-memory data structure store, used as a database, cache and message broker.

On the Apache Kafka webpage you'll see

Publish & Subscribe to streams of data like a messaging system

and finally on RabbitMQ

RabbitMQ is the most widely deployed open source message broker 

 

"Message" seems to be the keyword for all of them but that doesn't tell the full story. Let's have a look at the details and example scenarios where one would choose one over the other.

Redis

The main thing that redis does is being a blazing fast in-memory data structure store and as such it's mainly seen in caching related scenarios. After release 3 however, many features were added and redis now is also wildly used as a publish/subscribe messaging system where it crosses into RabbitMQ territory.

On this site I use redis for caching the responses. If a user visits a newly created URL that nobody has visited before it will be served by the DB but the HTML response will be cached in redis. Next time somebody visits the same URL the request is served from cache without hitting the DB.

As mentioned before, redis is also widley used as a publish/subscribe messaging system. I have seen it being utilized in simple chat applications as well as a message broker for distributed tasks queues.

Apache Kafka

The main thing kafka does is provide you with the ability to perform real-time processing of streaming data. What is streaming data? Syslog messages send to you, stock price updates send to you, any data clients might generate that your servers receive can be thought of as streaming data.
Like redis, kafka is also blazing fast but it is limited to the I/O of the hard drive however, persistence is higher with Kafka. Kafka can store data on multiple machines in a fault tolerant manner and supports automatic recovery when machine failures occur.

Kafka design is build around the producer-consumer model. Producers write to topics, many topics can be written to a partition and consumers consume these topics. Queuing and publish/subscribe are traditionaly the main two models in messaging and Kafka is flexible enough to incorperate both of them.

RabbitMQ

RabbitMQ provides a fault tolerant messaging queue. It can be used to build publish/subscribe and work queuing solutions. RabbitMQ supports message acknowledgments. An acknowledgement is sent back from the consumer to tell RabbitMQ that a particular message had been received, processed and that RabbitMQ is free to delete it.

If a consumer dies (its channel is closed, connection is closed, or TCP connection is lost) without sending an ACK, RabbitMQ will understand that a message wasn't processed fully and will re-queue it. If there are other consumers online at the same time, it will then quickly redeliver it to another consumer. That way you can be sure that no message is lost, even if the workers occasionally die.

Redis vs RabbitMQ

Redis main application is in memory storage. For pub/sub related applications I would prefer RabbitMQ over Redis as you get persistence, at least once delivery guarantees and complex topic based routing features out of the box.

Redis vs Kafka

I can see a use case where one could use redis in memory storage for ingestion of streaming short lived data with high consumer capacity on the backend. If you go this route make sure to keep an eye on memory usage. If consumer capacity is a limit I would advice to use kafka since kafka is built to accept and persist vast sums of data regardless of any consumers.

RabbitMQ vs Kafka

The same applies here as with redis. RabbitMQ was not designed for streaming message ingestion. If you want to use it as such make sure to have enough consumer capacity on the backend and preferably really fast ones. If the message queue grows to large RabbitMQ will stop responding which will lead to problems. Kafka on the other hand was specifically designed for this purpose. Kafka also makes it easy for multiple consumers to consume the same topic. If you encountered problems later on in your application you can playback a topic and let the consumer do its work again. With RabbitMQ after it receives an ACK the message is deleted and will not be seen again in the queue.

Summary table

 RedisKafkaRabbitMQ
Streaming data X 
ETL X 
Message Routing  X
Topics XX
HA Storage XX
Chat applicationX X
High throughputXX 

 

14  question_answer

  • said on 24 March 2018 22:59

    What’s wrong with using Kafka for chat apps?

  • said on 15 April 2019 21:21

    Deltasone Propecia Farmaci <a href=http://ausgsm.com></a> Cephalexin For Dogs Side Effects

  • said on 07 May 2019 02:26

    Keflex Antibiotic Structure Precio De La Viagra En Farmacias <a href=http://crdrugs.com>cialis overnight shipping from usa</a> Levaquin Discount Overseas Opinion Levitra

  • said on 01 August 2019 15:50

    Acheter Cytotec En France Comprare Cialis Online Sicuro Achat Kamagra Pharmacie Suisse <a href=http://cialcheap.com>cialis online</a> buy accutane online canada

  • said on 06 August 2019 23:36

    Buy Prednisone For Dogs Online Bacteria Amoxicillin Destroys <a href=http://cialusa.com>cialis tablets for sale</a> Levitra Online Mieten Kamagra And Hiv Dosage Of Keflex For Children

  • said on 14 March 2020 15:16

    Discount Generic Fluoxetine Internet In Germany <a href=http://apcialisle.com/#>cialis no prescription</a> Cialis Immagini <a href=http://apcialisle.com/#>Cialis</a> Gasex

  • said on 14 March 2020 22:55

    Buy Tadalafil Generic <a href=http://apcialisle.com/#>Cialis</a> Levitra Ohne Rezept Preisvergleich <a href=http://apcialisle.com/#>Buy Cialis</a> Cialis Without A

  • said on 23 February 2020 21:59

    Cockroaches most much relaxing on the north more than visceral octopodes where there is septenary onto between the miscegenation and the spotting <a href="http://btadalafil.com/">best ed pills</a> When a gink vice equivalent http://btadalafil.com/

  • said on 09 March 2020 00:33

    Levitra Cialis Viagra Comprar <a href=http://apcialisle.com/#>cialis 5mg</a> Chitosan <a href=http://apcialisle.com/#>cialis generic</a> Cialis Overnight

  • said on 11 March 2020 00:05

    viagra generic viagra generic <a href="https://pharm-usa-official.com/#">generic viagra </a> generic viagra cheap viagra bh88апрпhfa cheap viagra generic viagra <a href=https://pharm-usa-official.com/#>viagra online </a> generic viagra viagra online

  • said on 19 March 2020 08:47

    cheap viagra viagra generic <a href="http://octapod.ru/ #">buy viagra </a> buy viagra viagra 100mg viagra pills viagra 100mg <a href=http://topclub44.ru/>generic viagra </a> buy viagra viagra 100mg

  • said on 23 March 2020 23:54

    Viagra Levitra Differenze <a href=https://apcialisle.com/#>Cialis</a> Levaquin Versus Amoxicillin <a href=https://apcialisle.com/#>Cialis</a> Tomar Viagra O Levitra

  • said on 25 March 2020 07:49

    rhino instead of viagra viagra treatment for the heart problems <a href=" https://pharm-usa-official.com/# #">viagra generic </a> viagra meme generic names for cialis and viagra online usa viagra viagra efectos secundarios <a href=https://pharm-usa-official.com/#>levitra vs viagra </a> generic viagra extra dosage recommendations pfizer viagra https://fawandassociates.com/tax-payments-march-2020/?unapproved=2&moderation-hash=e1706ef857c3edb912d933838f3b2c77#comment-2 https://www.mcgintyspublakewood.com/2020/03/13/cbd-oils-12/?unapproved=24&moderation-hash=d59eaa929790e966ca1edd1720f2293e#comment-24 https://www.xl-hamster.nl/2020/03/20/fitness-mom-in-tight-leggings-plays-with-her-new-toy/?unapproved=3&moderation-hash=f091428f25fba5404e6500bc1682c718#comment-3 https://www.teaparty247.org/extinction-rebellion-co-founder-lets-the-cat-out-of-the-bag-its-not-really-about-the-climate-at-all/?unapproved=25574&moderation-hash=43e54d8ea6c8e8c04023794308ce74d3#comment-25574 https://citisportsonline.com/2020/03/08/chelsea-hammer-everton-at-home-to-stay-three-points-clear-in-fourth/?unapproved=1184&moderation-hash=1e0c4fccfa897a7b576a92e48653e579#comment-1184

  • said on 01 April 2020 13:10

    Zofran <a href=https://apcialisle.com/#>Cialis</a> Which Is Better Cipro Or Amoxicillin <a href=https://apcialisle.com/#>Cialis</a> Discount Programs For Doxycycline

Leave a comment

Your email address will not be published. It's only used for gravatars