Can Apache Kafka Replace a Database?

Can and should Apache Kafka replace a database? How long can and should I store data in Kafka? How can I query and process data in Kafka? These are common questions that come up more and more. This session explains the idea behind databases and different features like storage, queries, transactions, and processing to evaluate when Kafka is a good fit and when it is not.
The discussion includes different Kafka-native add-ons like Tiered Storage for long-term, cost-efficient storage and ksqlDB as event streaming database. The relation and trade-offs between Kafka and other databases are explored to complement each other instead of thinking about a replacement. This includes different options for pull and push-based bi-directional integration.
Key takeaways:
- Kafka can store data forever in a durable and high available manner
- Kafka has different options to query historical data
- Kafka-native add-ons like ksqlDB or Tiered Storage make Kafka more powerful than ever before to store and process data
- Kafka does not provide transactions, but exactly-once semantics
- Kafka is not a replacement for existing databases like MySQL, MongoDB or Elasticsearch
- Kafka and other databases complement each other; the right solution has to be selected for a problem
- Different options are available for bi-directional pull and push-based integration between Kafka and databases to complement each other
Slides:
www.slideshare.net/KaiWaehner...
Blog post:
www.kai-waehner.de/blog/2020/...

[@yusufnar]

I liked that. Quite informative.

[@KashishBhatia90]

One of the best Kafka related content on youtube. Thank you!

[@kosmologic6007]

Great explanation, thanks! I like especially database out with Kafka as a writer pipe and storage in different dbs (depending on the purpose).

[@sunilb128]

Thank you for creating this video. It broaden by prospective about Kafka. Super helpful !

[@jordangardiner8951]

Hi Kai! Firstly great video, after watching this I starting investigating how I could treat Kafka as a database and how KSQLdb could help support this.
I have a question about unique keys/auto increment which I'm hoping you can help me answer.
How would I do the following scenario (based on an app & relational DB approach) in Kafka & KSQLdb.
- The application creates an order and stores the user_id and fees in a table called order, the tables primary key is an auto incremented int starting at 0.
As the database is taking care of the order id the application does not need to manage this, this is what I am trying to replicate in Kafka/KSQLdb.
Any ideas on how to resolve this?
Cheers!

Sorry, I don't do much hands-on these days. The Confluent community (forum.confluent.io/) or Slack channel is likely the best location to discuss this.

[@helmudcole899]

I don't understand what version of Kafka is this talk about. It was said that community version doesn't have any features.

The whole talk is about Apache Kafka.
Confluent Tiered Storage is a component that is only available as part of the commercial Confluent Platform.