Real-time analytics and anomaly detection with Apache Kafka, Apache Flink, Grafana & QuestDB

How does a time-series database fit into your real-time streaming analytics projects? How do you integrate it with the tools you are already using?
In this video, we show you a demo in which we ingest temperature and status data from 100 sensors using Apache Kafka, then we detect anomalies in real-time with Apache Flink, and we store the results in QuestDB for further analytics and to power a real-time dashboard with Grafana.
We also show you some of the time-series queries you can run in QuestDB using SQL, and how they are simpler than their equivalent using a specialised database
Our GitHub: github.com/questdb/questdb
00:00 Intro
00:50 What is QuestDB
01:20 Data ingestion
02:13 Apache Kafka
02:46 Apache Flink to process in-stream data
05:23 Grafana and real-time data visualisation
06:03 Integrating QuestDB and Grafana using the PostgreSQL plugin
07:07 We start the demo!
07:34 Sending sensor data to Apache Kafka
09:54 Defining our tables in Apache Flink using the Table API with SQL
10:30 Integrating Apache Flink and QuestDB with the JDBC connector
11:17 Our first contact with the QuestDB console
12:20 Inserting data in QuestDB from Apache Kafka using Apache Flink
13:11 Detecting anomalies with the CEP Library of Apache Flink using SQL
13:34 Explanation of the MATCH_RECOGNIZE queries for pattern matching on continuous queries
15:55 Explanation of the MEASURES statement in continuous queries
17:00 Inserting data into QuestDB using continuous queries in Apache Flink
17:14 Real-time visualisation with Grafana reading from QuestDB
18:33 Exploring time-series queries in QuestDB
19:20 The LATEST ON statement to read recent data in QuestDB
21:50 CTEs and JOINS in QuestDB
23:52 Adding extra joins to query recent data in real-time across multiple tables
26:20 We speed up ingestion and we start processing several thousan events per second
28:07 Links and closing
To learn more, questdb.io, github.com/questdb, demo.questdb.io

[@oyeyemirafiuowolabi2347]

Great tutorial!
Please, can i have the link to the complete code?
I want to integrate with Apache Nifi.
Thank you.

[@QuestDB]

Thanks! Not sure this will help, but here you are all the materials I used. Apart from some (probably useless) shell scripts I used to start services on the machine, you can see the python code to send data to Kafka, and the source of the notebook I was using with all the queries to ingest and analyse the data. gist.github.com/javier/4be2baf2bc4f76358996b6d5dac9c1b5

[@oyeyemirafiuowolabi2347]

@@QuestDB Thank you for your reply. I would go through it & make good use of whatever can help in the code.