Realtime Streaming with Apache Flink | End to End Data Engineering Project

In this video, you will be building an end-to-end data engineering project using some of the most powerful technologies in the industry: Apache Flink, Kafka, Elasticsearch, and Docker. In this video, we dive deep into the world of real-time data processing and analytics, guiding you through every step of creating a robust, scalable data pipeline.

Timestamp
0:00 Introduction
0:55 The system architecture
08:00 Sales Analytics Data Generation
19:10 Producing Data into Kafka Broker
25:00 Setting up Apache Flink project
32:28 Consuming data from Kafka with Apache Flink
43:30 Starting Apache Flink on Mac
54:25 Writing Kafka Streams to Postgres Database
1:20:00 Aggregating Transactions per Category into Postgres
1:36:00 Aggregating Transactions Per Day into Postgres
1:39:46 Aggregating Transactions Per Month into Postgres
1:51:52 Writing Kafka Streams Data into Elasticsearch
2:05:00 Reindexing Data on Elasticsearch with Timestamp
2:10:52 Creating Streaming Dashboard on Elasticsearch
2:22:46 Realtime Dashboard Results
2:24:14 Recap
2:25:34 Outro

👦🏻 My Linkedin: https://www.linkedin.com/in/yusuf-ganiyu-b90140107/
🚀 Twitter: https://twitter.com/YusufOGaniyu
📝 Medium: https://medium.com/@yusuf.ganiyu

🌟 Please LIKE ❤️ and SUBSCRIBE for more AMAZING content! 🌟

🔗 Useful Links and Resources:
✅ Code: https://github.com/airscholar/FlinkCommerce.git
✅ Medium Article: https://medium.com/@yusuf.ganiyu/realtime-data-engineering-project-with-airflow-kafka-spark-cassandra-and-postgres-804bcd963974
✅ Docker Compose Documentation: https://docs.docker.com/compose/
✅ Apache Kafka Official Site: https://kafka.apache.org/
✅ Apache Flink Official Documentation: https://nightlies.apache.org/flink/flink-docs-stable/
✅ Confluent Docs: https://docs.confluent.io/home/overview.html
✅ Maven Repository: https://mvnrepository.com/

✨ Tags ✨
Big Data Engineering, Apache Flink, Kafka, Elasticsearch, Docker, Data Engineering, Realtime Data Processing, Big Data, Data Pipeline, Streaming Data, Data Analytics, Tech Tutorial, Data Science, Flink Streaming, Kafka Streaming, Elasticsearch Tutorial, Docker Containers, Data Engineering Project, Realtime Analytics, Big Data Technologies, Data Engineering Tutorial, Data Engineering Projects, Data Engineer

✨Hashtags✨
#ApacheFlink, #Kafka, #Elasticsearch, #Docker, #DataEngineering, #RealtimeData, #BigData, #DataPipeline, #TechTutorial, #DataScience, #StreamingData, #Flink, #KafkaStreams, #ElasticsearchTips, #DockerContainers, #DataEngineeringProjects, #RealtimeAnalytics, #BigDataTech, #LearnDataEngineering, #dataengineers