Realtime Streaming with Unstructured Data Engineering | Get Hired as an Experienced Data Engineer

Save
1 year ago
60
Technology Education Data Engineering Apache Airflow Kafka Apache Spark PostgreSQL Docker Docker Compose ETL Pipeline Data Pipeline

In this video you will be building a realtime streaming pipeline for unstructured data with different data types (TEXT, IMAGE, VIDEO, CSV, JSON, PDF) with over 600+ different datasets.

MORE DATA ENGINEERING VIDEOS AVAILABLE on datamasterylab.com

Like this video?
- Buy me a coffee: https://www.buymeacoffee.com/yusuf.ganiyu
- Support the channel: https://www.youtube.com/@codewithyu/join

Timestamps:
0:00 Introduction
1:50 System Architecture Overview
4:08 System Architecture Design
13:22 Setting up Spark Streaming for Unstructured Data
21:46 Handling multiple unstructured data types
24:31 Creating data schema
30:35 Creating custom user define functions for data extraction
51:14 Parsing and extracting text data
1:40:30 Structuring the results into a dataframe
1:46:15 Reading JSON structured files into the streams
1:49:47 Joining Structured and Unstructured Data Streams
1:52:50 Writing Data to AWS S3 Bucket
2:04:20 Creating AWS Glue Crawler for the data
2:08:25 Verifying the crawler results on Athena
2:11:36 Deploying Spark Streams to Spark Clusters
2:26:31 Verification of Results
2:29:40 Outro

👦🏻 My Linkedin: https://www.linkedin.com/in/yusuf-ganiyu-b90140107/
🚀 X(Twitter): https://x.com/YusufOGaniyu
📝 Medium: https://medium.com/@yusuf.ganiyu

🌟 Please LIKE ❤️ and SUBSCRIBE for more AMAZING content! 🌟

🔗 Useful Links and Resources:
✅ Source Code and Datasets: https://www.buymeacoffee.com/yusuf.ganiyu/source-code-real-time-streaming-pipelines-unstructured-data
✅ Docker Compose Documentation: https://docs.docker.com/compose/
✅ Apache Spark Official Site: https://spark.apache.org/
✅ Confluent Docs: https://docs.confluent.io/home/overview.html
✅ S3 Documentation: https://docs.aws.amazon.com/s3/
✅ AWS IAM Documentation: https://docs.aws.amazon.com/IAM/latest/UserGuide/introduction.html

✨ Tags ✨
Data Engineering, Apache Spark, Unstructured Data, Docker, Docker Compose, ETL Pipeline, Data Pipeline, Big Data, Streaming Data, Real-time Analytics, Kafka Connect, Spark Master, Spark Worker, Schema Registry, Control Center, Data Streaming

✨ Hashtags ✨
#DataEngineering #ApacheSpark #unstructureddata #Docker #ETLPipeline #DataPipeline #StreamingData #RealTimeAnalytics

Loading comments...