Description
Data Engineering course using Kafka and Spark Structured Streaming. In this course, you will learn how to build a streaming pipeline by integrating Kafka and Spark Structured Streaming. Let’s go through the details of what is covered in the course. First, we need to have a suitable environment for building a streaming pipeline using Kafka and Spark Structured Streaming on Hadoop or any other distributed file system. As part of the course, you will start by setting up a self-supporting lab with all key components such as Hadoop, Hive, Spark and Kafka on a single-node Linux-based system. After setting up the environment, you will review the details of getting started with Kafka. As part of this process, you will create a Kafka topic, generate messages into the topic, and also consume messages from the topic.
What you will learn in the Data Engineering using Kafka and Spark Structured Streaming course
- Setting up a self-supporting lab with Hadoop (HDFS and YARN), Hive, Spark and Kafka An overview of Kafka for building a streaming pipeline to receive data to Kafka topics using Kafka Connect using a file source Receiving data to HDFS using Kafka Connect with Using the HDFS 3 Connector Plugin Overview of Spark Structured Streaming to process data as part of a streaming pipeline Incremental data processing using Spark Structured Streaming using file source and file target Integrating Kafka and Spark Structured Streaming – Reading data from Kafka topics
This course is suitable for people who
- Experienced ETL developers who want to learn Kafka and Spark to build streaming pipelines Experienced PL/SQL developers who want to learn Kafka and Spark to build streaming pipelines Beginner or experienced data engineers who want Kafka and Spark to build a streaming pipeline
Data Engineering using Kafka and Spark Structured Streaming course specifications
- Publisher: Udemy
- Teacher: Durga Viswanatha Raju Gadiraju
- Training level: beginner to advanced
- Training duration: 9 hours and 26 minutes
Course topics on 12/2023
Course prerequisites
- Laptop with decent configuration
- Decent internet speed to watch the lessons
- Self Support lab (instructions will be provided as part of the course) or ITVersity labs
- Knowledge about Functional Programming (preferably Python or Scala)
- Knowledge or experience using Spark
Course images
Sample video of the course
Installation guide
After Extract, view with your favorite Player.
Subtitle: English
Quality: 720p
download link
File(s) password: www.downloadly.ir
File size
3.7 GB