GCP200DESTREAMING
Build Streaming Data Pipelines on Google Cloud
In this course you will get hands-on in order to work through real-world challenges faced when building streaming data pipelines. The primary focus is on managing continuous, unbounded data with Google Cloud products.
You will learn about the applications and common architectural patterns for real-time data processing across key scenarios: Streaming ETL, Streaming AI/ML, Streaming Application, and Reverse ETL.
What you will learn
- Ingest and manage streaming data using Pub/Sub and Managed Service for Apache Kafka.
- Build and deploy streaming data pipelines with Dataflow.
- Implement streaming data solutions for real-time analytics and application serving with BigQuery and Bigtable.
Prerequisites
- Proficiency in a common programming language like Python
- A strong understanding of SQL
- Data fundamentals like data modeling, formats, and ETL/ELT processes
- Familiarity with the Google Cloud Platform (GCP)
Target audience
- Data Engineers, Data Analysts, Data Architects
Training Program
4 modules to master the fundamentals
Objectives
- Introduce the course learning objectives, and the scenario that will be used to bring hands on learning to building streaming data pipelines
- Describe the concept of streaming data pipelines, challenges associated with it, and the role of these pipelines within the data engineering process
Topics covered
- →Fundamentals of building streaming data pipelines on Google Cloud
- →Course learning objectives and hands-on scenario
Objectives
- Learn about the various streaming use cases and their applications, including Streaming ETL, Streaming AI/ML, Streaming Application, and Reverse ETL
- Identify and describe common sample architectures for streaming data
Topics covered
- →Streaming data use cases and applications
- →Common architectural patterns for real-time data processing
- →Streaming ETL
- →Streaming AI/ML
- →Streaming Application
- →Reverse ETL
Objectives
- Define messaging concepts
- Use the console to create various Pub/Sub and Kafka elements
- Know when to use Pub/Sub or Managed Service for Apache Kafka
- Describe the Dataflow service and challenges with streaming data
- Build and deploy a streaming pipeline
- Explore various data ingestion methods into BigQuery
- Learn about BigQuery continuous queries and using BigQuery ETL and reverse ETL
- Configure Pub/Sub to BigQuery streaming
- Architect BigQuery into your streaming pipelines
- Establish a streaming pipeline from Dataflow to Bigtable
- Analyze the Bigtable continuous data stream for trends using BigQuery
- Synchronize the trends analysis back into the user-facing application
Topics covered
- →Messaging concepts with Pub/Sub and Managed Service for Apache Kafka
- →Dataflow service and challenges with streaming data
- →Building and deploying streaming pipelines
- →Data ingestion methods into BigQuery
- →BigQuery continuous queries and ETL/reverse ETL
- →Pub/Sub to BigQuery streaming configuration
- →Architecting BigQuery into streaming pipelines
- →Streaming pipeline from Dataflow to Bigtable
- →Analyzing continuous data streams for trends
- →Synchronizing trends analysis back into user-facing applications
Activities
4 hands-on labs
Objectives
- Summarize the course and what you learned about the various Google products
- Understand what you're enabled to do next as a result of completing the course
Topics covered
- →Comprehensive wrap-up of key concepts
- →Building resilient and robust streaming data pipelines
Quality Process
SFEIR Institute's commitment: an excellence approach to ensure the quality and success of all our training programs. Learn more about our quality approach
Teaching Methods Used
- Lectures / Theoretical Slides — Presentation of concepts using visual aids (PowerPoint, PDF).
- Technical Demonstration (Demos) — The instructor performs a task or procedure while students observe.
- Guided Labs — Guided practical exercises on software, hardware, or technical environments.
Evaluation and Monitoring System
The achievement of training objectives is evaluated at multiple levels to ensure quality:
- Continuous Knowledge Assessment : Verification of knowledge throughout the training via participatory methods (quizzes, practical exercises, case studies) under instructor supervision.
- Progress Measurement : Comparative self-assessment system including an initial diagnostic to determine the starting level, followed by a final evaluation to validate skills development.
- Quality Evaluation : End-of-session satisfaction questionnaire to measure the relevance and effectiveness of the training as perceived by participants.
Upcoming sessions
No date suits you?
We regularly organize new sessions. Contact us to find out about upcoming dates or to schedule a session at a date of your choice.
Register for a custom dateTrain multiple employees
- Volume discounts (multiple seats)
- Private or custom session
- On-site or remote