🏆SFEIR is the Google Cloud EMEA Training Partner of the Year 2025🤝New partnership: Official GitLab Training🤖New training: AI-Augmented Developer
🏆SFEIR is the Google Cloud EMEA Training Partner of the Year 2025🤝New partnership: Official GitLab Training🤖New training: AI-Augmented Developer
GCP300DATAFLOW

Serverless Data Processing with Dataflow Training

This training is intended for big data practitioners who want to further their understanding of Dataflow in order to advance their data processing applications. Beginning with foundations, this training explains how Apache Beam and Dataflow work together to meet your data processing needs without the risk of vendor lock-in. The section on developing pipelines covers how you convert your business logic into data processing applications that can run on Dataflow. This training culminates with a focus on operations, which reviews the most important lessons for operating a data application on Dataflow, including monitoring, troubleshooting, testing, and reliability.

Google Cloud
✓ Official training Google CloudLevel Advanced⏱️ 3 days (21h)

What you will learn

  • Demonstrate how Apache Beam and Dataflow work together to fulfill your organization's data processing needs.
  • Summarize the benefits of the Beam Portability Framework and enable it for your Dataflow pipelines.
  • Enable Shuffle and Streaming Engine, for batch and streaming pipelines respectively, for maximum performance.
  • Enable Flexible Resource Scheduling for more cost-efficient performance.
  • Select the right combination of IAM permissions for your Dataflow job.
  • Implement best practices for a secure data processing environment.
  • Select and tune the I/O of your choice for your Dataflow pipeline.
  • Use schemas to simplify your Beam code and improve the performance of your pipeline.
  • Develop a Beam pipeline using SQL and DataFrames.
  • Perform monitoring, troubleshooting, testing and CI/CD on Dataflow pipelines.

Prerequisites

  • Completed "Building Batch Data Pipelines"
  • Completed "Building Resilient Streaming Analytics Systems"

Target audience

  • Data Engineer, Data Analysts and Data Scientists aspiring to develop Data Engineering skills

Training Program

21 modules to master the fundamentals

Objectives
  • Introduce the course objectives.
  • Demonstrate how Apache Beam and Dataflow work together to fulfill your organization's data processing needs.
Topics covered
  • →Course Introduction
  • →Beam and Dataflow Refresher
Objectives
  • Summarize the benefits of the Beam Portability Framework.
  • Customize the data processing environment of your pipeline using custom containers.
  • Review use cases for cross-language transformations.
  • Enable the Portability framework for your Dataflow pipelines.
Topics covered
  • →Beam Portability
  • →Runner v2
  • →Container Environments
  • →Cross-Language Transforms
Activities

Quiz

Objectives
  • Enable Shuffle and Streaming Engine, for batch and streaming pipelines respectively, for maximum performance.
  • Enable Flexible Resource Scheduling for more cost-efficient performance.
Topics covered
  • →Dataflow
  • →Dataflow Shuffle Service
  • →Dataflow Streaming Engine
  • →Flexible Resource Scheduling
Activities

Quiz

Objectives
  • Select the right combination of IAM permissions for your Dataflow job.
  • Determine your capacity needs by inspecting the relevant quotas for your Dataflow jobs.
Topics covered
  • →IAM
  • →Quota
Activities

Quiz

Objectives
  • Select your zonal data processing strategy using Dataflow, depending on your data locality needs.
  • Implement best practices for a secure data processing environment.
Topics covered
  • →Data Locality
  • →Shared VPC
  • →Private IPs
  • →CMEK
Activities

Hands-on lab and quiz

Objectives
  • Review main Apache Beam concepts (Pipeline, PCollections, PTransforms, Runner, reading/writing, Utility PTransforms, side inputs), bundles and DoFn Lifecycle.
Topics covered
  • →Beam Basics
  • →Utility Transforms
  • →DoFn Lifecycle
Activities

Hands-on lab and quiz

Objectives
  • Implement logic to handle your late data.
  • Review different types of triggers.
  • Review core streaming concepts (unbounded PCollections, windows).
Topics covered
  • →Windows
  • →Watermarks
  • →Triggers
Activities

Hands-on lab and quiz

Objectives
  • Write the I/O of your choice for your Dataflow pipeline.
  • Tune your source/sink transformation for maximum performance.
  • Create custom sources and sinks using SDF.
Topics covered
  • →Sources and Sinks
  • →Text IO and File IO
  • →BigQuery IO
  • →PubSub IO
  • →Kafka IO
  • →Bigtable IO
  • →Avro IO
  • →Splittable DoFn
Activities

Quiz

Objectives
  • Introduce schemas, which give developers a way to express structured data in their Beam pipelines.
  • Use schemas to simplify your Beam code and improve the performance of your pipeline.
Topics covered
  • →Beam Schemas
  • →Code Examples
Activities

Hands-on lab and quiz

Objectives
  • Identify use cases for state and timer API implementations.
  • Select the right type of state and timers for your pipeline.
Topics covered
  • →State API
  • →Timer API
  • →Summary
Activities

Quiz

Objectives
  • Implement best practices for Dataflow pipelines.
Topics covered
  • →Schemas
  • →Handling unprocessable Data
  • →Error Handling
  • →AutoValue Code Generator
  • →JSON Data Handling
  • →Utilize DoFn Lifecycle
  • →Pipeline Optimizations
Activities

Hands-on lab and quiz

Objectives
  • Develop a Beam pipeline using SQL and DataFrames.
Topics covered
  • →Dataflow and Beam SQL
  • →Windowing in SQL
  • →Beam DataFrames
Activities

Hands-on lab and quiz

Objectives
  • Prototype your pipeline in Python using Beam notebooks.
  • Launch a job to Dataflow from a notebook.
Topics covered
  • →Beam Notebooks
Activities

Quiz

Objectives
  • Navigate the Dataflow Job Details UI.
  • Interpret Job Metrics charts to diagnose pipeline regressions.
  • Set alerts on Dataflow jobs using Cloud Monitoring.
Topics covered
  • →Job List
  • →Job Info
  • →Job Graph
  • →Job Metrics
  • →Metrics Explorer
Activities

Quiz

Objectives
  • Use the Dataflow logs and diagnostics widgets to troubleshoot pipeline issues.
Topics covered
  • →Logging
  • →Error Reporting
Activities

Quiz

Objectives
  • Use a structured approach to debug your Dataflow pipelines.
  • Examine common causes for pipeline failures.
Topics covered
  • →Troubleshooting Workflow
  • →Types of Troubles
Activities

Hands-on lab and quiz

Objectives
  • Understand performance considerations for pipelines.
  • Consider how the shape of your data can affect pipeline performance.
Topics covered
  • →Pipeline Design
  • →Data Shape
  • →Sources, Sinks, and External Systems
  • →Shuffle and Streaming Engine
Activities

Quiz

Objectives
  • Testing approaches for your Dataflow pipeline.
  • Review frameworks and features available to streamline your CI/CD workflow for Dataflow pipelines.
Topics covered
  • →Testing and CI/CD Overview
  • →Unit Testing
  • →Integration Testing
  • →Artifact Building
  • →Deployment
Activities

Hands-on labs and quiz

Objectives
  • Implement reliability best practices for your Dataflow pipelines.
Topics covered
  • →Introduction to Reliability
  • →Monitoring
  • →Geolocation
  • →Disaster Recovery
  • →High Availability
Activities

Quiz

Objectives
  • Using flex templates to standardize and reuse Dataflow pipeline code.
Topics covered
  • →Classic Templates
  • →Flex Templates
  • →Using Flex Templates
  • →Google-provided Templates
Activities

Hands-on labs and quiz

Objectives
  • Quick recap of training topics.
Topics covered
  • →Summary

Related Trainings

Google Cloud

Data Integration with Cloud Data Fusion

This 2-day course introduces learners to Google Cloud's data integration capability using Cloud Data Fusion. In this course, we discuss challenges with data integration and the need for a data integration platform (middleware). We then discuss how Cloud Data Fusion can help to effectively integrate data from a variety of sources and formats and generate insights. We take a look at Cloud Data Fusion's main components and how they work, how to process batch data and real time streaming data with visual pipeline design, rich tracking of metadata and data lineage, and how to deploy data pipelines on various execution engines.

2 d
Intermediate
Google Cloud

Managing a Data Mesh with Dataplex

Dataplex is an intelligent data fabric that enables organizations to centrally discover, manage, monitor, and govern their data across data lakes, data warehouses, and data marts. You can use Dataplex to build a data mesh architecture to decentralize data ownership among domain data owners. In this course, you will learn how to discover, manage, monitor, and govern your data across data lakes, data warehouses, and data marts through guided lectures and independent exercises using sample data.

2 d
Intermediate
View all trainings →

Upcoming sessions

No date suits you?

We regularly organize new sessions. Contact us to find out about upcoming dates or to schedule a session at a date of your choice.

Register for a custom date

Quality Process

SFEIR Institute's commitment: an excellence approach to ensure the quality and success of all our training programs. Learn more about our quality approach

Teaching Methods Used
  • Lectures / Theoretical Slides — Presentation of concepts using visual aids (PowerPoint, PDF).
  • Technical Demonstration (Demos) — The instructor performs a task or procedure while students observe.
  • Guided Labs — Guided practical exercises on software, hardware, or technical environments.
  • Quiz / MCQ — Quick knowledge check (paper-based or digital via tools like Kahoot/Klaxoon).
Evaluation and Monitoring System

The achievement of training objectives is evaluated at multiple levels to ensure quality:

  • Continuous Knowledge Assessment : Verification of knowledge throughout the training via participatory methods (quizzes, practical exercises, case studies) under instructor supervision.
  • Progress Measurement : Comparative self-assessment system including an initial diagnostic to determine the starting level, followed by a final evaluation to validate skills development.
  • Quality Evaluation : End-of-session satisfaction questionnaire to measure the relevance and effectiveness of the training as perceived by participants.

Frequently Asked Questions

Big data practitioners and data engineers who already build data processing applications and want to deepen their mastery of Dataflow and Apache Beam.
Experience with data processing and a working knowledge of Google Cloud are recommended. Familiarity with Apache Beam concepts is a plus, as this is an advanced (300) course.
You will learn how Apache Beam and Dataflow work together, develop batch and streaming pipelines, tune performance (Shuffle, Streaming Engine, Flexible Resource Scheduling), and apply security, testing, CI/CD and reliability best practices.
Yes. The course covers windows, watermarks and triggers, state and timers, sources and sinks, and the Streaming Engine for high-performance streaming pipelines.
It is an advanced (level 300) training lasting 3 days, alternating theory, demonstrations and hands-on labs.
Our training organizations SFEIR SAS and SFEIR-Est are Qualiopi certified for training activities. Contact us for a quote.

2,370€ excl. VAT

per learner

ReserveQuote