Workflow Orchestration with Cloud Composer
Cloud Composer is a fully managed workflow orchestration service built on Apache Airflow. Composer enables you to create, schedule, monitor, and manage workflow pipelines that span across clouds and on-premises data centers. In this course, you will learn about Apache Airflow and its implementation via Cloud Composer. You will learn how to provision Composer instances, create and manage Airflow DAGs on Composer, and perform tasks such as testing, debugging, and monitoring of Airflow DAGs.
What you will learn
- Explore Apache Airflow and Cloud Composer as workflow orchestration solutions.
- Create and manage Airflow DAGs following best practices.
- Test and debug Airflow DAGs.
- Monitor and observe Airflow DAGs on Cloud Composer.
Prerequisites
- Completion of "Building Batch Data Pipelines on Google Cloud" or equivalent knowledge of data analytics and engineering on Google Cloud.
Target audience
- Customers
Training Program
3 modules to master the fundamentals
Objectives
- Explore Apache Airflow and Cloud Composer.
- Provision Cloud Composer instances.
- Explore the Airflow and Composer UIs.
Topics covered
- →Data Engineer's need for Workflow Orchestration
- →Introduction to Apache Airflow
- →Cloud Composer
- →Environment Setup
- →Using the Composer and Airflow
Activities
Lab: Provisioning Cloud Composer
Objectives
- Write DAGs.
- Explore common Airflow operators.
- Manage triggers, dependencies, and flow control.
- Integrate Airflow with Google Cloud Services.
Topics covered
- →DAG structure and best practices
- →Common operators
- →Dependencies, trigger rules, and flow control
- →Integration of Airflow and Google Cloud Services
Activities
Lab: Assembling a Data Processing Workflow
Objectives
- Leverage advanced Airflow features.
- Debug DAGs.
- Observe and monitor your running DAGs.
Topics covered
- →Advanced Airflow features
- →Debugging DAGs
- →Performance and scalability
- →Security and Access Control
- →Observability and monitoring
Activities
Lab: Extending and Monitoring DAGs
Quality Process
SFEIR Institute's commitment: an excellence approach to ensure the quality and success of all our training programs. Learn more about our quality approach
- Lectures / Theoretical Slides — Presentation of concepts using visual aids (PowerPoint, PDF).
- Technical Demonstration (Demos) — The instructor performs a task or procedure while students observe.
- Guided Labs — Guided practical exercises on software, hardware, or technical environments.
- Quiz / MCQ — Quick knowledge check (paper-based or digital via tools like Kahoot/Klaxoon).
The achievement of training objectives is evaluated at multiple levels to ensure quality:
- Continuous Knowledge Assessment : Verification of knowledge throughout the training via participatory methods (quizzes, practical exercises, case studies) under instructor supervision.
- Progress Measurement : Comparative self-assessment system including an initial diagnostic to determine the starting level, followed by a final evaluation to validate skills development.
- Quality Evaluation : End-of-session satisfaction questionnaire to measure the relevance and effectiveness of the training as perceived by participants.
Train multiple employees
- Volume discounts (multiple seats)
- Private or custom session
- On-site or remote