Orchestrate BigQuery Workloads with Dataform
Dataform is a service for data analysts to develop, test, version control, and schedule complex SQL workflows for data transformation in BigQuery. In this course you will explore the components of Dataform core, learn how to define tables and dependencies in SQLX, document BigQuery tables and views, understand BigQuery security settings and how to manage these with Dataform, write assertions, execute SQL workflows, and explore additional advanced use cases.
What you will learn
- Understand the components of Dataform core.
- Create tables and views in BigQuery using Dataform.
- Document BigQuery tables and views.
- Understand BigQuery security settings using Dataform.
- Use assertions to validate data in Dataform workflows.
- Execute Dataform SQL workflows in an automated fashion.
Prerequisites
- Knowledge of SQL data analysis and BigQuery as discussed in BigQuery for Data Analysis.
Target audience
- Customers
Training Program
7 modules to master the fundamentals
Objectives
- Understand the components of Dataform core.
Topics covered
- →SQL workflow
- →Repositories and workspaces
- →Default files and folders
- →Compiled graphs
Objectives
- Create tables and views in BigQuery using Dataform.
Topics covered
- →Declare a data source.
- →Create a table.
- →Create an incremental table.
- →Set partitioning and clustering options.
- →Create an empty table.
- →Create an external BigLake table.
- →Create views and materialized views.
- →Define dependencies.
Objectives
- Document BigQuery tables and views.
Topics covered
- →Use column descriptions.
- →Use globally defined JavaScript constants.
- →Add labels.
Activities
Lab: Build SQL Workflows with Dependencies in Dataform
Objectives
- Understand BigQuery security settings using Dataform.
Topics covered
- →IAM dataset and table/view access
- →Column-level security
- →Row-level security
Objectives
- Use assertions to validate data in Dataform workflows.
Topics covered
- →Use built-in assertions.
- →Create manual assertions.
Activities
Lab: Work with Assertions and BigQuery Security Settings in Dataform
Objectives
- Execute Dataform SQL workflows in an automated fashion.
Topics covered
- →Dataform code lifecycle.
- →What happens during compilation.
- →Customize and schedule compilation results.
- →Execute workflows (UI, Cloud Scheduler, Cloud Composer).
- →Logging and monitoring.
Activities
Lab: Automate and Monitor SQL Workflow Executions in Dataform
Objectives
- Explore additional use cases for Dataform.
Topics covered
- →Create a BigLake table after file upload using Cloud Run functions.
- →Build a Machine Learning pipeline with BigQuery ML.
- →Work with Slowly Changing Dimensions Type 2.
Activities
Lab: Create a BigLake Table with Dataform Using Cloud Run Functions
Quality Process
SFEIR Institute's commitment: an excellence approach to ensure the quality and success of all our training programs. Learn more about our quality approach
- Lectures / Theoretical Slides — Presentation of concepts using visual aids (PowerPoint, PDF).
- Technical Demonstration (Demos) — The instructor performs a task or procedure while students observe.
- Guided Labs — Guided practical exercises on software, hardware, or technical environments.
- Quiz / MCQ — Quick knowledge check (paper-based or digital via tools like Kahoot/Klaxoon).
The achievement of training objectives is evaluated at multiple levels to ensure quality:
- Continuous Knowledge Assessment : Verification of knowledge throughout the training via participatory methods (quizzes, practical exercises, case studies) under instructor supervision.
- Progress Measurement : Comparative self-assessment system including an initial diagnostic to determine the starting level, followed by a final evaluation to validate skills development.
- Quality Evaluation : End-of-session satisfaction questionnaire to measure the relevance and effectiveness of the training as perceived by participants.
Train multiple employees
- Volume discounts (multiple seats)
- Private or custom session
- On-site or remote