DM250: Data Engineering & DataOps NEW

Agenda:

• Modern Data Architectures
  • Warehouse vs. Lakehouse
  • Medallion/Layering
  • Domain Boundaries
    
    
• Batch & Streaming
  • Event Models
  • Windowing Fundamentals
  • Late Data
  • Exactly once vs. At least once
    
    

• Pipeline Design
  • Ingestion Patterns
  • Transformation
  • Partitioning
  • Backfills
  • Idempotency
    
    
• Orchestration
  • DAG Design
  • Dependencies
  • Parameterization
  • Scheduling
  • Retries, SLAs
    
    
• Analytics Engineering (e.g. dbt Approach)
  • Modeling
  • Documentation
  • Tests
  • Semantic Layer Concepts
    
    

• CI/CD for Data
  • Git Flow
  • Build/Deploy Pipelines
  • Environments
  • Secrets
  • Artifacting
    
    

• Automated Testing
  • Unit/Integration
  • Schema Tests
  • DQ Checks
  • Contract Tests
  • Regression Strategies
    
    
• Observability
  • Logs/Metrics/Traces Approach
  • Pipeline Health
  • Freshness
  • Volume
  • Distribution
  • Cost
    
    
• Operations & Incident Response
  • Alert Design
  • On Call Basics
  • Runbooks
  • Postmortems
    
    
• Security & Governance Touchpoints
  • Access Patterns
  • Masking
  • Lineage Integration
  • Catalog Integration
    
    

• Capstone
  • End to End Blueprint
  • Operations and Quality Concept

Objectives:

• Select modern data architectures and outline them with clear justification (including batch/streaming classification)
• Design data pipelines so they are robust (idempotent), scalable, and maintainable
• Structure orchestration cleanly (DAG design, dependencies, backfills, SLAs)
• Build a CI/CD-capable delivery structure for data projects (repo structure, environments, releases)
• Implement a testing strategy for data (schema, contracts, data quality checks, regression)
• Define an observability concept (SLOs, alerts, monitoring metrics, cost indicators)
• Ensure operability (runbooks, incident procedures, postmortem improvements)
• “Build in” governance requirements technically (metadata/lineage, access, DQ as code)

Target audience:

The training DM250 Data Engineering & DataOps is targeted at:

• Data Engineers, Analytics Engineers, DataOps roles, Platform/Cloud Engineers with data focus
• MLOps/ML Engineers with responsibility for data feeds, Tech Leads for data platforms

Prerequisites:

To be able to follow the course content and learning pace in the DM250 Data Engineering & DataOps training, we recommend having attended the following courses beforehand:

• DM100 Fundamentals of Data Management & Data Governance
• DM200 Data Governance & Data Asset Management in Practice or equivalent foundational knowledge
• Solid foundational knowledge of SQL
• Basic knowledge of Python (or a comparable language) is very helpful
• Basic understanding of data pipelines/ETL/ELT as well as Git/version control is recommended

Description:

The course  DM250 Data Engineering & DataOps addresses the technical implementation of reliable, scalable, and operable data products—using the mindset and practices of DataOps. Participants work across the entire delivery chain: from modern data architectures (lakehouse approaches, streaming/event-driven, ELT/ETL patterns) through pipeline design and orchestration (e.g., Airflow principles, dbt workflows, or comparable tools) to CI/CD, automated testing, and observability. The focus is not on “tool training,” but on robust, transferable principles: version control, environment strategy, a data testing pyramid, data contracts, deployments, monitoring, incident handling, and continuous improvement.

A key outcome of the DM250 Data Engineering & DataOps workshop is the ability to deliver data pipelines like product software: reproducible, measurable, secure, and cost-efficient. This also includes alignment with governance (DM100 Fundamentals of Data Management & Data Governance / DM200 Data Governance & Data Asset Management in Practice): metadata/lineage, access models, quality rules, and documented responsibilities are treated as integral parts of engineering practice—not as “afterthoughts.” In a capstone scenario (e.g., “Batch + Streaming Ingestion → Transformation → Serving Layer”), the teams design an end-to-end solution including operational artifacts (runbooks, alerts, SLOs, release checklist).