The problem

SQL transformations, a DLT pipeline, and a few notebooks with business logic. They all run - until they don't, because a warehouse wasn't warm, a table wasn't refreshed, or a dependency silently failed. This playbook covers how to design one reliable workflow that coordinates SQL tasks, DLT pipelines, and notebooks with guardrails, observability, and CI/CD-friendly configs using Lakeflow Jobs.

What Lakeflow Jobs is

Lakeflow Jobs is Databricks' workflow engine for scheduling and orchestrating tasks - SQL queries/dashboards, DLT pipelines, notebooks, wheels/JARs, and control-flow steps (if/else, for-each). You define dependencies, retries, timeouts, and notifications; Jobs executes and tracks the lineage of runs.

Where it fits in Lakeflow:

• Connect - ingestion to the lake.
• Declarative Pipelines (DLT) - managed ETL with expectations.
• Jobs - orchestration that stitches everything together.

The mixed-mode pattern (SQL → DLT → Notebook)

Use case:

1. Warm up/validate with SQL (idempotent checks, quality queries).
2. Run the DLT pipeline for the bronze/silver/gold flow.
3. Post-process with a notebook for business rules, feature prep, or downstream triggers.

Task graph shape

[SQL: prechecks] ──▶ [DLT: pipeline run] ──▶ [Notebook: postprocess]
                          ▲
                          └── optional [SQL: backfill params]

• SQL task executes against a SQL Warehouse and can refresh dashboards or alerts.
• Pipeline task (DLT) runs a declarative pipeline as a job task.
• Notebook task handles Python/Scala logic or calls into external APIs.

Choosing the right task type

Need	Best task	Why
Data quality probes, idempotent transforms, dashboard refresh	SQL task	Simple, warehouse-optimized, good for lightweight checks and reporting.
Managed ETL with data quality expectations, SCDs, CDC, Auto Loader	Pipeline (DLT) task	Declarative, built-in expectations, optimized autoscaling, lineage + monitoring.
Custom logic, feature engineering, API calls, orchestration glue	Notebook task	Full flexibility, library reuse, quick iteration.
Reusable packaged code	Python wheel task	Versioned, testable, CI/CD-friendly entry points.

Minimal, production-friendly setup

1) Define tasks and dependencies

• SQL prechecks: validate source freshness, enforce row-count thresholds.
• DLT pipeline: run the bronze→silver→gold pipeline.
• Notebook postprocess: compute aggregates, update downstream tables, or call webhooks.

Configure tasks in the Jobs UI or as code (e.g., Databricks Asset Bundles with job task types) for repeatable deployments.

2) Bundle-style job snippet

# bundle.yml (excerpt)
resources:
  jobs:
    lakeflow_mixed_pipeline:
      name: lakeflow-mixed-sql-dlt-notebook
      tasks:
        - task_key: sql_prechecks
          sql:
            query: "SELECT count(*) FROM raw.orders WHERE _ingest_ts > now() - INTERVAL 1 HOUR"
            warehouse_id: ${var.sql_warehouse_id}
          max_retries: 2
          timeout_seconds: 600

        - task_key: dlt_pipeline
          depends_on: [sql_prechecks]
          pipeline_task:
            pipeline_id: ${var.dlt_pipeline_id}
          max_retries: 1
          timeout_seconds: 7200

        - task_key: nb_postprocess
          depends_on: [dlt_pipeline]
          notebook_task:
            notebook_path: "/Repos/team/analytics/notebooks/postprocess"
            base_parameters:
              run_date: "{{job.run_date}}"
          max_retries: 2
          timeout_seconds: 1800

3) Control flow and fan-out

• Add if/else for conditional branches (e.g., skip heavy DLT if no new data).
• Use for-each to parallelize runs per region/table.

Delta Live Tables notes for orchestration

• Include the pipeline directly as a Pipeline task in the workflow.
• DLT handles state, expectations, and autoscaling so the job graph stays clean.
• Monitor run history and events from the Jobs UI.

Keep pre-DLT checks in SQL and post-DLT enrichments in notebooks to isolate concerns and simplify debugging.

Observability and operations

• Run page and history: inspect DAG, durations, logs, and task-level retries.
• Notifications: Slack/email on failure or SLA breach.
• Reruns with repair: re-run failed tasks without redoing the entire pipeline.
• Auditability: use run IDs and export logs to your SIEM.

Best practices

1. Single source of truth in VCS. Use bundles or IaC to define jobs as code. Keep warehouse IDs, pipeline IDs, and paths parameterized per environment.
2. Verify with SQL prechecks. Short SQL probes catch most preventable failures before DLT spin-up.
3. Let DLT own the heavy lifting. Put core ETL in DLT, not notebooks, for built-in expectations, autoscaling, and lineage.
4. Keep notebooks thin. Limit notebooks to orchestration glue and business-specific transforms that don't fit declarative patterns.
5. Right-size retries and timeouts. Fewer, longer retries for warehouse spin-up; shorter, more frequent retries for flaky APIs. Configure at the task level.
6. SLA-aware scheduling. Use cron or triggers aligned with upstream availability; stagger fan-out tasks to avoid warehouse thrash.
7. Clear failure contracts. Fail fast on prechecks; emit structured errors. Downstream tasks should not run when inputs are invalid.
8. Observe first, optimize second. Start with conservative cluster policies; use run history to tune parallelism and warehouse sizing.

Common pitfalls and fixes

• Putting everything in notebooks. Fix: move repeatable ETL to DLT; keep notebooks for glue.
• Underusing SQL tasks. Fix: add SQL probes for freshness, counts, and schema checks - fast and cheap.
• No code-based definitions. Fix: adopt bundles for promotion across dev/test/prod with the same DAG definitions.
• Weak observability. Fix: rely on run history, notifications, and logs; standardize run tags and emit metrics.

Quick-start recipes

A. Lightweight daily batch

• SQL: REFRESH TABLE gold.sales_mv;
• DLT: Bronze→Silver sync from CDC.
• Notebook: rebuild semantic aggregates; notify BI.

B. Fan-out regional loads

Use for-each over ["us", "eu", "apac"] running the same DLT pipeline with a region param; postprocess consolidates to a global mart.

C. Packaged logic via wheel

Move complex postprocessing into a Python wheel task for versioning and unit tests; notebooks become thin wrappers if needed.

Realistic SQL precheck snippet

-- Fail early if source lag exceeds 90 minutes
SELECT
  CASE WHEN max(_ingest_ts) < (current_timestamp() - INTERVAL 90 MINUTES)
       THEN error('Source too stale') END AS _must_fail
FROM raw.orders;

Run this as the first SQL task; if it errors, the DAG stops before burning compute.

Monitoring playbook

• Per-task dashboards: track run duration, retry count, data volumes.
• Notifications: route failures to on-call; route SLA warnings to owners.
• Postmortems: use run IDs and logs to build a timeline within the Jobs UI.

Summary

Mixed-mode orchestration doesn't have to be fragile. Put SQL in front for fast checks, DLT in the middle for managed ETL, and notebooks/wheels at the edges for business logic. Define it as code, add guardrails, and observe closely. Start by converting one brittle notebook chain into a Lakeflow Jobs DAG and measure the failure-rate drop and runtime consistency, then scale the pattern across domains.