Jenkins is a self-hosted automation server that runs build, test, and deploy jobs triggered by code pushes, schedules, or upstream jobs. It predates most of the hosted CI services and survives because it runs anywhere a JVM runs, has a plugin for nearly every tool, and keeps the controller and the work entirely on infrastructure you own. For data teams that need a pipeline to reach a private Spark cluster, an on-prem warehouse, or a VPN-walled artifact store, that ownership is the reason Jenkins stays in the stack while hosted runners would need a network tunnel to even start.

Declarative pipelines as code

A modern Jenkins job lives in a Jenkinsfile checked into the repository, so the pipeline is versioned and reviewed with the code it builds. The declarative syntax fixes a top-level structure - agent, stages, steps, post - that the parser validates before anything runs, which catches a malformed pipeline at load time instead of three stages deep.

pipeline {
  agent { label 'spark-build' }
  environment {
    WAREHOUSE = credentials('snowflake-svc')
  }
  stages {
    stage('Build') {
      steps { sh 'python -m build' }
    }
    stage('dbt test') {
      steps {
        sh 'dbt deps && dbt build --target ci --fail-fast'
      }
    }
  }
  post {
    failure { slackSend channel: '#data-eng', message: "Build ${env.BUILD_NUMBER} failed" }
    always  { junit 'target/test-results/*.xml' }
  }
}

The environment block pulls secrets from the Jenkins credentials store rather than the repository, and credentials('snowflake-svc') binds both WAREHOUSE_USR and WAREHOUSE_PSW automatically for a username-password pair.

Controller and agents

Jenkins splits into a controller that schedules work and serves the UI, and agents that execute it. The controller should run no build steps of its own - a job that compiles on the controller can read its secrets and config. Agents are labeled by capability (spark-build, gpu, arm64) and the agent { label ... } directive routes each stage to a matching machine. Ephemeral agents that spin up per build, via the Kubernetes plugin spawning a pod per job, give a clean workspace every time and avoid the state drift that plagues long-lived static agents.

podTemplate(containers: [
  containerTemplate(name: 'dbt', image: 'ghcr.io/dbt-labs/dbt-snowflake:1.7', command: 'sleep', args: '99d')
]) {
  node(POD_LABEL) {
    container('dbt') {
      checkout scm
      sh 'dbt build --select state:modified+ --defer'
    }
  }
}

Where Jenkins fits

Choose Jenkins when the pipeline must run inside a private network, when an existing plugin already wraps a tool you depend on, or when you want the scheduler and history on hardware you control. The cost is operational: you patch the controller, manage agent capacity, and audit plugins, since a stale plugin is a common attack path. Teams that don't need the self-hosted control often find a hosted runner less work - see GitLab CI/CD or CircleCI for that comparison, and the broader infrastructure-as-code and deployment overview for the full set.

Frequently asked questions

Declarative or scripted pipeline? Declarative for almost everything - it validates structure up front and reads clearly. Drop into scripted (Groovy) only when you need dynamic stage generation or logic the declarative model can't express, and you can embed a script { } block inside a declarative stage for the rare case.

Why not run builds on the controller? The controller holds credentials, job config, and the build queue. A build executing there can read all of it and can take the scheduler down under load. Set the controller's executor count to zero and run everything on agents.

How does Jenkins handle secrets? Through the built-in credentials store, scoped globally or per-folder, surfaced in pipelines via the credentials() helper or withCredentials. Values are masked in logs. For higher assurance, back it with a HashiCorp Vault plugin so secrets never persist on the controller disk.

Is Jenkins still worth running over a hosted CI? When you control your own runners and network it usually is. The main reason to keep it is reaching infrastructure a hosted runner can't - a private cluster or an air-gapped registry. If your builds only touch public services, a hosted runner removes the patching and capacity burden.