Debugging Flaky Pipelines

Track down intermittent failures with replay, verbose logs, retries as a diagnostic signal, and by hunting the usual culprits: shared state, timing, and the network.

A flaky pipeline — one that passes and fails on the same commit — erodes trust faster than a consistently broken one, because people learn to just hit "rebuild" and stop reading failures. Chasing flakiness is a distinct skill.

Reproduce and iterate fast

Use the Replay feature to rerun a build with a tweaked Jenkinsfile without committing — perfect for adding debug output or bisecting a stage. Turn on options { timestamps() } and, temporarily, sh 'set -x' to see exactly where and when things diverge.

Know the usual suspects

Shared state — a reused workspace or a shared test database. Isolate with cleanWs() and ephemeral agents.
Timing / race conditions — tests that assume a service is ready. Replace fixed sleep calls with waitUntil polling.
Resource limits — an OOM-killed test process looks like a random failure. Check agent memory.
Network — registry pulls and external APIs. Wrap them in retry and watch whether retries "fix" it — if they do, the problem is the network, not your code.

Make flakiness visible

Record the retry count and quarantine known-flaky tests rather than letting them block everyone. A test that needs three retries to pass is a bug ticket, not a permanent retry(3).

Example

pipeline {
    agent any
    options { timestamps() }   // when did each line happen?
    stages {
        stage('Wait for service') {
            steps {
                // replace a blind sleep with real readiness polling
                timeout(time: 2, unit: 'MINUTES') {
                    waitUntil {
                        script {
                            def code = sh(
                                script: 'curl -sf http://localhost:8080/health',
                                returnStatus: true
                            )
                            return code == 0   // true once healthy
                        }
                    }
                }
            }
        }
        stage('Flaky integration test') {
            steps {
                // if this only passes on retry, the network/timing is the bug
                retry(3) {
                    sh 'set -x; npm run test:integration'
                }
            }
        }
    }
    post {
        always { cleanWs() }   // no stale state to poison the next run
    }
}

When to use it

A developer uses the Replay feature to edit and re-run the last failed build's Jenkinsfile without making a Git commit.
An engineer adds timestamps() and ansiColor() to the options block to track exactly which step is slow or hanging.
A flaky test is isolated by temporarily wrapping the step in retry(5) to confirm the failure rate and gather diagnostic logs.

More examples

Enable timestamps and verbose logging

Adds timestamps to every log line and enables shell trace mode to print each command before it runs.

pipeline {
  agent any
  options {
    timestamps()
    ansiColor('xterm')
  }
  stages {
    stage('Debug') {
      steps {
        sh 'set -x && ./run-tests.sh'  // -x echoes each command
      }
    }
  }
}

Retry to diagnose flakiness

Retries a flaky stage up to 5 times and archives the output log on failure to help diagnose the root cause.

stage('Flaky Integration Test') {
  options { retry(5) }
  steps {
    sh '''
      set -e
      echo "Attempt ${STAGE_NAME}"
      ./integration-test.sh 2>&1 | tee test-output.log
    '''
  }
  post {
    failure { archiveArtifacts 'test-output.log' }
  }
}

Replay a build from the Jenkins UI

Describes the Replay workflow for iteratively debugging a pipeline without creating Git commits.

# No code required — use the Jenkins UI:
# 1. Open the failed build page
# 2. Click 'Replay' in the left sidebar
# 3. Edit the Jenkinsfile in the text editor
# 4. Click Run to execute the modified pipeline
# Changes are NOT committed to Git
echo "Replay runs the edited script directly on the controller"

Discussion

Be the first to comment on this lesson.