Running in CI

Lint, type check and tests as separate steps, a cache keyed on the lockfile, and a red mark that names what broke.

Syntaxsubprocess.run(["pytest", "-q"], check=True) # raises CalledProcessError on non-zero

CI answers one question per run: is this commit safe to merge? It answers it usefully only when a failure points at a cause, which means separate steps with separate exit codes rather than one script that runs everything and reports one number.

Separate steps

Lint, type check and tests fail for unrelated reasons. As three steps the failed step names the reason; as one make check it names the makefile. Order them cheapest first so a formatting slip does not wait behind a ten-minute test suite.

Caching the install

Key the cache on a hash of the lockfile, never on the branch name: the key then changes exactly when the dependencies change and never otherwise. actions/setup-python@v6 with cache: pip does this for pip's download cache, given a cache-dependency-path. A cache keyed on something that never changes is worse than no cache at all, because it serves last month's resolution forever.

Failing for the right reason

Exit codes are the entire protocol between your tools and the runner. pytest returns 0 when everything passed, 1 when tests failed, 2 on interruption, 4 on a usage error, and 5 when it collected no tests at all. A step that only asks "did it crash" reads 5 as success and merges untested code. Never append || true, and use continue-on-error only where a step is genuinely advisory.

Shorthand

In Python glue code, subprocess.run(cmd, check=True) is the short form of inspecting .returncode yourself and raising CalledProcessError; check=False (the default) hands you the code to decide on.

Example

Try it yourself
Loading editor…
Press Run to execute the code.

When to use it

  • Tell at a glance whether a red build is a lint slip or a genuine test failure.
  • Stop paying for a full dependency install on every push without ever serving a stale set.
  • Catch a test suite that silently stopped collecting after a directory rename.
  • Keep flaky integration tests from blocking a pull request that only touched code they do not exercise.

More examples

The workflow these steps become

Each run: is its own step, so the failure annotation points at the tool that failed rather than at a wrapper script. --maxfail=1 stops the suite at the first failure, which is what you want on a pull request and not what you want on a nightly run.

Example · yaml
name: ci
on: [push, pull_request]

jobs:
  check:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v6
      - uses: actions/setup-python@v6
        with:
          python-version: "3.14"
          cache: pip
          # The cache key is the hash of this file. It changes when the
          # dependencies change, and at no other time.
          cache-dependency-path: requirements.txt

      - run: pip install --require-hashes -r requirements.txt
      - run: pip install --group dev

      # Three steps, three exit codes, three distinct reasons to be red.
      # Cheapest first: a missing comma should not wait for the test suite.
      - run: ruff check .
      - run: ruff format --check .
      - run: mypy src
      - run: pytest -q --maxfail=1

The same checks, runnable locally

Running every step and reporting all the failures is the right behaviour locally, where you want the whole list before you start fixing. CI keeps the steps separate instead, so its UI can tell you which one failed without you reading a log.

Example · python
#!/usr/bin/env python3
"""scripts/check.py - what CI runs, runnable on a laptop."""
import subprocess
import sys

STEPS = [
    ("lint", ["ruff", "check", "."]),
    ("format", ["ruff", "format", "--check", "."]),
    ("types", ["mypy", "src"]),
    ("tests", ["pytest", "-q"]),
]

def main():
    failures = []
    for name, cmd in STEPS:
        print(f"==> {name}: {' '.join(cmd)}")
        # check=False: we want the code, not an exception. The shorthand
        # subprocess.run(cmd, check=True) raises CalledProcessError instead.
        done = subprocess.run(cmd)
        if done.returncode != 0:
            failures.append(f"{name} (exit {done.returncode})")

    if failures:
        print("FAILED: " + ", ".join(failures))
        return 1
    print("all checks passed")
    return 0

if __name__ == "__main__":
    sys.exit(main())

Isolating a job that needs a real service

Two jobs run in parallel and fail independently, so a database that was slow to start does not mark your unit tests as broken. The service health check is the part people leave out, and it is the usual cause of integration tests that pass on a retry.

Example · yaml
jobs:
  unit:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v6
      - uses: actions/setup-python@v6
        with: { python-version: "3.14", cache: pip }
      - run: pip install -r requirements.txt && pip install --group dev
      - run: pytest -q -m "not integration"   # the suite that must never be flaky

  integration:
    runs-on: ubuntu-latest
    services:
      postgres:
        image: postgres:17
        env: { POSTGRES_PASSWORD: ci }
        # Without a health check the tests start before the database accepts
        # connections, and you get a flake that looks like your code.
        options: >-
          --health-cmd pg_isready --health-interval 5s --health-retries 10
        ports: ["5432:5432"]
    steps:
      - uses: actions/checkout@v6
      - uses: actions/setup-python@v6
        with: { python-version: "3.14", cache: pip }
      - run: pip install -r requirements.txt && pip install --group dev
      - run: pytest -q -m integration
        env:
          DATABASE_URL: postgres://postgres:ci@localhost:5432/postgres

Related

py quality pytestpy cloud depspy quality typingpy cloud clipy quality layout

Discussion

  • Be the first to comment on this lesson.