Speed up slow CI: caching and parallelism in GitHub Actions

A CI pipeline that takes twenty minutes changes how a team works. People batch changes to avoid waiting, context-switch while checks run, and merge with less confidence because re-running is painful. Usually nothing is wrong with the tests themselves. The pipeline is doing the same expensive work on every push, one step after another.

This article focuses on making an existing GitHub Actions workflow faster. If you are setting one up from scratch, start with our practical CI/CD pipeline with GitHub Actions and Docker and come back here. The levers are always the same: do less work (caching, skipping), do work at the same time (parallel jobs, matrices, sharding), and stop doing work nobody needs (cancelling stale runs). None of these require changing your tests or your tooling.

Measure before you optimise

Open a few recent runs and note how long each step takes. Typical patterns: dependency installation taking minutes on every run, a Docker build that rebuilds every layer, and a single test job running a suite that could be split. Fix the biggest step first. Also check queue time; if jobs wait a long time for runners, more parallelism will not help until you have more runner capacity.

It helps to separate two goals. Feedback time is how long a developer waits to learn whether a pull request is broken; total compute is what you pay for. Parallelism improves the first and usually increases the second slightly, while caching improves both. Ordering also matters: put fast checks like linting and type checks where they fail early, so a typo does not wait behind a ten-minute end-to-end suite.

Keep a note of the baseline so you can tell whether each change helped. The gh run list and gh run view commands, or the workflow usage view in the repository's Actions tab, make it easy to compare durations across runs without guessing.

Cache dependencies the right way

The setup actions for Node, Python, Go and Java have caching built in, keyed on your lockfile. For Node, actions/setup-node with cache: npm caches npm's download cache, and npm ci then installs from it without network round trips. This is safer than caching node_modules directly, which breaks on Node or OS upgrades and on packages with native builds.

For tools with their own build caches, such as Next.js (.next/cache), Turborepo, Nx, pytest or mypy, add an explicit actions/cache step. Use a key that changes when inputs change and a restore-keys prefix so a near-match is restored rather than nothing.

name: CI

on:
  pull_request:
  push:
    branches: [main]

# Cancel superseded runs on the same branch or PR, but never on main
concurrency:
  group: ${{ github.workflow }}-${{ github.ref }}
  cancel-in-progress: ${{ github.ref != 'refs/heads/main' }}

jobs:
  lint:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v5
      - uses: actions/setup-node@v5
        with:
          node-version: 22
          cache: npm
      - run: npm ci
      - run: npm run lint && npm run typecheck

  build:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v5
      - uses: actions/setup-node@v5
        with:
          node-version: 22
          cache: npm
      - uses: actions/cache@v4
        with:
          path: .next/cache
          key: nextjs-${{ runner.os }}-${{ hashFiles('package-lock.json') }}-${{ hashFiles('src/**') }}
          restore-keys: |
            nextjs-${{ runner.os }}-${{ hashFiles('package-lock.json') }}-
      - run: npm ci
      - run: npm run build

Lint, type checks and the build run as separate jobs in parallel instead of sequential steps in one job. Each job pays a small setup cost, but the wall-clock time becomes the slowest job rather than the sum of all of them.

Cache Docker layers

Docker builds on hosted runners start with an empty cache, so every layer is rebuilt by default. BuildKit can export its layer cache to the GitHub Actions cache backend and import it on the next run. Combined with a Dockerfile that copies the lockfile and installs dependencies before copying source, a typical code change only rebuilds the final layers.

  image:
    runs-on: ubuntu-latest
    permissions:
      contents: read
      packages: write
    steps:
      - uses: actions/checkout@v5
      - uses: docker/setup-buildx-action@v3
      - uses: docker/login-action@v3
        with:
          registry: ghcr.io
          username: ${{ github.actor }}
          password: ${{ secrets.GITHUB_TOKEN }}
      - uses: docker/build-push-action@v6
        with:
          context: .
          push: ${{ github.ref == 'refs/heads/main' }}
          tags: ghcr.io/${{ github.repository }}:${{ github.sha }}
          cache-from: type=gha
          cache-to: type=gha,mode=max

mode=max caches intermediate layers from multi-stage builds, not just the final stage, which is where dependency installation usually happens. For very large images, a registry cache (type=registry) avoids competing for the repository's Actions cache storage, which has a size limit and evicts least-recently-used entries.

Split tests with matrices and sharding

Once setup is cached, the test suite is usually the long pole. A matrix runs the same job several times with different parameters. Most modern runners can split a suite across machines: Playwright and Vitest support --shard=index/total, Jest supports --shard too, and pytest can be split with plugins like pytest-split.

  e2e:
    runs-on: ubuntu-latest
    strategy:
      fail-fast: false # let every shard finish so you see all failures
      matrix:
        shard: [1, 2, 3, 4]
    steps:
      - uses: actions/checkout@v5
      - uses: actions/setup-node@v5
        with:
          node-version: 22
          cache: npm
      - run: npm ci
      - run: npx playwright install --with-deps chromium
      - run: npx playwright test --shard=${{ matrix.shard }}/${{ strategy.job-total }} --reporter=blob
      - uses: actions/upload-artifact@v4
        if: ${{ !cancelled() }}
        with:
          name: blob-report-${{ matrix.shard }}
          path: blob-report
          retention-days: 3

A follow-up job can download the blob reports and run npx playwright merge-reports to produce one HTML report. Watch for uneven shards: if one shard consistently runs much longer, a few slow spec files are probably landing together, and splitting those files helps more than adding shards.

Matrices are also the right tool for genuine variations, such as testing against two Node versions or several database versions. Be deliberate about it: every matrix dimension multiplies job count. Run the full matrix on main or nightly, and a single representative combination on pull requests, using an include list or a condition on github.event_name.

Sharding only pays off if tests are independent. Suites that share a database, rely on execution order or write to the same files will produce flaky failures once split. Give each shard its own database or schema, use unique test data, and fix order dependencies before increasing the shard count.

Cancel superseded runs and right-size runners

The concurrency block in the first example cancels an in-progress run when a newer commit is pushed to the same branch or pull request. That frees runners for work that matters and stops queues from building up during busy review cycles. It deliberately does not cancel on main, where every commit should be built and deployed in order.

When jobs are CPU-bound, such as large TypeScript builds or compiling native code, larger runners can help, as can self-hosted runners with persistent caches. Measure cost against time saved; for many projects, caching and parallelism on standard runners get most of the benefit.

Key takeaways

• Measure step timings first and attack the slowest step.
• Use built-in dependency caching with npm ci, plus explicit caches for build tools.
• Cache Docker layers with BuildKit's type=gha cache and a dependency-first Dockerfile.
• Run independent checks as parallel jobs and shard long test suites with a matrix.
• Cancel superseded runs with concurrency, and skip workflows that changes cannot affect.

If slow pipelines are holding your team back, our DevOps team can tune CI/CD, build caching and deployment workflows. Let's talk.