CI/CD Pipeline

Design, debug, and optimize CI/CD pipelines across major platforms. Covers workflow creation, build optimization, caching, parallelization, secret management, and deployment strategies.

Workflow

Step 1: Assess the Project

Gather context about the project and its CI/CD needs:

• Detect platform: Look for .github/workflows/, .gitlab-ci.yml, Jenkinsfile, .circleci/config.yml.
• Identify stack: Check package.json, requirements.txt, go.mod, Cargo.toml, pom.xml, Gemfile for language and dependency info.
• Review existing config: If a pipeline exists, read it fully to understand current stages, jobs, and pain points.
• Check build times: If debugging performance, look at recent run logs or timing annotations.
• Review secrets: Check for references to secrets/environment variables that need configuration.

Step 2: Pipeline Design

Design or restructure the pipeline with these stages:

Standard Pipeline Stages:

1. Lint & Format — Fast feedback on code style (eslint, prettier, black, gofmt)
2. Type Check — Static analysis (TypeScript, mypy, flow)
3. Unit Tests — Fast, isolated tests
4. Build — Compile, bundle, or package
5. Integration Tests — Tests requiring databases, APIs, or services
6. Security Scan — Dependency audit, SAST, secret scanning
7. Deploy to Staging — Automated deploy to preview environment
8. E2E Tests — Full end-to-end tests against staging
9. Deploy to Production — Gated production deployment

GitHub Actions Structure:

name: CI/CD
on:
  push:
    branches: [main]
  pull_request:
    branches: [main]

concurrency:
  group: ${{ github.workflow }}-${{ github.ref }}
  cancel-in-progress: true

jobs:
  lint:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with:
          node-version: 20
          cache: 'npm'
      - run: npm ci
      - run: npm run lint

  test:
    runs-on: ubuntu-latest
    needs: lint
    strategy:
      matrix:
        node-version: [18, 20, 22]
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with:
          node-version: ${{ matrix.node-version }}
          cache: 'npm'
      - run: npm ci
      - run: npm test -- --coverage
      - uses: actions/upload-artifact@v4
        with:
          name: coverage-${{ matrix.node-version }}
          path: coverage/

  build:
    runs-on: ubuntu-latest
    needs: test
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-node@v4
        with:
          node-version: 20
          cache: 'npm'
      - run: npm ci
      - run: npm run build
      - uses: actions/upload-artifact@v4
        with:
          name: build
          path: dist/

  deploy:
    runs-on: ubuntu-latest
    needs: build
    if: github.ref == 'refs/heads/main'
    environment: production
    steps:
      - uses: actions/download-artifact@v4
        with:
          name: build
          path: dist/
      - name: Deploy
        run: echo "Deploy to production"

GitLab CI Structure:

stages:
  - lint
  - test
  - build
  - deploy

variables:
  NODE_VERSION: "20"

.node-setup:
  image: node:${NODE_VERSION}
  cache:
    key: ${CI_COMMIT_REF_SLUG}
    paths:
      - node_modules/

lint:
  extends: .node-setup
  stage: lint
  script:
    - npm ci
    - npm run lint

test:
  extends: .node-setup
  stage: test
  script:
    - npm ci
    - npm test -- --coverage
  coverage: '/All files[^|]*\|[^|]*\s+([\d\.]+)/'
  artifacts:
    reports:
      coverage_report:
        coverage_format: cobertura
        path: coverage/cobertura-coverage.xml

Step 3: Build Optimization

Apply these optimization techniques:

Caching strategies:

• Cache dependency directories (node_modules/, .venv/, ~/.m2/, ~/.cache/go-build/).
• Use lock file hashes as cache keys for deterministic invalidation.
• Layer caches: dependencies first (changes rarely), then build artifacts.

Parallelization:

• Run lint, type-check, and unit tests in parallel (no dependencies between them).
• Use matrix builds for multi-version testing.
• Split large test suites across parallel runners with test splitting tools.

Reducing redundant work:

• Use concurrency groups to cancel in-progress runs on the same PR.
• Skip jobs when only docs or non-code files changed using path filters.
• Use actions/cache with proper cache keys to avoid re-downloading dependencies.
• Build Docker images with multi-stage builds and layer caching.

Incremental builds:

• Use tools like turborepo, nx, or bazel for incremental/affected-only builds in monorepos.
• Cache build outputs between runs when possible.

Step 4: Debugging Failures

When a pipeline is failing:

1. Read the error output: Identify the exact command and error message.
2. Check for flaky tests: Look for tests that pass on retry (use --retry flags or rerun).
3. Environment differences: Compare CI environment with local (OS, Node version, env vars).
4. Resource limits: Check for OOM kills, disk space issues, or timeout limits.
5. Secret availability: Verify secrets are configured for the branch/environment.
6. Dependency issues: Check for version mismatches, network failures during install.
7. Permission issues: Verify the CI runner has access to required resources.

Common fixes:

• npm ci instead of npm install for reproducible installs.
• Pin action versions to full SHA hashes for security and reproducibility.
• Add --frozen-lockfile (pnpm) or --immutable (yarn) to prevent lockfile changes.
• Set timeout-minutes on jobs to prevent hung builds.

Step 5: Deployment Strategies

Blue-Green Deployment: Two identical environments; switch traffic after verification. Canary Deployment: Route a percentage of traffic to the new version, monitor, then promote. Rolling Deployment: Gradually replace instances with the new version. Feature Flags: Deploy code behind flags, enable incrementally.

Output Format

## CI/CD Pipeline Configuration

### Platform: <platform>
### Stack: <language/framework>

### Pipeline Architecture
<stage diagram or description>

### Configuration
<full YAML/Groovy pipeline config>

### Optimization Notes
- Estimated build time: <duration>
- Caching: <what is cached and expected hit rate>
- Parallelization: <which jobs run in parallel>

### Secrets Required
| Secret Name | Purpose | Where to Configure |
|-------------|---------|-------------------|
| <name> | <purpose> | <location> |

### Deployment Strategy
<strategy with rollback procedure>

Examples

User: "Set up GitHub Actions for my Node.js project" Action: Read package.json, detect test/lint/build scripts, create a complete workflow YAML with caching, matrix testing, and deployment.

User: "The build is failing on CI but works locally" Action: Read the workflow config and recent run logs, identify environment differences, suggest fixes.

User: "Our pipeline takes 20 minutes, can we speed it up?" Action: Analyze the current workflow, identify sequential jobs that can be parallelized, add caching, suggest path filters for skip conditions.

User: "Migrate from Jenkins to GitHub Actions" Action: Read the Jenkinsfile, map stages to GitHub Actions jobs, translate Groovy DSL to YAML, handle secret migration.

Best Practices

• Pin all action versions to full commit SHAs, not tags (prevents supply chain attacks).
• Use concurrency groups to cancel stale runs and save CI minutes.
• Add timeout-minutes to every job to prevent hung builds from consuming resources.
• Store secrets in the platform's secret manager, never in the repo or pipeline config.
• Use environment protection rules for production deployments (require approval).
• Add status badges to the README so build health is visible.
• Set up notifications (Slack, email) for failed builds on the main branch only.
• Keep pipelines DRY with reusable workflows (GitHub Actions) or YAML anchors (GitLab CI).
• Run security scans (dependency audit, SAST) on every PR, not just main.
• Use artifact uploads to pass build outputs between jobs instead of rebuilding.