Enterprise CI/CD Pipelines: From Commits to Production with Confidence

The testing pyramid: the foundation of every pipeline

The cost of a bug multiplies exponentially the later it's caught. A bug caught in unit tests costs 1x. In integration tests, 10x. In staging, 50x. In production, 500x. The testing pyramid defines the correct coverage distribution to maximize early detection at minimum execution cost.

• Unit tests (pyramid base, 70%): fast (<5 minutes for the full suite), no external dependencies, high business logic coverage. If they take more than 10 minutes, the team starts skipping them.
• Integration tests (30%): verify interactions with databases, external APIs, and other internal services. Use Docker containers for dependencies (testcontainers).
• E2E tests (10%): verify complete critical user flows. Only for the highest business-value happy paths — comprehensive E2E tests are slow and brittle.

Branch strategy: Trunk-Based Development for agile teams

Gitflow (develop, release, hotfix branches) was the standard a decade ago. For teams that want to deploy multiple times per day, Trunk-Based Development is the right model: all developers integrate to the main trunk frequently (ideally daily), and releases happen directly from trunk.

# Trunk-Based Development — simplified flow
main (trunk)
├── feature/TICKET-123-add-checkout    # short-lived branch (<2 days)
└── feature/TICKET-456-fix-cart        # merged to main via PR with mandatory CI

# NO: long-lived branches
# NO: develop, release, hotfix as integration model
# YES: feature flags to decouple deploy from release

The key to Trunk-Based Development is feature flags. Incomplete feature code is merged to trunk protected by a disabled flag. The feature is activated in production when ready, regardless of when the code was merged.

Production-grade GitLab CI pipeline

stages:
  - test
  - build
  - security
  - deploy-staging
  - integration-test
  - deploy-production

variables:
  IMAGE: ${CI_REGISTRY_IMAGE}:${CI_COMMIT_SHORT_SHA}

unit-tests:
  stage: test
  image: node:20-alpine
  script:
    - npm ci
    - npm run test:unit -- --coverage

build-image:
  stage: build
  image: docker:24
  services: [docker:24-dind]
  script:
    - docker build -t ${IMAGE} .
    - docker push ${IMAGE}
  only: [main]

trivy-scan:
  stage: security
  image: aquasec/trivy:latest
  script:
    - trivy image --exit-code 1 --severity HIGH,CRITICAL ${IMAGE}
  allow_failure: false
  only: [main]

deploy-production:
  stage: deploy-production
  environment: production
  when: manual
  script:
    - kubectl set image deployment/api api=${IMAGE} -n production
    - kubectl rollout status deployment/api -n production --timeout=5m
  only: [main]

Deployment Strategies: choosing based on risk

Rolling Update

Gradual deployment replacing pods one at a time. Low implementation cost. Rollback takes N minutes (rolling update time). Ideal for most low-risk changes to stateless services.

Blue/Green Deployment

Two identical environments (blue=current, green=new version). The switch is instantaneous at the load balancer level. Rollback in seconds. Cost: double infra during deployment. Ideal for high-impact changes where instant rollback justifies the cost.

Canary Deployment

Deploy to a small percentage of traffic (5%, 10%) before full rollout. Detects real production problems with minimal impact. Requires feature flags or traffic splitting at the ingress/service mesh. The most robust strategy for critical services with high user volume.

# Argo Rollouts — Canary with automated analysis
apiVersion: argoproj.io/v1alpha1
kind: Rollout
metadata:
  name: api
spec:
  strategy:
    canary:
      steps:
      - setWeight: 5    # 5% traffic to canary
      - pause: {duration: 10m}
      - analysis:
          templates:
          - templateName: success-rate
      - setWeight: 50
      - pause: {duration: 5m}
      - setWeight: 100

Secrets in pipelines: the most common mistake

Hardcoding secrets in pipelines (unmasked CI variables, kubectl apply with tokens in scripts) is the most common mistake in enterprise pipelines. The correct pattern: deploy credentials live in the cloud secret manager (AWS Secrets Manager, GCP Secret Manager, HashiCorp Vault), and the pipeline obtains them at runtime with temporary credentials via OIDC.