Sandbox testing in a GitHub Actions pipeline is straightforward to set up and surprisingly easy to get wrong.
The setup part is a few workflow steps. The getting-wrong part is what happens after the first few upstream service deployments, when the sandbox environment starts validating against mock files that no longer accurately represent the services they describe. The tests keep passing. The sandbox keeps reporting green. The production failures that follow look like deployment issues when they are actually sandbox currency issues.
This guide covers both parts: how to structure a GitHub Actions workflow for sandbox testing, and how to keep the sandbox environment current as the distributed system it validates against continues to change.
What Sandbox Testing Requires Before the Pipeline
Before writing any GitHub Actions YAML, the sandbox environment needs three things defined.
The service boundary being tested. Sandbox testing validates how a service behaves when interacting with its dependencies under conditions that approximate production. Define which service is under test and which dependencies it needs to interact with - upstream APIs, downstream services, external providers.
The behavioral representations of each dependency. In a GitHub Actions context, these are typically mock files or recorded traffic fixtures that represent how each dependency responds. These can be VCR cassettes, WireMock stubs, or any format the testing framework supports. The key property is that they need to reflect current dependency behavior - not historical behavior from when the fixtures were first written.
The test scenarios to run. The tests that exercise the service against its dependency representations. These validate whether the service handles current dependency behavior correctly.
With those three elements defined, the pipeline structure is straightforward.
Basic GitHub Actions Workflow Structure
A sandbox testing workflow in GitHub Actions runs after unit tests and before deployment to any shared environment. This placement catches integration failures at the point where they are cheapest to fix before any downstream impact.
name: Sandbox Testing
on:
push:
branches: [main, develop]
pull_request:
branches: [main]
jobs:
unit-tests:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- name: Set up environment
uses: actions/setup-node@v4
with:
node-version: '20'
- name: Install dependencies
run: npm ci
- name: Run unit tests
run: npm test
sandbox-tests:
runs-on: ubuntu-latest
needs: unit-tests
steps:
- uses: actions/checkout@v4
- name: Set up environment
uses: actions/setup-node@v4
with:
node-version: '20'
- name: Install dependencies
run: npm ci
- name: Start dependent services
run: docker-compose -f docker-compose.sandbox.yml up -d
- name: Wait for services to be ready
run: |
timeout 60 bash -c 'until curl -s http://localhost:8080/health; do sleep 2; done'
- name: Run sandbox tests
run: npm run test:sandbox
- name: Stop dependent services
if: always()
run: docker-compose -f docker-compose.sandbox.yml down
deploy:
runs-on: ubuntu-latest
needs: sandbox-tests
if: github.ref == 'refs/heads/main'
steps:
- name: Deploy to staging
run: echo "Deploy after sandbox tests pass"
The needs directive enforces the order. Unit tests run first. Sandbox tests run only when unit tests pass. Deployment runs only when sandbox tests pass. A failure at any stage stops the pipeline before the next stage begins.
Setting Up the Docker Compose Sandbox Environment
The docker-compose.sandbox.yml referenced in the workflow defines the services and dependency representations the sandbox tests run against.
version: '3.8'
services:
app:
build: .
ports:
- "3000:3000"
environment:
- NODE_ENV=sandbox
- PAYMENT_API_URL=http://payment-mock:4000
- NOTIFICATION_API_URL=http://notification-mock:4001
depends_on:
- payment-mock
- notification-mock
payment-mock:
image: wiremock/wiremock:latest
ports:
- "4000:8080"
volumes:
- ./sandbox/fixtures/payment:/home/wiremock/mappings
notification-mock:
image: wiremock/wiremock:latest
ports:
- "4001:8080"
volumes:
- ./sandbox/fixtures/notification:/home/wiremock/mappings
The application service points to mock services rather than real upstream APIs. The mock services load fixture files from the sandbox/fixtures directory. These fixtures are what the sandbox tests validate against.
This setup works correctly when the fixtures are current. The challenge begins when the real payment API or notification API deploys a change that the fixture files do not reflect.
The Sandbox Currency Problem in GitHub Actions Pipelines
In the workflow above, the fixture files in sandbox/fixtures/payment and sandbox/fixtures/notification were written at a point in time. They represent how the payment and notification services responded when the fixtures were created.
Those services keep deploying. The payment service may update its error response format. The notification service may add a required field. The fixture files do not update automatically when this happens. Someone on the team has to notice the upstream change, update the relevant fixture files, commit them, and push. Until that happens, the sandbox is validating against stale representations.
In a GitHub Actions context, this creates a specific failure mode: the sandbox tests pass in CI because they are running against the fixture files, which reflect old behavior. The deployment goes out. Production encounters the new behavior the fixture files did not capture. The production failure looks like a deployment issue. The root cause is a sandbox currency issue that the pipeline had no mechanism to surface.
The rate at which this problem compounds is proportional to how many upstream services the application integrates with and how frequently those services deploy. Two upstream services each deploying twice a week generate four potential fixture update events per week. Ten upstream services generate twenty. Manual fixture maintenance cannot scale to this rate reliably under delivery pressure.
Adding Traffic Capture to Keep Sandbox Fixtures Current
The structural fix for sandbox currency in a GitHub Actions pipeline is changing how fixture files are generated from manually authored specifications to recordings of real service behavior.
Rather than writing fixture files by hand and updating them after each upstream change, the pipeline captures real traffic between the service and its dependencies and generates fixtures from those observations. When an upstream service changes, the next capture reflects the updated behavior automatically.
Keploy supports this approach for API-driven services. It records real HTTP traffic between services during a capture run and generates test fixtures from those actual interactions. These fixtures can be committed to the repository and used in subsequent sandbox test runs, or regenerated on each pipeline run from fresh traffic observations.
Here is how this integrates into a GitHub Actions workflow:
name: Sandbox Testing With Traffic Capture
on:
push:
branches: [main, develop]
pull_request:
branches: [main]
schedule:
- cron: '0 6 * * *' # Daily fixture refresh at 6am UTC
jobs:
refresh-fixtures:
runs-on: ubuntu-latest
if: github.event_name == 'schedule'
steps:
- uses: actions/checkout@v4
with:
token: ${{ secrets.GITHUB_TOKEN }}
- name: Set up environment
uses: actions/setup-node@v4
with:
node-version: '20'
- name: Install dependencies
run: npm ci
- name: Install Keploy
run: |
curl --silent --location "https://github.com/keploy/keploy/releases/latest/download/keploy_linux_amd64.tar.gz" | tar xz -C /tmp
sudo mv /tmp/keploy /usr/local/bin
- name: Start real upstream services
run: docker-compose -f docker-compose.integration.yml up -d
env:
PAYMENT_API_URL: ${{ secrets.PAYMENT_API_URL }}
NOTIFICATION_API_URL: ${{ secrets.NOTIFICATION_API_URL }}
- name: Capture real service traffic
run: |
keploy record --app="npm start" \
--delay=5 \
--path=./sandbox/fixtures
env:
NODE_ENV: capture
- name: Run traffic generation script
run: npm run generate:sandbox-traffic
- name: Stop capture
run: keploy stop
- name: Commit updated fixtures
run: |
git config --local user.email "github-actions@github.com"
git config --local user.name "GitHub Actions"
git add sandbox/fixtures/
git diff --quiet && git diff --staged --quiet || git commit -m "chore: refresh sandbox fixtures from upstream traffic [skip ci]"
git push
unit-tests:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- name: Set up environment
uses: actions/setup-node@v4
with:
node-version: '20'
- name: Install dependencies
run: npm ci
- name: Run unit tests
run: npm test
sandbox-tests:
runs-on: ubuntu-latest
needs: unit-tests
steps:
- uses: actions/checkout@v4
- name: Set up environment
uses: actions/setup-node@v4
with:
node-version: '20'
- name: Install dependencies
run: npm ci
- name: Install Keploy
run: |
curl --silent --location "https://github.com/keploy/keploy/releases/latest/download/keploy_linux_amd64.tar.gz" | tar xz -C /tmp
sudo mv /tmp/keploy /usr/local/bin
- name: Run sandbox tests with captured fixtures
run: |
keploy test --app="npm start" \
--delay=5 \
--path=./sandbox/fixtures \
--coverage
env:
NODE_ENV: sandbox
- name: Upload coverage report
uses: actions/upload-artifact@v4
with:
name: sandbox-coverage
path: coverage/
deploy:
runs-on: ubuntu-latest
needs: sandbox-tests
if: github.ref == 'refs/heads/main'
steps:
- name: Deploy to staging
run: echo "Deploy after sandbox tests pass"
The refresh-fixtures job runs on a schedule - daily at 6am UTC in this example. It connects to real upstream services, captures actual traffic, generates updated fixture files, and commits them back to the repository. The sandbox-tests job then runs against the most recently captured fixtures rather than against fixtures that may be weeks outdated.
This workflow separates the two concerns that manual fixture maintenance conflates: when to update fixtures and whether current fixtures produce passing tests. The scheduled refresh handles the first concern automatically. The sandbox test run handles the second.
Handling Non-Deterministic Fields
Real traffic captures include non-deterministic fields - timestamps, request IDs, session tokens, generated identifiers - that change with every request. If the sandbox tests assert against these fields, they will fail on every run regardless of whether the underlying behavior has changed.
Keploy handles this through automatic noise detection: it identifies fields that vary across multiple observations of the same interaction and excludes those fields from test assertions automatically. This means the sandbox tests assert against the behavioral properties that matter - response structure, status codes, business logic outputs - rather than against the ephemeral values that differ between requests.
Explicit configuration is available for cases where automatic detection needs guidance:
# keploy.yml
noiseConfig:
global:
header:
- "x-request-id"
- "x-correlation-id"
body:
- "timestamp"
- "createdAt"
- "requestId"
This configuration tells Keploy to exclude these fields from sandbox test assertions across all captured interactions, preventing false positives from non-deterministic values while still validating the behavioral properties that indicate whether the integration is working correctly.
Managing Fixture Freshness Visibility
The scheduled refresh approach solves the currency problem but introduces a visibility gap: how do developers know how current the fixtures are?
Adding fixture timestamp tracking to the workflow surfaces this information explicitly:
- name: Record fixture refresh timestamp
run: |
echo "last_refreshed: $(date -u +%Y-%m-%dT%H:%M:%SZ)" > sandbox/fixtures/metadata.yml
echo "upstream_services:" >> sandbox/fixtures/metadata.yml
echo " payment: ${{ env.PAYMENT_SERVICE_VERSION }}" >> sandbox/fixtures/metadata.yml
echo " notification: ${{ env.NOTIFICATION_SERVICE_VERSION }}" >> sandbox/fixtures/metadata.yml
git add sandbox/fixtures/metadata.yml
The metadata.yml file committed alongside the fixtures records when they were last refreshed and which upstream service versions they were captured from. This gives developers a clear answer to the question "how current are these fixtures" without requiring investigation of git history.
A check in the sandbox test job can surface a warning when fixtures exceed a defined staleness threshold:
- name: Check fixture freshness
run: |
LAST_REFRESHED=$(grep 'last_refreshed' sandbox/fixtures/metadata.yml | cut -d' ' -f2)
HOURS_SINCE=$((( $(date -u +%s) - $(date -u -d "$LAST_REFRESHED" +%s) ) / 3600))
if [ $HOURS_SINCE -gt 48 ]; then
echo "::warning::Sandbox fixtures are ${HOURS_SINCE} hours old. Consider triggering a manual refresh."
fi
This check does not fail the pipeline - stale fixtures that still pass tests are better than no sandbox layer. It surfaces the staleness as a warning that developers can act on rather than discovering it through a production failure.
What This Pipeline Achieves
The complete workflow described above integrates sandbox testing as a required validation layer between unit tests and deployment, with automated fixture currency maintenance running on a daily schedule.
The sandbox tests catch the specific category of failure that unit tests cannot: failures that originate from changes in upstream service behavior that the unit tests had no knowledge of. The automated fixture refresh ensures that the sandbox environment reflects current upstream behavior rather than accumulating drift as services deploy independently.
The combination produces a pipeline where green means something more specific than it did with unit tests alone. A passing sandbox test run means the service works correctly against how its dependencies are actually currently behaving - not just against how a developer described them in a mock file at some earlier point.