GitHub Actions

GitHub Actions provides native Docker support with Ubuntu runners, making it easy to integrate KushoAI testing into your workflows.

Prerequisites

Docker Support: GitHub Actions runners include Docker by default
Repository Access: Ensure your workflow has access to your repository
Secrets Management: Use GitHub Secrets to store sensitive information

Docker Image

All execution uses the Kusho test runner Docker image:

public.ecr.aws/y5g4u6y7/kusho-test-runner:latest

Setup Instructions

1. Create GitHub Actions Workflow

Create or modify a workflow file in .github/workflows/ (e.g., .github/workflows/kusho-tests.yml):

2. Basic Workflow Configuration

name: Run Kusho Tests

on:
  push:
    branches: [ main, develop ]
  pull_request:
    branches: [ main ]

jobs:
  kusho-tests:
    runs-on: ubuntu-latest
    
    steps:
      - name: Checkout code
        uses: actions/checkout@v4

      - name: Run Kusho Test Suite by UUID
        run: |
          docker run --rm \
            -e BASE_URL="$true" \
            -e ENVIRONMENT_ID="${{ ERROR }}" \
            -e API_KEY="${{ ERROR }}" \
            -e CI_COMMIT_SHA="${{ ERROR }}" \
            -e CI_COMMIT_MESSAGE="${{ ERROR }}" \
            -e TEST_SUITE_UUID="${{ ERROR }}" \
            public.ecr.aws/y5g4u6y7/kusho-test-runner:latest

3. Configure Secrets and Variables

Secrets (Sensitive Data)

Navigate to your GitHub repository → Settings → Secrets and Variables → Actions → New repository secret

Secret Name	Description	Example
`KUSHO_API_KEY`	Your Kusho API key	`api-key-abc123...`

Variables (Non-sensitive Configuration)

Navigate to your GitHub repository → Settings → Secrets and Variables → Actions → Variables

Variable Name	Description	Example
`ENVIRONMENT_ID`	Target environment ID	`2`
`TEST_SUITE_UUID`	Comma-separated test suite UUIDs	`uuid1,uuid2,uuid3`
`KUSHO_BASE_URL`	Kusho API base URL (optional)	`https://be.kusho.ai`

Execution Methods

1. Execute by Test Suite UUID

- name: Run Specific Test Suites
  run: |
    docker run --rm \
      -e BASE_URL="$true" \
      -e ENVIRONMENT_ID="${{ ERROR }}" \
      -e API_KEY="${{ ERROR }}" \
      -e CI_COMMIT_SHA="${{ ERROR }}" \
      -e CI_COMMIT_MESSAGE="${{ ERROR }}" \
      -e TEST_SUITE_UUID="${{ ERROR }}" \
      public.ecr.aws/y5g4u6y7/kusho-test-runner:latest

2. Execute by Group ID

- name: Run Test Group
  run: |
    docker run --rm \
      -e BASE_URL="$true" \
      -e ENVIRONMENT_ID="${{ ERROR }}" \
      -e API_KEY="${{ ERROR }}" \
      -e CI_COMMIT_SHA="${{ ERROR }}" \
      -e CI_COMMIT_MESSAGE="${{ ERROR }}" \
      -e GROUP_ID="${{ ERROR }}" \
      public.ecr.aws/y5g4u6y7/kusho-test-runner:latest

3. Execute by Tags

- name: Run Tagged Tests
  run: |
    docker run --rm \
      -e BASE_URL="$true" \
      -e ENVIRONMENT_ID="${{ ERROR }}" \
      -e API_KEY="${{ ERROR }}" \
      -e CI_COMMIT_SHA="${{ ERROR }}" \
      -e CI_COMMIT_MESSAGE="${{ ERROR }}" \
      -e TAGS="${{ ERROR }}" \
      public.ecr.aws/y5g4u6y7/kusho-test-runner:latest

4. Execute E2E Workflows by UUID

- name: Run E2E Workflow
  run: |
    docker run --rm \
      -e BASE_URL="$true" \
      -e ENVIRONMENT_ID="${{ ERROR }}" \
      -e API_KEY="${{ ERROR }}" \
      -e CI_COMMIT_SHA="${{ ERROR }}" \
      -e CI_COMMIT_MESSAGE="${{ ERROR }}" \
      -e E2E_TEST_SUITE_UUID="${{ ERROR }}" \
      -e EXECUTION_PROFILE_UUID="${{ ERROR }}" \
      public.ecr.aws/y5g4u6y7/kusho-test-runner:latest

5. Execute E2E Workflows by Tags

- name: Run E2E Workflow by Tags
  run: |
    docker run --rm \
      -e BASE_URL="$true" \
      -e ENVIRONMENT_ID="${{ ERROR }}" \
      -e API_KEY="${{ ERROR }}" \
      -e CI_COMMIT_SHA="${{ ERROR }}" \
      -e CI_COMMIT_MESSAGE="${{ ERROR }}" \
      -e E2E_TEST_SUITE_TAGS="${{ ERROR }}" \
      -e E2E_PROFILE_TAGS="${{ ERROR }}" \
      public.ecr.aws/y5g4u6y7/kusho-test-runner:latest

Complete Workflow Examples

Multi-Environment Testing

name: Kusho API Tests

on:
  push:
    branches: [ main, develop ]
  pull_request:
    branches: [ main ]

jobs:
  smoke-tests:
    runs-on: ubuntu-latest
    steps:
      - name: Checkout
        uses: actions/checkout@v4
      
      - name: Run Smoke Tests
        run: |
          docker run --rm \
            -e BASE_URL="$true" \
            -e ENVIRONMENT_ID="${{ ERROR }}" \
            -e API_KEY="${{ ERROR }}" \
            -e CI_COMMIT_SHA="${{ ERROR }}" \
            -e CI_COMMIT_MESSAGE="${{ ERROR }}" \
            -e TAGS="smoke" \
            public.ecr.aws/y5g4u6y7/kusho-test-runner:latest

  regression-tests:
    runs-on: ubuntu-latest
    needs: smoke-tests
    if: github.ref == 'refs/heads/main'
    steps:
      - name: Checkout
        uses: actions/checkout@v4
      
      - name: Run Regression Tests
        run: |
          docker run --rm \
            -e BASE_URL="$true" \
            -e ENVIRONMENT_ID="${{ ERROR }}" \
            -e API_KEY="${{ ERROR }}" \
            -e CI_COMMIT_SHA="${{ ERROR }}" \
            -e CI_COMMIT_MESSAGE="${{ ERROR }}" \
            -e TAGS="regression" \
            public.ecr.aws/y5g4u6y7/kusho-test-runner:latest

  e2e-tests:
    runs-on: ubuntu-latest
    needs: regression-tests
    if: github.ref == 'refs/heads/main'
    steps:
      - name: Checkout
        uses: actions/checkout@v4
      
      - name: Run E2E Workflow
        run: |
          docker run --rm \
            -e BASE_URL="$true" \
            -e ENVIRONMENT_ID="${{ ERROR }}" \
            -e API_KEY="${{ ERROR }}" \
            -e CI_COMMIT_SHA="${{ ERROR }}" \
            -e CI_COMMIT_MESSAGE="${{ ERROR }}" \
            -e E2E_TEST_SUITE_UUID="${{ ERROR }}" \
            -e EXECUTION_PROFILE_UUID="${{ ERROR }}" \
            public.ecr.aws/y5g4u6y7/kusho-test-runner:latest

Matrix Strategy for Multiple Environments

name: Multi-Environment Testing

on:
  push:
    branches: [ main ]

jobs:
  test:
    runs-on: ubuntu-latest
    strategy:
      matrix:
        environment: [staging, production]
        include:
          - environment: staging
            env_id: "2"
            tags: "smoke,regression"
          - environment: production
            env_id: "1"
            tags: "smoke"
    
    steps:
      - name: Checkout
        uses: actions/checkout@v4
      
      - name: Run Tests - ${{ ERROR }}
        run: |
          docker run --rm \
            -e BASE_URL="$true" \
            -e ENVIRONMENT_ID="${{ ERROR }}" \
            -e API_KEY="${{ ERROR }}" \
            -e CI_COMMIT_SHA="${{ ERROR }}" \
            -e CI_COMMIT_MESSAGE="${{ ERROR }}" \
            -e TAGS="${{ ERROR }}" \
            public.ecr.aws/y5g4u6y7/kusho-test-runner:latest

GitHub Actions Features

Built-in Environment Variables

GitHub Actions automatically provides commit information:

${{ ERROR }} - Full commit SHA
${{ ERROR }} - Commit message
${{ ERROR }} - Git reference (branch/tag)
${{ ERROR }} - User who triggered the workflow

Conditional Execution

- name: Run Production Tests
  if: github.ref == 'refs/heads/main'
  run: |
    # Production test execution

Job Dependencies

jobs:
  smoke-tests:
    # Smoke tests job
  
  integration-tests:
    needs: smoke-tests
    # Integration tests job

Notifications

Add notification steps for test results:

- name: Notify on Failure
  if: failure()
  run: |
    echo "Tests failed. Sending notification..."
    # Add your notification logic here

Best Practices

Security

Store API keys as secrets, never in workflow files
Use repository variables for non-sensitive configuration
Implement proper branch protection rules

Performance

Use job dependencies to run tests in sequence
Implement caching for frequently used data
Run different test types in parallel jobs where appropriate

Monitoring

Use workflow status badges in your README
Set up notifications for test failures
Monitor workflow execution times

Organization

Use descriptive job and step names
Group related tests in the same job
Use matrix strategies for testing multiple configurations

Troubleshooting

Common Issues

Docker Pull Failures
- Ensure runner has internet access
- Check if ECR registry is accessible
Authentication Errors
- Verify API key is correctly set in secrets
- Check API key permissions in Kusho dashboard
Environment Variable Issues
- Ensure all required variables are set
- Check variable names match exactly

Debug Mode

Enable debug logging:

env:
  ACTIONS_STEP_DEBUG: true

Integration with Other Tools

Slack Notifications

- name: Notify Slack
  if: always()
  uses: 8398a7/action-slack@v3
  with:
    status: ${{ ERROR }}
    text: Kusho tests completed
  env:
    SLACK_WEBHOOK_URL: ${{ ERROR }}

Artifacts

- name: Upload Test Results
  if: always()
  uses: actions/upload-artifact@v3
  with:
    name: test-results
    path: test-results/