# How CI/CD Execution Works

## Docker-Based Test Execution

All CI/CD test execution in KushoAI is powered by Docker containers, providing consistent and reliable test execution across all supported platforms. The Docker test runner handles various execution modes and integrates seamlessly with your existing CI/CD workflows.

## Docker Image

The Kusho test runner is available as a public Docker image:
```
public.ecr.aws/y5g4u6y7/kusho-test-runner:latest
```

This image contains all necessary dependencies and tools to execute API tests and E2E workflows in any Docker-compatible environment.

## Execution Methods

### 1. API Test Suite Execution by UUID

Execute specific test suites using their unique identifiers:

```bash
docker run --rm \
  -e BASE_URL="https://be.kusho.ai" \
  -e ENVIRONMENT_ID="2" \
  -e API_KEY="your-api-key" \
  -e CI_COMMIT_SHA="commit-hash" \
  -e CI_COMMIT_MESSAGE="commit message" \
  -e TEST_SUITE_UUID="uuid1,uuid2,uuid3" \
  public.ecr.aws/y5g4u6y7/kusho-test-runner:latest
```

**Use Cases:**
- Testing specific APIs or endpoints.
- Targeted testing for specific features.
- Running critical path tests.

### 2. API Test Suite Execution by Group ID

Execute all test suites within a specific group:

```bash
docker run --rm \
  -e BASE_URL="https://be.kusho.ai" \
  -e ENVIRONMENT_ID="2" \
  -e API_KEY="your-api-key" \
  -e CI_COMMIT_SHA="commit-hash" \
  -e CI_COMMIT_MESSAGE="commit message" \
  -e GROUP_ID="87" \
  public.ecr.aws/y5g4u6y7/kusho-test-runner:latest
```

**Use Cases:**
- Running all tests for a specific service or module.
- Organized test execution by feature area.
- Team-based test organization.

### 3. API Test Suite Execution by Tags

Execute test suites filtered by specific tags:

```bash
docker run --rm \
  -e BASE_URL="https://be.kusho.ai" \
  -e ENVIRONMENT_ID="2" \
  -e API_KEY="your-api-key" \
  -e CI_COMMIT_SHA="commit-hash" \
  -e CI_COMMIT_MESSAGE="commit message" \
  -e TAGS="smoke,regression,critical" \
  public.ecr.aws/y5g4u6y7/kusho-test-runner:latest
```

**Use Cases:**
- Running smoke tests before deployment.
- Executing regression tests after major changes.
- Targeted testing based on test characteristics.

### 4. E2E Workflow Execution by UUID

Execute end-to-end workflows with execution profiles:

```bash
docker run --rm \
  -e BASE_URL="https://be.kusho.ai" \
  -e ENVIRONMENT_ID="2" \
  -e API_KEY="your-api-key" \
  -e CI_COMMIT_SHA="commit-hash" \
  -e CI_COMMIT_MESSAGE="E2E testing" \
  -e E2E_TEST_SUITE_UUID="e5d1b703-a798-4bd9-becb-8977bc2a1ec3" \
  -e EXECUTION_PROFILE_UUID="execution-profile-uuid" \
  public.ecr.aws/y5g4u6y7/kusho-test-runner:latest
```

**Use Cases:**
- Testing complete user journeys.
- Validating API integrations and data flow.
- Comprehensive workflow testing.

### 5. E2E Workflow Execution by Tags

Execute end-to-end workflows using tag-based filtering for both test suites and execution profiles:

```bash
docker run --rm \
  -e BASE_URL="https://be.kusho.ai" \
  -e ENVIRONMENT_ID="2" \
  -e API_KEY="your-api-key" \
  -e CI_COMMIT_SHA="commit-hash" \
  -e CI_COMMIT_MESSAGE="E2E testing with tags" \
  -e E2E_TEST_SUITE_TAGS="smoke,api,critical" \
  -e E2E_PROFILE_TAGS="fast,regression" \
  public.ecr.aws/y5g4u6y7/kusho-test-runner:latest
```

**Use Cases:**
- Smoke testing: Execute workflows tagged with "smoke" using "quick" profiles.
- Critical regression: Run "api" and "core" workflows with "critical" profiles.
- Nightly builds: Execute "full" workflows with "comprehensive" profiles.
- Flexible test execution based on workflow and profile characteristics.

## Environment Variables Reference

### Required Variables

| Variable | Description | Example |
|----------|-------------|---------|
| `API_KEY` | Authentication key for Kusho API | `"api-key-dadsadsadsadsaasa"` |
| `ENVIRONMENT_ID` | Target environment ID | `"2"` |

### Optional Variables

| Variable | Description | Default | Example |
|----------|-------------|---------|---------|
| `BASE_URL` | Kusho API base URL | `"https://be.kusho.ai"` | `"https://on-prem-kusho-backend.yourdomain.com"` |
| `EMAIL` | Email for notifications | None | `"user@example.com"` |
| `CI_COMMIT_SHA` | Git commit hash | None | `"a1b2c3d4"` |
| `CI_COMMIT_MESSAGE` | Git commit message | None | `"Fix user authentication"` |

### Execution Mode Variables (One Required)

| Variable | Description | Example |
|----------|-------------|---------|
| `TEST_SUITE_UUID` | Comma-separated test suite UUIDs | `"uuid1,uuid2"` |
| `GROUP_ID` | Test suite group ID | `"87"` |
| `TAGS` | Comma-separated tags for API test suites | `"smoke,regression"` |
| `E2E_TEST_SUITE_UUID` + `EXECUTION_PROFILE_UUID` | E2E workflow and profile UUIDs | See E2E UUID example above |
| `E2E_TEST_SUITE_TAGS` + `E2E_PROFILE_TAGS` | Tag-based E2E execution | `"smoke,api"` + `"fast,regression"` |

## Output and Exit Codes

### Test Result Output

The test runner provides detailed output for different execution modes:

**API Test Suite Output:**
- Test case descriptions.
- HTTP status codes with color coding.
- Test execution status (PASS/FAIL).
- Assertion results.

**E2E Test Suite Output:**
- Workflow information and progress.
- Combination execution details.
- Individual test case results with color coding:
  - **Status Codes**: Green (2xx), Yellow (4xx), Red (5xx), Gray (N/A)
  - **Assertions**: Green (PASS), Red (FAIL), Gray (N/A)

### Exit Codes

- `0`: All tests passed successfully.
- `1`: One or more tests failed, or an execution error occurred.

These exit codes integrate with CI/CD systems to determine build success or failure.

## E2E Workflow Execution Details

E2E test suites execute test cases in combinations based on the workflow structure:

1. **Workflow Graph**: Defines the sequence of test suites connected by data flow.
2. **Execution Profile**: Specifies which test cases to run from each test suite (refer [this section](/17-e2e/index.md/#execution-profiles) for more details).
3. **Cartesian Product**: Creates all possible combinations of test cases across connected test suites.

### Example E2E Execution

For a workflow: `TestSuite1 (2 test cases) → TestSuite2 (3 test cases)`

The system generates 6 execution combinations:
- TS1-TC1 → TS2-TC1
- TS1-TC1 → TS2-TC2  
- TS1-TC1 → TS2-TC3
- TS1-TC2 → TS2-TC1
- TS1-TC2 → TS2-TC2
- TS1-TC2 → TS2-TC3

Each combination represents a complete execution path through the workflow with proper data flow validation.

## Best Practices

### Security
- Store sensitive data (API keys) as platform secrets, never in code.
- Use environment-specific configurations.
- Implement proper access controls for test execution.

### Performance
- Use appropriate test selection (UUID, Group, Tags) to minimize execution time.
- Run critical tests early in pipelines for fast feedback.
- Implement parallel execution where supported.

### Monitoring
- Set up notifications for test failures.
- Track test execution trends over time.
- Monitor test execution duration and success rates.

### Organization
- Use descriptive commit messages for better test result tracking.
- Organize tests using groups and tags for flexible execution.
- Maintain execution profiles for consistent E2E testing.