# How CI/CD Execution Works

All CI/CD test execution in KushoAI is powered by Docker containers, providing consistent and reliable test execution across all supported platforms. The Docker test runner handles various execution modes and integrates seamlessly with your existing CI/CD workflows.

The Kusho test runner is available as a public Docker image:

public.ecr.aws/y5g4u6y7/kusho-test-runner:latest

This image contains all necessary dependencies and tools to execute API tests and E2E workflows in any Docker-compatible environment.

Execute specific test suites using their unique identifiers:

docker run --rm \
  -e BASE_URL="https://be.kusho.ai" \
  -e ENVIRONMENT_ID="2" \
  -e API_KEY="your-api-key" \
  -e CI_COMMIT_SHA="commit-hash" \
  -e CI_COMMIT_MESSAGE="commit message" \
  -e TEST_SUITE_UUID="uuid1,uuid2,uuid3" \
  public.ecr.aws/y5g4u6y7/kusho-test-runner:latest

Use Cases:

• Testing specific APIs or endpoints.
• Targeted testing for specific features.
• Running critical path tests.

Execute all test suites within a specific group:

docker run --rm \
  -e BASE_URL="https://be.kusho.ai" \
  -e ENVIRONMENT_ID="2" \
  -e API_KEY="your-api-key" \
  -e CI_COMMIT_SHA="commit-hash" \
  -e CI_COMMIT_MESSAGE="commit message" \
  -e GROUP_ID="87" \
  public.ecr.aws/y5g4u6y7/kusho-test-runner:latest

Use Cases:

• Running all tests for a specific service or module.
• Organized test execution by feature area.
• Team-based test organization.

Execute test suites filtered by specific tags:

docker run --rm \
  -e BASE_URL="https://be.kusho.ai" \
  -e ENVIRONMENT_ID="2" \
  -e API_KEY="your-api-key" \
  -e CI_COMMIT_SHA="commit-hash" \
  -e CI_COMMIT_MESSAGE="commit message" \
  -e TAGS="smoke,regression,critical" \
  public.ecr.aws/y5g4u6y7/kusho-test-runner:latest

Use Cases:

• Running smoke tests before deployment.
• Executing regression tests after major changes.
• Targeted testing based on test characteristics.

Execute end-to-end workflows with execution profiles:

docker run --rm \
  -e BASE_URL="https://be.kusho.ai" \
  -e ENVIRONMENT_ID="2" \
  -e API_KEY="your-api-key" \
  -e CI_COMMIT_SHA="commit-hash" \
  -e CI_COMMIT_MESSAGE="E2E testing" \
  -e E2E_TEST_SUITE_UUID="e5d1b703-a798-4bd9-becb-8977bc2a1ec3" \
  -e EXECUTION_PROFILE_UUID="execution-profile-uuid" \
  public.ecr.aws/y5g4u6y7/kusho-test-runner:latest

Use Cases:

• Testing complete user journeys.
• Validating API integrations and data flow.
• Comprehensive workflow testing.

Execute end-to-end workflows using tag-based filtering for both test suites and execution profiles:

docker run --rm \
  -e BASE_URL="https://be.kusho.ai" \
  -e ENVIRONMENT_ID="2" \
  -e API_KEY="your-api-key" \
  -e CI_COMMIT_SHA="commit-hash" \
  -e CI_COMMIT_MESSAGE="E2E testing with tags" \
  -e E2E_TEST_SUITE_TAGS="smoke,api,critical" \
  -e E2E_PROFILE_TAGS="fast,regression" \
  public.ecr.aws/y5g4u6y7/kusho-test-runner:latest

Use Cases:

• Smoke testing: Execute workflows tagged with "smoke" using "quick" profiles.
• Critical regression: Run "api" and "core" workflows with "critical" profiles.
• Nightly builds: Execute "full" workflows with "comprehensive" profiles.
• Flexible test execution based on workflow and profile characteristics.

The test runner provides detailed output for different execution modes:

API Test Suite Output:

• Test case descriptions.
• HTTP status codes with color coding.
• Test execution status (PASS/FAIL).
• Assertion results.

E2E Test Suite Output:

• Workflow information and progress.
• Combination execution details.
• Individual test case results with color coding:
  • Status Codes: Green (2xx), Yellow (4xx), Red (5xx), Gray (N/A)
  • Assertions: Green (PASS), Red (FAIL), Gray (N/A)

• 0: All tests passed successfully.
• 1: One or more tests failed, or an execution error occurred.

These exit codes integrate with CI/CD systems to determine build success or failure.

E2E test suites execute test cases in combinations based on the workflow structure:

1. Workflow Graph: Defines the sequence of test suites connected by data flow.
2. Execution Profile: Specifies which test cases to run from each test suite (refer this section for more details).
3. Cartesian Product: Creates all possible combinations of test cases across connected test suites.

For a workflow: TestSuite1 (2 test cases) → TestSuite2 (3 test cases)

The system generates 6 execution combinations:

• TS1-TC1 → TS2-TC1
• TS1-TC1 → TS2-TC2
• TS1-TC1 → TS2-TC3
• TS1-TC2 → TS2-TC1
• TS1-TC2 → TS2-TC2
• TS1-TC2 → TS2-TC3

Each combination represents a complete execution path through the workflow with proper data flow validation.

• Store sensitive data (API keys) as platform secrets, never in code.
• Use environment-specific configurations.
• Implement proper access controls for test execution.

• Use appropriate test selection (UUID, Group, Tags) to minimize execution time.
• Run critical tests early in pipelines for fast feedback.
• Implement parallel execution where supported.

• Set up notifications for test failures.
• Track test execution trends over time.
• Monitor test execution duration and success rates.

• Use descriptive commit messages for better test result tracking.
• Organize tests using groups and tags for flexible execution.
• Maintain execution profiles for consistent E2E testing.