Skip to main content

Skill: Integration Test Scenario Resolution

When to use this: when you need to understand how the Go CLI (deploy-camunda) resolves Helm values files for a specific integration test scenario — for example, to debug a CI failure, add a new scenario, or verify that a ci-test-config.yaml entry produces the correct helm install -f arguments.

What you need: familiarity with the CI config (ci-test-config.yaml) and the Helm values directory structure under charts/<version>/test/integration/scenarios/.

Outcome: full traceability from a CI config entry to the final ordered list of -f arguments passed to helm install, including how identity, persistence, platform, and feature layers are resolved.


How the Go CLI (deploy-camunda) resolves Helm values files for each integration test scenario, from the CI config entry to the final helm install -f arguments.


Table of Contents


Architecture Overview

Each integration test scenario produces an ordered list of Helm values files that are passed to helm install -f. The system uses a layered values approach: a base configuration is extended by identity, persistence, platform, and feature layers, then optionally by migrator and image-tag overrides.

There are two configuration approaches in ci-test-config.yaml:

  1. Legacy (name parsing) — The scenario name is parsed by MapScenarioToConfig() to derive identity, persistence, platform, and features. Example: name: keycloak-mt → identity=keycloak-external, features=[multitenancy].

  2. Selection + Composition (explicit) — The scenario entry includes explicit identity, persistence, and features fields that override name parsing. Example:

    name: keycloak-original
    identity: keycloak-external
    persistence: elasticsearch-external

Both approaches produce a DeploymentConfig that is resolved into file paths.


Data Flow

ci-test-config.yaml


CIScenario struct (matrix/config.go)


Entry struct (matrix/matrix.go — Generate())
├── Identity, Persistence, (propagated from CIScenario)
│ Features fields
├── Version, Flow, Platform
└── Scenario, Shortname, Auth, Exclude


RuntimeFlags (matrix/runner.go — executeEntry())
├── Identity, Persistence, (set from Entry fields)
│ Features
└── all other deploy flags


deploy.Execute() (deploy/deploy.go)


prepareScenarioValues()
├── MapScenarioToConfig(scenarioName) → DeploymentConfig (name-parsed)
├── Apply flag overrides: → DeploymentConfig (final)
│ if flags.Identity != "" → deployConfig.Identity = flags.Identity
│ if flags.Persistence != "" → deployConfig.Persistence = flags.Persistence
│ if len(flags.Features) > 0 → deployConfig.Features = flags.Features
│ if flags.TestPlatform != ""→ deployConfig.Platform = flags.TestPlatform
└── deployConfig.ResolvePaths(scenarioDir) → []string (ordered file list)


helm install -f common1.yaml -f common2.yaml -f layer1.yaml -f layer2.yaml ...

Key rule: Explicit identity/persistence/features fields in ci-test-config.yaml always override name-based derivation.


Layer Resolution Order

ResolvePaths() returns files in this order (see scripts/camunda-core/pkg/scenarios/scenarios.go:83):

OrderLayerFile(s)Condition
1Basevalues/base.yamlAlways (required)
2QA modifiervalues/base-qa.yamlQA == true
2Upgrade modifiervalues/base-upgrade.yamlUpgrade == true
3Identityvalues/identity/<identity>.yamlIdentity != ""
4Persistencevalues/persistence/<persistence>.yamlPersistence != ""
5Platformvalues/platform/<platform>.yamlPlatform != ""
6Featuresvalues/features/<feature>.yamlFor each feature, in order: multitenancy, rba, documentstore
7Migratorvalues-enable-migrator.yamlChart version starts with 13 AND flow != upgrade-patch
8Image tagsvalues/base-image-tags.yamlImageTags == true

Later files override earlier ones (standard Helm -f precedence).


Image Version Strategy (digest / latest / image-tags)

Three mutually exclusive strategies control which image versions are deployed. Only one is active per deployment:

StrategyOverlay fileWhen activeUse case
digest (default)values-digest.yamlNeither image-tags nor latest is activePR CI — pinned release digests for reproducibility
latestvalues-latest.yaml--use-latest flag on matrix runTesting against latest floating tags
image-tagsvalues/base-image-tags.yamlImageTags == true (see below)Nightly CI — SNAPSHOT builds via env var substitution

How ImageTags is activated

ImageTags is enabled per-scenario by setting image-tags: true in ci-test-config.yaml. All qa-* scenarios have this set because they always receive SNAPSHOT versions from nightly CI workflows. When active, ResolveFiles() includes base-image-tags.yaml and excludes values-digest.yaml.

Placeholder substitution in base-image-tags.yaml

The image-tags layer contains environment variable placeholders:

orchestration:
image:
tag: "$E2E_TESTS_ORCHESTRATION_IMAGE_TAG"
connectors:
image:
tag: "$E2E_TESTS_CONNECTORS_IMAGE_TAG"
# ... etc

These are resolved by values.Process() during deployment. The substitution uses the EnvOverrides map built by buildScenarioEnv():

  1. .env file — loaded via --env-file flag. In CI, the workflow converts the VALUES_CONFIG JSON to a .env file using jq, then passes it via --env-file. This provides the SNAPSHOT image tag values.
  2. Process environment — baseline snapshot of os.Environ().
  3. Scenario overrides — index prefixes, realm names, keycloak config, etc.

CI data flow

Nightly workflow (e2e repo)
└─ passes: values-config: {"E2E_TESTS_ORCHESTRATION_IMAGE_TAG":"8.8-SNAPSHOT",...}
└─ test-integration-runner.yaml
└─ sets: VALUES_CONFIG env var from input
└─ bash: jq converts VALUES_CONFIG JSON → /tmp/values-config.env
└─ runs: deploy-camunda matrix run --env-file /tmp/values-config.env ...
└─ image-tags: true in ci-test-config.yaml → includes base-image-tags.yaml
└─ buildScenarioEnv() loads .env file → envMap has E2E_TESTS_*_IMAGE_TAG
└─ values.Process(EnvOverrides=envMap)
└─ Substitutes $E2E_TESTS_ORCHESTRATION_IMAGE_TAG → "8.8-SNAPSHOT"

Scenario Name Parsing (MapScenarioToConfig)

When no explicit identity/persistence/features fields are set, the scenario name is parsed by MapScenarioToConfig() using substring matching:

Identity derivation

Scenario name containsIdentity assigned
keycloak-original (exact)keycloak-external (early return)
keycloak-mt, -mt-, multitenancykeycloak-external
oidc, entraoidc
basicbasic
hybridhybrid
(default)keycloak

Persistence derivation

Scenario name containsPersistence assigned
keycloak-original (exact)elasticsearch-external (early return)
elasticsearch (exact)elasticsearch-external (early return)
opensearchopensearch
rdbms + oraclerdbms-oracle
rdbmsrdbms
(default)elasticsearch-external

Platform derivation

Scenario name containsPlatform assigned
ekseks
openshift, rosaopenshift
(default)gke

Feature derivation

Scenario name containsFeature added
-mt, multitenancymultitenancy
rbarba
documentdocumentstore
mcpmcp

Other flags

Scenario name containsFlag set
qa- prefixQA = true
upgrade, -upgUpgrade = true

Selection + Composition Override

When a ci-test-config.yaml entry has explicit fields, these override the name-parsed values. The override happens in prepareScenarioValues() (deploy/deploy.go:1063-1073):

// MapScenarioToConfig runs first (name parsing)
deployConfig := scenarios.MapScenarioToConfig(scenarioCtx.ScenarioName)

// Then explicit fields override:
if flags.Identity != "" { deployConfig.Identity = flags.Identity }
if flags.Persistence != "" { deployConfig.Persistence = flags.Persistence }
if len(flags.Features) > 0 { deployConfig.Features = flags.Features }

This means a scenario like:

name: keycloak-original
identity: keycloak-external
persistence: elasticsearch-external

...first parses keycloak-original (which already returns the correct values via the early-return path), then applies the explicit overrides (which happen to match). The explicit fields serve as a safety net and documentation.


Available Layer Files by Version

Versions 8.2–8.5 use the legacy (non-layered) values system and do not have a values/ subdirectory structure. The layered system applies to 8.6+.

Legend

  • ✓ = file exists
  • ✗ = file does not exist
File8.68.78.88.98.10
Base
base.yaml
base-qa.yaml
base-upgrade.yaml
base-image-tags.yaml
Identity
identity/keycloak.yaml
identity/keycloak-external.yaml
identity/oidc.yaml
identity/basic.yaml
identity/hybrid.yaml
Persistence
persistence/elasticsearch.yaml
persistence/elasticsearch-external.yaml
persistence/opensearch.yaml
persistence/rdbms.yaml
persistence/rdbms-oracle.yaml
Platform
platform/gke.yaml
platform/eks.yaml
Features
features/multitenancy.yaml
features/rba.yaml
features/documentstore.yaml
features/mcp.yaml

Scenario Resolution Per Version

8.2 and 8.3

These versions use the legacy (non-layered) values system. They have a single base scenario and no case/scenario structure in ci-test-config.yaml.

# ci-test-config.yaml structure:
integration:
scenarios:
pr:
- name: base
enabled: true

There is no layered values/ directory. The legacy code path (processValues() + BuildValuesList()) is used. The Go CLI detects this via scenarios.HasLayeredValues() returning false.

Resolved values: determined entirely by the legacy BuildValuesList() logic.


8.4 and 8.5

First versions to use the case/scenario structure with auth, shortname, and exclude fields. Still use the legacy values system (no layered values/ directory).

8.4 Scenarios

TierScenarioAuthFlowPlatforms
PRelasticsearch (es)keycloak(all)(default)
Nightlyelasticsearch (es)keycloak(all)(default)
Nightlymultitenancy (mt)keycloak(all)(default)

8.5 Scenarios

TierScenarioAuthFlowPlatforms
PRelasticsearch (es)keycloakinstall, upgrade-patch(default)
Nightlyelasticsearch (es)keycloak(all)(default)
Nightlymultitenancy (mt)keycloak(all)(default)

Resolved values: Legacy path. The auth field selects which auth values file to process.


8.6

First version with the layered values system. The values/ directory has identity, persistence, platform, and features subdirectories.

Scenarios

TierScenarioShortnameAuthFlowPlatformsName-Parsed Config
PRelasticsearcheskeycloakinstall, upgrade-patchgke, eksidentity=keycloak, persistence=elasticsearch, platform=per-entry
Nightlyelasticsearcheskeycloak(all)(default=gke)identity=keycloak, persistence=elasticsearch, platform=gke
Nightlymultitenancymtkeycloak(all)(default=gke)identity=keycloak-external, persistence=elasticsearch, platform=gke, features=[multitenancy]
Nightlyopensearchoskeycloak(all)(default=gke)identity=keycloak, persistence=opensearch, platform=gke

None of these scenarios have explicit identity/persistence/features fields, so name parsing is the sole source of config.

Resolved Values Files

elasticsearch on GKE:

values/base.yaml
values/identity/keycloak.yaml
values/persistence/elasticsearch.yaml
values/platform/gke.yaml

elasticsearch on EKS:

values/base.yaml
values/identity/keycloak.yaml
values/persistence/elasticsearch.yaml
values/platform/eks.yaml

multitenancy on GKE:

values/base.yaml
values/identity/keycloak-external.yaml
values/persistence/elasticsearch.yaml
values/platform/gke.yaml
values/features/multitenancy.yaml

opensearch on GKE:

values/base.yaml
values/identity/keycloak.yaml
values/persistence/opensearch.yaml
values/platform/gke.yaml

What 8.6 Does NOT Have

  • No oidc.yaml, basic.yaml, hybrid.yaml identity layers
  • No base-upgrade.yaml
  • No documentstore.yaml feature

Note on elasticsearch-external.yaml

The persistence/elasticsearch-external.yaml file exists on 8.6 for consistency, but no scenario in ci-test-config.yaml currently uses it. The multitenancy scenario uses keycloak-external identity with internal elasticsearch persistence (same as 8.7+).


8.7

Adds OIDC identity, external Elasticsearch persistence, document store feature, and upgrade support.

Scenarios

TierScenarioShortnameAuthFlowPlatformsExplicit FieldsName-Parsed Config
PRelasticsearcheskeycloakinstall, upgrade-patchgke, eksidentity=keycloak, persistence=elasticsearch
PRkeycloak-mtkemtkeycloakinstall, upgrade-patch(gke)identity=keycloak-external, persistence=elasticsearch, features=[multitenancy]
PRoidcesoioidcinstall, upgrade-patch(gke)identity=oidc, persistence=elasticsearch
PRkeycloak-originalkeyckeycloakinstall, upgrade-patch(gke)identity=keycloak-external, persistence=elasticsearch-externalidentity=keycloak-external, persistence=elasticsearch-external (override matches early-return)
Nightlyelasticsearcheskeycloak(all)(gke)identity=keycloak, persistence=elasticsearch
Nightlymultitenancymtkeycloak(all)(gke)identity=keycloak-external, persistence=elasticsearch, features=[multitenancy]
Nightlyoidcoikeycloak(all)(gke)identity=oidc, persistence=elasticsearch
Nightlyopensearchoskeycloak(all)(gke)identity=keycloak, persistence=opensearch
Nightlyelasticsearch (enterprise)keycloak(all)(gke)identity=keycloak, persistence=elasticsearch
Nightlydocumentstoredocstrkeycloak(all)(gke)identity=keycloak, persistence=elasticsearch, features=[documentstore]

Note: keycloak-mt (PR) is disabled (enabled: false).

Resolved Values Files

elasticsearch on GKE:

values/base.yaml
values/identity/keycloak.yaml
values/persistence/elasticsearch.yaml
values/platform/gke.yaml

keycloak-mt on GKE:

values/base.yaml
values/identity/keycloak-external.yaml
values/persistence/elasticsearch.yaml
values/platform/gke.yaml
values/features/multitenancy.yaml

oidc on GKE:

values/base.yaml
values/identity/oidc.yaml
values/persistence/elasticsearch.yaml
values/platform/gke.yaml

keycloak-original on GKE:

values/base.yaml
values/identity/keycloak-external.yaml
values/persistence/elasticsearch-external.yaml
values/platform/gke.yaml

opensearch on GKE:

values/base.yaml
values/identity/keycloak.yaml
values/persistence/opensearch.yaml
values/platform/gke.yaml

documentstore on GKE:

values/base.yaml
values/identity/keycloak.yaml
values/persistence/elasticsearch.yaml
values/platform/gke.yaml
values/features/documentstore.yaml

multitenancy on GKE:

values/base.yaml
values/identity/keycloak-external.yaml
values/persistence/elasticsearch.yaml
values/platform/gke.yaml
values/features/multitenancy.yaml

What 8.7 Adds Over 8.6

  • identity/oidc.yaml — Microsoft Entra ID configuration
  • persistence/elasticsearch-external.yaml — shared CI Elasticsearch cluster
  • features/documentstore.yaml — AWS S3 document store
  • base-upgrade.yaml — Elasticsearch availability during upgrades

What 8.7 Does NOT Have

  • No identity/basic.yaml or identity/hybrid.yaml

8.8

Major restructuring: zeebe, operate, and tasklist are unified into a single orchestration component. Adds basic and hybrid identity modes.

Scenarios

TierScenarioShortnameAuthFlowPlatformsExplicit FieldsResolved IdentityResolved PersistenceResolved Features
PRelasticsearcheskekeycloak(all)gke, ekskeycloakelasticsearch
PRupgrade-migrationeske-upgmkeycloakupgrade-minor(gke)keycloakelasticsearch
PRelasticsearch-basicesbabasicinstall(gke)basicelasticsearch
PRoidcesoioidcupgrade-minor(gke)oidcelasticsearch
PRkeycloak-mtkemtkeycloakupgrade-minor(gke)keycloak-externalelasticsearch[multitenancy]
PRkeycloak-rbakerbakeycloakupgrade-minor(gke)keycloakelasticsearch[rba]
PRkeycloak-originalkeyckeycloakupgrade-minor(gke)identity=keycloak-external, persistence=elasticsearch-externalkeycloak-externalelasticsearch-external
PRelasticsearch (hybrid)eshyhybridinstall(gke)hybridelasticsearch
Nightlyelasticsearcheskeykeycloak(all)(gke)keycloakelasticsearch
Nightlyelasticsearch-basicesbabasic(all)(gke)basicelasticsearch
Nightlyelasticsearch-oidcesoioidc(all)(gke)oidcelasticsearch
Nightlymultitenancymtkekeycloak(all)(gke)keycloak-externalelasticsearch[multitenancy]
Nightlyopensearch (keycloak)oskekeycloak(all)(gke)keycloakopensearch
Nightlyopensearch (basic)osbabasic(all)(gke)basicopensearch
Nightlyidentity-migrationidmikekeycloak(all)(gke)keycloakelasticsearch
Nightlyelasticsearch (enterprise)elbaebasic(all)(gke)basicelasticsearch
Nightlydocumentstoredocstrkeycloak(all)(gke)keycloakelasticsearch[documentstore]
Nightlykeycloak-external-mimickcextmkeycloak(all)(gke)keycloak-externalelasticsearch

Note: elasticsearch-basic (PR) and elasticsearch-oidc (nightly) are disabled. opensearch with keycloak auth (nightly) is disabled.

Resolved Values Files

elasticsearch (keycloak) on GKE:

values/base.yaml
values/identity/keycloak.yaml
values/persistence/elasticsearch.yaml
values/platform/gke.yaml

elasticsearch-basic on GKE:

values/base.yaml
values/identity/basic.yaml
values/persistence/elasticsearch.yaml
values/platform/gke.yaml

elasticsearch (hybrid) on GKE:

values/base.yaml
values/identity/hybrid.yaml
values/persistence/elasticsearch.yaml
values/platform/gke.yaml

oidc on GKE:

values/base.yaml
values/identity/oidc.yaml
values/persistence/elasticsearch.yaml
values/platform/gke.yaml

keycloak-mt on GKE:

values/base.yaml
values/identity/keycloak-external.yaml
values/persistence/elasticsearch.yaml
values/platform/gke.yaml
values/features/multitenancy.yaml

keycloak-rba on GKE:

values/base.yaml
values/identity/keycloak.yaml
values/persistence/elasticsearch.yaml
values/platform/gke.yaml
values/features/rba.yaml

keycloak-original on GKE:

values/base.yaml
values/identity/keycloak-external.yaml
values/persistence/elasticsearch-external.yaml
values/platform/gke.yaml

opensearch (basic) on GKE:

values/base.yaml
values/identity/basic.yaml
values/persistence/opensearch.yaml
values/platform/gke.yaml

keycloak-external-mimic on GKE:

values/base.yaml
values/identity/keycloak-external.yaml
values/persistence/elasticsearch.yaml
values/platform/gke.yaml

upgrade-migration on GKE (flow=upgrade-minor):

values/base.yaml
values/identity/keycloak.yaml
values/persistence/elasticsearch.yaml
values/platform/gke.yaml

(Upgrade flag is set from flow containing "upgrade", which would include base-upgrade.yaml if Upgrade == true in the config.)

documentstore on GKE:

values/base.yaml
values/identity/keycloak.yaml
values/persistence/elasticsearch.yaml
values/platform/gke.yaml
values/features/documentstore.yaml

What 8.8 Adds Over 8.7

  • identity/basic.yaml — No IdP, basic auth only
  • identity/hybrid.yaml — Combined Identity + OIDC orchestration auth
  • Unit tests restructured: Management, Orchestration, Design (no more separate Zeebe/Operate/Tasklist)

8.9

Adds RDBMS persistence backends and a second keycloak-original entry.

Scenarios

TierScenarioShortnameAuthFlowPlatformsExplicit FieldsResolved IdentityResolved PersistenceResolved Features
PRelasticsearcheskekeycloak(all)gke, ekskeycloakelasticsearch
PRopensearchoskekeycloak(all)(gke)keycloakopensearch
PRupgrade-migrationeske-upgmkeycloakupgrade-minor(gke)keycloakelasticsearch
PRelasticsearch-basicesbabasicinstall(gke)basicelasticsearch
PRoidcesoioidcupgrade-minor(gke)oidcelasticsearch
PRkeycloak-mtkemtkeycloakupgrade-minor(gke)keycloak-externalelasticsearch[multitenancy]
PRkeycloak-rbakerbakeycloakupgrade-minor(gke)keycloakelasticsearch[rba]
PRkeycloak-originalkeyckeycloakupgrade-minor(gke)identity=keycloak-external, persistence=elasticsearch-externalkeycloak-externalelasticsearch-external
PRkeycloak-originalkeorgkeycloakinstall(gke)identity=keycloak-external, persistence=elasticsearch-externalkeycloak-externalelasticsearch-external
PRgateway-keycloakgatkckeycloakinstall(gke)keycloakelasticsearch
Nightlyelasticsearcheskeykeycloak(all)(gke)keycloakelasticsearch
Nightlyelasticsearch-basicesbabasic(all)(gke)basicelasticsearch
Nightlyelasticsearch-oidcesoioidc(all)(gke)oidcelasticsearch
Nightlymultitenancymtkekeycloak(all)(gke)keycloak-externalelasticsearch[multitenancy]
Nightlyopensearch (keycloak)oskekeycloak(all)(gke)keycloakopensearch
Nightlyopensearch (basic)osbabasic(all)(gke)basicopensearch
Nightlyidentity-migrationidmikekeycloak(all)(gke)keycloakelasticsearch
Nightlyelasticsearch (enterprise)elbaebasic(all)(gke)basicelasticsearch
Nightlydocumentstoredocstrkeycloak(all)(gke)keycloakelasticsearch[documentstore]
Nightlykeycloak-external-mimickcextmkeycloak(all)(gke)keycloak-externalelasticsearch

Note: upgrade-migration, elasticsearch-basic (PR), elasticsearch-oidc, and opensearch with keycloak (nightly) are disabled.

Resolved Values Files

All resolutions from 8.8 apply identically, plus:

opensearch (keycloak, PR) on GKE:

values/base.yaml
values/identity/keycloak.yaml
values/persistence/opensearch.yaml
values/platform/gke.yaml

keycloak-original (keorg, flow=install) on GKE:

values/base.yaml
values/identity/keycloak-external.yaml
values/persistence/elasticsearch-external.yaml
values/platform/gke.yaml

(Same files as the keyc entry — only the flow and shortname differ.)

gateway-keycloak on GKE:

values/base.yaml
values/identity/keycloak.yaml
values/persistence/elasticsearch.yaml
values/platform/gke.yaml

What 8.9 Adds Over 8.8

  • persistence/rdbms.yaml — PostgreSQL RDBMS secondary storage
  • persistence/rdbms-oracle.yaml — Oracle RDBMS secondary storage
  • Second keycloak-original entry (keorg) with flow: install
  • opensearch enabled in PR tier
  • gateway-keycloak scenario
  • Documentation comments in ci-test-config.yaml describing the Selection + Composition approach

8.10

Includes MCP (Model Context Protocol) feature support for Self-Managed MCP cluster tools testing (introduced in 8.9).

Scenarios

All scenarios from 8.9 apply, including the mcp feature scenario.

8.10 Layer Notes

  • features/mcp.yaml — enables the MCP gateway via orchestration.extraConfiguration

Resolved Values Files

mcp on GKE:

values/base.yaml
values/identity/keycloak.yaml
values/persistence/elasticsearch.yaml
values/platform/gke.yaml
values/features/mcp.yaml

qa-elasticsearch-mcp on GKE:

values/base.yaml
values/base-qa.yaml
values/identity/keycloak.yaml
values/persistence/elasticsearch.yaml
values/platform/gke.yaml
values/features/mcp.yaml

Known Discrepancies

The following scenarios rely on name parsing and have auth fields that do not align with the derived identity. These work correctly today because the layered path ignores the auth field (it uses identity instead), but adding explicit identity fields would improve clarity:

VersionScenarioauth fieldDerived IdentityNotes
8.7–8.9keycloak-mtkeycloakkeycloak-externalName parsing maps -mt to external keycloak
8.6–8.9multitenancykeycloakkeycloak-externalName parsing maps multitenancy to external keycloak
8.8–8.9keycloak-external-mimickeycloakkeycloak-externalName parsing maps keycloak-external substring
8.8–8.9elasticsearch (hybrid)hybridhybridauth and derived identity happen to align
8.8–8.9elasticsearch-basicbasicbasicauth and derived identity happen to align

For 8.6+ the auth field is effectively ignored in the layered code path.