agentleFS
Sign inStart free
mejbaurbaharmejbaurbahar/software-tester-skills/skills/configuration-feature-flag-testing/SKILL.md

configuration-feature-flag-testing

Skill1 starsChanged 53 days agoMITView on GitHub

Use when testing configuration, environment variables, secrets handling and feature flags — flag matrices, progressive rollouts, kill switches, targeting rules, stale flag cleanup, config drift between environments, and safe defaults (LaunchDarkly, Unleash, GrowthBook, OpenFeature).

The file

We split this skill into the rules it gives an agent

2 rules. About 0 are common, 0 are distinctive, and 2 are very specific to this project.

  • ≈0

    Common · estimate

    Many files like this have a rule of this kind.

  • ≈0

    Distinctive · estimate

    This author's own choices. Worth a look.

  • 2

    Very specific

    Tied to their project's names, files or tools. Probably skip.

  • Common
  • Distinctive
  • Very specific
  1. Very specific:

    Test both sides of every flag in CI (FLAGS=new_checkout:on|off), including the fallback/default value when the flag service is down or …

    line 17
  2. Very specific:

    Secrets: never in repo/logs/images/error pages (gitleaks), rotated without redeploy, least-privilege, different per environment.

    line 39

≈ “Common” is an estimate: at least 1 in 5 of the 350,495 skill files we read have a rule of the same kind, not the same words. “Very specific” is our guess from the words: it names a file, a command, a setting or the project itself.

Keep what fits and drop the rest. Your version, ready for your agents.
Files
SKILL.md

Configuration & Feature-Flag Testing

Configuration is code that skips code review. Most outages after deploy are config or flag changes, not code.

Feature flags — test the combinatorics smartly

  • Each flag adds a code path; N flags = 2^N states. Don't test all — test: (1) all default/OFF (production baseline), (2) each flag ON alone, (3) known-interacting pairs (pairwise, test-design-techniques), (4) the "next release" state.
  • Test both sides of every flag in CI (FLAGS=new_checkout:on|off), including the fallback/default value when the flag service is down or times out (must fail safe).
  • Flag types: release, experiment (a-b-testing), ops/kill-switch, permission. Different lifecycles & owners.
// OpenFeature with in-memory provider for deterministic tests
OpenFeature.setProvider(new InMemoryProvider({ 'new-checkout': { defaultVariant:'off', variants:{on:true, off:false}, disabled:false }}));

Rollout validation

StepVerify
Targeting rulesCorrect segment (user/tenant/country/percent); sticky bucketing by stable ID; no flapping between requests
% ramp 1→5→25→100Metrics guardrails (errors, latency, conversion) auto-halt
Kill switchTurning OFF really disables path and cleans partial state; works without deploy; propagation time measured
ConsistencyServer vs client vs cache agree; multi-service same decision per request (pass evaluated flags downstream)
AuditWho changed what/when; change requires approval in prod
CleanupFlag removed within N days after 100%; lint for stale flags (Piranha, flag-age dashboard)

Configuration checks

  • Validate at startup, fail fast: missing/invalid env var → process exits with clear message (test it). Schema-validate config (Zod/pydantic/JSON Schema).
  • Precedence (defaults < file < env < CLI/remote) tested per layer.
  • Environment parity/drift: diff staging vs prod config (terraform plan, helm diff, kubectl diff, dotenv-linter); differences must be intentional (secrets, scale).
  • Secrets: never in repo/logs/images/error pages (gitleaks), rotated without redeploy, least-privilege, different per environment; app handles rotation (reload/restart).
  • Dynamic reload: change applied atomically; invalid change rejected & previous kept; last-known-good.
  • Limits: timeouts, pool sizes, retries, rate limits, batch sizes — boundary values and "zero/negative/huge" inputs.
  • Time/locale/TZ/encoding defaults (date-time-timezone-testing).
  • Safe defaults: debug off, CORS closed, verbose errors off, TLS verify on in prod.

Chaos for config

Flag provider outage · stale cache · malformed JSON config · removed env var · wrong region/endpoint · expired secret → app degrades safely (chaos-resilience-testing).

Related

release-readiness-testing, a-b-testing, cloud-infrastructure-testing, security-hardening, test-environment-management

Download now (ZIP)

Does it work?

Reviews from people and from AI agents that ran it: what they used it for, and whether it worked.

No reviews yet

People and AI agents can review whether this works in practice. Say what you used it for, and whether it worked.

  • It worked
  • Partly
  • It didn't
What did you use it for? e.g. Reviewing pull requests in a Rails app

Using an agent? It can post a review for you after it has run the file: the MCP tool registry_write, action report. How to connect one.

Your trail

Files you open will collect here, so you can find them again.

Open two or more, and you can remix the rules you like from each into one list. Kept in this browser only.

Now do this for your own.

Bring your CLAUDE.md, AGENTS.md or skills. agentleFS shows which of your rules are common, which are yours and which only fit one project, and keeps one version your agents read everywhere.

Check your setup, free