stress-test
Skill314 starsChanged 5 months agoApache-2.0View on GitHub
RBG controller stress test - deploy kwok in existing cluster, run create/update/delete load tests with real controller pod, collect pprof profiling and logs, generate analysis report with recommendations
18 other projects use a version of this.
Adopted by
Most recently taken up by
- arndvs/ctrlshfttheir own version, first seen Oct 15, 2026
- adam-paterson/skillstheir own version, first seen Oct 11, 2026
- shakacode/react_on_railstheir own version, first seen Oct 8, 2026
- ai-analyst-lab/ai-analysttheir own version, first seen Sep 10, 2026
- arusli1/askillstheir own version, first seen Sep 2, 2026
- ddmsolutions/claude-skillstheir own version, first seen Aug 27, 2026
Used with
Projects with it also use ai-act-readiness, c-level-agents and 10 more.
Other files people use alongside this one.
Files found in the same public projects as a version of this one, more often than they turn up anywhere else. Each project counts once, large collections count for less, and this file's own repository is left out.
Hussain-ElfalahSkillai-act-readiness
Also in 3 projects that use this
Hussain-ElfalahSkillc-level-agents
Also in 3 projects that use this
MicrosoftSkillinbox-setup
Set up the Inbox agent with recommended VS Code settings and gh CLI configuration
Also in 3 projects that use this
- +9 more from Hussain-Elfalah
Other versions18 projects use a version of this.
Word for word · 0
None found.
Edited · 0
None found.
Wrote their own version · 18
- shakacode/react_on_rails★ 5.2ksame name
- ai-analyst-lab/ai-analyst★ 304same name
- machina-exm/film-studio-skills★ 137same name
- okikusan-public/stock_skills★ 134same name
- gbasin/stress-test-skill★ 44same name
- Hassaan146/claude-skills★ 18same name
- ilya-a11y/growthband-outbound-skills★ 13same name
- erhwenkuo/stock_skills★ 3same name
- Hussain-Elfalah/Agentic-Skills★ 2same name
- Recusive/Skills★ 2same name
8 more not listed
How matches are found. Word for word: the exact same text. Edited: shares passages with this file (“99% same” means 99% of its five-word phrases also appear here; 10% or more counts). Their own version: similar meaning, or, weakest, the same name. We can’t tell which project had it first. Projects are counted among the 200 most-starred candidates, and mentions are links to the repository we found, up to 50 per site, so both counts are a floor.
The file
Show its rules and text
We split this skill into the rules it gives an agent
17 rules. 0 are common, 11 are distinctive, and 6 are very specific to this project.
0
Common
Most of the 16 edited versions we read keep these.
11
Distinctive
This author's own choices. Worth a look.
6
Very specific
Tied to their project's names, files or tools. Probably skip.
- Common
- Distinctive
- Very specific
- Distinctive: line 31
Use AskUserQuestion to gather parameters:
- Very specific: line 121
If pprof is not available (controller image without pprof support), omit the --pprof-addr flag.
- Distinctive: line 138
Test Scenario: all configured parameters
- Distinctive: line 168
After the Go client produces output files, perform expert analysis by reading the output data.
- Very specific: line 172
Read the following files from {output-dir} and analyze each dimension:
- Very specific: line 176
Read {output-dir}/summary.json and {output-dir}/timing-create.csv.
- Distinctive: line 187
Latency trend: Read the timing CSV and check if latency increases over time (monotonically growing = reconcile queue saturation, the …
- Very specific: line 254
Read {output-dir}/errors.log (first 50 lines) and the error classification from the HTML report.
- Very specific: line 288
Read the pprof top-N text files: {output-dir}/cpu-create-top.txt, {output-dir}/heap-create-top.txt, {output-dir}/goroutine-create-top.txt, …
- Very specific: line 290
If pprof files are not available, skip this section and note "Profiling not available — deploy controller with --enable-pprof=true."
- Distinctive: line 298
If total CPU sample time is < 1% of profile duration, the controller is I/O bound, not CPU bound.
- Distinctive: line 300
Heap analysis: Check the top memory allocators:
- Distinctive: line 303
If total heap > 1GB for < 1000 RBGs, flag as high memory usage
- Distinctive: line 306
Goroutine analysis: Check total goroutine count:
- Distinctive: line 372
No OOM or excessive memory growth
- Distinctive: line 409
When the controller manages many objects, low QPS/Burst causes client-side throttling visible as:
- Distinctive: line 422
Reconcile queue depth: if growing, controller can't keep up
“Common” is measured: kept by at least 6 in 10 of the 16 edited versions of this file we could read. “Very specific” is our guess from the words: it names a file, a command, a setting or the project itself.
Files
RBG Controller Stress Test
This skill orchestrates a full stress test of the RBG controller. It supports two modes:
- Manual mode: User defines all parameters, skill runs tests and outputs analysis report
- Auto-tuning mode: User defines the workload scenario, skill runs multiple rounds and recommends optimal controller resource/QPS/burst configuration
Architecture
The stress test uses in-cluster kwok to simulate workload pods:
- kwok controller + stage-fast deployed via Helm in the existing cluster
- Fake nodes created with
type=kwoklabel andkwok.x-k8s.io/nodetaint - RBG controller runs as a real Pod (via Helm) on real nodes with configurable resources
- Stress test workload pods schedule to fake nodes and are simulated as Ready by kwok
- This allows testing controller performance without needing real GPU/compute resources
Prerequisites
- A Kubernetes cluster accessible via current kubeconfig
helm,kubectl,go,curlavailable in PATH- RBG CRDs and controller already deployed (or will be deployed by this skill)
Workflow
Phase 1: Ask User for Scenario
Use AskUserQuestion to gather parameters:
Question 1 - Test Mode:
- Manual mode (user sets all params, get report)
- Auto-tuning mode (auto-iterate to find optimal config)
Question 2 - Controller Resources (affects scheduling and memory):
- 4c/8g (small)
- 8c/16g (medium, recommended)
- 16c/32g (large)
- 32c/64g (extra-large)
Question 3 - Controller Parameters:
- max-concurrent-reconciles: 5, 10, 20, 50
- kube-api-qps: 20 (default), 50, 100, 200
- kube-api-burst: 30 (default), 100, 200, 400
Question 4 - RBG Workload Template:
- Roles per RBG: 2, 3, 5
- LeaderWorkerPattern roles: 0, 1, 2
- LWS size (pods per instance): 2, 4, 8
- Total RBG instances: 10, 50, 100, 500, 1000
Question 5 - QPS for Operations:
- Create QPS: 1, 5, 10, 20
- Update QPS: 1, 5, 10, 20
- Delete QPS: 1, 5, 10, 20
Question 6 - In-place Update:
- Yes (update image to trigger InPlaceIfPossible)
- No (update annotations to trigger RollingUpdate with RecreatePod)
Phase 2: Setup Environment
- Deploy kwok in-cluster (installs kwok controller + stage-fast + creates fake nodes):
FAKE_NODE_COUNT=5 bash test/stress/scripts/setup-kwok.sh - Deploy/upgrade RBG controller via Helm with user-specified resources and params:
CONTROLLER_CPU=8 CONTROLLER_MEMORY=16Gi \ MAX_RECONCILES=10 KUBE_API_QPS=50 KUBE_API_BURST=100 \ PPROF_ENABLED=true \ bash test/stress/scripts/deploy-controller.sh - Verify controller is ready:
kubectl rollout status deploy/rbgs-controller-manager -n rbgs-system --timeout=120s
Phase 3: Run Stress Test
Execute the Go stress test client:
go run ./test/stress/ \
--namespace=stress-test \
--total-rbgs=100 \
--roles-per-rbg=3 \
--lws-roles=1 \
--lws-size=4 \
--create-qps=5 \
--update-qps=5 \
--delete-qps=5 \
--in-place-update=true \
--use-kwok-nodes=true \
--pprof-addr=localhost:6060 \
--output-dir=/tmp/rbg-stress-results \
--timeout=30mThe stress client runs three phases sequentially: Create -> Update -> Delete. Each phase collects CPU profile (concurrently) and timing data to CSV files.
Phase 4: Profiling (integrated, requires pprof-enabled controller)
Pprof collection is integrated into the Go stress client. When --pprof-addr is specified:
- CPU profile (30s) is collected during each phase (concurrent with workload)
- Heap, allocs, goroutine snapshots are collected after each phase
- Top-N text reports are generated via
go tool pprof
Prerequisites:
# Setup port-forward to controller's pprof port
kubectl port-forward -n rbgs-system deploy/rbgs-controller-manager 6060:6060 &Then pass --pprof-addr=localhost:6060 to the stress client.
If pprof is not available (controller image without pprof support), omit the --pprof-addr flag.
Alternatively, use the standalone script for manual collection:
PHASE=create OUTPUT_DIR=/tmp/rbg-stress-results \
bash test/stress/scripts/collect-profiling.shPhase 5: Report Generation (automatic)
The Go stress test client automatically:
- Collects controller logs (scoped to the test period via
--since-time) - Classifies errors (reconcile, conflict, throttle, timeout, panic, workload-type)
- Loads pprof top-N text reports if available
- Generates an HTML report at
{output-dir}/report.html
The HTML report includes:
- Test Scenario: all configured parameters
- Controller Configuration: image, resources, startup args, node
- Issues & Findings: auto-detected problems (high latency, throttling, panics, etc.)
- Performance Results Table: P50/P90/P99/Max/Avg per phase, actual QPS achieved
- Per-phase Latency Distribution: visual bar chart with hover details
- Controller Log Analysis: classified error counts + sample errors
- Profiling (pprof): CPU top functions, heap allocators, allocs, goroutine stacks
Output files:
{output-dir}/
├── report.html # Main HTML report (open in browser)
├── summary.json # Machine-readable stats
├── timing-create.csv # Raw create latencies
├── timing-update.csv # Raw update latencies
├── timing-delete.csv # Raw delete latencies
├── controller-full.log # Full controller logs during test
├── errors.log # Filtered error lines
├── cpu-{phase}.prof # CPU pprof binary (if pprof enabled)
├── heap-{phase}.prof # Heap pprof binary
├── allocs-{phase}.prof # Allocs pprof binary
├── goroutine-{phase}.prof # Goroutine pprof binary
├── cpu-{phase}-top.txt # CPU top-N text report
├── heap-{phase}-top.txt # Heap top-N text report
├── allocs-{phase}-top.txt # Allocs top-N text report
└── goroutine-{phase}-top.txt # Goroutine top-N text reportPhase 6: Model-Driven Analysis
After the Go client produces output files, perform expert analysis by reading the output data. Present the analysis as a structured report to the user.
IMPORTANT: This is not just a pass-through of the data — you must apply Kubernetes controller domain knowledge to interpret the results and produce actionable recommendations.
Read the following files from {output-dir} and analyze each dimension:
6A. Performance & QPS Analysis
Read {output-dir}/summary.json and {output-dir}/timing-create.csv.
Analyze:
- Throughput gap: For each phase, compare
actual_qpsagainst the target QPS (from--create-qps/--update-qps/--delete-qps). Calculate the percentage gap. If actual < 90% of target, the controller is falling behind. - Latency profile: Classify the latency pattern:
- P50 ≈ P99 → uniform, healthy processing
- P99 >> P50 (>5x) → tail latency, likely queue buildup or GC pauses
- P99 > 10s → severe queue saturation, reconcile workers are backlogged
- Max >> P99 → outliers, possibly due to leader election or API server hiccups
- Latency trend: Read the timing CSV and check if latency increases over time (monotonically growing = reconcile queue saturation, the submission rate exceeds the controller's processing capacity).
- Phase comparison: Compare create vs update vs delete latencies. Create is always the heaviest (must create sub-resources: Deployments/LWS + Services + RoleInstanceSets). Update and delete should be fast (<500ms). If update is slow, it suggests the controller re-reconciles too many objects on annotation change.
Format the output as:
## Performance Analysis
### Throughput
| Phase | Target QPS | Actual QPS | Gap | Verdict |
|--------|-----------|------------|--------|--------------|
| create | 10 | 6.95 | -30.5% | ⚠ Behind |
| update | 10 | 10.0 | 0% | ✓ On target |
### Latency Distribution
- Create: P50=Xms, P99=Yms (ratio: Z) — [interpretation]
- ...
### Latency Trend
[Is latency stable or growing? What does this mean?]6B. Controller Tuning Recommendations
Based on the performance data, controller configuration (from {output-dir}/report.html or the controller args), and log analysis, recommend specific parameter changes.
Decision tree:
- If actual QPS < target QPS AND no throttle errors AND CPU < 80%:
→ Bottleneck is
max-concurrent-reconciles. Recommend increasing it. Rule of thumb: for N RBGs with M roles, at leastN*M / target_P99_secondsworkers needed. - If throttle errors > 0 OR
Throttling request tookin logs: → Bottleneck iskube-api-qps/kube-api-burst. Recommend increasing both. Rule of thumb:kube-api-qpsshould be at leastmax-concurrent-reconciles × 3(each reconcile makes ~3 API calls). - If conflict errors > 5% of total operations:
→ Too many concurrent reconciles competing on the same objects. Recommend decreasing
max-concurrent-reconcilesor adding jitter. - If memory usage in heap profile > 500MB for < 1000 RBGs: → Potential memory inefficiency. Check if informer cache is unbounded.
- If P99 latency > 30s but CPU is low and no throttling: → Controller might be I/O bound (waiting on API server). Check API server latency or consider enabling watch bookmarks.
Provide concrete recommendations:
## Tuning Recommendations
### Current Configuration
- max-concurrent-reconciles: 50
- kube-api-qps: 200
- kube-api-burst: 400
- CPU limit: 8 cores
- Memory limit: 16Gi
### Recommendations
1. **[parameter]**: Change from X to Y. Reason: [evidence from the data].
2. ...
### Estimated Impact
[What improvement to expect from these changes]6C. Controller Log Analysis
Read {output-dir}/errors.log (first 50 lines) and the error classification from the HTML report.
Analyze:
- Error categorization: Group errors by type and identify the root cause for each category:
unsupported workload type: /→ Missingrbg.workloads.x-k8s.io/role-workload-typeannotation on rolesthe object has been modified→ Optimistic lock conflict, normal under high concurrency, becomes a problem if > 10% of operationsThrottling request took Xs→ Client-side rate limiting, increasekube-api-qpscontext deadline exceeded→ Reconcile took too long, increase timeout or reduce work per reconcileFailed to create workload reconciler→ Likely workload type or CRD issuepanic/runtime error→ Critical bug, needs immediate investigation
- Error ratio: Calculate errors as a percentage of total reconcile operations. < 1% is acceptable (transient conflicts). > 5% indicates a systemic issue.
- Error pattern over time: Are errors concentrated at the beginning (startup/cache-sync) or throughout (systemic)?
Format:
## Log Analysis
### Error Summary
| Category | Count | % of Operations | Severity | Root Cause |
|-------------------|-------|-----------------|----------|------------------|
| Workload Type | 422 | 14% | Medium | Missing annotation|
### Key Findings
- [Finding 1]
- [Finding 2]
### Actionable Items
1. [What to fix and how]6D. Profiling Analysis
Read the pprof top-N text files: {output-dir}/cpu-create-top.txt, {output-dir}/heap-create-top.txt, {output-dir}/goroutine-create-top.txt, {output-dir}/allocs-create-top.txt.
If pprof files are not available, skip this section and note "Profiling not available — deploy controller with --enable-pprof=true."
Analyze:
- CPU hotspots: Categorize top CPU consumers:
- Infrastructure (expected):
crypto/tls,runtime.*(GC),net/http,encoding/json— normal overhead - Controller logic (tunable): reconciler functions, status update, dependency resolution — high % here means the controller logic itself is the bottleneck
- Serialization (reducible):
k8s.io/apimachinery/pkg/runtime,Unmarshal,Marshal— indicates heavy object churn - If total CPU sample time is < 1% of profile duration, the controller is I/O bound, not CPU bound.
- Infrastructure (expected):
- Heap analysis: Check the top memory allocators:
runtime.allocm→ goroutine stack memory, scales withmax-concurrent-reconcilesk8s.io/apimachinery/...Unmarshal→ informer cache deserializing K8s objects- If total heap > 1GB for < 1000 RBGs, flag as high memory usage
- Note: heap profile shows in-use memory, not peak. For peak, check container metrics.
- Goroutine analysis: Check total goroutine count:
- < 100 goroutines: healthy
- 100-500: normal for high-concurrency controllers
1000: potential goroutine leak, check for unbounded goroutine creation
Format:
## Profiling Analysis
### CPU Profile (create phase)
- Total sample time: X of Y seconds (Z% CPU utilization)
- Top consumers: [categorized list]
- Verdict: [CPU bound / I/O bound / balanced]
### Memory Profile
- Heap in-use: XMB
- Top allocators: [list with interpretation]
- Verdict: [memory efficient / needs attention]
### Goroutine Profile
- Active goroutines: N
- Verdict: [healthy / needs investigation]6E. Executive Summary
Synthesize all findings into a brief summary:
## Summary
**Test**: [N] RBGs × [M] roles at [Q] QPS — [all succeeded / X failed]
**Performance**: [One sentence on whether the controller met the throughput target]
**Bottleneck**: [What is the primary bottleneck — worker count / API QPS / CPU / memory]
**Top 3 Recommendations**:
1. [Most impactful change]
2. [Second most impactful]
3. [Third]
**Overall Assessment**: [Ready for production at this scale / Needs tuning / Critical issues found]Phase 7: Teardown (optional)
# Remove fake nodes and stress test namespace
bash test/stress/scripts/teardown-kwok.sh
# To also uninstall kwok:
UNINSTALL_KWOK=true bash test/stress/scripts/teardown-kwok.shAuto-Tuning Mode (if selected)
In auto-tuning mode, repeat Phase 2-6 multiple rounds with different configurations:
Round 1: User's initial config (baseline) Round 2: Increase kube-api-qps/burst (test if API throttling is bottleneck) Round 3: Increase max-concurrent-reconciles (test if worker count is bottleneck) Round 4: Optimal combination based on Round 2+3 findings
Compare results across rounds and recommend the configuration that achieves:
- Lowest P99 latency
- No OOM or excessive memory growth
- Minimal API throttle errors
- Best resource efficiency (latency per CPU-core)
Background Knowledge
Workload Type Annotation
The stress test templates must include rbg.workloads.x-k8s.io/role-workload-type annotation on each role:
- Standalone roles:
"apps/v1/Deployment" - LWS roles:
"leaderworkerset.x-k8s.io/v1/LeaderWorkerSet"
Without this annotation, the controller falls back to RoleInstanceSet type which may not be supported by older controller images, causing transient "unsupported workload type: /" errors and slow reconciliation.
In-Cluster KWOK Architecture
Real Nodes (ACK) Fake Nodes (kwok)
┌─────────────────────┐ ┌─────────────────────┐
│ RBG Controller Pod │ │ kwok-node-0..N │
│ (real, Helm-managed)│ │ (simulated by kwok) │
│ │ │ │
│ kwok Controller Pod │ │ Workload Pods │
│ (simulates pods) │ │ (auto Ready by kwok) │
└─────────────────────┘ └─────────────────────┘
│ ▲
│ watches pods on │ schedules to
│ fake nodes │ (toleration + nodeSelector)
└────────────────────────────┘controller-runtime Client QPS/Burst
The controller uses client-go rest.Config which has:
QPS: sustained queries-per-second to the API server (default 20 for client-go, often 5 for controller-runtime)Burst: maximum burst above QPS (default 30)
When the controller manages many objects, low QPS/Burst causes client-side throttling visible as:
Throttling request took Xslog messages- Increased reconcile latency without high CPU usage
rate: Wait(n=1) would exceed context deadlineerrors
max-concurrent-reconciles
Controls how many reconcile loops run in parallel per controller. Higher values:
- Pros: faster throughput, better utilization of API quota
- Cons: more memory, more concurrent API calls, more conflicts
Key Metrics to Watch
- Reconcile queue depth: if growing, controller can't keep up
- API request latency: high = API server overloaded or network issues
- Memory growth: linear growth during test = potential leak
- Conflict errors: high = too many concurrent reconciles competing on same objects
Does it work?
Reviews from people and from AI agents that ran it: what they used it for, and whether it worked.
No reviews yet
People and AI agents can review whether this works in practice. Say what you used it for, and whether it worked.
- It worked
- Partly
- It didn't
Using an agent? It can post a review for you after it has run the file: the MCP tool registry_write, action report. How to connect one.
Your trail
Files you open will collect here, so you can find them again.
Open two or more, and you can remix the rules you like from each into one list. Kept in this browser only.
Now do this for your own.
Bring your CLAUDE.md, AGENTS.md or skills. agentleFS shows which of your rules are common, which are yours and which only fit one project, and keeps one version your agents read everywhere.

