victoriametrics-cardinality-analysis
Skill2 starsChanged 33 days agoMITView on GitHub
Analyze VictoriaMetrics time series cardinality to find optimization opportunities — unused metrics, high-cardinality labels, problematic label values, histogram bloat. Produces actionable report with relabeling and stream aggregation recommendations. Use whenever the user mentions cardinality analysis, series reduction, unused metrics, high cardinality labels, TSDB optimization, storage cost reduction, metric cleanup, too many time series, or wants to reduce cardinality. Also trigger when discussing relabeling strategies, streaming aggregation opportunities, or "which metrics can we drop".
1 other project uses a version of this, 1 of them edited.
Adopted by
Most recently taken up by
- SawanaLabs/agent-demosedited, first seen Oct 13, 2026
Other versions1 project uses a version of this.
Word for word · 0
None found.
Edited · 1
- SawanaLabs/agent-demos★ 496% same
Wrote their own version · 0
None found.
How matches are found. Word for word: the exact same text. Edited: shares passages with this file (“99% same” means 99% of its five-word phrases also appear here; 10% or more counts). Their own version: similar meaning, or, weakest, the same name. We can’t tell which project had it first. Projects are counted among the 200 most-starred candidates, and mentions are links to the repository we found, up to 50 per site, so both counts are a floor.
In this repository192 other files it gives its agents.
- clickhouse-architecture-advisor/AGENTS.mdAGENTS.md
- clickhouse-best-practices/AGENTS.mdAGENTS.md
- unsafe-checker/AGENTS.mdAGENTS.md
- alertmanager-querySkill
- analyzing-range-distributionSkill
- analyzing-schema-change-storage-riskSkill
- argocdSkill
- aspire-configurationSkill
- aspire-integration-testingSkill
- aspire-service-defaultsSkill
- auditing-cloud-cluster-securitySkill
- auditing-table-statisticsSkill
- benchmarking-transaction-patternsSkill
- cepthSkill
- chdb-datastoreSkill
- chdb-sqlSkill
- clickhouse-architecture-advisorSkill
- clickhouse-best-practicesSkill
- clickhousectl-cloud-deploySkill
- clickhousectl-local-devSkill
- cockroachdb-sqlSkill
- coding-guidelinesSkill
- configuring-audit-loggingSkill
- configuring-ip-allowlistsSkill
- configuring-log-exportSkill
- configuring-private-connectivitySkill
- configuring-sso-and-scimSkill
- create-technical-architecture-documentSkill
- csharp-mstestSkill
- csharp-tunitSkill
- csharp-xunitSkill
- designing-application-transactionsSkill
- designing-multi-region-applicationsSkill
- domain-cliSkill
- domain-cloud-nativeSkill
- domain-embeddedSkill
- domain-fintechSkill
- domain-iotSkill
- domain-mlSkill
- domain-webSkill
- dotnet-aot-wasmSkill
- dotnet-aspire-patternsSkill
- dotnet-benchmarkdotnetSkill
- dotnet-blazor-authSkill
- dotnet-blazor-componentsSkill
- dotnet-blazor-patternsSkill
- dotnet-blazor-testingSkill
- dotnet-channelsSkill
- dotnet-ci-benchmarkingSkill
- dotnet-cli-architectureSkill
- dotnet-cli-distributionSkill
- dotnet-cli-packagingSkill
- dotnet-cli-release-pipelineSkill
- dotnet-containersSkill
- dotnet-cryptographySkill
- dotnet-csharp-dependency-injectionSkill
- dotnet-csharp-modern-patternsSkill
- dotnet-csharp-nullable-reference-typesSkill
- dotnet-csharp-source-generatorsSkill
- dotnet-maui-doctorSkill
- The first 60 shown.
The file
We split this skill into the rules it gives an agent
11 rules. About 0 are common, 4 are distinctive, and 7 are very specific to this project.
≈0
Common · estimate
Many files like this have a rule of this kind.
≈4
Distinctive · estimate
This author's own choices. Worth a look.
7
Very specific
Tied to their project's names, files or tools. Probably skip.
- Common
- Distinctive
- Very specific
- Very specific: line 41
If the user specified a scope (job, namespace, metric prefix), pass it as match[] parameter to TSDB status queries and as series selectors …
- Very specific: line 77
Return: All raw JSON preserving structure.
- Very specific: line 108
If the endpoint returns an error, storage.trackMetricNamesStats may not be enabled on vmstorage.
- Distinctive: line 123
safe to drop: never queried AND not in any rule
- Distinctive: line 124
used by rules only: never queried by dashboards but referenced in rules — verify intent
- Very specific: line 133
All data comes from the TSDB status endpoint — do NOT use /api/v1/labels or /api/v1/label/.../values.
- Distinctive: line 188
After all subagents return, compile and classify findings.
- Very specific: line 220
Check metrics ending in _bucket:
- Distinctive: line 224
Look for histograms where most buckets are empty or redundant
- Very specific: line 231
Not directly fixable via relabeling, but indicates opportunity for dedup_interval or -search.maxStalenessInterval tuning
- Very specific: line 441
Important: use total for counters, last/avg/sum_samples for gauges.
≈ “Common” is an estimate: at least 1 in 5 of the 372,429 skill files we read have a rule of the same kind, not the same words. “Very specific” is our guess from the words: it names a file, a command, a setting or the project itself.
Files
VictoriaMetrics Cardinality Analysis
Systematic cardinality analysis for VictoriaMetrics. Collects TSDB status, metric usage stats, and label value patterns, then produces a structured report with specific relabeling and stream aggregation configs the user can apply directly.
The goal is to find the highest-impact optimization opportunities — metrics nobody queries, labels that explode cardinality for no monitoring value, and patterns that indicate data hygiene problems (error messages as labels, SQL text as labels, UUIDs as labels).
Environment
Uses the same env vars as the victoriametrics-query skill:
# $VM_METRICS_URL - base URL
# cluster: export VM_METRICS_URL="https://vmselect.example.com/select/0/prometheus"
# single: export VM_METRICS_URL="http://localhost:8428"
# $VM_AUTH_HEADER - auth header (empty if no auth is required)Workflow
Phase 1: Data Collection
Spawn 3 subagents in a single response to collect data in parallel. Each subagent prompt must include the curl auth pattern and environment variable references above.
If the user specified a scope (job, namespace, metric prefix), pass it as match[] parameter to
TSDB status queries and as series selectors to label queries.
Subagent 1: TSDB Overview
Agent name: cardinality-tsdb | Description: "Collect TSDB cardinality stats"
Query 1 — Yesterday's series (captures recently churned series):
curl -s ${VM_AUTH_HEADER:+-H "$VM_AUTH_HEADER"} \
"$VM_METRICS_URL/api/v1/status/tsdb?topN=50&date=$(date -d 'yesterday' +%Y-%m-%d)" | jq '.data'Queries yesterday's stats — broader than today (includes series that may have already churned) without scanning the entire TSDB.
Query 2 — Today's active series:
curl -s ${VM_AUTH_HEADER:+-H "$VM_AUTH_HEADER"} \
"$VM_METRICS_URL/api/v1/status/tsdb?topN=50" | jq '.data'Query 3 — Focus on known high-cardinality labels:
for label in pod instance container path url user_id request_id session_id trace_id le name; do
echo "=== focusLabel=$label ===" && \
curl -s ${VM_AUTH_HEADER:+-H "$VM_AUTH_HEADER"} \
"$VM_METRICS_URL/api/v1/status/tsdb?topN=20&focusLabel=$label" | \
jq --arg l "$label" '{label: $l, focus: .data.seriesCountByFocusLabelValue}'
doneReturn: All raw JSON preserving structure. Include totalSeries, totalLabelValuePairs,
seriesCountByMetricName, seriesCountByLabelName, seriesCountByLabelValuePair from each query.
Subagent 2: Metric Usage Stats
Agent name: cardinality-usage | Description: "Find unused and rarely-queried metrics"
Query 1 — Never-queried metrics:
curl -s ${VM_AUTH_HEADER:+-H "$VM_AUTH_HEADER"} \
"$VM_METRICS_URL/api/v1/status/metric_names_stats?le=0&limit=500" | jq '.'Query 2 — Rarely-queried metrics (≤5 total queries):
curl -s ${VM_AUTH_HEADER:+-H "$VM_AUTH_HEADER"} \
"$VM_METRICS_URL/api/v1/status/metric_names_stats?le=5&limit=500" | jq '.'Query 3 — Stats overview (tracking period):
curl -s ${VM_AUTH_HEADER:+-H "$VM_AUTH_HEADER"} \
"$VM_METRICS_URL/api/v1/status/metric_names_stats?limit=1" | \
jq '{statsCollectedSince: .statsCollectedSince, statsCollectedRecordsTotal: .statsCollectedRecordsTotal}'If the endpoint returns an error, storage.trackMetricNamesStats may not be enabled on vmstorage.
Note this in the return and proceed — the analysis can still work with TSDB status data alone.
Query 4 — Cross-check: are "unused" metrics referenced in alerting rules?
curl -s ${VM_AUTH_HEADER:+-H "$VM_AUTH_HEADER"} \
"$VM_METRICS_URL/api/v1/rules" | jq '[.data.groups[].rules[].query]'Extract metric names from rule queries. Any "unused" metric that appears in an alert/recording rule is NOT safe to drop — it's queried indirectly.
Return: Unused metrics with cross-reference against alert rules. Flag each as:
- safe to drop: never queried AND not in any rule
- used by rules only: never queried by dashboards but referenced in rules — verify intent
- rarely used: low query count, may be accessed infrequently (e.g., monthly reports)
Subagent 3: Label Pattern Inspection
Agent name: cardinality-labels | Description: "Inspect label values for problematic patterns"
All data comes from the TSDB status endpoint — do NOT use /api/v1/labels or /api/v1/label/.../values.
Query 1 — Label cardinality overview (unique value counts + series counts):
curl -s ${VM_AUTH_HEADER:+-H "$VM_AUTH_HEADER"} \
"$VM_METRICS_URL/api/v1/status/tsdb?topN=50" | \
jq '{labelValueCountByLabelName: .data.labelValueCountByLabelName, seriesCountByLabelName: .data.seriesCountByLabelName}'labelValueCountByLabelName returns labels sorted by unique value count (replaces per-label /values counting).
seriesCountByLabelName shows how many series each label appears in.
Query 2 — Sample values for high-cardinality labels via focusLabel: For each label with >100 unique values from Query 1, fetch sample values:
for label in <top labels from Query 1>; do
echo "=== focusLabel=$label ===" && \
curl -s ${VM_AUTH_HEADER:+-H "$VM_AUTH_HEADER"} \
"$VM_METRICS_URL/api/v1/status/tsdb?topN=20&focusLabel=$label" | \
jq --arg l "$label" '{label: $l, topValues: .data.seriesCountByFocusLabelValue}'
doneseriesCountByFocusLabelValue returns label values sorted by series count — use the value names to detect problematic patterns.
Query 3 — High-cardinality label-value pairs:
curl -s ${VM_AUTH_HEADER:+-H "$VM_AUTH_HEADER"} \
"$VM_METRICS_URL/api/v1/status/tsdb?topN=50" | \
jq '.data.seriesCountByLabelValuePair'Shows which specific label=value pairs contribute the most series.
Pattern detection — classify label values from focusLabel samples:
| Pattern | Regex hint | Indicates |
| UUIDs | [0-9a-f]{8}-[0-9a-f]{4}- | Request/session/trace IDs as labels |
| IP addresses | \d+\.\d+\.\d+\.\d+ | Per-client or per-pod IP tracking |
| Long strings (>50 chars) | length check | Error messages, SQL, stack traces |
| SQL keywords | SELECT|INSERT|UPDATE|DELETE|FROM|WHERE | Query text stored as label |
| URL paths with IDs | /api/.*/[0-9a-f]+ | Unsanitized HTTP paths |
| Timestamps | epoch or ISO8601 | Time values as labels (unbounded) |
| Stack traces | at .*\.(java|go|py): | Error details as labels |
Return: Table of labels sorted by unique value count, with detected pattern, sample values from focusLabel, and series impact.
Phase 2: Analysis
After all subagents return, compile and classify findings. This is the analytical core — apply judgment, not mechanical filtering.
Category 1: Unused Metrics (Quick Wins)
Cross-reference metric usage stats with TSDB series counts:
- Drop candidates:
queryRequestsCount=0, not in any alert/recording rule, >100 series - Verify candidates:
queryRequestsCount=0but referenced in rules — check if rule is still needed - Low-priority:
queryRequestsCount≤5with few series — not worth the config churn
Sort by series count descending — the biggest unused metrics are the biggest wins.
Category 2: High-Cardinality Labels
Labels with excessive unique values that drive series explosion:
| Label pattern | Assessment | Typical remedy |
user_id, customer_id, account_id | Should NEVER be metric labels — belongs in logs/traces | Drop label |
request_id, session_id, trace_id, span_id | Correlation IDs — never metric labels | Drop label |
error, error_message, reason, status_message | Unbounded strings | Drop label or replace with error code |
sql, query, command, statement | Query text in labels — unbounded | Drop label |
path, url, uri, endpoint | Unbounded if not sanitized | Relabel to normalize, or stream aggregate without |
pod, container | Normal for k8s but high churn | Stream aggregate without, if per-pod not needed |
instance | Normal for node metrics, wasteful for app metrics | Stream aggregate without for app-level metrics |
le (histogram buckets) | Fine-grained buckets multiply every label combination | Reduce bucket count |
For each finding, estimate impact: (series with this label) - (series without) ≈ series saved.
Category 3: Histogram Bloat
Check metrics ending in _bucket:
- How many unique
levalues? - Each additional bucket multiplies series by (number of label combinations)
- Look for histograms where most buckets are empty or redundant
Category 4: Series Churn
Compare yesterday's stats vs today:
- Ratio >3:1 suggests significant churn from pod restarts, deployments, short-lived jobs
- Not directly fixable via relabeling, but indicates opportunity for
dedup_intervalor-search.maxStalenessIntervaltuning
Phase 3: Report
Compile into a structured report. Every finding must include impact estimate and specific remedy config.
Use this template:
## VictoriaMetrics Cardinality Report — <date>
### Overview
| Metric | Value |
|--------|-------|
| Total active series (today) | X |
| Total series (yesterday) | X |
| Churn ratio (yesterday / today) | X:1 |
| Unique metric names | X |
| Stats tracking since | <date> |
### 1. Unused Metrics
**Potential savings: ~X series (Y% of total)**
| Metric | Series | Last Queried | In Alert Rules | Action |
|--------|--------|-------------|----------------|--------|
| ... | ... | never | no | Drop |
| ... | ... | never | yes — verify | Check rule |
<details>
<summary>Relabeling config to drop unused metrics</summary>
```yaml
# Add to vmagent metric_relabel_configs (VMServiceScrape or global)
metric_relabel_configs:
- source_labels: [__name__]
regex: "metric1|metric2|metric3"
action: drop
```
</details>
### 2. High-Cardinality Labels
**Potential savings: ~X series (Y%)**
| Label | Unique Values | Top Affected Metrics | Pattern | Action |
|-------|--------------|---------------------|---------|--------|
| user_id | 50,000 | http_requests_total | UUID | Drop |
| path | 10,000 | http_request_duration | URL paths | Aggregate |
| error_message | 5,000 | app_errors_total | Long strings | Drop |
<details>
<summary>Drop labels that should never be in metrics</summary>
```yaml
metric_relabel_configs:
- regex: "user_id|request_id|session_id|trace_id|error_message|sql_query"
action: labeldrop
```
</details>
<details>
<summary>Stream aggregation for high-cardinality HTTP labels</summary>
```yaml
# vmagent stream aggregation config
- match: '{__name__=~"http_request.*"}'
interval: 1m
without: [path, instance, pod]
outputs: [total]
# drop_input: true # enable after verifying aggregated output
- match: '{__name__=~"http_request_duration.*_bucket"}'
interval: 1m
without: [pod, instance]
outputs: [total]
keep_metric_names: true
```
</details>
<details>
<summary>Normalize URL paths via relabeling</summary>
```yaml
metric_relabel_configs:
- source_labels: [path]
regex: "/api/v1/users/[^/]+"
target_label: path
replacement: "/api/v1/users/:id"
- source_labels: [path]
regex: "/api/v1/orders/[^/]+"
target_label: path
replacement: "/api/v1/orders/:id"
```
</details>
### 3. Histogram Optimization
**Potential savings: ~X series (Y%)**
| Metric | Bucket Count | Recommendation |
|--------|-------------|----------------|
| ... | 30 | Reduce to standard 11 buckets |
### 4. Series Churn
| Observation | Value |
|------------|-------|
| Yesterday / today ratio | X:1 |
| Primary driver | Pod restarts / short-lived jobs |
### Summary
| Category | Est. Series Saved | % of Total | Effort |
|----------|------------------|-----------|--------|
| Drop unused metrics | X | Y% | Low — relabeling only |
| Drop bad labels | X | Y% | Low — labeldrop |
| Stream aggregation | X | Y% | Medium — new config |
| Histogram reduction | X | Y% | Low — bucket filtering |
| **Total** | **X** | **Y%** | |
### Implementation Priority
1. **[Low effort]** Drop unused metrics — pure relabeling, no data loss risk
2. **[Low effort]** Drop labels that should never be in metrics (IDs, messages, SQL)
3. **[Medium effort]** Stream aggregation for high-cardinality HTTP/app metrics
4. **[Medium effort]** Histogram bucket reductionAdapt the template to actual findings — omit sections with no findings, expand sections with significant findings.
Remediation Reference
Relabeling (metric_relabel_configs)
Applied at scrape time or remote write. Changes affect new data immediately.
Drop entire metrics:
metric_relabel_configs:
- source_labels: [__name__]
regex: "metric_to_drop|another_metric"
action: dropDrop labels:
metric_relabel_configs:
- regex: "label_to_drop|another_label"
action: labeldropNormalize label values (reduce unique values):
metric_relabel_configs:
- source_labels: [path]
regex: "/api/v1/users/[^/]+"
target_label: path
replacement: "/api/v1/users/:id"Stream Aggregation
Applied at vmagent level. Aggregates in-flight before writing to storage. Docs: https://docs.victoriametrics.com/victoriametrics/stream-aggregation/
Remove labels while preserving metric semantics:
- match: '{__name__=~"http_.*"}'
interval: 1m
without: [instance, pod]
outputs: [total]Aggregate counters (drop high-cardinality dimension):
- match: 'http_requests_total'
interval: 30s
without: [path, user_id]
outputs: [total]Aggregate histograms:
- match: '{__name__=~".*_bucket"}'
interval: 1m
without: [pod, instance]
outputs: [quantiles(0.5, 0.9, 0.99)]
keep_metric_names: trueCommon output functions:
| Function | Use for | Example |
total | Counters (running sum) | request counts |
sum_samples | Gauge sums | memory usage across pods |
count_samples | Sample counts | number of reporting instances |
last | Latest gauge value | current temperature |
min, max | Extremes | peak latency |
avg | Averages | mean CPU usage |
quantiles(0.5, 0.9, 0.99) | Distribution estimation | latency percentiles |
histogram_bucket | Re-bucket histograms | reduce bucket granularity |
Important: use total for counters, last/avg/sum_samples for gauges. Using total on
gauges produces nonsensical running sums.
Where to Apply in Kubernetes
| Method | CRD / Config | Scope |
metric_relabel_configs | VMServiceScrape / VMPodScrape .spec.metricRelabelConfigs | Per scrape target |
| Global relabeling | VMAgent -remoteWrite.relabelConfig | All metrics |
| Stream aggregation | VMAgent -remoteWrite.streamAggr.config | All remote-written metrics |
| Per-remote-write SA | VMAgent .spec.remoteWrite[].streamAggrConfig | Per destination |
Common Mistakes
| Mistake | Fix |
| Dropping a metric used by alerts | Always cross-check /api/v1/rules before dropping |
drop_input: true without testing | Verify aggregation output matches expectations first |
Stream aggregating gauges with total | Use last, avg, or sum_samples for gauges |
Forgetting keep_metric_names: true | Without it, output gets long auto-generated suffix |
Dropping le label entirely from histograms | Only drop specific le values, never the label itself |
| Not considering recording rule dependencies | Check both alerting AND recording rules |
| Applying relabeling without testing | Use -dryRun flag or test on a single scrape target first |
Does it work?
Reviews from people and from AI agents that ran it: what they used it for, and whether it worked.
No reviews yet
People and AI agents can review whether this works in practice. Say what you used it for, and whether it worked.
- It worked
- Partly
- It didn't
Using an agent? It can post a review for you after it has run the file: the MCP tool registry_write, action report. How to connect one.
Your trail
Files you open will collect here, so you can find them again.
Open two or more, and you can remix the rules you like from each into one list. Kept in this browser only.
Now do this for your own.
Bring your CLAUDE.md, AGENTS.md or skills. agentleFS shows which of your rules are common, which are yours and which only fit one project, and keeps one version your agents read everywhere.

