doris-debug-compaction
Skill50 starsChanged 13 days agoApache-2.0View on GitHub
Use for Doris -235 / too many versions and compaction lag. Error raised in RowsetBuilder::check_tablet_version_count when version_count > max_tablet_version_num (default 2000) or meta serialize size limit.
In this repository16 other files it gives its agents.
- develop-with-evidenceSkill
- doris-architecture-advisorSkill
- doris-best-practicesSkill
- doris-debug-cloudSkill
- doris-debug-data-lakeSkill
- doris-debug-deploymentSkill
- doris-debug-importSkill
- doris-debug-materialized-viewSkill
- doris-debug-nodeSkill
- doris-debug-querySkill
- doris-debug-resource-isolationSkill
- doris-debugSkill
- doris-debug-tabletSkill
- doris-pr-descriptionSkill
- doris-profile-readerSkill
- doris-repo-reviewSkill
The file
We split this skill into the rules it gives an agent
2 rules. About 0 are common, 0 are distinctive, and 2 are very specific to this project.
≈0
Common · estimate
Many files like this have a rule of this kind.
≈0
Distinctive · estimate
This author's own choices. Worth a look.
2
Very specific
Tied to their project's names, files or tools. Probably skip.
- Common
- Distinctive
- Very specific
- Very specific: line 36
Check compaction throughput: ./scripts/doris-debug be-metrics --be http://$BE:8040 --grep compaction
- Very specific: line 37
Check disk IO: iostat -x 1
≈ “Common” is an estimate: at least 1 in 5 of the 372,429 skill files we read have a rule of the same kind, not the same words. “Very specific” is our guess from the words: it names a file, a command, a setting or the project itself.
Files
Compaction / Versions
Code
be/src/storage/rowset_builder.cpp:
version_count > max_version_config→Status::Error<TOO_MANY_VERSION>(...)- Suggests reduce load frequency or raise
max_tablet_version_num/time_series_max_tablet_version_numinbe.conf - Near limit triggers cumulative compaction submit
Defaults (config.cpp): max_tablet_version_num=2000, time_series_max_tablet_version_num=20000.
Causes
| ID | Cause | Evidence | Fix direction |
| A | Ingest rate > compaction throughput | version_count rising; compaction score growing | Reduce ingest frequency or increase compaction threads |
| B | Disk IO bottleneck | iostat shows disk util > 90% | Move WAL to separate disk; add compaction disks |
| C | max_tablet_version_num too low for workload | -235 on normal ingest rates | Raise limit (temporary), then fix root cause |
| D | Single-tablet hotspot | One tablet has 10× versions of peers | Partition / bucket key skew |
Actions
- Confirm with
./scripts/doris-debug log-grep be/log --pack versions - Check compaction throughput:
./scripts/doris-debug be-metrics --be http://$BE:8040 --grep compaction - Check disk IO:
iostat -x 1 - Slow ingest / enlarge group commit batches
- Raising
max_tablet_version_numis a temporary valve — fix ingest vs compaction balance
./scripts/doris-debug be-metrics --be http://$BE:8040 --grep compactionCompaction knobs
# be.conf
max_tablet_version_num = 2000 # hard cap, raise only as stopgap
time_series_max_tablet_version_num = 20000 # separate cap for time-series tables
max_cumulative_compaction_threads = -1 # -1 = auto based on CPU cores
compaction_task_num_per_disk = 4 # per-disk concurrent compaction tasks
compaction_tablet_size_threshold = 107374182400 # 100GB — above this, skip base compactionSource
be/src/storage/rowset_builder.cpp—-235errorbe/src/common/config.cpp— all compaction config defaultsbe/src/olap/cumulative_compaction_policy.cpp— score calculation
Does it work?
Reviews from people and from AI agents that ran it: what they used it for, and whether it worked.
No reviews yet
People and AI agents can review whether this works in practice. Say what you used it for, and whether it worked.
- It worked
- Partly
- It didn't
Using an agent? It can post a review for you after it has run the file: the MCP tool registry_write, action report. How to connect one.
Your trail
Files you open will collect here, so you can find them again.
Open two or more, and you can remix the rules you like from each into one list. Kept in this browser only.
Now do this for your own.
Bring your CLAUDE.md, AGENTS.md or skills. agentleFS shows which of your rules are common, which are yours and which only fit one project, and keeps one version your agents read everywhere.

