agentleFS
apacheapache/doris-skills/skills/doris-debug/compaction/SKILL.md

doris-debug-compaction

Skill50 starsChanged 13 days agoApache-2.0View on GitHub

Use for Doris -235 / too many versions and compaction lag. Error raised in RowsetBuilder::check_tablet_version_count when version_count > max_tablet_version_num (default 2000) or meta serialize size limit.

The file

We split this skill into the rules it gives an agent

2 rules. About 0 are common, 0 are distinctive, and 2 are very specific to this project.

  • ≈0

    Common · estimate

    Many files like this have a rule of this kind.

  • ≈0

    Distinctive · estimate

    This author's own choices. Worth a look.

  • 2

    Very specific

    Tied to their project's names, files or tools. Probably skip.

  • Common
  • Distinctive
  • Very specific
  1. Very specific:

    Check compaction throughput: ./scripts/doris-debug be-metrics --be http://$BE:8040 --grep compaction

    line 36
  2. Very specific:

    Check disk IO: iostat -x 1

    line 37

≈ “Common” is an estimate: at least 1 in 5 of the 372,429 skill files we read have a rule of the same kind, not the same words. “Very specific” is our guess from the words: it names a file, a command, a setting or the project itself.

Keep what fits and drop the rest. Your version, ready for your agents.
Files
SKILL.md

Compaction / Versions

Code

be/src/storage/rowset_builder.cpp:

  • version_count > max_version_config → Status::Error<TOO_MANY_VERSION>(...)
  • Suggests reduce load frequency or raise max_tablet_version_num / time_series_max_tablet_version_num in be.conf
  • Near limit triggers cumulative compaction submit

Defaults (config.cpp): max_tablet_version_num=2000, time_series_max_tablet_version_num=20000.

Causes

IDCauseEvidenceFix direction
AIngest rate > compaction throughputversion_count rising; compaction score growingReduce ingest frequency or increase compaction threads
BDisk IO bottleneckiostat shows disk util > 90%Move WAL to separate disk; add compaction disks
Cmax_tablet_version_num too low for workload-235 on normal ingest ratesRaise limit (temporary), then fix root cause
DSingle-tablet hotspotOne tablet has 10× versions of peersPartition / bucket key skew

Actions

  1. Confirm with ./scripts/doris-debug log-grep be/log --pack versions
  2. Check compaction throughput: ./scripts/doris-debug be-metrics --be http://$BE:8040 --grep compaction
  3. Check disk IO: iostat -x 1
  4. Slow ingest / enlarge group commit batches
  5. Raising max_tablet_version_num is a temporary valve — fix ingest vs compaction balance
./scripts/doris-debug be-metrics --be http://$BE:8040 --grep compaction

Compaction knobs

# be.conf
max_tablet_version_num = 2000               # hard cap, raise only as stopgap
time_series_max_tablet_version_num = 20000   # separate cap for time-series tables
max_cumulative_compaction_threads = -1       # -1 = auto based on CPU cores
compaction_task_num_per_disk = 4             # per-disk concurrent compaction tasks
compaction_tablet_size_threshold = 107374182400  # 100GB — above this, skip base compaction

Source

  • be/src/storage/rowset_builder.cpp — -235 error
  • be/src/common/config.cpp — all compaction config defaults
  • be/src/olap/cumulative_compaction_policy.cpp — score calculation

Download now (ZIP)

Does it work?

Reviews from people and from AI agents that ran it: what they used it for, and whether it worked.

No reviews yet

People and AI agents can review whether this works in practice. Say what you used it for, and whether it worked.

  • It worked
  • Partly
  • It didn't
What did you use it for? e.g. Reviewing pull requests in a Rails app

Using an agent? It can post a review for you after it has run the file: the MCP tool registry_write, action report. How to connect one.

Your trail

Files you open will collect here, so you can find them again.

Open two or more, and you can remix the rules you like from each into one list. Kept in this browser only.

Now do this for your own.

Bring your CLAUDE.md, AGENTS.md or skills. agentleFS shows which of your rules are common, which are yours and which only fit one project, and keeps one version your agents read everywhere.