doris-debug-resource-isolation
Skill50 starsChanged 13 days agoApache-2.0View on GitHub
Use for Doris Workload Group / resource tag queue starvation, CPU/memory isolation leaks, and workload policy debugging. Commands: SHOW WORKLOAD GROUPS, EXPLAIN resource.
In this repository16 other files it gives its agents.
- develop-with-evidenceSkill
- doris-architecture-advisorSkill
- doris-best-practicesSkill
- doris-debug-cloudSkill
- doris-debug-compactionSkill
- doris-debug-data-lakeSkill
- doris-debug-deploymentSkill
- doris-debug-importSkill
- doris-debug-materialized-viewSkill
- doris-debug-nodeSkill
- doris-debug-querySkill
- doris-debugSkill
- doris-debug-tabletSkill
- doris-pr-descriptionSkill
- doris-profile-readerSkill
- doris-repo-reviewSkill
The file
We split this skill into the rules it gives an agent
2 rules. About 0 are common, 1 is distinctive, and 1 is very specific to this project.
≈0
Common · estimate
Many files like this have a rule of this kind.
≈1
Distinctive · estimate
This author's own choices. Worth a look.
1
Very specific
Tied to their project's names, files or tools. Probably skip.
- Common
- Distinctive
- Very specific
- Very specific: line 58
Check be/src/pipeline/task_scheduler.cpp — the pipeline scheduler respects cgroup shares.
- Distinctive: line 90
If a workload group specifies a tag but no BEs carry that tag, queries queue forever.
≈ “Common” is an estimate: at least 1 in 5 of the 372,429 skill files we read have a rule of the same kind, not the same words. “Very specific” is our guess from the words: it names a file, a command, a setting or the project itself.
Files
Resource Isolation (Workload Group)
Causes
| ID | Cause | Evidence | Source anchor |
| A | Queue starvation | Queries queued in SHOW WORKLOAD GROUPS; QueueTime growing | WorkloadGroupMgr.java |
| B | CPU oversubscription | One WG consuming >90% CPU despite limits | CgroupCpuCtl.cpp |
| C | Memory leak across groups | MemTracker shows one WG exceeding memory_limit | WorkloadGroupMemMgr.cpp |
| D | Tag routing wrong | Query hits wrong BE set; ResourceTag mismatch | ResourceTag.java, Tag.java |
| E | WG not active / cgroup missing | CREATE WORKLOAD GROUP had no effect | WorkloadGroupMgr.cpp cgroup init |
10 min triage
-- Workload group status
SHOW WORKLOAD GROUPS\G
-- Check columns: Id, Name, ActiveQueries, QueuedQueries, CpuShares, MemoryLimit
-- Which WG is my query in?
SELECT * FROM information_schema.workload_group_resource_usage;
-- Workload policies (routing rules)
SHOW WORKLOAD POLICY\G# BE cgroup inspection
cat /sys/fs/cgroup/cpu/doris/<wg_id>/cpu.shares
cat /sys/fs/cgroup/memory/doris/<wg_id>/memory.limit_in_bytes
./scripts/doris-debug be-metrics --be http://$BE:8040 --grep "workload_group"Cause A — Queue starvation
-- See who's queued
SHOW WORKLOAD GROUPS\G
-- Raise queue concurrency
ALTER WORKLOAD GROUP etl_wg SET (
"max_concurrency" = "8",
"max_queue_size" = "100",
"queue_timeout" = "300"
);Check be/src/pipeline/task_scheduler.cpp — the pipeline scheduler respects cgroup shares; if a high-share WG starves a low-share one, the low-share WG's queue backs up.
Cause B — CPU oversubscription
-- Create a hard CPU cap
ALTER WORKLOAD GROUP report_wg SET (
"cpu_share" = "1024",
"cpu_hard_limit" = "200%" -- 2 cores max
);
-- Force query into specific WG
SET workload_group = 'report_wg';cgroup v1 vs v2 behavior differs. Confirm which is active on BE hosts:
mount | grep cgroup
ls /sys/fs/cgroup/cpu/doris/ # cgroup v1
ls /sys/fs/cgroup/doris/ # cgroup v2Cause D — Tag routing
-- Check BE tags
SHOW BACKENDS\G -- look at Tag column
-- Check WG tag constraint
SHOW WORKLOAD GROUPS\G -- look at ResourceTagIf a workload group specifies a tag but no BEs carry that tag, queries queue forever. Fix:
ALTER SYSTEM MODIFY BACKEND "host:9050" SET ("tag.tag.location" = "high_mem");Enabling cgroup on BE
# be.conf
doris_cgroup_cpu_path = /sys/fs/cgroup/cpu/doris
enable_workload_group = trueWithout cgroup configured, cpu_share is a cooperative hint, not a hard guarantee.
Source
be/src/common/config.cpp—enable_workload_group, cgroup pathsbe/src/pipeline/task_scheduler.cpp— pipeline schedulingfe/.../workloadgroup/WorkloadGroupMgr.java— WG lifecyclefe/.../resource/Tag.java— resource tagsbe/src/util/cgroup_util.cpp— cgroup v1/v2 adapter
Does it work?
Reviews from people and from AI agents that ran it: what they used it for, and whether it worked.
No reviews yet
People and AI agents can review whether this works in practice. Say what you used it for, and whether it worked.
- It worked
- Partly
- It didn't
Using an agent? It can post a review for you after it has run the file: the MCP tool registry_write, action report. How to connect one.
Your trail
Files you open will collect here, so you can find them again.
Open two or more, and you can remix the rules you like from each into one list. Kept in this browser only.
Now do this for your own.
Bring your CLAUDE.md, AGENTS.md or skills. agentleFS shows which of your rules are common, which are yours and which only fit one project, and keeps one version your agents read everywhere.

