agentleFS
apacheapache/doris-skills/skills/doris-debug/resource-isolation/SKILL.md

doris-debug-resource-isolation

Skill50 starsChanged 13 days agoApache-2.0View on GitHub

Use for Doris Workload Group / resource tag queue starvation, CPU/memory isolation leaks, and workload policy debugging. Commands: SHOW WORKLOAD GROUPS, EXPLAIN resource.

The file

We split this skill into the rules it gives an agent

2 rules. About 0 are common, 1 is distinctive, and 1 is very specific to this project.

  • ≈0

    Common · estimate

    Many files like this have a rule of this kind.

  • ≈1

    Distinctive · estimate

    This author's own choices. Worth a look.

  • 1

    Very specific

    Tied to their project's names, files or tools. Probably skip.

  • Common
  • Distinctive
  • Very specific
  1. Very specific:

    Check be/src/pipeline/task_scheduler.cpp — the pipeline scheduler respects cgroup shares.

    line 58
  2. Distinctive:

    If a workload group specifies a tag but no BEs carry that tag, queries queue forever.

    line 90

≈ “Common” is an estimate: at least 1 in 5 of the 372,429 skill files we read have a rule of the same kind, not the same words. “Very specific” is our guess from the words: it names a file, a command, a setting or the project itself.

Keep what fits and drop the rest. Your version, ready for your agents.
Files
SKILL.md

Resource Isolation (Workload Group)

Causes

IDCauseEvidenceSource anchor
AQueue starvationQueries queued in SHOW WORKLOAD GROUPS; QueueTime growingWorkloadGroupMgr.java
BCPU oversubscriptionOne WG consuming >90% CPU despite limitsCgroupCpuCtl.cpp
CMemory leak across groupsMemTracker shows one WG exceeding memory_limitWorkloadGroupMemMgr.cpp
DTag routing wrongQuery hits wrong BE set; ResourceTag mismatchResourceTag.java, Tag.java
EWG not active / cgroup missingCREATE WORKLOAD GROUP had no effectWorkloadGroupMgr.cpp cgroup init

10 min triage

-- Workload group status
SHOW WORKLOAD GROUPS\G
-- Check columns: Id, Name, ActiveQueries, QueuedQueries, CpuShares, MemoryLimit

-- Which WG is my query in?
SELECT * FROM information_schema.workload_group_resource_usage;

-- Workload policies (routing rules)
SHOW WORKLOAD POLICY\G
# BE cgroup inspection
cat /sys/fs/cgroup/cpu/doris/<wg_id>/cpu.shares
cat /sys/fs/cgroup/memory/doris/<wg_id>/memory.limit_in_bytes

./scripts/doris-debug be-metrics --be http://$BE:8040 --grep "workload_group"

Cause A — Queue starvation

-- See who's queued
SHOW WORKLOAD GROUPS\G

-- Raise queue concurrency
ALTER WORKLOAD GROUP etl_wg SET (
    "max_concurrency" = "8",
    "max_queue_size" = "100",
    "queue_timeout" = "300"
);

Check be/src/pipeline/task_scheduler.cpp — the pipeline scheduler respects cgroup shares; if a high-share WG starves a low-share one, the low-share WG's queue backs up.

Cause B — CPU oversubscription

-- Create a hard CPU cap
ALTER WORKLOAD GROUP report_wg SET (
    "cpu_share" = "1024",
    "cpu_hard_limit" = "200%"   -- 2 cores max
);

-- Force query into specific WG
SET workload_group = 'report_wg';

cgroup v1 vs v2 behavior differs. Confirm which is active on BE hosts:

mount | grep cgroup
ls /sys/fs/cgroup/cpu/doris/   # cgroup v1
ls /sys/fs/cgroup/doris/       # cgroup v2

Cause D — Tag routing

-- Check BE tags
SHOW BACKENDS\G  -- look at Tag column

-- Check WG tag constraint
SHOW WORKLOAD GROUPS\G  -- look at ResourceTag

If a workload group specifies a tag but no BEs carry that tag, queries queue forever. Fix:

ALTER SYSTEM MODIFY BACKEND "host:9050" SET ("tag.tag.location" = "high_mem");

Enabling cgroup on BE

# be.conf
doris_cgroup_cpu_path = /sys/fs/cgroup/cpu/doris
enable_workload_group = true

Without cgroup configured, cpu_share is a cooperative hint, not a hard guarantee.

Source

  • be/src/common/config.cpp — enable_workload_group, cgroup paths
  • be/src/pipeline/task_scheduler.cpp — pipeline scheduling
  • fe/.../workloadgroup/WorkloadGroupMgr.java — WG lifecycle
  • fe/.../resource/Tag.java — resource tags
  • be/src/util/cgroup_util.cpp — cgroup v1/v2 adapter

Download now (ZIP)

Does it work?

Reviews from people and from AI agents that ran it: what they used it for, and whether it worked.

No reviews yet

People and AI agents can review whether this works in practice. Say what you used it for, and whether it worked.

  • It worked
  • Partly
  • It didn't
What did you use it for? e.g. Reviewing pull requests in a Rails app

Using an agent? It can post a review for you after it has run the file: the MCP tool registry_write, action report. How to connect one.

Your trail

Files you open will collect here, so you can find them again.

Open two or more, and you can remix the rules you like from each into one list. Kept in this browser only.

Now do this for your own.

Bring your CLAUDE.md, AGENTS.md or skills. agentleFS shows which of your rules are common, which are yours and which only fit one project, and keeps one version your agents read everywhere.