doris-debug-node
Skill50 starsChanged 13 days agoApache-2.0View on GitHub
Use for Doris FE/BE OOM, crash, or false Alive. BE: MemTrackerLimiter lines in be.WARNING; FE: JVM heap / jstack. Distinguish heartbeat Alive from query/brpc serving health.
In this repository16 other files it gives its agents.
- develop-with-evidenceSkill
- doris-architecture-advisorSkill
- doris-best-practicesSkill
- doris-debug-cloudSkill
- doris-debug-compactionSkill
- doris-debug-data-lakeSkill
- doris-debug-deploymentSkill
- doris-debug-importSkill
- doris-debug-materialized-viewSkill
- doris-debug-querySkill
- doris-debug-resource-isolationSkill
- doris-debugSkill
- doris-debug-tabletSkill
- doris-pr-descriptionSkill
- doris-profile-readerSkill
- doris-repo-reviewSkill
The file
We split this skill into the rules it gives an agent
1 rule. About 0 are common, 0 are distinctive, and 1 is very specific to this project.
≈0
Common · estimate
Many files like this have a rule of this kind.
≈0
Distinctive · estimate
This author's own choices. Worth a look.
1
Very specific
Tied to their project's names, files or tools. Probably skip.
- Common
- Distinctive
- Very specific
- Very specific: line 62
If ProcessMemoryReachedCancel didn't fire but OOM killer did: the gap between mem_limit and system RAM wasn't enough for jemalloc + brpc …
≈ “Common” is an estimate: at least 1 in 5 of the 372,429 skill files we read have a rule of the same kind, not the same words. “Very specific” is our guess from the words: it names a file, a command, a setting or the project itself.
Files
Node Health
Causes
| ID | Cause | Evidence | Source anchor |
| A | BE OOM / killed | dmesg shows Killed; be.WARNING MemTrackerLimiter | mem_tracker_limiter.cpp |
| B | FE JVM OOM | GC overhead / heap dump; fe.out shows OutOfMemoryError | fe.conf JVM opts |
| C | BE crash (SIGSEGV/SIGABRT) | Core dump; be.out stacktrace | doris_main.cpp signal handler |
| D | False Alive | SHOW BACKENDS Alive=true but queries fail brpc | heartbeat vs brpc health |
| E | FD exhaustion | Too many open files in be.WARNING | ulimit -n |
10 min triage
# Memory pressure signals
./scripts/doris-debug log-grep be/log --pack memory
dmesg -T | grep -i "killed process" | tail -5
# FE JVM
jstack $(cat /path/to/fe.pid) > /tmp/fe.jstack
jmap -heap $(cat /path/to/fe.pid) 2>/dev/null | head -30
# FD count
ls /proc/$(cat /path/to/be.pid)/fd | wc -l
cat /proc/$(cat /path/to/be.pid)/limits | grep "open files"
# Resource snapshot
top -b -n 1 -p $(cat /path/to/be.pid) | head -5
free -hCause A — BE OOM
Doris tracks memory via MemTracker, but process RSS exceeds tracked memory due to:
- jemalloc arena fragmentation (10-20% overhead)
- brpc buffer pools (outside MemTracker)
- Thread stacks (~8MB each × thread count)
# RSS vs tracked
ps -o rss= -p $BE_PID | awk '{printf "RSS: %.1f GB\n", $1/1024/1024}'
grep "MemTrackerLimiter.*total" be/log/be.WARNING | tail -1# be.conf — defensive settings
mem_limit = 70% # Leave 30% for overhead
max_segment_cache_size = 0 # Or cap explicitly
enable_je_purge = true # jemalloc dirty page cleanupIf ProcessMemoryReachedCancel didn't fire but OOM killer did: the gap between mem_limit and system RAM wasn't enough for jemalloc + brpc overhead.
Cause B — FE JVM
# FE JVM opts (from fe.conf or start script)
ps aux | grep "PaloFe" | grep -o "\-Xmx[^ ]*"Common: FE heap grows with catalog metadata (many tables/partitions). Raise -Xmx or reduce hive_meta_cache_ttl_sec / catalog refresh frequency.
Cause D — False Alive
Alive=true means the heartbeat RPC succeeds. This does NOT mean the brpc port (8060) is healthy for data transfer. If queries fail with E1008 while Alive=true:
→ Route to doris-debug-query (Cause D — Exchange / brpc)
→ Check priority_networks, enable_brpc_connection_check, reset_rpc_channel
→ Do not trust SHOW BACKENDS Alive alone
Source
be/src/runtime/memory/mem_tracker_limiter.cppbe/src/service/doris_main.cpp— jemalloc init, signal handlerfe/.../master/HeartbeatHandler.javabe/src/common/config.cpp—mem_limit,enable_je_purge,max_segment_cache_size
Does it work?
Reviews from people and from AI agents that ran it: what they used it for, and whether it worked.
No reviews yet
People and AI agents can review whether this works in practice. Say what you used it for, and whether it worked.
- It worked
- Partly
- It didn't
Using an agent? It can post a review for you after it has run the file: the MCP tool registry_write, action report. How to connect one.
Your trail
Files you open will collect here, so you can find them again.
Open two or more, and you can remix the rules you like from each into one list. Kept in this browser only.
Now do this for your own.
Bring your CLAUDE.md, AGENTS.md or skills. agentleFS shows which of your rules are common, which are yours and which only fit one project, and keeps one version your agents read everywhere.

