agentleFS
apacheapache/doris-skills/skills/doris-debug/node/SKILL.md

doris-debug-node

Skill50 starsChanged 13 days agoApache-2.0View on GitHub

Use for Doris FE/BE OOM, crash, or false Alive. BE: MemTrackerLimiter lines in be.WARNING; FE: JVM heap / jstack. Distinguish heartbeat Alive from query/brpc serving health.

The file

We split this skill into the rules it gives an agent

1 rule. About 0 are common, 0 are distinctive, and 1 is very specific to this project.

  • ≈0

    Common · estimate

    Many files like this have a rule of this kind.

  • ≈0

    Distinctive · estimate

    This author's own choices. Worth a look.

  • 1

    Very specific

    Tied to their project's names, files or tools. Probably skip.

  • Common
  • Distinctive
  • Very specific
  1. Very specific:

    If ProcessMemoryReachedCancel didn't fire but OOM killer did: the gap between mem_limit and system RAM wasn't enough for jemalloc + brpc …

    line 62

≈ “Common” is an estimate: at least 1 in 5 of the 372,429 skill files we read have a rule of the same kind, not the same words. “Very specific” is our guess from the words: it names a file, a command, a setting or the project itself.

Keep what fits and drop the rest. Your version, ready for your agents.
Files
SKILL.md

Node Health

Causes

IDCauseEvidenceSource anchor
ABE OOM / killeddmesg shows Killed; be.WARNING MemTrackerLimitermem_tracker_limiter.cpp
BFE JVM OOMGC overhead / heap dump; fe.out shows OutOfMemoryErrorfe.conf JVM opts
CBE crash (SIGSEGV/SIGABRT)Core dump; be.out stacktracedoris_main.cpp signal handler
DFalse AliveSHOW BACKENDS Alive=true but queries fail brpcheartbeat vs brpc health
EFD exhaustionToo many open files in be.WARNINGulimit -n

10 min triage

# Memory pressure signals
./scripts/doris-debug log-grep be/log --pack memory
dmesg -T | grep -i "killed process" | tail -5

# FE JVM
jstack $(cat /path/to/fe.pid) > /tmp/fe.jstack
jmap -heap $(cat /path/to/fe.pid) 2>/dev/null | head -30

# FD count
ls /proc/$(cat /path/to/be.pid)/fd | wc -l
cat /proc/$(cat /path/to/be.pid)/limits | grep "open files"

# Resource snapshot
top -b -n 1 -p $(cat /path/to/be.pid) | head -5
free -h

Cause A — BE OOM

Doris tracks memory via MemTracker, but process RSS exceeds tracked memory due to:

  • jemalloc arena fragmentation (10-20% overhead)
  • brpc buffer pools (outside MemTracker)
  • Thread stacks (~8MB each × thread count)
# RSS vs tracked
ps -o rss= -p $BE_PID | awk '{printf "RSS: %.1f GB\n", $1/1024/1024}'
grep "MemTrackerLimiter.*total" be/log/be.WARNING | tail -1
# be.conf — defensive settings
mem_limit = 70%              # Leave 30% for overhead
max_segment_cache_size = 0   # Or cap explicitly
enable_je_purge = true       # jemalloc dirty page cleanup

If ProcessMemoryReachedCancel didn't fire but OOM killer did: the gap between mem_limit and system RAM wasn't enough for jemalloc + brpc overhead.

Cause B — FE JVM

# FE JVM opts (from fe.conf or start script)
ps aux | grep "PaloFe" | grep -o "\-Xmx[^ ]*"

Common: FE heap grows with catalog metadata (many tables/partitions). Raise -Xmx or reduce hive_meta_cache_ttl_sec / catalog refresh frequency.

Cause D — False Alive

Alive=true means the heartbeat RPC succeeds. This does NOT mean the brpc port (8060) is healthy for data transfer. If queries fail with E1008 while Alive=true:

→ Route to doris-debug-query (Cause D — Exchange / brpc) → Check priority_networks, enable_brpc_connection_check, reset_rpc_channel → Do not trust SHOW BACKENDS Alive alone

Source

  • be/src/runtime/memory/mem_tracker_limiter.cpp
  • be/src/service/doris_main.cpp — jemalloc init, signal handler
  • fe/.../master/HeartbeatHandler.java
  • be/src/common/config.cpp — mem_limit, enable_je_purge, max_segment_cache_size

Download now (ZIP)

Does it work?

Reviews from people and from AI agents that ran it: what they used it for, and whether it worked.

No reviews yet

People and AI agents can review whether this works in practice. Say what you used it for, and whether it worked.

  • It worked
  • Partly
  • It didn't
What did you use it for? e.g. Reviewing pull requests in a Rails app

Using an agent? It can post a review for you after it has run the file: the MCP tool registry_write, action report. How to connect one.

Your trail

Files you open will collect here, so you can find them again.

Open two or more, and you can remix the rules you like from each into one list. Kept in this browser only.

Now do this for your own.

Bring your CLAUDE.md, AGENTS.md or skills. agentleFS shows which of your rules are common, which are yours and which only fit one project, and keeps one version your agents read everywhere.