LogQL Query Generator

Mindset

LogQL queries are cheap to write and expensive to run wrong. A query that reads fine in isolation can take down a Loki cluster in production because stream selectors and filter ordering determine how much raw data gets scanned before anything is discarded — the query planner does not reorder your pipeline for you. Treat query construction as a performance decision first and a syntax exercise second: every stage you add either narrows the data Loki has to touch or forces it to touch more of it.

When to Use: creating new LogQL queries, log-filter or metric queries for dashboards, Loki alerting rules, or troubleshooting a production Loki instance where filter ordering or label cardinality is suspected as the cause of slow or timing-out queries.

When NOT to Use: writing PromQL for a Prometheus-native metric (LogQL only applies to Loki log streams, not scraped metrics), or when the user already has a working, performant query and only wants a syntax explanation — in that case answer directly rather than running the full planning workflow.

Interactive Query Planning Workflow

CRITICAL: Always engage the user in collaborative planning before generating queries.

Stages 1-3: Gather Requirements (use AskUserQuestion)

Ask about: goal (error analysis, alerting, debugging), use case, log sources (labels, format), query type (log/metric), filtering needs, parsing method, aggregation, and time range.

Stage 4: Plan, Validate & Consult References

Before generating code, present a plain-English plan and confirm with the user via AskUserQuestion:

## LogQL Query Plan

**Goal**: [Description]
**Query Structure**:
1. Select streams: `{label="value"}`
2. Filter lines: [operations]
3. Parse logs: [parser]
4. Aggregate: [function]

**Does this match your intentions?**

Once confirmed, MANDATORY: consult references before generating. Do NOT rely on prior knowledge.

Local References (Read tool)

Example paths (relative to this skill's own directory, wherever it is installed):

Read("assets/common_queries.logql")
Read("references/best_practices.md")

External Documentation (context7 MCP / WebSearch)

Use when local references don't cover the topic:

Stage 5: Generate Query

Best Practices

1. Specific Stream Selectors: {namespace="prod", app="api", level="error"} not just {namespace="prod"}
2. Filter Order: Line filter → parse → label filter (fastest to slowest)
3. Parser Performance: pattern > logfmt > json > regexp

Core Query Patterns

Log Filtering:

{job="app"} |= "error" |= "timeout"        # Contains both
{job="app"} |~ "error|fatal|critical"       # Regex match
{job="app"} != "debug"                      # Exclude

JSON/logfmt Parsing:

{app="api"} | json | level="error" | status_code >= 500
{app="app"} | logfmt | caller="database.go"

Pattern Extraction:

{job="nginx"} | pattern "<ip> - - [<_>] \"<method> <path>\" <status> <size>"

Metrics:

# Rate
rate({job="app"} | json | level="error" [5m])

# Count by label
sum by (app) (count_over_time({namespace="prod"} | json [5m]))

# Error percentage
sum(rate({app="api"} | json | level="error" [5m])) / sum(rate({app="api"}[5m])) * 100

# Latency percentiles
quantile_over_time(0.95, {app="api"} | json | unwrap duration [5m])

# Top N
topk(10, sum by (error_type) (count_over_time({job="app"} | json | level="error" [1h])))

Formatting:

{job="app"} | json | line_format "{{.level}}: {{.message}}"
{job="app"} | json | label_format env="{{.environment}}"

IP Filtering (prefer label filter after parsing for precision):

{job="nginx"} | logfmt | remote_addr = ip("192.168.4.0/24")

Stage 5a: Incremental Query Building (Educational/Debugging)

When to use this stage:

• User is learning LogQL
• Complex multi-stage queries
• Debugging query issues
• User explicitly requests step-by-step explanation

Present the query construction incrementally:

## Building Your Query Step-by-Step

### Step 1: Stream Selector (verify logs exist)

```logql
{app="api"}

Test this first to confirm logs are flowing

Step 2: Add Line Filter (fast pre-filtering)

{app="api"} |= "error"

Reduces data before parsing

Step 3: Add Parser (extract fields)

{app="api"} |= "error" | json

Now you can filter on extracted labels

Step 4: Add Label Filter (precise filtering)

{app="api"} |= "error" | json | level="error"

Final filter on parsed data

Step 5: Add Aggregation (if metric query)

sum(count_over_time({app="api"} |= "error" | json | level="error" [5m]))

Complete metric query

**Use AskUserQuestion** to offer incremental mode:
- Option: "Show step-by-step construction" vs "Show final query only"

### Stage 6: Provide Usage

1. **Final Query** with explanation
2. **How to Use**: Grafana panel, Loki alerting rules, `logcli query`, HTTP API
3. **Customization**: Labels to modify, thresholds to tune

## Advanced Techniques

### Multiple Parsers

```logql
{app="api"} | json | regexp "user_(?P<user_id>\\d+)"

Unwrap for Numeric Metrics

sum(sum_over_time({app="api"} | json | unwrap duration [5m]))

Pattern Match Operators (Loki 3.0+, 10x faster than regex)

{service_name=`app`} |> "<_> level=debug <_>"

Logical Operators

{app="api"} | json | (status_code >= 400 and status_code < 500) or level="error"

Offset Modifier

sum(rate({app="api"} | json | level="error" [5m])) - sum(rate({app="api"} | json | level="error" [5m] offset 1d))

Label Operations

{app="api"} | json | keep namespace, pod, level
{app="api"} | json | drop pod, instance

Note: LogQL has no dedup or distinct operators. Use metric aggregations like sum by (field) for programmatic deduplication.

Loki 3.x Key Features

Structured Metadata

High-cardinality data without indexing (trace_id, user_id, request_id):

# Filter AFTER stream selector, NOT in it
{app="api"} | trace_id="abc123" | json | level="error"

Query Acceleration (Bloom Filters)

Place structured metadata filters BEFORE parsers:

# ACCELERATED
{cluster="prod"} | detected_level="error" | logfmt | json
# NOT ACCELERATED
{cluster="prod"} | logfmt | json | detected_level="error"

approx_topk (Probabilistic)

approx_topk(10, sum by (endpoint) (rate({app="api"}[5m])))

vector() for Alerting

sum(count_over_time({app="api"} | json | level="error" [5m])) or vector(0)

Automatic Labels

• service_name: Auto-populated from container name
• detected_level: Auto-detected when discover_log_levels: true (stored as structured metadata)

Function and Parser Quick Reference

For comprehensive function and parser documentation, see references/function_reference.md:

• Log range aggregations: rate(), count_over_time(), bytes_rate(), absent_over_time()
• Unwrapped aggregations: sum_over_time(), quantile_over_time(), etc.
• Aggregation operators: sum, topk, approx_topk, with by/without grouping
• Parsers: json, logfmt with options
• Template functions for line_format/label_format

Alerting Rules

# Alert when error rate exceeds 5%
(sum(rate({app="api"} | json | level="error" [5m])) / sum(rate({app="api"}[5m]))) > 0.05

# With vector() to avoid "no data"
sum(rate({app="api"} | json | level="error" [5m])) or vector(0) > 10

Anti-Patterns

NEVER use a broad stream selector without label filters

• WHY: {job="app"} on high-volume streams reads all log lines before any filtering, causing massive I/O and query timeouts in production Loki.
• BAD: {job="app"} | json | level="error"
• GOOD: {job="app", namespace="prod"} |= "error" | json | level="error" — line filter before parser reduces data volume

NEVER place parsers before line filters

• WHY: Parsers (json, logfmt, regexp) are CPU-intensive. Placing a fast line filter (|=, !=, |~) before the parser pre-filters data and reduces parsing overhead by up to 10x.
• BAD: {app="api"} | json | level="error"
• GOOD: {app="api"} |= "error" | json | level="error"

NEVER use regexp when logfmt or json applies

• WHY: Parser performance order is: pattern > logfmt > json > regexp. Regexp is 5-20x slower than structured parsers and breaks on format changes.
• BAD: {app="api"} | regexp "level=(?P<level>[^ ]+)"
• GOOD: {app="api"} | logfmt (if logs are logfmt-formatted)

NEVER put high-cardinality values in stream selectors

• WHY: Loki indexes stream selectors. High-cardinality labels (user_id, trace_id, request_id) in stream selectors create millions of streams, causing index bloat and degraded query performance.
• BAD: {user_id="12345", app="api"} | json
• GOOD: {app="api"} | trace_id="abc123" | json — use structured metadata for high-cardinality values (Loki 3.x)

NEVER use count_over_time when rate is needed for alerting

• WHY: count_over_time returns a raw count, not a rate. Alert thresholds set on raw counts break when the time range changes. Use rate() for stable per-second thresholds.
• BAD: count_over_time({app="api"} | json | level="error" [5m]) > 100
• GOOD: rate({app="api"} | json | level="error" [5m]) > 2 (2 errors/second)

Error Handling

Guidelines

1. Always plan interactively - Present plain-English plan before generating
2. Use AskUserQuestion - Gather requirements and confirm plans
3. Consult references - See Stage 4 for mandatory reference consultation (local + external)
4. Offer incremental building - See Stage 5a for step-by-step construction
5. Explain queries - What it does, how to interpret results
6. Prioritize performance - Specific selectors, filter early, simpler parsers

Version Notes

• Loki 3.0+: Bloom filters, structured metadata, pattern match operators (|>, !>)
• Loki 3.3+: approx_topk function
• Loki 3.5+: Promtail deprecated (use Grafana Alloy)
• Loki 3.6+: Horizontally scalable compactor, Loki UI as Grafana plugin

Deprecations: Promtail (use Alloy), BoltDB store (use TSDB with v13 schema)

References

• Common Queries — comprehensive query examples covering filtering, parsing, metrics, alerting, and Loki 3.x patterns
• Best Practices — 39+ LogQL best practices, performance optimization, and anti-patterns
• Function Reference — quick reference tables for all LogQL functions and parsers