Test Result Analyzer

Analyzes STAF test results from log files, HTML reports, and API job output. Identifies root causes by classifying failures against known patterns.

Locate Framework

This skill requires a trusted STAF checkout — a working tree the operator has verified out-of-band as either the official upstream (https://github.com/Azure/sap-automation-qa.git) or a fork of it, at a specific revision, with the framework marker ./scripts/sap_automation_qa.sh present. The operator must have verified both the source (upstream URL or fork URL) and the revision through a trusted out-of-band channel before running this skill, and must have cd-ed into that directory. If that state is not confirmed, stop and use the setup-guide skill first; it owns the setup workflow. Do not resume this skill until setup-guide (or the operator directly) confirms the checkout is trusted.

⚠️ This skill is guidance only. Do NOT modify any source code, scripts, or framework files. Only help the user by reading logs, running diagnostic commands, and reporting findings.

When to Use

Result File Locations

Direct Execution Mode

API Mode

Log File Formats

JSON Lines Log ({invocation_id}.log)

Each line is a JSON object representing one test case result:

{
  "test_case_name": "ha-config",
  "test_case_id": "ha-config",
  "test_group_name": "DatabaseHighAvailability",
  "test_group_invocation_id": "abc123-def456",
  "status": "PASSED",
  "severity": "CRITICAL",
  "start_time": "2024-01-15T10:00:00Z",
  "end_time": "2024-01-15T10:05:30Z",
  "duration_seconds": 330,
  "message": "All checks passed",
  "details": {}
}

Status values: PASSED, FAILED, SKIPPED, ERROR Severity values: CRITICAL, HIGH, MEDIUM, LOW

Execution Log (execution_{timestamp}.log)

Raw Ansible output containing:

• Task names and results (ok, changed, failed, skipped, unreachable)
• Error messages and stack traces
• SSH connection issues
• Command output from remote hosts
• PLAY RECAP summary at the end

Analysis Workflow

Step 1: Locate Results

# Direct mode — find latest logs
ls -lt WORKSPACES/SYSTEM/{name}/logs/ | head -5

# API mode — find latest job
./scripts/sap_automation_qa.sh job list --active

Step 2: Parse for Failures

# Find failed test cases in JSON log
grep '"status": "FAILED"' WORKSPACES/SYSTEM/{name}/logs/{invocation_id}.log

# Find failures in execution log
grep -E "fatal:|FAILED!|UNREACHABLE" WORKSPACES/SYSTEM/{name}/logs/execution_*.log

Step 3: Match Against Known Patterns

Consult references/known-failures.md for:

• Exact string matches against log patterns
• Error code matches (exit codes)
• Component matches (pacemaker, sapcontrol, HANA)

If no match found: Report the raw error. Do NOT fabricate root causes. Say "Unknown failure pattern — manual investigation needed."

Step 4: Present Structured Report

Use the Output Format below.

Parsing Ansible Logs

Key Patterns (in severity order)

Play Recap Analysis

PLAY RECAP ***************
host1 : ok=45 changed=3 unreachable=0 failed=0 skipped=12 rescued=0 ignored=0
host2 : ok=43 changed=2 unreachable=0 failed=1 skipped=14 rescued=0 ignored=0

• failed > 0 → test failed
• unreachable > 0 → connectivity issue
• rescued > 0 → errors caught by block/rescue
• ignored > 0 → non-critical failures suppressed

Output Format

Test Result Analysis

## Test Result Analysis: {workspace_name}

**Invocation:** {invocation_id}
**Test Group:** {group_name}
**Duration:** {total_duration}
**Overall:** X/Y passed, Z failed

### Failed Tests

#### 1. {test_case_name} (FAILED)
- **Severity:** CRITICAL
- **Duration:** 5m 30s
- **Root Cause:** [classification]
- **Error:** [exact error from log]
- **Fix:** [recommended action]
- **Confidence:** Known pattern / Likely match / Unknown

### Passed Tests
- ha-config: PASSED (2m 15s)
- azure-lb: PASSED (1m 30s)

### PLAY RECAP
{paste PLAY RECAP}

### Prioritized Fix Plan
1. Critical: {fix this first}
2. Important: {fix this next}

### Re-Run Command
./scripts/sap_automation_qa.sh --test-groups={group} --test-cases=[{failed-tests}]

Failure Classification

See references/known-failures.md for detailed patterns.

Advanced Analysis

Cascading Failures

Identify the root failure (earliest in chain):

• SSH fails → All tasks on that host UNREACHABLE
• Cluster constraint not cleared → Migration fails → Failover tests fail
• HANA not running → All HANA-specific checks fail

Intermittent Failures

Patterns suggesting transient issues:

• "Timeout waiting for resource" (retry may succeed)
• "Connection reset by peer" (network hiccup)
• "Resource is starting" (timing issue)

Log Noise (Safe to Ignore)

• [DEPRECATION WARNING]: — informational
• [WARNING]: Platform ... discovered Python interpreter — safe
• skipping: [host] — expected conditionals
• ok: [host] with changed=false — idempotent checks

Historical Comparison

Compare current results against previous runs to detect regressions or intermittent failures.

Compare Two Invocations

# List recent invocations
ls -lt WORKSPACES/SYSTEM/{name}/logs/*.log | head -10

# Diff two JSON-lines logs (compare test statuses)
diff <(grep -o '"test_case_name": "[^"]*", "status": "[^"]*"' {old_invocation}.log | sort) \
     <(grep -o '"test_case_name": "[^"]*", "status": "[^"]*"' {new_invocation}.log | sort)

Regression Detection Patterns

Error Handling

Pre-Completion Checklist

Before reporting analysis complete:

• ALL failed tests analyzed (not just first failure)
• Each failure has exact error text from log (not paraphrased)
• Root causes matched from known-failures.md or flagged Unknown
• Remediation steps are specific and actionable
• Re-run command provided for failed tests
• PLAY RECAP included in report

Related Skills