agentleFS
samsam007bsamsam007b/claude-code-standards-kit/skills/continuous-testing-guardian/SKILL.md

continuous-testing-guardian

Skill2 starsChanged 8 months agoMITView on GitHub

Implements a continuous testing system with chaos engineering to automatically detect regressions introduced by code changes (AI or human)

  • Reads credentials

The file

We split this skill into the rules it gives an agent

5 rules. About 0 are common, 5 are distinctive, and 0 are very specific to this project.

  • ≈0

    Common · estimate

    Many files like this have a rule of this kind.

  • ≈5

    Distinctive · estimate

    This author's own choices. Worth a look.

  • 0

    Very specific

    Tied to their project's names, files or tools. Probably skip.

  • Common
  • Distinctive
  • Very specific
  1. Distinctive:

    Alerts immediately: when a change breaks a functional chain

    line 14
  2. Distinctive:

    Prevents technical debt: before it accumulates

    line 15
  3. Distinctive:

    Add tests for each bug found: Every bug becomes a new test

    line 959
  4. Distinctive:

    Refine chaos scenarios: Add new failure modes

    line 960
  5. Distinctive:

    Review test relevance: Remove obsolete tests

    line 962

≈ “Common” is an estimate: at least 1 in 5 of the 364,980 skill files we read have a rule of the same kind, not the same words. “Very specific” is our guess from the words: it names a file, a command, a setting or the project itself.

Keep what fits and drop the rest. Your version, ready for your agents.
Files
SKILL.md

Continuous Testing Guardian - "Mais Codeur" System

Goal

Build a continuous testing system that:

  1. Automatically walks through all critical user flows
  2. Detects invisible regressions introduced by Claude Code
  3. Alerts immediately when a change breaks a functional chain
  4. Prevents technical debt before it accumulates

Concept: The "Mais Codeur"

Inspired by Chaos Engineering (Netflix Chaos Monkey) but adapted to the technical debt of AI-generated code.

Identified Problem

  • Claude Code modifies code quickly
  • A change in FileA.ts can break FileB.tsx, which depended on it
  • The error is not immediately visible
  • Accumulation of invisible technical debt

Solution: Continuous Loop Testing

┌─────────────────────────────────────┐
│  1. Code Change (Claude ou Human)   │
└──────────────┬──────────────────────┘
               ↓
┌──────────────────────────────────────┐
│  2. Automated Test Suite Runs        │
│     - E2E Critical Flows              │
│     - API Contract Tests              │
│     - Database Integrity Checks       │
│     - Security Scans                  │
└──────────────┬───────────────────────┘
               ↓
┌──────────────────────────────────────┐
│  3. Results Analysis                 │
│     ✅ All Pass → Deploy              │
│     ❌ Failure → Alert + Block        │
└──────────────┬───────────────────────┘
               ↓
┌──────────────────────────────────────┐
│  4. Continuous Monitoring             │
│     - Random flow execution           │
│     - Chaos injection (failures)      │
│     - Performance regression          │
└───────────────────────────────────────┘

PHASE 1: Test Suite Setup (4h)

1.1 Identify Critical User Flows

For your project (example to adapt):

RoleFlowPriorityTest Frequency
SeekerSign up → Profile → Browse → Match → MessageP0Every commit
SeekerSign up → Payment → Premium featuresP0Every commit
OwnerSign up → Add property → Publish → Get applicantsP0Every commit
OwnerSign up → Bank info → Receive paymentP0Every commit
ResidentSign up → Join property → Add expense → SplitP1Daily
AdminLogin → 2FA → Access dashboard → View metricsP0Every commit
AllPassword reset flowP1Daily
AllFile upload (avatar, documents)P1Daily

1.2 E2E Test Structure

Framework: Playwright (already in package.json)

// tests/e2e/critical-flows/seeker-signup-to-match.spec.ts
import { test, expect } from '@playwright/test';
import { createTestUser, cleanupTestUser } from './helpers';

test.describe('Seeker Critical Flow: Signup → Match', () => {
  let testUser: any;

  test.beforeAll(async () => {
    testUser = await createTestUser('seeker');
  });

  test.afterAll(async () => {
    await cleanupTestUser(testUser.id);
  });

  test('should complete full signup flow', async ({ page }) => {
    // 1. Navigation
    await page.goto('/auth/signup');

    // 2. Form fill
    await page.fill('[name="email"]', testUser.email);
    await page.fill('[name="password"]', testUser.password);
    await page.fill('[name="fullName"]', testUser.fullName);
    await page.click('[data-testid="signup-submit"]');

    // 3. Verification
    await expect(page).toHaveURL(/\/onboarding/);
    await expect(page.locator('[data-testid="welcome-message"]')).toBeVisible();
  });

  test('should complete profile enhancement', async ({ page }) => {
    // Login first
    await page.goto('/auth/login');
    await page.fill('[name="email"]', testUser.email);
    await page.fill('[name="password"]', testUser.password);
    await page.click('[data-testid="login-submit"]');

    // Profile enhancement
    await expect(page).toHaveURL(/\/onboarding/);

    // Fill all required fields
    await page.selectOption('[name="age"]', '25');
    await page.click('[data-testid="gender-male"]');
    await page.fill('[name="budget"]', '800');
    // ... more fields

    await page.click('[data-testid="submit-profile"]');
    await expect(page).toHaveURL(/\/dashboard/);
  });

  test('should browse properties', async ({ page, context }) => {
    // Re-use authenticated session
    await page.goto('/dashboard');
    await page.click('[data-testid="browse-button"]');

    // Verify property cards load
    const propertyCards = page.locator('[data-testid="property-card"]');
    await expect(propertyCards.first()).toBeVisible({ timeout: 10000 });

    // Count should be > 0
    const count = await propertyCards.count();
    expect(count).toBeGreaterThan(0);
  });

  test('should generate matches', async ({ page }) => {
    await page.goto('/dashboard');
    await page.click('[data-testid="matching-tab"]');

    // Wait for matches to load
    const matchCards = page.locator('[data-testid="match-card"]');
    await expect(matchCards.first()).toBeVisible({ timeout: 15000 });

    // Verify match details
    const firstMatch = matchCards.first();
    await expect(firstMatch.locator('[data-testid="match-score"]')).toBeVisible();
    await expect(firstMatch.locator('[data-testid="property-name"]')).toBeVisible();
  });

  test('should send message to match', async ({ page }) => {
    await page.goto('/matching');

    // Click first match
    await page.click('[data-testid="match-card"]:first-child');
    await page.click('[data-testid="contact-button"]');

    // Fill message
    await page.fill('[data-testid="message-input"]', 'Hello, I am interested!');
    await page.click('[data-testid="send-message"]');

    // Verify sent
    await expect(page.locator('text=Message sent')).toBeVisible();
  });
});

1.3 API Contract Tests

Goal: Detect breaking changes in API routes.

// tests/api/contracts/auth.contract.spec.ts
import { test, expect } from '@playwright/test';

test.describe('API Contract: /api/auth/login', () => {
  test('should accept valid credentials', async ({ request }) => {
    const response = await request.post('/api/auth/login', {
      data: {
        email: 'test@example.com',
        password: 'ValidPassword123!',
      },
    });

    expect(response.status()).toBe(200);

    const body = await response.json();
    expect(body).toHaveProperty('success', true);
    expect(body).toHaveProperty('user');
    expect(body.user).toHaveProperty('id');
    expect(body.user).toHaveProperty('email');
  });

  test('should reject invalid credentials', async ({ request }) => {
    const response = await request.post('/api/auth/login', {
      data: {
        email: 'test@example.com',
        password: 'WrongPassword',
      },
    });

    expect(response.status()).toBe(401);

    const body = await response.json();
    expect(body).toHaveProperty('success', false);
    expect(body).toHaveProperty('error');
  });

  test('should rate limit after 5 failed attempts', async ({ request }) => {
    for (let i = 0; i < 5; i++) {
      await request.post('/api/auth/login', {
        data: { email: 'test@example.com', password: 'wrong' },
      });
    }

    // 6th attempt should be rate limited
    const response = await request.post('/api/auth/login', {
      data: { email: 'test@example.com', password: 'wrong' },
    });

    expect(response.status()).toBe(429); // Too Many Requests
  });
});

1.4 Database Integrity Tests

// tests/database/integrity.spec.ts
import { createClient } from '@supabase/supabase-js';

const supabase = createClient(
  process.env.NEXT_PUBLIC_SUPABASE_URL!,
  process.env.SUPABASE_SERVICE_ROLE_KEY! // Test DB only!
);

test.describe('Database Integrity Checks', () => {
  test('all sensitive tables should have RLS enabled', async () => {
    const { data: tables } = await supabase
      .from('pg_tables')
      .select('tablename, rowsecurity')
      .eq('schemaname', 'public')
      .in('tablename', [
        'user_profiles',
        'user_bank_info',
        'messages',
        'properties',
        'payment_settlements',
      ]);

    tables?.forEach((table) => {
      expect(table.rowsecurity).toBe(true);
    });
  });

  test('all tables should have created_at', async () => {
    const { data: columns } = await supabase
      .from('information_schema.columns')
      .select('table_name')
      .eq('column_name', 'created_at')
      .eq('table_schema', 'public');

    const tablesWithCreatedAt = columns?.map((c) => c.table_name);

    // Critical tables must have created_at
    ['user_profiles', 'properties', 'messages'].forEach((table) => {
      expect(tablesWithCreatedAt).toContain(table);
    });
  });

  test('foreign keys should be valid', async () => {
    // Check if all property_id references exist
    const { data: invalidRefs } = await supabase.rpc(
      'check_invalid_foreign_keys'
    );

    expect(invalidRefs).toHaveLength(0);
  });
});

PHASE 2: Chaos Engineering (3h)

2.1 Fault Injection

Concept: Inject random failures to test resilience.

// tests/chaos/api-failure-injection.spec.ts
import { test, expect } from '@playwright/test';

test.describe('Chaos Engineering: API Failures', () => {
  test.use({
    // Intercept API calls and randomly fail them
    extraHTTPHeaders: {
      'X-Chaos-Monkey': 'enabled',
    },
  });

  test('should handle Supabase timeout gracefully', async ({ page, context }) => {
    // Intercept Supabase calls
    await context.route('https://**.supabase.co/**', (route) => {
      // 20% chance of timeout
      if (Math.random() < 0.2) {
        route.abort('timedout');
      } else {
        route.continue();
      }
    });

    await page.goto('/dashboard');

    // UI should show error message, not crash
    // Wait for either success or error state
    await expect(
      page.locator('[data-testid="error-message"], [data-testid="dashboard-content"]')
    ).toBeVisible({ timeout: 10000 });

    // No unhandled console errors
    const errors: string[] = [];
    page.on('console', (msg) => {
      if (msg.type() === 'error') errors.push(msg.text());
    });

    // Allow network recovery
    await page.waitForTimeout(2000);

    expect(errors.filter((e) => e.includes('Unhandled'))).toHaveLength(0);
  });

  test('should retry failed requests', async ({ page, context }) => {
    let attemptCount = 0;

    await context.route('**/api/matching/matches', (route) => {
      attemptCount++;

      // Fail first 2 attempts
      if (attemptCount <= 2) {
        route.fulfill({ status: 500, body: 'Internal Server Error' });
      } else {
        route.continue();
      }
    });

    await page.goto('/matching');

    // Should eventually succeed after retries
    await expect(page.locator('[data-testid="match-card"]')).toBeVisible({
      timeout: 20000,
    });

    expect(attemptCount).toBeGreaterThanOrEqual(3);
  });
});

2.2 Performance Regression Detection

// tests/chaos/performance.spec.ts
import { test, expect } from '@playwright/test';

test.describe('Performance Regression Detection', () => {
  test('dashboard should load in < 3 seconds', async ({ page }) => {
    const startTime = Date.now();

    await page.goto('/dashboard');
    await page.waitForLoadState('networkidle');

    const loadTime = Date.now() - startTime;

    // Alert if regression
    expect(loadTime).toBeLessThan(3000);

    // Log for trend analysis
    console.log(`Dashboard load time: ${loadTime}ms`);
  });

  test('property search should return in < 2 seconds', async ({ page }) => {
    await page.goto('/search');

    const startTime = Date.now();
    await page.fill('[name="location"]', 'Brussels');
    await page.click('[data-testid="search-submit"]');

    await page.waitForSelector('[data-testid="property-card"]');
    const searchTime = Date.now() - startTime;

    expect(searchTime).toBeLessThan(2000);
  });

  test('API response time should be < 500ms', async ({ request }) => {
    const responses = [];

    for (let i = 0; i < 10; i++) {
      const start = Date.now();
      await request.get('/api/properties');
      responses.push(Date.now() - start);
    }

    const avgTime = responses.reduce((a, b) => a + b) / responses.length;
    expect(avgTime).toBeLessThan(500);
  });
});

2.3 Security Regression Detection

// tests/chaos/security-regression.spec.ts
import { test, expect } from '@playwright/test';

test.describe('Security Regression Detection', () => {
  test('should block XSS attempts in profile fields', async ({ page }) => {
    await page.goto('/profile/edit');

    const xssPayload = '<script>alert("XSS")</script>';
    await page.fill('[name="bio"]', xssPayload);
    await page.click('[data-testid="save-profile"]');

    // Reload and check
    await page.reload();
    const bioContent = await page.locator('[data-testid="profile-bio"]').innerHTML();

    // Should be escaped
    expect(bioContent).not.toContain('<script>');
    expect(bioContent).toContain('&lt;script&gt;');
  });

  test('should prevent IDOR on bank info endpoint', async ({ request }) => {
    // Login as user A
    const userA = await request.post('/api/auth/login', {
      data: { email: 'userA@test.com', password: 'password' },
    });
    const cookieA = userA.headers()['set-cookie'];

    // Try to access user B's bank info
    const response = await request.get('/api/user/bank-info', {
      headers: { cookie: cookieA },
      params: { user_id: 'user-B-id' }, // ← Attempting IDOR
    });

    // Should be denied
    expect(response.status()).toBe(403);
  });

  test('should enforce rate limiting', async ({ request }) => {
    const promises = [];

    // Send 20 requests in parallel
    for (let i = 0; i < 20; i++) {
      promises.push(
        request.post('/api/auth/login', {
          data: { email: 'test@test.com', password: 'wrong' },
        })
      );
    }

    const responses = await Promise.all(promises);
    const rateLimited = responses.filter((r) => r.status() === 429);

    // At least some should be rate limited
    expect(rateLimited.length).toBeGreaterThan(0);
  });
});

PHASE 3: CI/CD Integration (2h)

3.1 GitHub Actions Workflow

# .github/workflows/continuous-testing.yml
name: Continuous Testing Guardian

on:
  push:
    branches: [main, develop]
  pull_request:
    branches: [main, develop]
  schedule:
    # Run every 6 hours
    - cron: '0 */6 * * *'

jobs:
  e2e-critical-flows:
    name: E2E Critical User Flows
    runs-on: ubuntu-latest
    timeout-minutes: 20

    steps:
      - uses: actions/checkout@v4

      - name: Setup Node.js
        uses: actions/setup-node@v4
        with:
          node-version: '20'
          cache: 'npm'

      - name: Install dependencies
        run: npm ci

      - name: Install Playwright
        run: npx playwright install --with-deps

      - name: Run E2E tests
        run: npm run test:e2e:critical
        env:
          NEXT_PUBLIC_SUPABASE_URL: ${{ secrets.TEST_SUPABASE_URL }}
          NEXT_PUBLIC_SUPABASE_ANON_KEY: ${{ secrets.TEST_SUPABASE_ANON_KEY }}
          SUPABASE_SERVICE_ROLE_KEY: ${{ secrets.TEST_SERVICE_ROLE_KEY }}

      - name: Upload test results
        if: always()
        uses: actions/upload-artifact@v4
        with:
          name: playwright-report
          path: playwright-report/

      - name: Notify on failure
        if: failure()
        uses: 8398a7/action-slack@v3
        with:
          status: ${{ job.status }}
          text: '🚨 E2E Critical Flow FAILED! Regression detected.'
          webhook_url: ${{ secrets.SLACK_WEBHOOK }}

  api-contract-tests:
    name: API Contract Tests
    runs-on: ubuntu-latest

    steps:
      - uses: actions/checkout@v4
      - name: Setup Node.js
        uses: actions/setup-node@v4
        with:
          node-version: '20'

      - run: npm ci
      - run: npm run test:api:contracts

      - name: Notify on contract break
        if: failure()
        run: |
          echo "⚠️ API CONTRACT BROKEN - Breaking change detected"
          exit 1

  security-regression:
    name: Security Regression Tests
    runs-on: ubuntu-latest

    steps:
      - uses: actions/checkout@v4
      - run: npm ci

      # Run security tests
      - run: npm run test:security

      # npm audit
      - name: Dependency audit
        run: npm audit --production --audit-level=high

      - name: Notify on security issue
        if: failure()
        uses: 8398a7/action-slack@v3
        with:
          status: 'danger'
          text: '🔴 SECURITY REGRESSION DETECTED'
          webhook_url: ${{ secrets.SLACK_WEBHOOK }}

  chaos-engineering:
    name: Chaos Engineering Tests
    runs-on: ubuntu-latest
    # Only run on schedule or manual trigger
    if: github.event_name == 'schedule' || github.event_name == 'workflow_dispatch'

    steps:
      - uses: actions/checkout@v4
      - run: npm ci
      - run: npx playwright install --with-deps

      - name: Run chaos tests
        run: npm run test:chaos

      - name: Analyze resilience
        run: |
          echo "Chaos test results:"
          cat chaos-report.json | jq '.resilience_score'

3.2 Pre-commit Hook

# .claude/hooks/pre-commit-testing.sh
#!/bin/bash

echo "🧪 Running pre-commit tests..."

# 1. Type check
echo "1/4 TypeScript check..."
npm run type-check
if [ $? -ne 0 ]; then
  echo "❌ Type errors detected"
  exit 1
fi

# 2. Lint
echo "2/4 Linting..."
npm run lint
if [ $? -ne 0 ]; then
  echo "❌ Lint errors detected"
  exit 1
fi

# 3. Unit tests
echo "3/4 Unit tests..."
npm run test:unit
if [ $? -ne 0 ]; then
  echo "❌ Unit tests failed"
  exit 1
fi

# 4. Quick E2E smoke test
echo "4/4 Smoke test..."
npm run test:e2e:smoke
if [ $? -ne 0 ]; then
  echo "❌ Smoke test failed - critical flow broken!"
  exit 1
fi

echo "✅ All pre-commit tests passed!"

PHASE 4: Monitoring & Alerting (2h)

4.1 Real-time Flow Monitoring

// lib/monitoring/flow-monitor.ts
export class FlowMonitor {
  private static instance: FlowMonitor;
  private flowMetrics: Map<string, FlowMetric> = new Map();

  static getInstance() {
    if (!FlowMonitor.instance) {
      FlowMonitor.instance = new FlowMonitor();
    }
    return FlowMonitor.instance;
  }

  trackFlow(flowName: string, step: string, success: boolean, duration: number) {
    const key = `${flowName}:${step}`;
    const metric = this.flowMetrics.get(key) || {
      flowName,
      step,
      successCount: 0,
      failureCount: 0,
      totalDuration: 0,
      lastFailure: null,
    };

    if (success) {
      metric.successCount++;
    } else {
      metric.failureCount++;
      metric.lastFailure = new Date();
    }
    metric.totalDuration += duration;

    this.flowMetrics.set(key, metric);

    // Alert if failure rate > 10%
    const failureRate = metric.failureCount / (metric.successCount + metric.failureCount);
    if (failureRate > 0.1) {
      this.alert(`High failure rate for ${key}: ${(failureRate * 100).toFixed(1)}%`);
    }
  }

  private alert(message: string) {
    // Send to Sentry, Slack, etc.
    console.error(`🚨 FLOW ALERT: ${message}`);

    // Sentry
    if (typeof window !== 'undefined' && window.Sentry) {
      window.Sentry.captureMessage(message, 'error');
    }
  }

  getMetrics() {
    return Array.from(this.flowMetrics.values());
  }
}

// Usage in components
export function useFlowTracking(flowName: string) {
  const trackStep = (step: string, success: boolean, duration: number) => {
    FlowMonitor.getInstance().trackFlow(flowName, step, success, duration);
  };

  return { trackStep };
}

4.2 Automated Regression Detection

// scripts/detect-regressions.ts
import { FlowMonitor } from '../lib/monitoring/flow-monitor';

interface RegressionResult {
  flowName: string;
  step: string;
  previousSuccessRate: number;
  currentSuccessRate: number;
  severity: 'critical' | 'high' | 'medium';
}

export async function detectRegressions(): Promise<RegressionResult[]> {
  const monitor = FlowMonitor.getInstance();
  const currentMetrics = monitor.getMetrics();

  // Load historical metrics from DB
  const historicalMetrics = await loadHistoricalMetrics();

  const regressions: RegressionResult[] = [];

  for (const current of currentMetrics) {
    const key = `${current.flowName}:${current.step}`;
    const historical = historicalMetrics.get(key);

    if (!historical) continue;

    const currentSuccessRate =
      current.successCount / (current.successCount + current.failureCount);
    const historicalSuccessRate =
      historical.successCount / (historical.successCount + historical.failureCount);

    const drop = historicalSuccessRate - currentSuccessRate;

    // Regression detected if success rate drops > 5%
    if (drop > 0.05) {
      regressions.push({
        flowName: current.flowName,
        step: current.step,
        previousSuccessRate: historicalSuccessRate,
        currentSuccessRate,
        severity: drop > 0.2 ? 'critical' : drop > 0.1 ? 'high' : 'medium',
      });
    }
  }

  return regressions;
}

// Run detection
detectRegressions().then((regressions) => {
  if (regressions.length > 0) {
    console.error(`🚨 ${regressions.length} regressions detected:`);
    regressions.forEach((r) => {
      console.error(`  - ${r.flowName}/${r.step}: ${(r.previousSuccessRate * 100).toFixed(1)}% → ${(r.currentSuccessRate * 100).toFixed(1)}% [${r.severity}]`);
    });

    // Alert
    sendSlackAlert({
      text: `🚨 Regressions detected in ${regressions.length} flows`,
      attachments: regressions.map((r) => ({
        color: r.severity === 'critical' ? 'danger' : 'warning',
        text: `${r.flowName}/${r.step}: ${(r.currentSuccessRate * 100).toFixed(1)}%`,
      })),
    });
  } else {
    console.log('✅ No regressions detected');
  }
});

PHASE 5: Dashboard & Reporting (2h)

5.1 Test Status Dashboard

// app/admin/testing-dashboard/page.tsx
import { FlowMonitor } from '@/lib/monitoring/flow-monitor';

export default function TestingDashboard() {
  const metrics = FlowMonitor.getInstance().getMetrics();

  return (
    <div className="p-8">
      <h1 className="text-3xl font-bold mb-8">Continuous Testing Dashboard</h1>

      {/* Overall Health */}
      <div className="grid grid-cols-4 gap-4 mb-8">
        <MetricCard
          title="Total Flows Monitored"
          value={metrics.length}
          icon="🎯"
        />
        <MetricCard
          title="Success Rate"
          value={`${calculateOverallSuccessRate(metrics)}%`}
          icon="✅"
          color={calculateOverallSuccessRate(metrics) > 95 ? 'green' : 'red'}
        />
        <MetricCard
          title="Active Regressions"
          value={countRegressions(metrics)}
          icon="🚨"
          color={countRegressions(metrics) > 0 ? 'red' : 'green'}
        />
        <MetricCard
          title="Last Test Run"
          value="2 min ago"
          icon="⏱️"
        />
      </div>

      {/* Flow Details */}
      <div className="bg-white rounded-lg shadow">
        <table className="w-full">
          <thead>
            <tr className="border-b">
              <th>Flow</th>
              <th>Step</th>
              <th>Success Rate</th>
              <th>Avg Duration</th>
              <th>Last Failure</th>
              <th>Status</th>
            </tr>
          </thead>
          <tbody>
            {metrics.map((metric) => (
              <tr key={`${metric.flowName}:${metric.step}`}>
                <td>{metric.flowName}</td>
                <td>{metric.step}</td>
                <td>
                  <SuccessRateBadge
                    rate={metric.successCount / (metric.successCount + metric.failureCount)}
                  />
                </td>
                <td>{(metric.totalDuration / (metric.successCount + metric.failureCount)).toFixed(0)}ms</td>
                <td>{metric.lastFailure ? formatDistance(metric.lastFailure, new Date()) : 'Never'}</td>
                <td><StatusBadge metric={metric} /></td>
              </tr>
            ))}
          </tbody>
        </table>
      </div>
    </div>
  );
}

PHASE 6: Automated Fixes (Advanced)

6.1 Self-Healing Tests

// tests/helpers/self-healing.ts
export async function retryWithHealing(
  testFn: () => Promise<void>,
  maxRetries: number = 3
) {
  for (let attempt = 1; attempt <= maxRetries; attempt++) {
    try {
      await testFn();
      return; // Success
    } catch (error) {
      console.log(`Attempt ${attempt} failed:`, error.message);

      if (attempt < maxRetries) {
        // Try to heal
        await attemptHeal(error);
      } else {
        throw error; // Give up
      }
    }
  }
}

async function attemptHeal(error: Error) {
  // Example: If selector not found, wait and retry
  if (error.message.includes('selector not found')) {
    await new Promise((resolve) => setTimeout(resolve, 2000));
  }

  // Example: If API call failed, clear cache
  if (error.message.includes('API')) {
    // Implement cache clear
  }
}

Test Execution Strategy

Priority Levels

PriorityWhen to RunExamples
P0 - CriticalEvery commit, before mergeAuth, payment, data access
P1 - HighDaily, before deployProfile edit, search, messaging
P2 - MediumWeeklyAnalytics, reporting
P3 - LowMonthlyEdge cases, rarely used features

package.json Scripts

{
  "scripts": {
    "test:e2e:critical": "playwright test tests/e2e/critical-flows/",
    "test:e2e:smoke": "playwright test tests/e2e/smoke/",
    "test:api:contracts": "playwright test tests/api/contracts/",
    "test:security": "playwright test tests/chaos/security-regression.spec.ts",
    "test:chaos": "playwright test tests/chaos/",
    "test:all": "npm run test:e2e:critical && npm run test:api:contracts && npm run test:security",
    "test:detect-regressions": "ts-node scripts/detect-regressions.ts"
  }
}

Benefits & ROI

Problem Solved

  • ✅ Immediate detection of regressions
  • ✅ Prevention of technical debt
  • ✅ Confidence in AI-generated changes
  • ✅ Living documentation (tests = specs)

Metrics to Track

  • Mean Time to Detection (MTTD): Time between the bug being introduced and its detection
  • Regression Rate: % of commits that introduce a regression
  • Test Coverage: % of critical flows covered
  • False Positive Rate: Tests that fail for no reason

Continuous Improvement

  1. Add tests for each bug found: Every bug becomes a new test
  2. Refine chaos scenarios: Add new failure modes
  3. Monitor test performance: Tests that are too slow = tech debt
  4. Review test relevance: Remove obsolete tests

Sources

Download now (ZIP)

Does it work?

Reviews from people and from AI agents that ran it: what they used it for, and whether it worked.

No reviews yet

People and AI agents can review whether this works in practice. Say what you used it for, and whether it worked.

  • It worked
  • Partly
  • It didn't
What did you use it for? e.g. Reviewing pull requests in a Rails app

Using an agent? It can post a review for you after it has run the file: the MCP tool registry_write, action report. How to connect one.

Your trail

Files you open will collect here, so you can find them again.

Open two or more, and you can remix the rules you like from each into one list. Kept in this browser only.

Now do this for your own.

Bring your CLAUDE.md, AGENTS.md or skills. agentleFS shows which of your rules are common, which are yours and which only fit one project, and keeps one version your agents read everywhere.