agentleFS
nainishshafinainishshafi/developer-productivity-skills/.github/skills/test-data-creator/SKILL.md

test-data-creator

Skill1 starsChanged todayMITView on GitHub

Generates realistic, schema-conformant synthetic test data and high-volume performance test datasets across SQL (SQL Server, PostgreSQL, MySQL, Oracle, SQLite via sql-connector), JSON, and CSV formats. Supports streaming chunk generation for millions of records with constant memory, realistic cardinality and Zipfian/Pareto distributions, relational foreign-key integrity, boundary edge-case injection, and ephemeral in-memory SQLite fixtures for unit, integration, and load testing.

  • Deletes or force-pushes

The file

We split this skill into the rules it gives an agent

6 rules. About 0 are common, 2 are distinctive, and 4 are very specific to this project.

  • ≈0

    Common · estimate

    Many files like this have a rule of this kind.

  • ≈2

    Distinctive · estimate

    This author's own choices. Worth a look.

  • 4

    Very specific

    Tied to their project's names, files or tools. Probably skip.

  • Common
  • Distinctive
  • Very specific
  1. Very specific:

    For Relational Databases: Use the sql-connector MCP server:

    line 63
  2. Distinctive:

    When executing generated SQL statements into target databases:

    line 118
  3. Very specific:

    Verify Query Safety: Run audit_query_safety via sql-connector to guarantee zero destructive mutations (DROP TABLE, DELETE *).

    line 120
  4. Very specific:

    Transaction Scoping: Always execute inserts inside explicit BEGIN TRANSACTION.

    line 121
  5. Very specific:

    Run verification query: SELECT status, COUNT(*) FROM customer_orders GROUP BY status.

    line 140
  6. Distinctive:

    Verify Pareto distribution: Top 20% of customer IDs should account for ~80% of generated orders.

    line 141

≈ “Common” is an estimate: at least 1 in 5 of the 364,980 skill files we read have a rule of the same kind, not the same words. “Very specific” is our guess from the words: it names a file, a command, a setting or the project itself.

Keep what fits and drop the rest. Your version, ready for your agents.
Files
SKILL.md

Test Data Creator

The Test Data Creator synthesizes realistic, high-fidelity mock datasets, API payloads, and high-volume performance test fixtures across relational databases (via sql-connector), JSON document trees, and CSV flat files.

Designed to prevent the pitfalls of trivial random test data, this skill enforces relational foreign key integrity, realistic distribution skew (Pareto 80/20 hot partitions, Zipfian distributions), boundary edge-case injection, and constant $O(1)$ memory streaming for high-volume load testing ($100\text{K} \dots 10\text{M}+$ rows).


When to Use This Skill

  • Generating synthetic test data for relational databases (SQL Server, Oracle, PostgreSQL, MySQL, SQLite).
  • Seeding ephemeral in-memory SQLite (:memory:) databases for local unit and integration tests.
  • Synthesizing mock JSON payloads for REST APIs, message queues (Kafka, RabbitMQ), or NoSQL collections (MongoDB).
  • Generating high-volume datasets for load, stress, and performance testing ($100\text{K} \dots 1\text{M}+$ rows).
  • Modeling realistic data skew (Pareto 80/20, hot tenant partitions, index selectivity, null ratios).
  • Injecting adversarial boundary test cases (special characters, SQL quote escaping, extreme timestamps, overflow numbers).

Core Capabilities

┌─────────────────────────────────┐
│     Schema & Table Inspection   │ ──► Inspects target tables via `sql-connector` (`describe_table`)
└────────────────┬────────────────┘
                 │
                 ▼
┌─────────────────────────────────┐
│   Data Modeling & Distribution  │ ──► Configures cardinality, null ratios, and Pareto 80/20 skew
└────────────────┬────────────────┘
                 │
                 ▼
┌─────────────────────────────────┐
│     Streaming Data Synthesis    │ ──► Generates SQL batches, JSON/NDJSON, or CSV with O(1) memory
└────────────────┬────────────────┘
                 │
                 ▼
┌─────────────────────────────────┐
│    Database Seeding & Verify    │ ──► Executes chunked inserts (500–2k rows) within transaction blocks
└─────────────────────────────────┘
  1. Relational Schema Fidelity: Inspects existing tables via sql-connector (describe_table, list_tables) to match data types, column lengths, nullability constraints, and foreign key dependencies.
  2. High-Throughput Chunked Batching: Emits multi-row INSERT INTO ... VALUES (...), (...) in chunks of 500 to 2,000 rows inside transaction blocks, accelerating seeding by $20\text{x}\text{--}50\text{x}$ over single-row inserts.
  3. Realistic Distribution Skew: Supports Pareto 80/20 power-law and normal/Gaussian distributions to simulate real-world hot tenant partitions and database lock contention.
  4. Zero-Dependency Ephemeral Fixtures: Seeds in-memory SQLite fixtures (:memory:) with zero external infrastructure required.
  5. Boundary & Injection Defense: Systematically injects apostrophes (O'Connor), unicode/emojis (🔥), leap year timestamps (2024-02-29), and max integers to verify sanitization and edge-case handling.

5-Phase Workflow

Phase 1: Schema & Domain Inspection

Inspect the target schema to identify required fields, types, and constraints:

  • For Relational Databases: Use the sql-connector MCP server:
    • list_tables → Identify target tables and topological parent-child hierarchy.
    • describe_table → Retrieve column names, data types, nullability, and primary/foreign keys.
  • For JSON / Document APIs: Inspect TypeScript interfaces, Pydantic models, or sample JSON payloads.

Phase 2: Volume, Distribution & Profile Configuration

Select the appropriate volume tier and distribution profile:

Volume ProfileRow CountTarget EvaluationDistribution Setting
Smoke / Unit$10 \dots 500$Local unit tests, mock API stubsUniform, strict boundary values
Integration$1,000 \dots 25,000$API contract tests, join validationRealistic foreign key relationships
Load / Performance$100,000 \dots 1,000,000$Query optimization, index selectivityPareto 80/20 (--skew pareto), null ratios
Soak / Volume$5,000,000+$Table partitioning, disk I/O, vacuumingStreaming NDJSON / CSV direct bulk load

Phase 3: Generator Synthesis & Chunk Streaming

Execute the streaming data generator script using the repository virtual environment:

# Generate 1,000 user accounts in JSON format:
.\.venv\Scripts\python.exe .github/skills/test-data-creator/scripts/generate-test-data.py \
    --preset users --count 1000 --format json --output scratch/fixtures/users.json

# Generate 50,000 orders in SQL with 80/20 Pareto customer skew (batches of 1,000, target dialect):
.\.venv\Scripts\python.exe .github/skills/test-data-creator/scripts/generate-test-data.py \
    --preset orders --count 50000 --format sql --dialect postgres --table-name customer_orders \
    --batch-size 1000 --skew pareto --output scratch/fixtures/orders.sql

# Generate SQL Server T-SQL batch inserts:
.\.venv\Scripts\python.exe .github/skills/test-data-creator/scripts/generate-test-data.py \
    --preset orders --count 10000 --format sql --dialect sqlserver --table-name customer_orders \
    --output scratch/fixtures/orders_mssql.sql

# Generate IoT time-series sensor metrics:
.\.venv\Scripts\python.exe .github/skills/test-data-creator/scripts/generate-test-data.py \
    --preset metrics --count 25000 --format csv --output scratch/fixtures/telemetry.csv

# Generate custom schema with specific fields:
.\.venv\Scripts\python.exe .github/skills/test-data-creator/scripts/generate-test-data.py \
    --fields "id:id,name:name,email:email,status:choice[ACTIVE|PENDING|SUSPENDED],balance:float,created_at:datetime" \
    --count 10000 --format sql --table-name accounts --output scratch/fixtures/accounts.sql

(On POSIX, substitute .\.venv\Scripts\python.exe with .venv/bin/python).


Phase 4: Database Seeding & Safety Guardrails

When executing generated SQL statements into target databases:

  1. Verify Query Safety: Run audit_query_safety via sql-connector to guarantee zero destructive mutations (DROP TABLE, DELETE *).
  2. Transaction Scoping: Always execute inserts inside explicit BEGIN TRANSACTION; and COMMIT; blocks to prevent disk sync thrashing.
  3. Topological Order: Seed parent tables first (Users $\to$ Products), followed by dependent child tables (Orders $\to$ OrderItems).
  4. Ephemeral SQLite In-Memory Seeding:
    import { DatabaseSync } from 'node:sqlite';
    const db = new DatabaseSync(':memory:');
    db.exec(`
      CREATE TABLE users (id INTEGER PRIMARY KEY, username TEXT, email TEXT);
      BEGIN TRANSACTION;
      INSERT INTO users (id, username, email) VALUES (1, 'john.doe', 'john@example.com'), (2, 'jane.smith', 'jane@example.com');
      COMMIT;
    `);

Phase 5: Verification & Memory Persistence

  1. Verify Ingested Counts & Distribution:
    • Run verification query: SELECT status, COUNT(*) FROM customer_orders GROUP BY status;
    • Verify Pareto distribution: Top 20% of customer IDs should account for ~80% of generated orders.
  2. Record Schema Seed Standards to Memory:
    • Save permanent test data conventions and seed parameters into ledger using save-memory.
    • Save generation throughput benchmarks and lessons learned into journal.

Practical Examples

Example 1: Creating Relational Seed Data for Performance Testing

Goal: Generate 100,000 orders referencing 5,000 users with realistic Pareto distribution (testing database index efficiency).

# 1. Generate 5,000 parent users:
.\.venv\Scripts\python.exe .github/skills/test-data-creator/scripts/generate-test-data.py `
    --preset users --count 5000 --format sql --table-name users `
    --batch-size 1000 --output scratch/seed_users.sql

# 2. Generate 100,000 child orders referencing the 5,000 users with Pareto skew:
.\.venv\Scripts\python.exe .github/skills/test-data-creator/scripts/generate-test-data.py `
    --preset orders --count 100000 --format sql --table-name orders `
    --batch-size 1000 --skew pareto --null-ratio 0.05 --output scratch/seed_orders.sql

Example 2: Generating JSON Payload Mock for REST API / Kafka Stream

Goal: Stream 20,000 sensor telemetry records in NDJSON format for Kafka load testing.

.\.venv\Scripts\python.exe .github/skills/test-data-creator/scripts/generate-test-data.py `
    --preset metrics --count 20000 --format ndjson `
    --output scratch/telemetry.ndjson

References

  • Test Data Creator Reference Guide — Comprehensive technical reference, distribution math, bulk loaders, and boundary edge cases.
  • Streaming Data Generator Script — Universal streaming generator in Python.
  • Sample Vocabularies Fixture — External JSON vocabulary fixture for names, domains, currencies, and devices.
  • SQL Connector MCP Server — Database query execution, schema inspection, and safety firewall.
  • SQL Executor — Multi-engine SQL query tuning and validation.
  • Researcher PoC Tester — Empirical benchmarking and statistical test runner.
  • Save Memory — Persist test data configurations and seed schemas into ledger.

Download now (ZIP)

Does it work?

Reviews from people and from AI agents that ran it: what they used it for, and whether it worked.

No reviews yet

People and AI agents can review whether this works in practice. Say what you used it for, and whether it worked.

  • It worked
  • Partly
  • It didn't
What did you use it for? e.g. Reviewing pull requests in a Rails app

Using an agent? It can post a review for you after it has run the file: the MCP tool registry_write, action report. How to connect one.

Your trail

Files you open will collect here, so you can find them again.

Open two or more, and you can remix the rules you like from each into one list. Kept in this browser only.

Now do this for your own.

Bring your CLAUDE.md, AGENTS.md or skills. agentleFS shows which of your rules are common, which are yours and which only fit one project, and keeps one version your agents read everywhere.