agentleFS
kmarrrahulkmarrrahul/dataweave-ai-playground/.agents/skills/project-expert/SKILL.md

project-expert

Skill2 starsChanged 5 months agoMITView on GitHub

Deep technical knowledge base for the DataWeave AI Playground project, covering backend (FastAPI), frontend (Alpine/Tailwind/CodeMirror), APIs, and development considerations.

  • Sends data out

1 other project uses a version of this.

Adopted by

Most recently taken up by

  • Other versions1 project uses a version of this.

    Word for word · 0

    None found.

    Edited · 0

    None found.

    Wrote their own version · 1

    1. felixgeelhaar/skills★ 1same name

    How matches are found. Word for word: the exact same text. Edited: shares passages with this file (“99% same” means 99% of its five-word phrases also appear here; 10% or more counts). Their own version: similar meaning, or, weakest, the same name. We can’t tell which project had it first. Projects are counted among the 200 most-starred candidates, and mentions are links to the repository we found, up to 50 per site, so both counts are a floor.

  • In this repositoryOne other file it gives its agents.

The file

We split this skill into the rules it gives an agent

12 rules. About 0 are common, 4 are distinctive, and 8 are very specific to this project.

  • ≈0

    Common · estimate

    Many files like this have a rule of this kind.

  • ≈4

    Distinctive · estimate

    This author's own choices. Worth a look.

  • 8

    Very specific

    Tied to their project's names, files or tools. Probably skip.

  • Common
  • Distinctive
  • Very specific
  1. Very specific:

    Security: Ensure API keys and sensitive payloads are not logged in plain text.

    line 96
  2. Very specific:

    State Synchronization: Always establish two-way binding.

    line 195
  3. Very specific:

    Performance: Do not call .setValue() on every single character stream from the AI if it causes lag.

    line 196
  4. Distinctive:

    Correction Loop: If the output mismatches or throws a syntax error, the backend constructs a new hidden prompt containing the error logs …

    line 229
  5. Distinctive:

    Cost: Ensure strict circuit breakers are in place.

    line 235
  6. Distinctive:

    Security Context: When running DataWeave scripts generated by an AI, the backend invokes the script inside a restricted subprocess or a …

    line 246
  7. Distinctive:

    When a new AI SDK drops (e.g., a new local Ollama model):

    line 259
  8. Very specific:

    Backend Integration: Create app/providers/ollama_service.py.

    line 260
  9. Very specific:

    Router Registration: Update app/routers/agent.py to route requests with provider: 'ollama' to your new service.

    line 261
  10. Very specific:

    Frontend UI: Add the provider to the index.html navigation bar.

    line 262
  11. Very specific:

    Testing: Use the Bruno collection (bruno-collection/) to test the SSE stream independently of the UI.

    line 263
  12. Very specific:

    Read CONTRIBUTING.md before making large architectural changes.

    line 270

≈ “Common” is an estimate: at least 1 in 5 of the 364,980 skill files we read have a rule of the same kind, not the same words. “Very specific” is our guess from the words: it names a file, a command, a setting or the project itself.

Keep what fits and drop the rest. Your version, ready for your agents.
Files
SKILL.md

Project Expert: DataWeave AI Playground

This skill provides comprehensive technical context on the DataWeave AI Playground project. It is designed to act as a deep knowledge base covering the full stack architecture, frontend integrations, backend APIs, and important considerations for development and debugging.

1. Project Overview & Architecture

DataWeave AI Playground is a self-hosted, vendor-agnostic AI Playground template designed to autonomously generate, explain, and debug DataWeave 2.0 transformation scripts.

Core Architecture

  • Backend: Python 3 + FastAPI framework, served via Uvicorn.
  • Frontend: HTML/Jinja2 templates, styled with Tailwind CSS, reactive state managed by Alpine.js.
  • Editors: CodeMirror 5 is used for the multi-pane interface (Tabbed Inputs, Script, Output).
  • AI Integration: Seamless integration with multiple LLM SDKs (GitHub Copilot, Google Antigravity, Anthropic Claude, OpenAI Codex).
  • Communication: Uses Server-Sent Events (SSE) for streaming AI reasoning and generation processes to the client.
  • Containerization: The entire application is containerized using Docker, allowing ephemeral executions and isolated environments.

The Value Proposition

Unlike standard chat interfaces (like ChatGPT or Claude Web), this playground is specifically tailored for DataWeave transformation development. It exposes the underlying "thinking steps" of the agent, providing transparency into how the AI interprets the source schema, applies mapping rules, and verifies the syntax.


2. Backend Stack (FastAPI)

The backend is built around FastAPI and follows a modular structure within the app/ directory.

Directory Structure Detailed

  • app/main.py: The entry point for the FastAPI application. Sets up CORS, static file serving, template rendering, and includes all routers.
  • app/routers/:
    • agent.py: Handles the core AI streaming endpoints (/api/run, /api/followup).
    • dataweave.py: Manages the local or remote execution of DataWeave scripts without AI involvement.
    • auth.py: Manages session keys and provider authentication flows.
    • core.py: Core utility endpoints like /api/health and /api/version.
    • skills.py: Exposes endpoints for managing agent skills.
  • app/providers/: Houses the integration logic for different AI providers:
    • antigravity_service.py
    • claude_service.py
    • codex_service.py
    • copilot_service.py
    • cursor_service.py
  • app/schemas/: Pydantic models for request/response validation. Ensures strong typing and automated OpenAPI documentation.
  • app/core/:
    • prompts.py: Contains the system prompts and few-shot examples used across different LLMs.
    • mcp_config.py: Model Context Protocol configuration.
  • app/skills/: Custom agent skills, such as local syntax verification.

Key API Endpoints & Payloads

POST /api/run

Starts an agentic DataWeave generation task. Request Payload Example:

{
  "provider": "antigravity",
  "model": "gemini-1.5-pro",
  "input_payload": "{\"user\": {\"name\": \"John Doe\", \"age\": 30}}",
  "expected_output": "{\"fullName\": \"John Doe\"}",
  "prompt": "Map the user data to a simplified profile object.",
  "auto_recover": true,
  "__auth_token": "your_encrypted_or_plain_token_here"
}

Note: The frontend encrypts this entire JSON payload (and the __auth_token) via a bitwise XOR using APP_SESSION_KEY before sending it to the server if encryptNetworkPayload is enabled. Response: text/event-stream (SSE).

POST /api/dataweave/transform

Executes a DataWeave script against an input payload. Request Payload Example:

{
  "script": "%dw 2.0\noutput application/json\n---\n{ fullName: payload.user.name }",
  "input_data": "{\"user\": {\"name\": \"John Doe\", \"age\": 30}}"
}

Response Example:

{
  "success": true,
  "output": "{\n  \"fullName\": \"John Doe\"\n}",
  "execution_time_ms": 45
}

Backend Considerations

  • Streaming Responses: The backend heavily relies on SSE for real-time AI feedback. When adding new providers, ensure they yield chunks correctly. For instance, using Python's asynchronous generators:
    async def generate_stream():
        for chunk in llm_response_stream:
            yield f"data: {json.dumps({'content': chunk})}\n\n"
  • Provider Agnosticism: The app uses a BYOK (Bring Your Own Key) model. The core logic should not contain provider-specific code; all such logic belongs in app/providers/.
  • Security: Ensure API keys and sensitive payloads are not logged in plain text. Use environment variables and the protection.json configuration for sanitization.

3. Frontend Stack (Alpine.js + Tailwind CSS)

The frontend is a single-page application (SPA) architecture built without heavy virtual DOM frameworks like React or Vue, opting instead for Alpine.js for lightweight, attribute-based reactivity.

Alpine.js State Management Deep Dive

The global state is managed in app.js which composes multiple mixins. This pattern keeps the codebase maintainable.

Mixin Example (static/js/mixins/dataweave.js):

window.dataweaveMixin = function() {
  return {
    inputPayload: '{}',
    dwScript: '%dw 2.0\noutput application/json\n---\n{}',
    outputPayload: '{}',
    isExecuting: false,
    
    initDataWeave() {
      // Initialize CodeMirror and bind to Alpine state
      this.inputEditor = CodeMirror.fromTextArea(document.getElementById('input-area'), {
        mode: "javascript", theme: "material", lineNumbers: true
      });
      this.inputEditor.on('change', (cm) => {
        this.inputPayload = cm.getValue();
      });
    },
    
    async runTransform() {
      this.isExecuting = true;
      try {
        const response = await fetch('/api/dataweave/transform', {
          method: 'POST',
          headers: { 'Content-Type': 'application/json' },
          body: JSON.stringify({ script: this.dwScript, input_data: this.inputPayload })
        });
        const result = await response.json();
        this.outputPayload = result.output;
        this.outputEditor.setValue(result.output);
      } finally {
        this.isExecuting = false;
      }
    }
  }
}

Main Store Aggregation (app.js):

window.appStore = function () {
  return {
    ...authMixin(),
    ...skillsMixin(),
    ...dataweaveMixin(),
    
    init() {
      console.log('App initialized.');
      this.initAuth();
      this.$nextTick(() => { this.initDataWeave(); });
    }
  };
};

This state is attached to the DOM using x-data="appStore()" on the <body> element.

Styling (Tailwind CSS)

  • Dark Mode Context: The application defaults to dark mode. Always use dark variant classes (e.g., dark:bg-slate-800).
  • Custom Theme Variables: Custom colors (brand, surface, slate) are defined in index.html and mapped to native CSS variables.
    :root {
      --color-surface-900: 15 22 41;
    }
  • Responsive Layout: The main layout is designed to be fully responsive using flexbox. The 3 panes collapse into a single column on mobile devices and expand to three columns on desktops.

4. CodeMirror 5 Integration

The application uses CodeMirror 5 to power the multi-pane intelligent editor.

Pane Configurations

  • Pane 1: Tabbed Inputs
    • Purpose: Source data schema payloads, configurable with multiple tabs for arbitrary DataWeave identifiers (payload, vars, attributes).
    • Mode: Dynamically sets to javascript/json, xml, or text/plain based on the selected format (JSON, XML, CSV, TXT).
    • Behavior: Editable. Triggers auto-save or updates Alpine state on change.
  • Pane 2: DataWeave Script
    • Purpose: The transformation logic.
    • Mode: Usually plain text or a custom basic DataWeave highlighter.
    • Behavior: Receives streamed updates from the AI. Requires careful handling of cursor position to avoid jumping during streaming.
  • Pane 3: Output / Expected
    • Purpose: The resulting data or the target shape.
    • Mode: Dynamic based on format selection (JSON, XML, CSV, TXT).
    • Behavior: Can be toggled between "Read-only Output" and "Editable Expected Output".

Integration Nuances

  1. Lifecycle Hooks: CodeMirror instances must be created in Alpine's $nextTick to guarantee the underlying <textarea> elements are rendered.
  2. State Synchronization: Always establish two-way binding. When Alpine state changes (e.g., from an SSE stream), update CodeMirror using .setValue(). When the user types, capture the change event to update Alpine.
  3. Performance: Do not call .setValue() on every single character stream from the AI if it causes lag. Buffer chunks and apply updates to the CodeMirror instance in larger batches, or append lines instead of replacing the entire content.
  4. Resizing: CodeMirror does not always adapt gracefully to CSS flexbox changes. If a pane is hidden and then shown via x-show, invoke editor.refresh() to recalculate line heights and scrollbars.

5. Server-Sent Events (SSE) & Markdown Rendering

Real-time feedback is a cornerstone of the application's UX.

SSE Handling (streaming.js)

The application uses the Fetch API and ReadableStream to process SSE, rather than the older EventSource API, because fetch allows sending POST requests with JSON bodies.

Stream Parsing Logic:

  1. A POST request is made to /api/run.
  2. The response body is read chunk-by-chunk using response.body.getReader().
  3. A TextDecoder decodes the Uint8Array chunks into strings.
  4. The strings are split by double newlines (\n\n) to parse individual SSE messages.
  5. The data: prefix is stripped, and the payload is JSON-parsed.

Markdown Rendering

  • Marked.js: AI reasoning is often returned in Markdown format. marked.js converts this to HTML dynamically.
  • Note: Mermaid diagram support has been explicitly removed from the project to optimize token usage and simplify the UI. Do not attempt to re-introduce mermaid diagrams unless requested.

6. Flagship Feature: Autonomous Auto-Recover

The "Auto-Recover" mechanism sets this playground apart from standard text-based LLM interfaces.

How it Works

  1. Initial Generation: The AI generates a DataWeave script based on the Input Payload and Expected Output.
  2. Local Execution: The backend autonomously executes this script against the Input Payload.
  3. Validation: The actual output is compared against the Expected Output.
  4. Correction Loop: If the output mismatches or throws a syntax error, the backend constructs a new hidden prompt containing the error logs and asks the AI to fix the script.
  5. Termination: This loop runs for a maximum of 3 iterations.

Development Considerations for Auto-Recover

  • State Management: The frontend must accurately display the current state of this loop (e.g., "Validating...", "Syntax Error Detected, Retrying (1/3)").
  • Latency: Because this loop involves multiple sequential LLM calls, latency can be high. The UI must aggressively stream reasoning steps so the user doesn't assume the application has frozen.
  • Cost: Ensure strict circuit breakers are in place. An infinite loop here would drain provider API credits rapidly.

7. Deployment & Containerization

DataWeave AI Playground is designed for both local development and production-grade Kubernetes deployments.

Docker Strategy

  • Dockerfile: Uses a multi-stage build to keep the final image lightweight.
  • Dockerfile.dwcli: A specialized image intended solely for executing DataWeave CLI commands securely in a sandbox.
  • Security Context: When running DataWeave scripts generated by an AI, the backend invokes the script inside a restricted subprocess or a dedicated sandboxed container to prevent remote code execution (RCE) vulnerabilities.

Kubernetes (k8s/)

The provided manifests configure the application for scalability:

  • deployment.yaml: Defines replicas, readiness/liveness probes, and resource constraints (CPU/Memory).
  • service.yaml: Exposes the pods internally.
  • ingress.yaml: Configures routing from the outside world (often referencing an Nginx controller or an AWS ALB).

8. Development Workflows & Testing

Adding a New Integration Provider

When a new AI SDK drops (e.g., a new local Ollama model):

  1. Backend Integration: Create app/providers/ollama_service.py. Implement the standard interface methods (stream_generation, get_models).
  2. Router Registration: Update app/routers/agent.py to route requests with provider: 'ollama' to your new service.
  3. Frontend UI: Add the provider to the index.html navigation bar. Provide a custom SVG icon. Update auth.js to manage the endpoint URL and optional API key.
  4. Testing: Use the Bruno collection (bruno-collection/) to test the SSE stream independently of the UI.

Testing Strategy

  • Unit Tests: Located in tests/. Currently focuses on testing the provider abstractions and payload parsing.
  • Integration Tests: Covered in INTEGRATION_TESTS.md. Run these to ensure that the 3-pane editor and backend streams synchronize correctly.

Contributing

Read CONTRIBUTING.md before making large architectural changes. Ensure black and flake8 are run to maintain Python code style, and check Alpine.js components for memory leaks (especially around CodeMirror instances not being destroyed on DOM removal).

Download now (ZIP)

Does it work?

Reviews from people and from AI agents that ran it: what they used it for, and whether it worked.

No reviews yet

People and AI agents can review whether this works in practice. Say what you used it for, and whether it worked.

  • It worked
  • Partly
  • It didn't
What did you use it for? e.g. Reviewing pull requests in a Rails app

Using an agent? It can post a review for you after it has run the file: the MCP tool registry_write, action report. How to connect one.

Your trail

Files you open will collect here, so you can find them again.

Open two or more, and you can remix the rules you like from each into one list. Kept in this browser only.

Now do this for your own.

Bring your CLAUDE.md, AGENTS.md or skills. agentleFS shows which of your rules are common, which are yours and which only fit one project, and keeps one version your agents read everywhere.