transform

Intent

Transform data from one format or schema to another while tracking what information is preserved, modified, or lost during conversion. Ensure output conforms to target schema with full provenance.

Success criteria:

• Output conforms to target schema
• All fields mapped correctly or explicitly marked as lost
• Transformation is deterministic and reproducible
• Loss/distortion explicitly documented
• Evidence anchors trace source to output

Compatible schemas:

• schemas/output_schema.yaml

Inputs

Inputs not listed here are outside the capability's contract: a workflow step that binds them gets W604 (STANDARD v1.1 §5.5).

Procedure

1. Parse source data: Load and validate input
   • Identify source format (JSON, YAML, XML, CSV, etc.)
   • Parse into internal representation
   • Detect encoding and special characters
   • Record source structure for provenance
2. Analyze target schema: Understand destination requirements
   • Load target schema specification (options.schema, if given)
   • Identify required vs optional fields
   • Note type constraints and validations
   • Map nested structures
3. Build transformation map: Match source to target
   • Apply explicit mappings if provided (options.mapping)
   • Infer mappings for matching field names
   • Identify fields requiring type conversion
   • Flag unmappable source fields
4. Execute transformation: Apply mappings
   • Transform each field according to mapping
   • Apply type conversions (string to number, etc.)
   • Handle nested objects recursively
   • Apply default values for missing required fields
5. Track losses: Document what was not preserved
   • List source fields not in output
   • Note precision losses in numeric conversions
   • Record truncations or format changes
   • Document semantic changes
6. Validate output: Confirm target schema conformance
   • Check all required fields present
   • Validate types match schema
   • Run any schema-defined constraints
   • Verify structural integrity
7. Ground output: Attach provenance
   • Link output fields to source locations
   • Document transformation rules applied
   • Record validation results

Output Contract

Return a structured object:

output: object|array  # Transformed data, a mapping or a list of records
transformation:
  input_type: string  # Format of source data
  output_type: string  # Format of output data
  mapping_applied: string  # Reference to mapping specification
validation:
  input_valid: boolean  # Was source valid?
  output_valid: boolean  # Does output match target schema?
  schema_ref: string  # Target schema reference
  errors: array[string]  # Validation errors if any
losses:
  - field: string  # Source field that was lost
    reason: string  # Why it was not preserved
    severity: info | warning | error  # Impact level
changes:  # Applied transformations
  - source_field: string
    target_field: string
    conversion: string  # What transformation was applied
    reversible: boolean  # Can be converted back?
confidence: 0..1  # Confidence in transformation correctness
evidence_anchors: ["source:field", "schema:rule"]
assumptions: []  # Explicit assumptions

Field Definitions

Examples

Example 1: JSON to YAML with Schema Validation

Input:

input:
  user_id: 12345
  user_name: "john_doe"
  email_address: "john@example.com"
  created_timestamp: 1705312800
  metadata:
    source: "api"
    version: "2.1"
source_format: json
target_format: yaml
options:
  schema:
    type: object
    required: [id, name, email, created_at]
    properties:
      id: { type: integer }
      name: { type: string }
      email: { type: string, format: email }
      created_at: { type: string, format: date-time }
      tags: { type: array, items: { type: string } }
  mapping:
    user_id: id
    user_name: name
    email_address: email
    created_timestamp: created_at

Output:

output:
  id: 12345
  name: "john_doe"
  email: "john@example.com"
  created_at: "2024-01-15T10:00:00Z"
  tags: []
transformation:
  input_type: "json"
  output_type: "yaml"
  mapping_applied: "explicit_mapping_v1"
validation:
  input_valid: true
  output_valid: true
  schema_ref: "user_schema_v2"
  errors: []
losses:
  - field: "metadata.source"
    reason: "No mapping to target schema"
    severity: info
  - field: "metadata.version"
    reason: "No mapping to target schema"
    severity: info
changes:
  - source_field: "created_timestamp"
    target_field: "created_at"
    conversion: "unix_epoch to ISO8601 datetime"
    reversible: true
  - source_field: "tags"
    target_field: "tags"
    conversion: "default empty array added"
    reversible: false
confidence: 0.9
evidence_anchors:
  - "source:user_id -> target:id"
  - "schema:required_fields_present"
  - "conversion:timestamp_to_iso8601"
assumptions:
  - "Unix timestamp is in seconds, not milliseconds"
  - "Timezone is UTC"

Evidence pattern: Field-by-field mapping documented, schema validation confirms conformance.

Example 2: CSV to Structured Object

Input:

input: "name,age,city\nAlice,30,NYC\nBob,25,LA"
source_format: csv
target_format: json
options:
  schema:
    type: object
    properties:
      records:
        type: array
        items:
          type: object
          properties:
            full_name: { type: string }
            age_years: { type: integer }
            location: { type: string }
  mapping:
    name: full_name
    age: age_years
    city: location

Output:

output:
  records:
    - full_name: "Alice"
      age_years: 30
      location: "NYC"
    - full_name: "Bob"
      age_years: 25
      location: "LA"
transformation:
  input_type: "csv"
  output_type: "json"
  mapping_applied: "csv_to_object_mapping"
validation:
  input_valid: true
  output_valid: true
  schema_ref: "person_array_schema"
  errors: []
losses: []
changes:
  - source_field: "age"
    target_field: "records[].age_years"
    conversion: "string to integer"
    reversible: true
  - source_field: "(rows)"
    target_field: "records"
    conversion: "CSV rows wrapped as an array under records"
    reversible: true
confidence: 0.95
evidence_anchors:
  - "source:row_count=2"
  - "schema:all_required_present"
assumptions:
  - "CSV uses comma delimiter"
  - "First row is header"
  - "No quoted fields with commas"

Verification

• Output validates against target schema
• All required fields are present
• Type conversions are correct
• Losses are explicitly documented
• Transformation is reproducible

Verification tools: Read (for schema validation), Grep (for pattern matching)

Safety Constraints

• mutation: false
• requires_checkpoint: false
• requires_approval: false
• risk: low

Capability-specific rules:

• Always document data loss explicitly
• Never silently drop fields without recording
• Validate output against schema before returning
• Flag precision loss in numeric conversions
• If transformation is ambiguous, ask for clarification
• Never transform sensitive data without noting it

Composition Patterns

Commonly follows:

• receive - Transform incoming messages to canonical form
• retrieve - Transform retrieved data to expected format
• inspect - Understand source before transformation

Commonly precedes:

• send - Format data before external transmission
• integrate - Prepare data for merging
• validate - Check transformed output

Anti-patterns:

• Never transform without documenting losses
• Never assume field type without verification
• Avoid chained transforms without intermediate validation

Workflow references:

• See reference/composition_patterns.md#digital-twin-sync-loop for transform in data pipeline
• See reference/composition_patterns.md#enrichment-pipeline for transform context