CLAUDE.md

This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.

What This Project Is

liquid-spec is a test suite and CLI for testing Liquid template implementations. Its purpose is to act as a harness for the gradual construction of a full, production-ready Liquid implementation: start with trivial passthrough specs, then progressively add variables, filters, control flow, partials, compatibility quirks, and finally production/theme recordings. It captures test cases from the reference Shopify/liquid implementation and can verify that any Liquid implementation produces correct output.

CLI Usage

# Install the gem (from GitHub, not published to rubygems.org)
gem install specific_install
gem specific_install https://github.com/Shopify/liquid-spec

# The top level stays focused on init, docs, run, and bench.
liquid-spec help

# Inspection, comparison, generated coverage, and contributor checks live here.
liquid-spec tools help

# Generate starter adapters. With no FILE, init creates both:
#   specs/adapter.rb and specs/adapter-jsonrpc.rb
liquid-spec init
liquid-spec init my_adapter.rb
liquid-spec init --jsonrpc

# Run specs with an adapter
liquid-spec run examples/liquid_ruby.rb
liquid-spec run my_adapter.rb

# Back-compat shorthand still works, but prefer the explicit run command
liquid-spec my_adapter.rb

# Filter by test name / run a specific suite / verbose output
liquid-spec run my_adapter.rb -n assign
liquid-spec run my_adapter.rb -s liquid_ruby
liquid-spec run my_adapter.rb -v

# List available specs and suites
liquid-spec run my_adapter.rb -l
liquid-spec run my_adapter.rb --list-suites

# Inspect/debug a specific spec in detail
liquid-spec tools inspect my_adapter.rb -n "case.*empty"

# Quick YAML-spec eval against an adapter, usually with reference comparison
cat my_spec.yml | liquid-spec tools eval my_adapter.rb --compare
liquid-spec tools eval my_adapter.rb --spec=my_spec.yml --compare

# Generated differential coverage after the recorded ramp is green
liquid-spec tools mutate my_adapter.rb --around=for_loops
liquid-spec tools fuzz my_adapter.rb --seed=1234 --json
liquid-spec tools stress my_adapter.rb --depth=64

# Benchmarks, cross-adapter matrices, reports, feature docs, implementer docs
liquid-spec bench
liquid-spec bench my_adapter.rb
liquid-spec tools matrix --all
liquid-spec tools report
liquid-spec tools features
liquid-spec tools check       # runs every verifier in scripts/verifiers/
liquid-spec docs curriculum

Default runner output

The standard (liquid-spec run adapter.rb; shorthand liquid-spec adapter.rb still works) run prints the lowest-complexity failures — the "next best specs to work on" — followed by a single stats line. Preamble, per-suite progress, and skipped-suite lines are verbose-only (-v):

Next best specs to work on:

1) [c=70] ForTagTest#test_iterate_with_each...
   ...

Complexity level cleared: 5 of 1000, 1234 passes, 56 failures, 12 skipped.

The level is the highest complexity level present in the run that is below the first failing level (0 when the first level fails; the max level when nothing fails) — a real level from the ramp, not an interpolated number. skipped is omitted when zero. Gem-owned file paths in failure locations are shortened to <gem>/.../<basename>:<line>.

When every run spec passes (0 failures, level = max), a Congrats addendum prints 3 suggested paths forward: implement an optional feature that actually had specs skipped this run (listed dynamically from the skipped-features set, filtered through Features::FEATURE_DOCS to :optional, excluding Ruby-interop :unnecessary ones), run liquid-spec bench <adapter> for performance, or use a matrix tool / contribute back to liquid-spec.

Spec Quality Gates

Run the contributor check gate when changing complexity scores, hints, early-ramp specs, feature tags, or docs that describe those rules:

rake check

rake check includes the spec-quality gate and the mechanical feature/error-mode audits. The spec-quality portion currently enforces:

• complexity scores must be <= 1000
• every spec with effective complexity <= 220 must have an effective hint

• every spec with complexity <= 220 must have a spec-LEVEL hint (spec.hint), not just suite/file boilerplate. Existing generated/bulk specs without one are grandfathered in test/spec_hint_baseline.txt; any NEW spec at c<=220 lacking a spec-level hint fails the gate, and stale baseline entries (specs that gained a hint or were removed) also fail. After intentionally adding hints, shrink the baseline:

  ruby -Ilib scripts/generate_spec_hint_baseline.rb

rake check — spec verifiers

liquid-spec tools check runs all verifiers in scripts/verifiers/ in-process; rake check is the contributor-task equivalent. Each verifier is a standalone Ruby script that prints findings (never modifies files) and returns 0 on success or non-zero on violations. Verifiers marked # advisory: true in their header are non-blocking — they report known debt but don't fail the overall check.

liquid-spec tools check  # run all verifiers through the CLI
rake check               # equivalent contributor task
rake prepush             # run unit tests then all verifiers (standard pre-push gate)

You can also run individual verifiers directly:

ruby -Ilib scripts/verifiers/ruby_type_tags.rb      # Ruby-content + instantiate: drops carry a ruby feature tag + complexity > 100
ruby -Ilib scripts/verifiers/lax_mode_declared.rb   # lax-dependent specs declare error_mode: lax (auto-tags lax_parsing)
ruby -Ilib scripts/verifiers/spec_schema.rb         # spec YAML structure: valid fields, required fields, known features, complexity range
ruby -Ilib scripts/verifiers/lax_placement.rb       # advisory: lax-only specs should live in the liquid_ruby_lax suite
ruby -Ilib scripts/verifiers/jsonrpc_portability.rb # specs don't send non-primitive Ruby types over JSON-RPC
ruby -Ilib scripts/verifiers/cross_mode_compatibility.rb # multi-mode error_mode specs produce consistent results
ruby -Ilib scripts/verifiers/parse_mode_annotation.rb # specs without error_mode behave identically across parse modes
ruby -Ilib scripts/verifiers/spec_name_collisions.rb # advisory: no two specs share the same name
ruby -Ilib scripts/verifiers/filesystem_extensions.rb # advisory: filesystem keys should have valid extensions
ruby -Ilib scripts/verifiers/minimum_complexity.rb  # specs with advanced features sit above the beginner ramp
ruby -Ilib scripts/verifiers/sequential_complexity.rb # opted-in curated files advance exactly one level per spec
ruby -Ilib scripts/verifiers/strict_mode_compatibility.rb # strict/strict2-only specs must need their single-mode restriction

Blocking verifiers (must be green before pushing):

• ruby_type_tags — specs with Ruby-specific content (Hash#inspect, non-string keys, instantiate: drops) must declare a ruby feature tag and sit above complexity 100.
• lax_mode_declared — specs that need lax mode must declare error_mode: lax (the gem auto-tags lax_parsing, so lax-opt-out adapters skip it).
• spec_schema — every spec YAML file must be well-formed: valid YAML, correct top-level structure, required fields (name, template, expected or errors), complexity in 0..1000, features from the known set, valid error_mode (single or array of strict/strict2/lax), valid errors sub-keys (parse_error, render_error, output), valid generate field definitions, no unknown fields.
• minimum_complexity — specs with advanced features must sit above the beginner ramp: Ruby content (ruby_types/ruby_drops/binary_data) and instantiate: drops ≥ 100; drops ≥ 200; template_factory and Shopify-specific features ≥ 200; render_errors: true and error_mode: strict2 ≥ 100.
• sequential_complexity — curated files with _metadata.sequential_complexity: true must advance exactly one complexity level per spec. This prevents one-lesson ramps from collapsing back into round-number buckets.
• strict_mode_compatibility — a strict- or strict2-only spec must actually differ in the other mode; otherwise it declares [strict2, strict] and runs for adapters supporting either strictness.
• jsonrpc_portability — specs must not send non-primitive Ruby types (non-string keys, symbols, unregistered objects) over JSON-RPC, which would trigger _rpc_drop markers. Time/Date/DateTime are whitelisted (sent as ISO 8601 strings).
• cross_mode_compatibility — specs with array error_mode (e.g. [lax, strict]) must produce consistent results in all declared modes.
• parse_mode_annotation — specs without error_mode must behave identically across all parse modes (lax, strict, strict2). Specs that behave differently must declare error_mode to indicate which modes they're compatible with. Specs with errors: must always declare error_mode (syntax errors are mode-dependent).

Advisory verifiers (report known debt, don't block push):

• lax_placement — lax-only specs belong in specs/liquid_ruby_lax/, not specs/liquid_ruby/. Reports misplaced ones; move them with scripts/move_spec.rb.
• spec_name_collisions — duplicate names can shadow another spec and make it unreachable.
• filesystem_extensions — committed filesystem keys should use a recognized extension (.liquid, .svg, .json, .css, .js, .scss, .html, or .txt).

Error-mode policy these enforce:

• A spec that needs lax mode must declare error_mode: lax (the gem auto-tags it lax_parsing, so lax-opt-out adapters skip it). lax_mode_declared catches any that forgot.
• Lax-only specs belong in specs/liquid_ruby_lax/, not specs/liquid_ruby/. lax_placement reports misplaced ones; move them with scripts/move_spec.rb.
• Specs exercising a lax-vs-strict2 difference that matters declare error_mode: strict2 (auto-tagged strict2_parsing).
• Specs compatible with multiple modes can declare error_mode: [lax, strict] (array form). The first mode is primary (used for testing); only the primary mode generates feature tags.
• Any spec with errors: must declare error_mode — syntax errors are mode-dependent. parse_mode_annotation catches any that forgot. When you introduce a new cross-cutting rule, add a verifier script in scripts/verifiers/ rather than a one-off check, so the rule stays enforced. Each verifier defines a module with a run class method returning 0 or non-zero, and ends with exit ModuleName.run if $PROGRAM_NAME == __FILE__ so it can run standalone. Mark non-blocking checks with # advisory: true in the header.

Dumb Adapter Ramp Audits

When changing early complexity scores or adding beginner specs, play dumb and verify the harness still behaves like an implementation curriculum:

# Source echo adapter: should pass only raw text, then fail on first object output
liquid-spec run /tmp/echo_adapter.rb --list-passed

# Always-empty adapter: may pass many empty-output specs accidentally; check max complexity
liquid-spec run /tmp/empty_adapter.rb --json --list-passed > /tmp/empty-results.json

# Always-raise adapters: should fail at complexity 0 with clear Error + Hint output
liquid-spec run /tmp/raise_compile_adapter.rb
liquid-spec run /tmp/raise_render_adapter.rb

Use --list-passed to inspect accidental passes and --json for tooling. Prefer Complexity level cleared (or JSON max_complexity_reached) over raw pass count when judging partial or deliberately naive adapters.

Deterministic date/time specs

Every spec execution surface places the clock 2024-01-01 00:01:58 UTC in the render register current_time. Registers are passed beside assigns but are only for filters and drops, never direct template lookup. Specs using now or today must assert output for that exact instant. Never make a matrix test pass merely because adapters agree on the host's current date.

Result Logging

Each test run appends results to /tmp/liquid-spec-results.jsonl with the format:

[run_id, version, source_file, test_name, complexity, "success|fail|error"]

• run_id: Unique timestamp for this run (e.g., "20260106_124439")
• version: liquid-spec version (e.g., "1.2.0")
• source_file: Path to the spec YAML file
• test_name: Name of the individual spec
• complexity: Complexity score (default 1000 if not set)
• status: "success", "fail", or "error"

This data can be analyzed later to:

• Identify specs that consistently pass early (suggesting lower complexity)
• Find specs that frequently fail (may need better hints or higher complexity)
• Track implementation progress over time
• Tune complexity scores based on real-world implementation order

Writing an Adapter

An adapter defines how your Liquid implementation compiles and renders templates:

#!/usr/bin/env ruby
require "liquid/spec/cli/adapter_dsl"

LiquidSpec.setup do |ctx|
  require "my_liquid"
  # ctx is a hash for storing adapter state (environment, file_system, etc.)
  # ctx[:environment] = MyLiquid::Environment.new
end

LiquidSpec.configure do |config|
  config.suite = :liquid_ruby  # or :all, :basics, :liquid_ruby_lax, :partials, :parser_errors, :benchmarks, :shopify_theme_dawn, ...

  # Parse modes implemented by this adapter. The highest supported strictness
  # is used for ordinary specs: strict2, then strict, then lax.
  config.error_modes = [:strict2]

  # Render errors raise by default. Add :inline only when supported.
  config.render_error_modes = [:raise]

  # Declare what your adapter cannot support yet. Specs requiring these
  # features are skipped so you can build incrementally.
  config.missing_features = [:shopify_tags, :shopify_filters]
end

LiquidSpec.compile do |ctx, source, parse_options|
  ctx[:template] = MyLiquid::Template.parse(source, **parse_options)
end

LiquidSpec.render do |ctx, assigns, render_options|
  # assigns: the environment variables for the template
  # render_options: { registers:, strict_errors:, error_mode: }
  ctx[:template].render(assigns, **render_options)
end

The ctx hash is passed to all blocks and can store adapter state like custom environments, file systems, or translations. This enables adapters that need isolated Liquid environments with custom tags/filters.

Adapters generated by liquid-spec init include a direct-execution guard that re-launches the CLI:

./my_adapter.rb
# equivalent to running the adapter through liquid-spec
liquid-spec run my_adapter.rb

Hand-written adapters can either include the same guard from lib/liquid/spec/cli/init.rb or be run explicitly with liquid-spec run.

Development Commands

# Run example adapter locally
ruby -I../liquid/lib -Ilib bin/liquid-spec run examples/liquid_ruby.rb

# Run with verbose output
ruby -I../liquid/lib -Ilib bin/liquid-spec run examples/liquid_ruby.rb -v

# Run specific tests
ruby -I../liquid/lib -Ilib bin/liquid-spec run examples/liquid_ruby.rb -n assign

# Test the CLI
ruby -Ilib bin/liquid-spec help
ruby -Ilib bin/liquid-spec init test.rb

# Generate specs from Shopify/liquid
bundle exec rake generate

Architecture

CLI Components

• bin/liquid-spec - Entry point
• lib/liquid/spec/cli.rb - Core command dispatcher (init, docs, run, bench, tools)
• lib/liquid/spec/cli/tools.rb - Dispatcher for inspection, comparison, reporting, verifier, and adversarial utilities
• lib/liquid/spec/cli/init.rb - Generates adapter templates and implementation guidance
• lib/liquid/spec/cli/runner.rb - Runs specs with an adapter
• lib/liquid/spec/cli/adapter_dsl.rb - DSL for LiquidSpec.setup/configure/compile/render
• lib/liquid/spec/cli/eval.rb - One-off YAML spec evaluation and reference comparison
• lib/liquid/spec/cli/bench.rb / benchmark.rb - Benchmark command and timing harness
• lib/liquid/spec/cli/matrix.rb - Cross-adapter comparison runner
• lib/liquid/spec/cli/inspect.rb - Detailed spec inspection/debugging
• lib/liquid/spec/cli/docs.rb / features.rb / report.rb - Implementer docs, feature inventory, benchmark reports
• lib/liquid/spec/cli/check.rb / lib/liquid/spec/verifiers.rb - CLI check command and shared all-verifier runner

Spec Components

• Liquid::Spec::LazySpec - Single lazily materialized YAML spec case, including environment/filesystem setup
• Liquid::Spec::SpecLoader - Loads specs from suites, YAML files, and additional spec globs
• Liquid::Spec::Suite - Suite configuration/discovery loaded from suite.yml
• Liquid::Spec::AdapterRunner - Compiles/renders specs through adapter DSL blocks or JSON-RPC adapters
• Liquid::Spec::API - Programmatic helpers for loading/running specs

Directory Structure

• bin/ - CLI executable
• examples/ - Example adapters (liquid_ruby.rb, liquid_ruby_lax.rb, liquid_ruby_shopify.rb, json_rpc_ruby_liquid.rb, JIT variants); liquid-c examples are legacy explicit opt-ins and are excluded from defaults
• lib/liquid/spec/cli/ - CLI implementation
• specs/basics/ - Curated beginner ramp specs
• specs/liquid_ruby/ - Core Liquid specs from Shopify/liquid
• specs/liquid_ruby_lax/ - Lax-mode reference behavior
• specs/parser_errors/ - Parser/error-mode matrix specs
• specs/partials/ - Include/render/filesystem-focused specs
• specs/benchmarks/ - Timing benchmark specs
• specs/shopify_production_recordings/ - Recorded specs from Shopify production
• specs/shopify_theme_dawn/ - Shopify Dawn theme specs
• tasks/ - Rake tasks for spec generation

Spec Format

YAML Structure

YAML spec files support two formats:

Simple format (array of specs):

---
- name: test_one
  template: "{{ foo }}"
  expected: "bar"

Extended format (with metadata):

---
_metadata:
  hint: "These specs require lax mode"
  required_options:
    error_mode: :lax
specs:
- name: test_one
  template: "{{ foo }}"
  expected: "bar"

Hints

IMPORTANT: Hints are critical for helping implementers understand what they need to do to make a spec succeed. Every non-trivial spec should have a hint that explains either:

1. What to implement - Concrete steps or logic needed to pass the spec
2. Why the behavior exists - The higher-order reason for unexpected or quirky behavior
3. Common pitfalls - What implementers typically get wrong

Hints are displayed in yellow when the spec fails, making them the first thing implementers see when debugging.

- name: empty_string_is_empty
  template: "{% if x == empty %}empty{% else %}not{% endif %}"
  environment: { x: "" }
  expected: "empty"
  complexity: 100
  hint: |
    Recognize 'empty' as a keyword representing the empty state.
    An empty string "" should equal the 'empty' keyword. Create
    an EmptyLiteral node during parsing. During evaluation, compare
    the variable's value against emptiness: empty strings, empty
    arrays, and empty hashes all equal 'empty'.

- name: int_size_returns_byte_size
  template: "{{ num | size }}"
  environment: { num: 42 }
  expected: "8"
  complexity: 150
  hint: |
    QUIRK: int | size returns 8 because Ruby's Integer#size returns
    the byte representation size (8 bytes on 64-bit systems), NOT
    the number of digits. This is surprising but matches liquid-ruby.

What makes a good hint:

• Actionable - Tells implementers what code to write or what behavior to implement
• Explains "why" - For quirky behavior, explains why Liquid works this way
• Flags quirks - Marks surprising behavior with "QUIRK:" prefix so implementers know it's intentional
• Progressive - For complex features, hints guide through the implementation steps

Use hints to communicate:

• Dependencies that must be loaded (e.g., ActiveSupport)
• Implementation-specific behaviors being tested
• Known edge cases or platform differences (mark with "QUIRK:")
• Step-by-step guidance for implementing the feature
• Why a spec exists and what behavior it validates

Source-Level Metadata

YAML files can include a _metadata section for settings that apply to all specs in the file:

• hint: A global hint displayed when any spec in this file fails
• required_options: Options that are automatically applied to all specs (e.g., error_mode: :lax)
• sequential_complexity: Opts a curated file into the verifier requiring each spec to use the next integer complexity

Spec-level settings override source-level settings. For example, a spec with its own error_mode will use that instead of the required_options.error_mode.

Good Specs

Good specs preserve the project goal: help someone build a production-ready Liquid implementation gradually. A spec should teach one behavior at the right time, fail with an actionable message, and point to implementation guidance when the behavior is not obvious.

Evolve the curriculum when agents get stuck

A repeated agent failure is evidence about the curriculum, not necessarily just a weak agent. Do not hand-fix the latest generated implementation or weaken correct acceptance behavior. Diagnose why the general rule was hard to discover, then make the durable fix in liquid-spec. Depending on the failure:

• Improve the hint when it describes the expected output but not the semantic rule or implementation strategy.
• Move the complexity score when the spec appears before its prerequisites or so late that it no longer teaches anything new.
• Add a missing prerequisite spec when the jump combines multiple new concepts. Give the prerequisite a smaller, focused template and place it immediately before the harder interaction.
• Add or improve an implementer guide when the rule is cross-cutting or too large for a failure hint, then link the spec to it with doc:.
• Add a verifier when the same spec-authoring mistake can recur mechanically. A verifier turns review advice into a permanent quality gate.
• Split an overly broad spec into smaller lessons when one failure can have several unrelated causes. Each resulting spec should isolate one new behavior and retain a useful regression boundary.

After evolving a spec, replay the dumb-adapter audits and the surrounding complexity levels. The change is good only if it makes the next implementation step clearer without letting source-echo, always-empty, or special-cased implementations advance falsely.

Ramp discipline

• Treat complexity as an ordinal curriculum position, not a bucket. First-contact specs for a feature must be tiny, gentle, hinted, and placed on their own level. Give each subsequent independently failable behavior the next free integer (N, N+1, N+2) before combinations and edge cases. Specs may share a level only when they are true breadth variants requiring the same implementation step; do not pile different lessons onto round scores such as 180, 250, or 300.
• Keep the 0-50 band boring: passthrough, literals, missing variables, simple variable lookup, a few simple filters, and assign.
• Keep whitespace control ({{-, -}}, {%-, -%}), drop/to_liquid boundaries, generated filter matrices, parser recovery, date/time quirks, and filesystem/security quirks out of the beginner band.
• Generated specs should not flood the early ramp. Prefer curated beginner specs early; generated compatibility breadth generally starts at 120+ or much later.
• If a dumb adapter that returns the input, returns "", or raises for everything passes a spec unexpectedly, either the spec is too weak or the complexity/hint needs review.
• When judging naive adapters, prefer Complexity level cleared over total pass count. An always-empty adapter can pass later specs whose correct output is empty, but it should not advance through the contiguous ramp.

Error specs: prefer raised errors over inline errors

Most specs that exercise an error path should let the error raise and match it with errors:. Do not set render_errors: true (inline error rendering) unless the spec is specifically testing inline-error display behaviour — and those specs must declare features: [inline_errors].

Two raised-error forms exist:

• errors: parse_error: — the template fails to parse (syntax errors, unknown tags in strict mode, unclosed blocks, …).
• errors: render_error: — the template parses but rendering raises (division by zero, invalid filter arguments, accessing a non-drop object, include-inside-render, …).

- name: division_by_zero_lax
  template: "{{ 10 | divided_by: 0 }}"
  errors:
    render_error:
      - divided by 0
  complexity: 200
  error_mode: lax

- name: unknown_tag_strict_parse_error
  template: "{% nonexistent_tag %}content{% endnonexistent_tag %}"
  error_mode: strict
  errors:
    parse_error:
      - Unknown tag
  complexity: 300

Matching error patterns

Each entry under parse_error: / render_error: is matched against:

1. The full exception message
2. The "core" message (text after the last ): or : )
3. The exception class name (e.g. NoMethodError, Liquid::ArgumentError)

A string pattern is a case-insensitive, literal substring (special characters are escaped). A Regexp pattern (!ruby/regexp /.../i) is used as-is, so metacharacters like |, .+, and anchors work. All listed patterns must match for the spec to pass. Use multiple patterns to pin down the error precisely without coupling to exact wording:

- name: sec_wide_open_object_name_raises_no_to_liquid
  template: "{{ o.name }}"
  environment:
    o:
      instantiate:WideOpenObject: {}
  errors:
    render_error:
      - NoMethodError        # the error class name
      - to_liquid            # what's missing from the message
  complexity: 220

- name: error_in_partial_line_3_precise
  template: "{% render 'multiline' %}"
  filesystem:
    multiline.liquid: |
      first line
      second line
      {{ bad | divided_by: 0 }}
  environment:
    bad: 99
  errors:
    render_error:
      - multiline            # partial name appears in the error
      - line 3               # line number within the partial
      - divided by 0         # the underlying error
  complexity: 550
  error_mode: lax

A Regexp pattern (used as-is, unlike string patterns which are literal substrings) — useful for alternation without doubling up specs:

- name: regexp_render_error_match
  template: "{{ 10 | divided_by: 0 }}"
  errors:
    render_error:
      - !ruby/regexp /ZeroDivision|divided by/i
  complexity: 200

Best practices for error patterns:

• Match the error class name (NoMethodError, Liquid::SyntaxError, …) to assert the kind of error, not just its message text.
• Match one stable substring of the message that captures the meaningful part (e.g. divided by 0, to_liquid, Unknown tag). Avoid matching the full message — wording varies across implementations.
• Match location info when relevant — partial name and line N — but keep those as separate patterns so an implementation that lacks one of them fails clearly on that specific pattern.
• Use a Regexp (!ruby/regexp) when you need alternation or anchors rather than a literal substring. Note: backslash escapes inside the YAML regexp literal must be doubled (write \\d, not \d), because YAML processes the scalar before the regexp is compiled. Prefer backslash-free patterns where possible (e.g. !ruby/regexp /ZeroDivision|divided by/i).
• Never set render_errors: true for new specs. New specs should always let errors raise and match them with errors: parse_error: / errors: render_error:. The inline form (render_errors: true with errors: output:) is legacy and should not be added to new specs.

Suite Configuration

Each suite directory contains a suite.yml file that configures the suite:

---
# Human-readable name
name: "Liquid Ruby"
description: "Core Liquid template engine behavior"

# Whether this suite runs by default (when suite: :all)
default: true

# Context hint displayed when any spec in this suite fails
hint: |
  These specs test core Liquid functionality.
  Failures indicate missing or incorrect Liquid behavior.

# Required features/capabilities for this suite.
# Specs are skipped when the adapter lists any of these in config.missing_features.
features:
  - core

# Default complexity for specs without explicit complexity (see `liquid-spec docs complexity`)
# Specs without complexity default to 1000 unless this is set
minimum_complexity: 1000

# Default options applied to all specs (can be overridden per-file or per-spec)
defaults:
  render_errors: false

Complexity Scoring

Each spec should have a complexity field indicating implementation difficulty. Lower scores = simpler features to implement first. Specs without explicit complexity default to 1000 or the suite's minimum_complexity. Complexity is capped at 1000; do not score specs above 1000.

Run liquid-spec docs complexity for the full guide with examples. See SPECS.md for guidelines on writing effective specs.

Available Suites

Current builtin suites (liquid-spec run ADAPTER --list-suites):

• basics (default: true) - Curated beginner ramp covering fundamental Liquid behavior.
• liquid_ruby (default: true) - Core Liquid specs from Shopify/liquid integration tests.
• liquid_ruby_lax (default: true) - Lax-mode reference behavior; requires :lax_parsing.
• parser_errors (default: false) - Parser/error-mode matrix specs; requires :strict2_parsing.
• partials (default: false) - Include/render/filesystem-focused correctness and timing specs; requires :core.
• benchmarks (default: false) - Timing specs, normally run with the core liquid-spec bench command.
• shopify_production_recordings (default: true) - Recorded Shopify production specs; requires :inline_errors.
• shopify_theme_dawn (default: true) - Real-world templates from Shopify Dawn; requires Shopify-specific tags/objects/filters.

Suites can also be discovered from local project ./specs/<name>/suite.yml directories when invoking liquid-spec from another project.

Features

Adapters declare parse and render error behavior positively, and other unsupported features through missing_features. Specs without error_mode run once at the highest supported strictness. Explicit multi-mode specs run only at their highest compatible strict mode (strict2, then strict). An explicitly declared lax mode runs separately. Raised render errors are the default; inline-error specs require render_error_modes to include :inline.

LiquidSpec.configure do |config|
  config.error_modes = [:strict2]
  config.render_error_modes = [:raise]
  # Empty means "try every spec". Add unsupported capabilities here.
  config.missing_features = [:shopify_tags]
end

Available Features

Feature selection is denylist-based. Leave missing_features empty to try everything, or add unsupported capabilities while the implementation is still growing. Use liquid-spec tools features (backed by lib/liquid/spec/cli/features.rb) as the source of truth for the current feature inventory and recommendations.

Common features to list in missing_features:

• :drops - Adapter cannot support the standard test drop library yet (see docs/test_drops.md)
• Parse-mode and inline-error feature tags are derived from error_modes and render_error_modes; do not maintain them manually in missing_features.
• :self_environment_shadowing - Adapter does not implement the compatibility behavior where an environment variable named self shadows the synthetic SelfDrop
• :ruby_types / :ruby_drops / :drop_class_output / :binary_data - Adapter cannot consume or reproduce Ruby-specific values/output
• :template_factory - Adapter cannot support template factory/artifact callbacks
• :strict2_blank_body_errors - Aspirational "NEW STRICT2 CONTRACT" (strict2 surfaces inline errors even for blank block bodies); liquid-ruby 5.13 does not implement this, so opt out until your adapter deliberately adopts it

Shopify-specific features:

• :shopify_tags - Shopify-specific tags (schema, style, section)
• :shopify_objects - Shopify-specific objects (section, block)
• :shopify_filters - Shopify-specific filters (asset_url, image_url)
• :shopify_includes, :shopify_blank, :shopify_error_handling, :shopify_error_format, :shopify_string_access - Shopify platform/theme behavior beyond portable Liquid
• :shopify_resource_limits - Shopify render score tracking and cumulative limit enforcement across partials. Implementation plumbing, not Liquid semantics; recursion/stack-depth specs are core.

JSON-RPC adapters that can't support the standard test drops yet should set config.missing_features = [:drops, :ruby_types, :ruby_drops, :binary_data] (plus any Shopify capabilities they lack).

JSON-RPC Adapter Setup Notes

JSON-RPC is the main path for non-Ruby Liquid implementations. Keep it especially well documented and tested.

• Generate with liquid-spec init --jsonrpc my_adapter.rb. Running liquid-spec init with no filename generates both specs/adapter.rb and specs/adapter-jsonrpc.rb.
• The generated JSON-RPC adapter is executable/self-launching; use --command=... when you need to point it at a separate server command.
• The server implements initialize, compile, render, and quit over newline-delimited JSON-RPC on stdin/stdout.
• Server logs must go to stderr, never stdout.
• The Ruby adapter controls spec selection with config.missing_features; server-reported features are informational.
• Minimal JSON-RPC implementations should usually skip Ruby/transport-specific specs: :drops, :ruby_types, :ruby_drops, :drop_class_output, :self_environment_shadowing, :binary_data, :template_factory, plus Shopify-specific features.
• Implement the standard test drops (see docs/test_drops.md) to enable :drops. Standard drops use _instantiate markers — no RPC callbacks needed.
• Prefer result.error for Liquid parse/render errors; legacy JSON-RPC errors -32000/-32001 are accepted for compatibility.
• Render receives options.strict_errors; when it is false, render errors should become inline Liquid error output.

Run JSON-RPC checks with:

ruby -Ilib -I$LIQUID -Itest -e 'require "test_helper"; require File.expand_path("test/json_rpc_test.rb")'
ruby -Ilib -I$LIQUID bin/liquid-spec run examples/json_rpc_ruby_liquid.rb -n '^empty_template$|^literal_passthrough$' --json

The Eval Tool

The liquid-spec tools eval command is the primary tool for testing individual templates and discovering behavioral differences. Always use --compare to validate your implementation against the reference liquid-ruby.

Why Use Eval?

1. Quick iteration - Test a single template without running the full suite
2. Discover differences - --compare shows exactly where your implementation differs from the reference
3. Generate specs - Results are automatically saved to /tmp/liquid-spec-{date}.yml for contribution
4. Debug failures - Understand why a specific template produces unexpected output

Basic Usage

eval reads a YAML spec from stdin or --spec=FILE; it does not take inline --liquid/assigns/expected flags. Use a small YAML spec even for one-liners:

# Quick test with automatic comparison to reference
cat <<'EOF' | liquid-spec tools eval adapter.rb --compare
name: test_upcase_literal
complexity: 40
template: "{{ 'hello' | upcase }}"
expected: "HELLO"
hint: "The upcase filter should uppercase a string literal."
EOF

# Test with environment variables
cat <<'EOF' | liquid-spec tools eval adapter.rb --compare
name: test_array_size
complexity: 40
template: "{{ x | size }}"
environment:
  x: [1, 2, 3]
expected: "3"
hint: "The size filter returns array length."
EOF

# Or read from a file
liquid-spec tools eval adapter.rb --spec=my_test.yml --compare

Using --compare (Recommended)

The --compare flag runs your template against the reference liquid-ruby implementation first, then compares results. This is the best way to find behavioral differences:

cat <<'EOF' | liquid-spec tools eval adapter.rb --compare
name: test_filter
template: "{{ 'hi' | upcase }}"
expected: "HI"
complexity: 40
hint: "The upcase filter should convert ASCII letters to uppercase."
EOF

# When implementations differ, you'll see a failure explaining how the reference
# output differed from your adapter's output.

When to use --compare:

• Testing any new feature implementation
• Debugging a failing spec
• Exploring edge cases
• Generating specs for contribution

YAML Spec Input

For complex tests, use YAML input via stdin or file:

# From stdin (heredoc) - great for multi-line templates
cat <<EOF | liquid-spec tools eval adapter.rb --compare
name: test_for_loop_with_break
hint: "break should exit the loop immediately"
complexity: 120
template: |
  {% for i in (1..5) %}
    {% if i == 3 %}{% break %}{% endif %}
    {{ i }}
  {% endfor %}
expected: |
  
    1
  
    2
  
EOF

# From a YAML file
liquid-spec tools eval adapter.rb --spec=my_test.yml --compare

Spec YAML Format

name: test_descriptive_name          # Required: unique identifier
hint: "Explain what this tests"      # Recommended: helps debug failures
complexity: 75                       # Recommended: see `liquid-spec docs complexity`
template: "{{ x | upcase }}"         # Required: the Liquid template
expected: "HELLO"                    # Optional with --compare: auto-filled from reference
environment:                         # Optional: variables available to template
  x: hello

Auto-Save and Contribution

Every eval run saves results to /tmp/liquid-spec-{date}.yml. When a difference is detected, you'll see:

============================================================
  DIFFERENCE DETECTED
============================================================

This spec reveals a behavioral difference worth documenting.
Please contribute it: https://github.com/Shopify/liquid-spec

Review the saved specs and contribute interesting ones to liquid-spec!

Programmatic API

You can also use eval from Ruby code for automated testing:

require 'liquid/spec/cli/adapter_dsl'

# Run a spec and get results
LiquidSpec.evaluate("adapter.rb", <<~YAML, compare: true)
  name: test_upcase_filter
  hint: "upcase should convert to uppercase"
  complexity: 40
  template: "{{ 'hello' | upcase }}"
  expected: "HELLO"
YAML

Tips for Effective Testing

1. Always name your specs - Use descriptive names like test_for_break_in_nested_loop
2. Add hints - Explain what behavior you're testing and why it matters
3. Set complexity - Helps organize specs by implementation difficulty
4. Use --compare first - Let the reference fill in expected before asserting
5. Test edge cases - Empty arrays, nil values, unusual inputs
6. Save interesting failures - Differences reveal implementation gaps worth documenting

Filesystem in Specs

Specs that use {% include %} or {% render %} need a filesystem: field. This is always a simple hash mapping filenames to their content:

- name: test_render_partial
  template: "{% render 'greeting' %}"
  expected: "Hello!"
  filesystem:
    greeting.liquid: "Hello!"

- name: test_include_with_variable
  template: "{% include 'card' %}"
  expected: "Product: $99"
  environment:
    title: "Product"
    price: 99
  filesystem:
    card.liquid: "{{ title }}: ${{ price }}"

Filesystem keys must include a recognized extension. Use greeting.liquid, not greeting; the filesystem_extensions verifier enforces this for committed specs.

Object Instantiation in Specs

Specs can include custom Ruby objects (drops, etc.) in their environment. We use a simple instantiate:ClassName format - no YAML tags like !ruby/object.

Format

Any string value starting with instantiate: triggers object creation:

environment:
  # Simple: no arguments
  my_drop: "instantiate:CountingDrop"

  # With arguments: value after the key becomes constructor args
  user:
    instantiate: StringDrop
    value: "hello"

  # Hash arguments passed to constructor
  renderer:
    instantiate: StubExceptionRenderer
    raise_internal_errors: false

How It Works

1. Spec loader detects instantiate:ClassName pattern
2. Looks up ClassName in Liquid::Spec::ClassRegistry
3. Calls registry[klass].call(remaining_hash) to create instance
4. If class not found in registry, raises hard error

ClassRegistry

Classes must be registered before use:

# In lib/liquid/spec/deps/liquid_ruby.rb
Liquid::Spec::ClassRegistry.register("CountingDrop") { |p| CountingDrop.new(p) }
Liquid::Spec::ClassRegistry.register("StringDrop") { |p| StringDrop.new(p) }

The lambda receives whatever hash/value was alongside instantiate: and must return the instance.

Available Classes

See lib/liquid/spec/deps/liquid_ruby.rb for all registered classes:

• CountingDrop, ToSDrop, TestDrop - Test drops
• StringDrop, IntegerDrop, BooleanDrop - Value drops
• StubExceptionRenderer - Exception handling
• StubTemplateFactory - Template factories
• SafeBuffer - HTML-safe strings (requires ActiveSupport)

Adding New Classes

1. Define the class in lib/liquid/spec/deps/liquid_ruby.rb or lib/liquid/spec/test_drops.rb
2. Register it: Liquid::Spec::ClassRegistry.register("MyClass") { |p| MyClass.new(p) }
3. Use in specs: my_var: { instantiate: MyClass, arg1: value1 }

Editing YAML Specs with yq

If the yq tool is available, prefer it for bulk YAML edits. You can write bash scripts that make multiple edits in one go:

#!/bin/bash
# Bulk update complexity scores in a spec file

FILE="specs/liquid_ruby_lax/variable_type_filters.yml"

# Update metadata minimum_complexity
yq -i '._metadata.minimum_complexity = 500' "$FILE"

# Update all specs with complexity 150 to 500
yq -i '(.specs[] | select(.complexity == 150)).complexity = 500' "$FILE"

# Add 350 to all complexity values under 200
yq -i '(.specs[].complexity | select(. < 200)) += 350' "$FILE"

# Set a value in suite config
yq -i '.minimum_complexity = 500' specs/liquid_ruby_lax/suite.yml

Common yq patterns for spec files:

# Read a value
yq '.specs[0].name' file.yml

# Update matching specs by name
yq -i '(.specs[] | select(.name == "test_foo")).complexity = 100' file.yml

# Add a hint to all specs missing one
yq -i '(.specs[] | select(.hint == null)).hint = "TODO: add hint"' file.yml

# Delete a field from all specs
yq -i 'del(.specs[].some_field)' file.yml

# Count specs
yq '.specs | length' file.yml

Landing the Plane (Session Completion)

When ending a work session, you MUST complete ALL steps below. Work is NOT complete until git push succeeds.

MANDATORY WORKFLOW:

1. File issues for remaining work - Create issues for anything that needs follow-up
2. Run quality gates (if code changed) - rake prepush runs unit tests then all verifiers
3. Update issue status - Close finished work, update in-progress items
4. PUSH TO REMOTE - This is MANDATORY:

   git pull --rebase
   git push
   git status  # MUST show "up to date with origin"

5. Clean up - Clear stashes, prune remote branches
6. Verify - All changes committed AND pushed
7. Hand off - Provide context for next session

CRITICAL RULES:

• Work is NOT complete until git push succeeds
• NEVER stop before pushing - that leaves work stranded locally
• NEVER say "ready to push when you are" - YOU must push
• If push fails, resolve and retry until it succeeds