Vespa Schema Authoring

Overview

A Vespa schema (.sd file) defines a document type: its fields, indexing behavior, matching rules, and ranking configuration. Schemas live under schemas/ in a Vespa application package.

This skill applies when the user needs to create, modify, or troubleshoot .sd schema files -- including field definitions, indexing pipelines, match modes, tensor/HNSW config, rank profiles, structs, fieldsets, or document references.

For deeper detail, load docs/field-types.md, docs/schema-patterns.md, docs/tensors-and-hnsw.md, docs/example-ecommerce-schema.md, or docs/gotchas-extended.md from this skill's directory as needed.

Quick Reference

Minimal Skeleton

schema my_doc {
    document my_doc {
        field title type string {
            indexing: summary | index
            match: text
        }
    }
    fieldset default { fields: title }
    rank-profile default {
        first-phase { expression: nativeRank(title) }
    }
}

The outer schema name and inner document name must match.

Field Types

Primitive Types

Reference and Tensor Types

Collection Types

• array<T> -- Ordered list. Supports index and attribute.
• weightedset<T> -- Set with integer weights. T must be string, int, or long.
• map<K, V> -- Key-value map. K must be string, int, or long. Index sub-fields via struct-field.

field tags type array<string> {
    indexing: summary | index
}
field scores type weightedset<string> {
    indexing: summary | attribute
}
field properties type map<string, string> {
    indexing: summary
    struct-field key   { indexing: attribute   match: exact }
    struct-field value { indexing: attribute }
}

Indexing Pipeline

A field's indexing statement is a |-separated pipeline of directives.

• index -- Disk-backed inverted index. Required for match: text and linguistic search. Supports stemming, normalization, proximity data.
• attribute -- In-memory column store. Required for sorting, grouping, filtering, and access in ranking expressions.
• summary -- Include in search result output. Without this, field values are not returned.

Common Combinations

field title type string       { indexing: summary | index }              # text search + results
field category type string    { indexing: summary | attribute            # filter + results
                                match: exact   rank: filter }
field description type string { indexing: summary | index | attribute }  # search + filter + results
field quality type double     { indexing: attribute }                    # ranking only
field html type string        { indexing: summary }                     # retrieval only

Attribute Modifiers

Pick by workload, not by default:

Input Expressions

Transform data inline: lowercase, normalize, tokenize, to_int, to_long, to_string, to_array, set_language.

field title_lower type string {
    indexing: input title | lowercase | summary | index
}

Match Modes

field body type string     { indexing: summary | index   match: text }
field sku type string      { indexing: summary | index   match: exact }
field suggest type string  { indexing: summary | index   match: prefix }
field cjk type string      { indexing: summary | index   match { gram   gram-size: 2 } }

Note: match: word and match: exact require index. For attribute-only exact matching, use fast-search and query with the = prefix operator.

Tensor Fields and HNSW Configuration

Tensor type syntax: tensor<value-type>(dimension-list). Indexed dims x[N], mapped dims x{}, mixed tensor<float>(cat{}, x[128]). Configure HNSW under index { hnsw { ... } } on a tensor attribute field.

Default skeleton for an embedding field. Do not add paged here — it is strongly discouraged with HNSW (see Attribute Modifiers table above):

field embedding type tensor<float>(x[768]) {
    indexing: attribute | index
    attribute {
        distance-metric: prenormalized-angular   # for already-normalized vectors; use `angular` otherwise
    }
    index { hnsw { max-links-per-node: 16  neighbors-to-explore-at-insert: 200 } }
}

For the full distance-metric table, HNSW parameter guidance, and a complete tensor-field example, load docs/tensors-and-hnsw.md.

Struct Types

Define composite types with struct inside document. Use struct-field to index inner fields.

struct address {
    field street type string {}
    field city type string {}
    field country type string {}
}
field warehouse type address {
    indexing: summary
    struct-field city    { indexing: attribute   match: exact }
    struct-field country { indexing: attribute   match: exact }
}
field locations type array<address> {
    indexing: summary
    struct-field city { indexing: attribute }
}

Structs can be used inside array<> and map<> collections.

Fieldsets

Combine fields for unified search. Declared at schema level, outside document.

fieldset default {
    fields: title, body, description
}

All fields in a fieldset must be the same type (string) with compatible match/indexing modes.

Rank Profiles

Structure and Key Elements

rank-profile hybrid inherits default {
    inputs {
        query(query_embedding) tensor<float>(x[384])
        query(text_weight) double: 0.7
    }
    function text_score() {
        expression: bm25(title) * 2 + bm25(body)
    }
    function vector_score() {
        expression: closeness(field, embedding)
    }
    first-phase {
        expression: text_score
    }
    second-phase {
        expression: query(text_weight) * normalize_linear(text_score) + (1 - query(text_weight)) * normalize_linear(vector_score)
        rerank-count: 200
    }
    match-features {
        text_score
        vector_score
    }
}

Rank profiles support inheritance -- child profiles override or extend parent definitions via inherits.

Schema Evolution Warnings

Reserved field names (cause conflicts): documentid, relevance, sddocname.

Type changes: Changing a field type (e.g., string to int) requires reindexing and validation-overrides.xml. Widening (int to long) is safe; narrowing is not.

Removing fields: First remove all references in rank profiles, fieldsets, and summaries, then remove the field. Stop feeding the field before removal.

Adding fields: Safe on a running system. Existing documents get empty/default values. Attribute fields consume memory even when empty.

For a comprehensive production example combining BM25, semantic search, structs, maps, weightedsets, and phased ranking (default/bm25_ranking/semantic/hybrid/personalized profiles), load docs/example-ecommerce-schema.md.

Gotchas and Common Mistakes

1. Forgetting index: enable-bm25. The bm25(field) rank feature returns 0 without it.

# Wrong -- bm25(title) returns 0
field title type string { indexing: summary | index }
# Correct
field title type string { indexing: summary | index   index: enable-bm25 }

2. match: exact without index. The match directive controls how the index processes terms. On attribute-only fields it has no effect. Use fast-search + the = query prefix for attribute exact matching.

3. Schema/document name mismatch. Both names must be identical or deployment fails.

4. Tensor dimension mismatch. Query and document tensors must share exact dimension names and sizes (x[384] in both, not d[384] vs x[384]).

For gotchas 5–12 (memory/paging, type changes, reserved names, raw storage, weightedset type limits, references + import field, fast-access/fast-search restrictions), load docs/gotchas-extended.md.

Validation Checks

Apply only the checks that match what you actually changed:

• Wrote a new schema? Confirm outer schema and inner document names match.
• Added a field used in bm25()? Confirm index: enable-bm25 is set on that field.
• Added or changed a tensor / a rank-profile inputs block? Confirm document-side and query-side dimension names and sizes agree (x[384] on both, not d[384] vs x[384]).
• Added a rank profile, fieldset, or summary that references field names? Confirm those fields exist in the document.

For incremental edits to an existing schema, do not re-validate sections you did not touch. Trust the existing file. Re-reading or restating unchanged content adds turns without changing the output.