Adamant Generator Development

Patterns and workflows for developing custom generators for Adamant framework components. Generators produce Ada source, YAML types, HTML docs, and ground system artifacts from YAML model files.

Architecture Overview

Adamant's code generation pipeline:

YAML Model File -> Schema Validation -> Python Model Object -> Jinja2 Template -> Generated Output

Discovery chain:

1. env/set_python_path.sh adds every directory containing __init__.py to PYTHONPATH
2. redo/util/meta.py scans PYTHONPATH for directories named generators/ or generator/
3. All .py files in those directories are imported
4. meta.get_all_subclasses_of(generator_base) finds all generator classes
5. Each generator's input_file_regex() determines which YAML files it processes
6. The generator database maps (input_file, generator) -> output_file for redo

Directory Structure

Component-local generators live inside the component directory:

src/components/<component_name>/
  gen/
    __init__.py              # REQUIRED: empty file, makes gen/ a Python package
    generators/              # NO __init__.py here!
      <generator_name>.py    # Generator classes
    models/                  # NO __init__.py here!
      <model_name>.py        # Model classes (parse YAML into Python objects)
    schemas/                 # NO __init__.py here!
      <schema_name>.yaml     # Pykwalify schemas for YAML validation
    templates/               # NO __init__.py here!
      <template_dir>/
        name.ads             # Jinja2 templates ("name" is replaced with model name)
    doc/                     # Optional: LaTeX documentation for the generator
      <generator_name>.tex

CRITICAL: __init__.py goes ONLY in gen/, NEVER in subdirectories. Adding __init__.py to gen/models/ or gen/generators/ makes them Python packages that shadow the framework's models and generators packages (which contain models.base, models.exceptions, generators.basic, etc.), causing ModuleNotFoundError on any redo build. The framework's 15 existing generators confirm this: none have __init__.py in subdirectories.

All 15 framework components with custom generators follow this exact structure.

YAML File Naming Convention

CRITICAL: YAML model files follow a strict naming pattern that determines how they are discovered and processed:

[specific_name.]model_name.model_type.yaml

• model_type = the type string used for regex matching (e.g., ccsds_router_table, parameter_table)
• model_name = typically the assembly name (for assembly submodels) or component name
• specific_name = optional qualifier when multiple instances of the same model type exist

Examples:

linux_example.ccsds_router_table.yaml          # model_name=linux_example, model_type=ccsds_router_table
test_parameter_table.test_assembly.parameter_table.yaml  # specific_name=test_parameter_table, model_name=test_assembly
linux_example.product_packets.yaml             # model_name=linux_example, model_type=product_packets

The model_type in the filename MUST match model_class.__name__ in the generator (which becomes input_file_regex = r".*\.<model_type>\.yaml$").

Generator Class Patterns

Pattern 1: Basic Generator (most common)

For generators that produce a single output file from a single YAML input. Inherit from both basic_generator and generator_base:

import os.path
from models import my_model
from base_classes.generator_base import generator_base
from generators.basic import basic_generator


class my_output_ads(basic_generator, generator_base):
    def __init__(self):
        this_file_dir = os.path.dirname(os.path.realpath(__file__))
        template_dir = os.path.join(this_file_dir, ".." + os.sep + "templates")
        basic_generator.__init__(
            self,
            model_class=my_model.my_model,
            template_filename="name.ads",
            template_dir=template_dir,
        )

    def output_filename(self, input_filename):
        dirname, specific_name, model_name, *ignore = self._split_input_filename(
            input_filename
        )
        if not specific_name:
            build_dir = self._get_default_build_dir()
            a = self.template_basename.rsplit("name", maxsplit=1)
            output_filename = (model_name + "_my_suffix").join(a)
            return dirname + os.sep + build_dir + os.sep + output_filename
        return basic_generator.output_filename(self, input_filename)

    def generate(self, input_filename):
        m = my_model.my_model(input_filename)
        m.load_assembly()  # if assembly submodel
        print(m.render(self.template, template_path=self.template_dir))

Pattern 2: Assembly-Aware Generator (needs assembly context)

When the generator needs to resolve references against the assembly (component names, connector indices, IDs):

def generate(self, input_filename):
    # Load the assembly model
    dirname, view_name, assembly_name, *ignore = self._split_input_filename(
        input_filename
    )
    assembly_model_name = model_loader.get_model_file_path(
        assembly_name, model_types="assembly"
    )
    if not assembly_model_name:
        error.error_print(
            "Could not find a model file for assembly '" + assembly_name + "'."
        )
        error.abort()
    a = assembly.assembly(assembly_model_name)

    # Load the submodel and resolve against assembly
    r = my_model.my_model(input_filename)
    r.resolve_against_assembly(a)
    print(r.render(self.template, template_path=self.template_dir))

Pattern 3: Assembly Generator Extension (extending assembly templates)

For generators that produce assembly-level outputs with component-local templates (e.g., event_to_text):

from generators.assembly import assembly_generator

class my_assembly_output_ads(assembly_generator, generator_base):
    def __init__(self):
        this_file_dir = os.path.dirname(os.path.realpath(__file__))
        template_dir = os.path.join(this_file_dir, ".." + os.sep + "templates")
        assembly_generator.__init__(
            self, "name_my_output.ads", template_dir=template_dir
        )

Pattern 4: Ided Suite Override (custom commands/events/data_products/faults)

When a component needs to generate its own commands, events, data products, or faults that depend on the assembly-level configuration YAML:

from generators.basic import add_basic_generators_to_module
from generators.ided_suite import fault_templates, command_templates, data_product_templates

# Override the standard ided_suite generators with custom model classes
add_basic_generators_to_module(
    my_custom_faults.my_custom_faults, fault_templates, module=globals()
)
add_basic_generators_to_module(
    my_custom_commands.my_custom_commands, command_templates, module=globals()
)

This reuses standard templates (events/name_events.ads, etc.) but with a custom model class that computes different content.

Model Class Patterns

Component Submodel (for component-scoped generators)

Use component_submodel when the generator's YAML is scoped to a single component and needs to verify against that component's connectors, parameters, or other features. The YAML filename uses the component name: component_name.model_type.yaml.

from models.component import component_submodel
from models.exceptions import ModelException

class my_component_config(component_submodel):
    def __init__(self, filename):
        this_file_dir = os.path.dirname(os.path.realpath(__file__))
        schema_dir = os.path.join(this_file_dir, ".." + os.sep + "schemas")
        super(my_component_config, self).__init__(
            filename, schema_dir + "/my_schema.yaml"
        )

    def load(self):
        """Parse YAML data."""
        super(my_component_config, self).load()  # sets self.component = None
        # Parse self.data here...

    def set_component(self, component):
        """Called when the component model is loaded. Validate against component here."""
        super(my_component_config, self).set_component(component)
        # self.component is now set -- verify connectors, features, etc.
        for connector in self.component.connectors:
            # connector.name, connector.kind, connector.type, connector.count
            pass
        # Raise ModelException if validation fails

The generator calls load_component() to trigger the lifecycle:

def generate(self, input_filename):
    m = my_component_config(input_filename)
    m.load_component()  # triggers set_component() callback
    print(m.render(self.template, template_path=self.template_dir))

Do NOT use base with manual model_loader calls for component-scoped generators -- use component_submodel which handles caching, dependency tracking, and the set_component() lifecycle automatically.

Assembly Submodel (most component generators use this)

Assembly submodels are loaded by the assembly model during its load process. The assembly calls set_assembly() on each submodel, giving it access to the full assembly context.

from models.assembly import assembly_submodel
from models.exceptions import ModelException

class my_model(assembly_submodel):
    def __init__(self, filename):
        this_file_dir = os.path.dirname(os.path.realpath(__file__))
        schema_dir = os.path.join(this_file_dir, ".." + os.sep + "schemas")
        super(my_model, self).__init__(
            filename, schema_dir + "/my_schema.yaml"
        )

    def load(self):
        """Called during __init__. Parse YAML data into Python objects."""
        super(my_model, self).load()  # sets self.assembly = None
        self.name = None
        self.description = None

        # Access raw YAML via self.data (dict from pykwalify-validated YAML)
        if self.specific_name:
            self.name = ada.formatType(self.specific_name)
        else:
            self.name = ada.formatType(self.model_name) + "_My_Suffix"

        if "description" in self.data:
            self.description = self.data["description"]

        # Parse YAML entries into model objects
        for entry_data in self.data["entries"]:
            # ... build Python objects from YAML data

    def set_assembly(self, assembly):
        """Called by assembly model during load. Resolve assembly references here."""
        super(my_model, self).set_assembly(assembly)
        # Now self.assembly is set -- resolve component names, connector indices, IDs
        self._resolve_references()

Key Model Base Class Members

Available in all models inheriting from base:

• self.data -- parsed YAML content (dict)
• self.model_name -- extracted from filename (e.g., "linux_example")
• self.model_type -- extracted from filename (e.g., "parameter_table")
• self.specific_name -- optional prefix from filename (or None)
• self.full_filename -- absolute path to YAML file
• self.dependencies -- list of file paths this model depends on
• self.render(template, template_path) -- render Jinja2 template with self.dict

Available in assembly submodels:

• self.assembly -- the parent assembly model (set via set_assembly())
• self.load_assembly() -- explicitly load the assembly (called by generators, not during normal assembly load)

Assembly Model API (for resolving references)

# Get a component by its instance name
comp = self.assembly.get_component_with_name("My_Component_Instance")
# comp.instance_name, comp.name (type name), comp.full_filename
# comp.connectors, comp.commands, comp.parameters, comp.events, comp.faults, comp.data_products

# Get a connector by name
connector = comp.connectors.of_name("Ccsds_Space_Packet_T_Send")

# Get connections on a connector
connections = connector.get_connections()
# Each connection: c.to_component, c.to_connector, or None/"ignore"

Schema Patterns (Pykwalify)

Schemas validate YAML input before model loading:

---
type: map
mapping:
  # Required string field
  instance_name:
    type: str
    required: True
  # Optional description
  description:
    type: str
    required: False
  # Required list with minimum 1 entry
  entries:
    seq:
      - type: map
        mapping:
          name:
            type: str
            required: True
          value:
            type: int
            required: True
          mode:
            type: str
            enum: ['option_a', 'option_b', 'option_c']
            required: False
    range:
      min: 1
    required: True

Template Patterns (Jinja2)

Templates use Jinja2 with access to all model object members via self.__dict__:

-- Generated file. DO NOT EDIT.
{% if description %}
{{ printMultiLine(description, '-- ') }}
{% endif %}
package {{ name }} is

{% for entry in entries %}
   -- Entry for {{ entry.name }}
   {{ entry.name }}_Value : constant := {{ entry.value }};
{% endfor %}

end {{ name }};

Template naming: name in the template filename is replaced with the model name in the output filename. Example: template name.ads with model name linux_example_my_table produces linux_example_my_table.ads.

Ada float literals: YAML float values that are integer-valued (e.g., 85.0 in YAML) may lose their decimal point when formatted in Python (repr(85.0) -> '85.0' but "%.6g" % 85 -> '85'). Ada requires float literals to always have a decimal point (85.0 not 85). Use a helper:

def _to_ada_float(value):
    s = repr(float(value))
    if "." not in s and "e" not in s and "E" not in s:
        s = s + ".0"
    return s

Store the Ada-formatted string in the model object (e.g., self.value_ada = _to_ada_float(self.value)) and reference it in the template.

Template location: Templates go in gen/templates/<template_subdir>/ where <template_subdir> matches the model type or a descriptive name.

Output file placement rules (from basic_generator._get_default_build_dir()):

• .ads, .adb files -> build/src/ (unless name ends in -implementation, -implementation-tester, main, test -> build/template/)
• .tex files -> doc/build/tex/
• .html files -> build/html/
• .yaml files -> build/yaml/
• Other extensions -> build/<extension>/

Utility Functions

from util import ada
ada.formatVariable("my_component")  # -> "My_Component" (Ada variable naming)
ada.formatType("my_type")           # -> "My_Type" (Ada type naming)
ada.isTypePrimitive("Natural")      # -> True

from util import model_loader
model = model_loader.try_load_model_by_name("assembly_name", model_types="assembly")
path = model_loader.get_model_file_path("component_name", model_types="component")

from util import error
error.error_print("message")
error.error_abort("fatal message")

Framework Components with Custom Generators (Reference Catalog)

15 of 59 framework components have custom generators:

Creating a New Generator: Checklist

1. Define the YAML schema in gen/schemas/<model_type>.yaml
   • Use pykwalify format
   • Required fields, types, enums, ranges
2. Create the model class in gen/models/<model_name>.py
   • Inherit from assembly_submodel if assembly context is needed
   • Implement load() to parse self.data
   • Implement set_assembly() if resolving assembly references
   • Add resolved data to self.dependencies for redo tracking
3. Create Jinja2 templates in gen/templates/<template_dir>/
   • name in filename is replaced with model name
   • Access all model members directly in template
4. Create the generator class in gen/generators/<generator_name>.py
   • Inherit from basic_generator and generator_base
   • Point to model class and template
   • Override output_filename() for custom naming
   • Override generate() if assembly loading is needed
5. Create empty __init__.py in gen/
   • REQUIRED for PYTHONPATH discovery
   • set_python_path.sh uses git ls-files to find __init__.py -- files must be committed (not just staged)
   • After committing new gen/__init__.py, run this EXACT sequence to make the generator discoverable:

     # Step 1: Commit the gen/ files on host
     git add src/components/<name>/gen/ && git commit -m "Add generator"
     # Step 2: Sync container git index + rebuild PYTHONPATH (SINGLE COMMAND)
     bash docker/adamant_env.sh exec "cd /home/user/<project> && git read-tree HEAD && rm -f /tmp/.*_env_snapshot"
     # Step 3: Next exec will do full activation with updated PYTHONPATH

   • Do NOT skip git read-tree HEAD -- the container's git index does not auto-sync from host commits due to the bind mount. Without this, git ls-files won't see new files even after commit.
   • Verify discovery: adamant_env.sh exec "echo \$PYTHONPATH | tr ':' '\n' | grep <component>"
6. Create the YAML model file in the assembly directory
   • Named: [specific.]assembly_name.<model_type>.yaml
   • Must be in a directory with .all_path
7. Refresh the environment: Run adamant_env.sh refresh to rebuild the cached environment snapshot (picks up the new __init__.py), then redo clear_cache to invalidate stale generator registry
8. Test: Run redo and verify the generated output appears in build/src/

Common Patterns in Existing Generators

Resolving Component Connector Indices

Many generators need to translate component names into connector output indices (1-based). Pattern from ccsds_router_table:

# Get the component's output connector
connector = component_model.connectors.of_name("Output_T_Send")
connections = connector.get_connections()
connected_components = {}
for idx, c in enumerate(connections):
    if c is not None and c != "ignore":
        connected_components[c.to_component.instance_name] = idx + 1

Generating YAML Type Definitions (Cascading Dependency Chain)

Some generators produce YAML type definitions (records, enums) that feed back into the type system. These go to build/yaml/ and are picked up by the standard type generators.

When a downstream generator (e.g., Ada spec) needs to USE a type produced by an upstream generator (e.g., YAML enum), the downstream generator must:

1. Declare the upstream output in depends_on() so redo builds it first
2. The downstream Ada TEMPLATE must with and use the generated package

Example: enum generator produces build/yaml/<name>_phases.enums.yaml which becomes <Name>_Phases Ada package with Phase_Name.E enum type. The Ada spec template must reference it:

with {{ name }}_Phases; use {{ name }}_Phases;
-- ...
Phase : Phase_Name.E;  -- field using the generated enum

The depends_on() in the Ada spec generator ensures redo builds the enum YAML first, which triggers the standard enum generator to produce the .ads, which is then available for the Ada spec to with:

class my_record_yaml(basic_generator, generator_base):
    def __init__(self):
        # ...
        basic_generator.__init__(
            self,
            model_class=my_model.my_model,
            template_filename="name_record.record.yaml",
            template_dir=template_dir,
        )

Overriding Standard Ided Suite Generators

When a component needs assembly-aware commands/events/faults/data_products, create a custom model that inherits from the standard model but adds assembly resolution:

# In gen/models/my_custom_commands.py
from models.commands import commands

class my_custom_commands(commands):
    def __init__(self, filename):
        # Custom loading that computes commands from assembly config
        ...

Then in the generator, use add_basic_generators_to_module to register the custom model with standard templates.

Error Handling

Use ModelException for user-facing errors in models:

from models.exceptions import ModelException, throw_exception_with_filename

@throw_exception_with_filename
def resolve(self):
    if not self.assembly.get_component_with_name(name):
        raise ModelException(
            'Component "' + name + '" does not exist in assembly "' + self.assembly.name + '".'
        )

The @throw_exception_with_filename decorator adds the YAML filename to error messages. Apply it to methods, NOT classes -- applying it to a class replaces the class with a wrapper function, breaking model_type detection and generator discovery.

Anti-Patterns

• Do NOT use rm -rf on build directories -- use redo clean or redo clean_all
• Do NOT generate files outside build/ -- redo owns the build directory lifecycle
• Do NOT hardcode paths -- use os.path and the model's full_file_dir
• Do NOT forget __init__.py in gen/ -- without it, generators won't be discovered
• Do NOT put __init__.py in gen/generators/, gen/models/, or other subdirs -- they shadow the framework's packages and break all imports
• Do NOT forget adamant_env.sh refresh -- after adding __init__.py, the environment snapshot is stale; adamant_env.sh refresh rebuilds it without restarting the container. Follow with redo clear_cache.
• Do NOT use same model_type string as an existing model -- causes regex collisions
• Do NOT forget to add dependencies -- redo needs self.dependencies for incremental builds
• Do NOT use print() for debugging -- generators capture stdout as output; use sys.stderr.write()
• Model .py changes are NOT auto-tracked by redo -- redo tracks the generator .py file and the input YAML, but NOT the model .py file (loaded dynamically). After modifying model code, run redo clean in the component dir or touch the YAML file to force regeneration
• Do NOT forget git commit before refresh -- set_python_path.sh uses git ls-files which only sees committed (or staged) files. Untracked __init__.py files are invisible to the Python path builder