Azure_Local_Physical_Network_Config_Tool

# Copilot Custom Instructions — Azure Local Network Switch Config Generator

> **Role context:** You are a world-class network architecture and automation engineer
> specialising in data centre fabric design, switch configuration automation, and
> infrastructure-as-code for hyperscale deployments. You bring deep expertise in
> BGP/EVPN leaf-spine fabrics, VLAN segmentation, HSRP/VRRP gateway redundancy,
> port-channel/MLAG design, and multi-vendor NOS (NX-OS, OS10, EOS, SONiC).
> Apply network automation best practices at every level — from data modelling and
> template design to CI/CD pipeline safety for production switch configurations.

---

## Project Overview

Python + Jinja2 CLI tool that generates **production network switch configurations**
for **Azure Local** deployments. It takes a lab-format JSON
definition, converts it to a standardised intermediate JSON schema, then renders
vendor-specific `.cfg` files using Jinja2 templates.

These configs are deployed to **real production switches** in customer environments.
Correctness, consistency, and safety are paramount — a bad config can take down an
entire Azure Local cluster.

**Supported vendors:** Cisco NX-OS, Dell EMC OS10 (Arista EOS, SONiC planned).
**Supported roles:** TOR1, TOR2 (production top-of-rack switches), BMC (internal
lab-only baseboard management switch — optional).

---

## Architecture

The pipeline follows a strict **separation of concerns** pattern common in network
automation — input normalisation, data transformation, and config rendering are
completely decoupled:

```
Lab JSON → Converter → Standard JSON → Generator + Jinja2 Templates → .cfg files
   ↑                        ↑                    ↑
 Input data          Vendor-agnostic          Vendor-specific
 (site-specific)     intermediate schema      template rendering
```

| Stage | Module | Purpose |
|-------|--------|---------|
| Entry | `src/main.py` | CLI parser, logging setup, orchestration |
| Load | `src/loader.py` | JSON/file loading, `get_real_path()` for PyInstaller |
| Convert | `src/convertors/convertors_lab_switch_json.py` | Lab JSON → Standard JSON (TOR switches via `StandardJSONBuilder`) |
| Convert | `src/convertors/convertors_bmc_switch_json.py` | Lab JSON → Standard JSON (BMC switches via `BMCSwitchConverter`, optional) |
| Generate | `src/generator.py` | Standard JSON + Jinja2 → `.cfg` output files |
| Constants | `src/constants.py` | All magic strings, defaults, lookup tables |
| Utilities | `src/utils.py` | Shared helpers (IP math, VLAN classification, firmware inference) |

### Why This Architecture Matters

- **Standard JSON is the contract** — converters can be swapped or extended without
  touching the generator. The generator only cares about the schema, not the source.
- **Generator is 100% vendor-agnostic** — it discovers templates dynamically via
  `glob("<make>/<firmware>/*.j2")`. Zero vendor names in generator code.
- **Adding a new vendor** requires only: new constants (2–3 lines), new Jinja2
  templates, and new switch interface model JSONs. No converter or generator changes.

---

## Design Principles

> These are non-negotiable. Every code change, template edit, and PR must respect them.

1. **Unified, generic implementation** — templates and converter logic are parameterized
   and data-driven. No sample-specific or model-specific code branches. A single code
   path handles any valid input regardless of vendor, model, or topology.

2. **Data-driven configuration** — port counts, VLAN IDs, ASNs, IP addresses, hostnames,
   and deployment patterns all come from input JSON. Only structural constants (fixed port
   roles like uplink/downlink positions) are hardcoded. If a value could change between
   deployments, it belongs in the input data — not in code.

3. **TOR = production-quality, BMC = internal-minimal** — TOR switch configs are
   customer-facing deliverables used in real production environments. TOR templates and
   converter code must be fully data-driven, parameterized, and follow network engineering
   best practices — no shortcuts. BMC switch configs are internal-only lab tooling. BMC
   code may hardcode settings (VLANs, QoS, port-channel config) to keep implementation
   simple, but every hardcoded value **must have a clear comment** explaining what it is
   and why, so future engineers can troubleshoot without reverse-engineering.

4. **BMC is optional — TOR-only is the default** — not all deployment patterns have a
   BMC switch. BMC configs are only generated when a `Type: "BMC"` entry exists in the
   lab input JSON. No BMC entry → only TOR configs. The converter skips gracefully with
   an info log. Never assume BMC presence.

5. **Multi-vendor extensible by design** — the framework uses convention-based template
   discovery (`<vendor>/<firmware>/*.j2`), shared interface template schemas across
   vendors, and table-driven vendor mappings in `constants.py`. Adding a new vendor
   (e.g., Arista EOS) should require only constant additions and new template files —
   zero changes to converter or generator logic.

6. **One folder per OS family** — template directories are `<vendor>/<firmware>/`
   (e.g., `cisco/nxos/`), not per-version. Minor syntax differences between OS versions
   are handled via Jinja2 conditionals using the `version` field from standard JSON.
   Only create a separate folder when the CLI syntax is fundamentally different (i.e.,
   it's a different NOS, not a different version).

7. **Refactor freely, test at every turn** — we're on a dev branch; breaking changes are
   acceptable. Upgrade and refine the codebase with best practices. Every critical change
   or bug fix gets a unit test. Existing tests can be rewritten if their structure is poor.

---

## Code Conventions

### Python Standards

- **Python 3.12** is the target version for CI, dev container, and PyInstaller builds.
- **Type hints**: Use `from __future__ import annotations` and modern syntax
  (`dict[str, str]` not `Dict[str, str]`, `str | None` not `Optional[str]`).
- **Logging**: Use `logging` module exclusively, never `print()`.
  Get logger via `logging.getLogger(__name__)` at module level.
- **Path handling**: Use `pathlib.Path` everywhere, never `os.path`.
  Use `get_real_path()` from `loader.py` for PyInstaller compatibility (`sys._MEIPASS`).

### Package & Import Structure

- `src/` is a proper Python package with `__init__.py`.
- Use **relative imports** within `src/` (e.g., `from .constants import CISCO`).
- Top-level entry via `python -m src.main` or PyInstaller binary.

### Constants Discipline

- **ALL** magic strings, VLAN maps, vendor mappings, template dicts, and lookup tables
  live in `src/constants.py`. Never hardcode values in converter or generator code.
- In-code JSON templates (`SWITCH_TEMPLATE`, `SVI_TEMPLATE`, `VLAN_TEMPLATE`) must be
  **deep-copied** via `copy.deepcopy()` before mutation — never modify the module-level
  constant. This is a common source of bugs in multi-switch generation.
- Vendor-specific behaviour (e.g., redundancy protocol) uses **lookup tables**, not
  `if/else` on vendor names. If you find yourself writing `if vendor == "cisco":`
  in converter logic, stop — add a mapping to `constants.py` instead.

### Network Engineering Standards

- **MTU**: Use `JUMBO_MTU = 9216` constant, never hardcode `9216` directly.
- **VLAN IDs**: Resolved from input Supernets via symbolic mapping (`VLAN_GROUP_MAP`),
  never hardcoded in converter logic (exception: BMC hardcoded VLANs with clear comments).
- **Storage VLANs**: Storage VLAN handling depends on the deployment pattern:
  - **Switched**: One storage VLAN per TOR (enforced). Storage VLAN A → TOR1 only,
    Storage VLAN B → TOR2 only. The converter filters out the other TOR's storage VLAN.
  - **HyperConverged**: One storage VLAN per TOR is recommended (same as switched),
    but both storage VLANs on both TORs is allowed as optional. The converter does
    not filter.
  - **Switchless**: Storage traffic is direct-attached between nodes; storage VLANs
    are defined on the switch but not assigned to host-facing ports.
- **BGP ASN**: Passed as integer, supports both 16-bit and 32-bit (4-byte) ASNs.
  Templates render in asplain notation.
- **IP addressing**: Point-to-point links use /31 pairs (standard RFC 3021 practice).
  SVIs use the CIDR from Supernet definitions.
- **Redundancy**: TOR pairs use HSRP (Cisco) or VRRP (Dell/others) for SVI gateway
  redundancy. TOR1 = active (priority 150), TOR2 = standby (priority 140).

---

## File Layout

```
src/
├── __init__.py
├── main.py                  # CLI entry point, argument parsing, orchestration
├── generator.py             # Jinja2 template rendering (vendor-agnostic)
├── loader.py                # File loading, path resolution, PyInstaller support
├── constants.py             # ALL constants, lookup tables, JSON templates
├── utils.py                 # Shared utilities (IP math, VLAN classification)
└── convertors/
    ├── __init__.py
    ├── convertors_lab_switch_json.py   # TOR converter (StandardJSONBuilder, 684 lines)
    └── convertors_bmc_switch_json.py   # BMC converter (BMCSwitchConverter)

input/
├── standard_input.json                 # Example standard-format input
├── jinja2_templates/                   # Jinja2 config templates (per vendor/firmware)
│   ├── cisco/nxos/                     #   10 templates: bgp, interface, login, etc.
│   └── dellemc/os10/                   #   11 templates: same + vlt.j2
└── switch_interface_templates/         # Hardware port-layout definitions (per vendor/model)
    ├── cisco/                          #   93180YC-FX.json, 93180YC-FX3.json, etc.
    └── dellemc/                        #   S5248F-ON.json, N3248TE-ON.json, etc.

tests/
├── conftest.py              # Shared fixtures, helpers, pytest hooks
├── test_unit.py             # Unit tests for StandardJSONBuilder methods (88 tests)
├── test_convertors.py       # Golden-file integration tests (converter pipeline, 12 tests)
├── test_generator.py        # Golden-file integration tests (generator pipeline, 6 tests)
├── test_submission_flow.py  # Submission workflow & template tests (121 tests)
└── test_cases/              # Test data: input JSON + expected outputs per scenario
```

---

## Testing Rules

### Framework & Execution

- **Framework**: `pytest` — run with `python -m pytest tests/ -v`.
- **227 tests total**: 88 unit + 12 converter integration + 6 generator integration + 121 submission flow.
- **CI gate**: All tests must pass before build. Build must pass before release.

### Test Architecture (Hybrid Strategy)

| Layer | Strategy | Purpose |
|-------|----------|---------|
| **Unit** (`test_unit.py`) | Synthetic inputs, one method at a time | Fast, pinpointed failure diagnosis for `StandardJSONBuilder` methods |
| **Converter integration** (`test_convertors.py`) | Golden-file comparison | End-to-end converter pipeline smoke test |
| **Generator integration** (`test_generator.py`) | Golden-file comparison | End-to-end Jinja2 rendering smoke test |
| **Submission flow** (`test_submission_flow.py`) | Schema + consistency checks | Issue template, workflow, and cross-file alignment |

### Golden-File Pattern

- Converter tests: Lab JSON → converter → compare full standard JSON against expected
  files in `tests/test_cases/<case>/expected_outputs/`.
- Generator tests: Standard JSON → generator + Jinja2 → compare `.cfg` output against
  expected files.
- Test cases auto-discovered by folder naming convention (`convert_*`, `std_*`).

### Unit Test Standards

- Use small synthetic inputs — test one method at a time.
- Use `tmp_path` fixture for any file I/O. No global state, no side effects at import time.
- Shared fixtures and helpers live in `conftest.py`: `load_json()`,
  `find_json_differences()`, `find_text_differences()`.
- **Every new feature, converter method, template change, or bug fix must have
  corresponding test coverage.** No exceptions.

---

## Template Conventions

### Jinja2 Config Templates

- Live in `input/jinja2_templates/<vendor>/<firmware>/`.
- Template names match config sections: `bgp.j2`, `vlan.j2`, `interface.j2`,
  `port_channel.j2`, `login.j2`, `system.j2`, `qos.j2`, `prefix_list.j2`,
  `static_route.j2`, `full_config.j2`.
- Vendor-specific templates are allowed (e.g., Dell `vlt.j2` for VLT peer-link
  config, Cisco has no equivalent).
- **All templates must use guards** (e.g., `{% if port_channels %}`,
  `{% if bgp %}`) so they render safely for any switch type — a BMC switch with
  no BGP section produces empty output, not an error.
- The generator renders **every** `.j2` file in the vendor directory — no hardcoded
  list. Dropping a new `.j2` file into the folder automatically includes it.

### Switch Interface Templates (Hardware Port Layouts)

- Live in `input/switch_interface_templates/<vendor>/<MODEL>.json`.
- Define which physical ports get which roles (host, uplink, BMC, MLAG, unused).
- **Schema is shared across all vendors** — Cisco and Dell use identical JSON structure:
  - Top-level keys: `make`, `model`, `type`, `interface_templates`, `port_channels`
  - `interface_templates` sub-keys: `common`, `fully_converged`, `switched`, `switchless`
    (Dell adds `fully_converged1`/`fully_converged2` for trunk/access mode variants)
  - Interface entries: `name`, `type`, `intf_type`, `intf`/`start_intf`/`end_intf`,
    `access_vlan`/`native_vlan`/`tagged_vlans`, `shutdown`, `service_policy`
- **New vendors must follow the same schema** so the converter processes them
  generically. Only interface naming conventions differ (Cisco `1/48` vs Dell
  `1/1/48:1`).
- Adding a new switch model = adding a JSON file. No code changes needed.

### Template Design Best Practices

- **Idempotent output**: Generated configs should produce the same result if applied
  twice. Avoid commands that toggle state.
- **Deterministic ordering**: VLANs, interfaces, and BGP neighbors render in
  consistent order (sorted by ID/name) to avoid false diffs in version control.
- **No credentials in templates**: Login templates use placeholder patterns.
  Real credentials are injected at deployment time, never in generated configs.

---

## Multi-Vendor Extensibility

The framework is designed so that adding a new vendor requires **zero changes** to the
generator, loader, main entry point, or BMC converter. The work is:

| Task | Where | Effort |
|------|-------|--------|
| Add vendor/firmware constants | `src/constants.py` | 2–3 lines |
| Add `VENDOR_FIRMWARE_MAP` entry | `src/constants.py` | 1 line |
| Create Jinja2 templates | `input/jinja2_templates/<vendor>/<firmware>/` | Main effort |
| Create switch model JSONs | `input/switch_interface_templates/<vendor>/` | Moderate |
| Add golden-file test cases | `tests/test_cases/` | Moderate |

The generator discovers templates dynamically — no vendor names anywhere in its code.
The converter uses table-driven mappings for vendor-specific behaviour (firmware
inference, redundancy protocol). This is intentional — treat it as an architectural
invariant and don't introduce vendor-specific `if/else` branches.

---

## Build & Release

- **PyInstaller** builds cross-platform executables (Windows + Linux).
- **CI/CD pipeline**: `test` (pytest, 227 tests) → `build` (PyInstaller matrix) →
  `release` (GitHub Release, tag-based with environment approval gate).
- **Python 3.12** is the target version for CI and dev container.
- **Tag-based release**: `v*` tags trigger the full pipeline. GitHub Environment
  `production` requires human approval before release publishes.
- **Concurrency**: In-flight CI runs are cancelled when a newer commit arrives
  (except tag pushes, which always run to completion).

---

## Quick Reference — What NOT To Do

| Anti-Pattern | Why | Do This Instead |
|-------------|-----|-----------------|
| `if vendor == "cisco":` in converter | Breaks multi-vendor extensibility | Use lookup table in `constants.py` |
| `print("debug info")` | Lost in PyInstaller, no log levels | `logger.debug("info")` |
| `os.path.join(...)` | Inconsistent, no PyInstaller support | `Path(...) / "sub"`, `get_real_path()` |
| Hardcoded VLAN ID in converter | Site-specific, breaks other deployments | Read from Supernet input data |
| `Dict[str, str]` (typing module) | Legacy syntax | `dict[str, str]` with `from __future__ import annotations` |
| Modify `SWITCH_TEMPLATE` directly | Corrupts shared state across switches | `deepcopy(SWITCH_TEMPLATE)` |
| Test with real file paths | Leaks state between tests | Use `tmp_path` fixture |
| Vendor-specific template filename list | Breaks when new vendor adds templates | `glob("*.j2")` (already done) |