nic-structure

---
name: nic-structure
description: 'NIC architecture, resource processing pipeline, template systems, and key type definitions. Use when exploring the codebase, understanding data flow, debugging config generation, or working on controller logic.'
---

# NIC Architecture and Structure

## Repository Layout

```text
cmd/nginx-ingress/              Main binary entry point
pkg/apis/configuration/v1/
  types.go                      CRD struct definitions (source of truth)
  zz_generated.deepcopy.go      Auto-generated DeepCopy (never edit)
pkg/apis/configuration/validation/
  policy.go                     ValidatePolicy entry point
  virtualserver.go              VirtualServer/VSR validation
pkg/client/                     Auto-generated typed clients, informers, listers
internal/k8s/
  controller.go                 Informer setup, sync loop, task dispatch
  policy.go                     syncPolicy handler
  handlers.go                   Event handler factories
  configuration.go              In-memory resource state
  secrets/                      Secret store and validation
  policies/policy_refs.go       Policy reference conversion
internal/configs/
  configurator.go               Orchestrator: merge config, render, write, reload
  virtualserver.go              VirtualServer -> version2 config generation
  ingress.go                    Ingress -> version1 config generation
  transportserver.go            TransportServer -> version2 stream config generation
  policy.go                     generatePolicies() dispatcher + add*Config() methods
  annotations.go                Annotation constants + parseAnnotations()
  config_params.go              ConfigParams struct + defaults
  configmaps.go                 ConfigMap -> ConfigParams merge
  dos.go                        DoS protection config generation
  common.go                     Shared config utilities
  warnings.go                   Warning accumulation types
  validation_results.go         validationResults type (isError + warnings)
  commonhelpers/                Shared template helper functions (v1 + v2)
  oidc/                         OIDC config files (openid_connect.js, oidc_common.conf)
  njs/                          NJS scripts (apikey_auth.js)
  version1/                     Ingress template structs + .tmpl files
    __snapshots__/              Snapshot golden files
  version2/                     VirtualServer/TS template structs + .tmpl files
    __snapshots__/              Snapshot golden files
internal/nginx/                 NGINX process manager, reload, rollback, version detection
internal/metrics/               Prometheus metrics collectors and listeners
internal/telemetry/             Usage telemetry collection and export
internal/certmanager/           cert-manager integration controller
internal/externaldns/           ExternalDNS integration controller
charts/nginx-ingress/           Helm chart (values.yaml, schema, templates)
charts/tests/                   Helm snapshot tests (terratest + go-snaps)
tests/suite/                    Python integration tests (pytest)
tests/data/                     Test YAML manifests by feature
config/crd/bases/               Generated CRD YAML (from controller-gen)
deploy/                         Pre-built CRD YAML bundles (crds.yaml, crds-nap-*.yaml)
hack/                           update-codegen.sh, verify-codegen.sh
```

---

## Architectural Layers

Each layer has a strict ownership boundary. Identify the correct layer before placing any change.

| Layer | Package(s) | Owns |
| --- | --- | --- |
| Data model | `pkg/apis/configuration/v1/` | CRD struct definitions, generated DeepCopy |
| Validation | `pkg/apis/configuration/validation/`, `internal/k8s/validation.go` | CRD field validation (kubebuilder markers), Ingress annotation validation |
| Controller | `internal/k8s/` | Event handling, in-memory state, secret resolution, sync handlers, status updates |
| Config generation | `internal/configs/`, `version1/`, `version2/` | Extended resource → NGINX config struct → template render → file write |
| Process management | `internal/nginx/` | NGINX process lifecycle, reload, rollback |

**Layer crossing rules — violations cause architectural drift:**

- Config generation (`internal/configs/`) must NOT call the k8s API or access `SecretStore` directly — it receives pre-resolved role-qualified `SecretReference` via extended resources. Controller-side WAF bundle resolution and the dedicated PLM `KubeClientSecretSource` are explicit exceptions to the normal extended-resource flow.
- Controller (`internal/k8s/`) must NOT generate NGINX config text or render templates.
- Data model (`types.go`) must NOT import `internal/configs` or `internal/k8s`.
- Validation layer must NOT trigger NGINX reloads or update k8s status.

---

## Generated Artifacts — never hand-edit

Every entry below is produced by a command. Regenerate and commit the output after changing the source.

| Artifact | Source | Command | Diffed by CI |
| --- | --- | --- | --- |
| `pkg/apis/**/zz_generated.deepcopy.go`, `pkg/client/**` | `pkg/apis/**/types.go` | `make update-codegen` | yes (`pkg/**`) |
| `config/crd/bases/*.yaml` | kubebuilder markers in `pkg/apis/**` | `make update-crds` | yes |
| `deploy/crds.yaml`, `deploy/crds-nap-*.yaml` | `config/crd/**` via kustomize | `make update-crds` | **no** |
| `docs/crd/*.md` | `config/crd/bases` via `hack/generate-crd-docs.go` | `make update-crds` (runs `update-crd-docs`) | **no** |
| `charts/nginx-ingress/crds` | **symlink** to `config/crd/bases/` | nothing — never edit | n/a |
| `internal/telemetry/*_generated.go`, `data.avdl` | `Data` / `NICResourceCounts` in `internal/telemetry/exporter.go` | `make telemetry-schema` | yes |
| `internal/configs/version1/__snapshots__/**` | `version1/*.tmpl` + fixtures in `template_test.go` | `make test-update-snaps` | via `unit-tests` |
| `internal/configs/version2/__snapshots__/**` | `version2/*.tmpl` + fixtures in `templates_test.go` | `make test-update-snaps` | via `unit-tests` |
| `charts/tests/__snapshots__/**` | chart templates + `charts/tests/testdata/*.yaml` | `make test-update-snaps` | via `unit-tests` |

Two traps:

- `verify-codegen` diffs only `config/crd/bases` after `make update-crds`. Uncommitted `deploy/crds*.yaml` or `docs/crd/` changes pass CI silently.
- Snapshot files only re-record the *existing* fixtures. A template change with no matching fixture produces an empty diff and zero coverage — see `nic-testing` for the required sequence.

---

## Resource Processing Pipeline

```text
kubectl apply -f resource.yaml
  -> K8s API Server persists resource
  -> Informer detects Add/Update/Delete event
      [handlers.go: createXxxHandlers()]
  -> Event handler enqueues task onto syncQueue
      [controller.go: AddSyncQueue()]
  -> Controller dispatches task
      [controller.go: sync() -> syncVirtualServer() / syncIngress() / syncSecret() / syncPolicy() / ...]
  -> Build / update in-memory state, returning []ResourceChange
      [configuration.go: AddOrUpdateVirtualServer() / AddOrUpdateIngress()]
      Validation (CRD fields):          pkg/apis/configuration/validation/
      Validation (Ingress annotations): internal/k8s/validation.go
  -> Find affected resources (fans out when a secret or policy changes)
      [configuration.go: FindResourcesForSecret() / FindResourcesForPolicy()]
  -> Resolve secret references  <-- controller layer resolves; configurator only consumes paths
      [controller.go: createVirtualServerEx() / createIngressEx() and add*SecretRefs() -> secretStore.GetSecret(key, role)]
      On a store miss, the resolver reads the namespace informer cache.
      Valid file-backed roles are materialized under /etc/nginx/secrets on first successful resolution. 
      Later secret updates revalidate and rewrite roles that have already been resolved.
  -> Build extended resources
      [controller.go: createVirtualServerEx()   -> VirtualServerEx]
                     [createIngressEx()          -> IngressEx]
                     [createTransportServerEx()  -> TransportServerEx]
  -> Configurator generates NGINX config  [internal/configs/configurator.go: AddOrUpdateVirtualServer()]
      HTTP path:   GenerateVirtualServerConfig() [virtualserver.go]    -> version2.VirtualServerConfig
                   generateNginxCfg()            [ingress.go]          -> version1.IngressNginxConfig
      Stream path: generateTransportServerConfig(...) [transportserver.go] -> *version2.TransportServerConfig
      Policies:    generatePolicies() -> add*Config() -> policiesCfg   [policy.go]
      OSS vs Plus: Configurator.isPlus flag; Plus-only policies = OIDC, WAF
                   Template level: nginx.virtualserver.tmpl vs nginx-plus.virtualserver.tmpl
  -> Template executor renders NGINX config text
      [version1.TemplateExecutor / version2.TemplateExecutorV2;
       TransportServer uses ExecuteTransportServerTemplate(...)]
  -> NginxManager writes files + reloads NGINX
      [internal/nginx/: Manager.CreateConfig() + Manager.Reload()]
  -> Update resource status + emit Kubernetes events  [happens AFTER reload returns]
      [controller.go: updateVirtualServerStatusAndEvents() / updateIngressStatusAndEvents()]
      [k8s/status.go: statusUpdater.UpdateVirtualServerStatus()]
      Startup optimisation: status updates deferred to pendingVSStatus slices during !isNginxReady;
      flushed in background via flushPendingStatusesAsync() after first reload.
```

---

## Secret Store

The secret store (`internal/k8s/secrets/`) is role-driven. A reference site selects a `SecretRole` for each secret. Kubernetes `Secret.type` does not determine validation, materialization, or reload behavior.

**Phase 1 — reference-gated caching** (`syncSecret()` and `SecretStore.AddOrUpdateSecret()`):
`syncSecret()` finds direct references and Policy references through `policySecretIndex`. Referenced and special Secrets are cached; unreferenced Secrets are evicted. `preSyncSecrets()` temporarily primes the store during startup, and the informer-backed resolver loads newly referenced Secrets on demand.

**Phase 2 — lazy role resolution** (`SecretStore.GetSecret()`):
For an existing Secret, `GetSecret()` validates and caches the result by `(namespace/name, role)`. Valid file-backed roles are materialized under `/etc/nginx/secrets/` using role-specific filenames. Missing lookups return an error reference with the expected path but are not cached.

**Secret roles** (`internal/k8s/secrets/validation.go`):
Kubernetes `Secret.type` is not used for validation, so any type is accepted. The `Opaque` type is recommended, or `kubernetes.io/tls` for TLS secrets. Legacy `nginx.org/*` and `nginx.com/*` types remain accepted.

| Role | Required or Recognized Keys | Used for |
| --- | --- | --- |
| `RoleTLS` | `tls.crt`, `tls.key` | TLS server certs |
| `RoleCA` | `ca.crt`; optional `ca.crl` | CA cert (mTLS / upstream trust) |
| `RoleJWK` | `jwk` | JWT validation keys |
| `RoleHtpasswd` | `htpasswd` | HTTP Basic auth |
| `RoleOIDC` | `client-secret` | OIDC client secret |
| `RoleAPIKey` | Client IDs and credentials | API key auth |
| `RoleLicense` | `license.jwt` | NGINX Plus license |
| `RoleWAFBundle` | `token`, or `username` and `password`; optional `ca.crt` | Bundle-fetch credentials |

**Special secrets** Default-server TLS, wildcard TLS, license, management client certificate, and management trusted CA are selected by configured references. One Secret can satisfy multiple roles; NIC validates and writes every applicable representation and performs the strongest required reload once.

**Key invariant**: Extended resources carry `map[secrets.SecretRefKey]*secrets.SecretReference`, keyed by namespaced Secret and role. Standard config generation may consume `Path`, `CRLPath`, `Error`, and role-specific data, but must not inspect `Secret.type` or call `SecretStore.GetSecret()`.

---

## Two Template Systems

| Pipeline | Resources | Package | Templates |
| --- | --- | --- | --- |
| Version 1 | Ingress | `internal/configs/version1/` | `nginx.ingress.tmpl`, `nginx-plus.ingress.tmpl` |
| Version 2 | VirtualServer, VSR, TS | `internal/configs/version2/` | `nginx.virtualserver.tmpl`, `nginx-plus.virtualserver.tmpl` |

- Version 1: `IngressNginxConfig` with multiple `Server` blocks per config
- Version 2: `VirtualServerConfig` with single `Server` block per config
- Main templates (`nginx.tmpl`, `nginx-plus.tmpl`) produce global `nginx.conf`
- Both share `generatePolicies()` in `internal/configs/policy.go`

---

## Policy System

Policies are mutually exclusive: each Policy CR has exactly ONE non-nil field in `PolicySpec`.

**Types**: AccessControl, RateLimit, JWTAuth, ExternalAuth, BasicAuth, IngressMTLS, EgressMTLS, OIDC, WAF, APIKey, Cache, CORS.

**Application levels (VirtualServer)**:

- `spec.policies` -- server-level (all routes unless overridden)
- `route.policies` -- route-level (overrides spec-level)
- `subroute.policies` -- VirtualServerRoute subroute-level

**Ingress**: Policies referenced via `IngressEx.Policies` map. Annotations are Ingress-only, never on VS/VSR.

---

## Key Types

**`policiesCfg`** (`internal/configs/policy.go`): Aggregation struct holding resolved policies per context (Allow/Deny slices, RateLimit, JWTAuth, ExternalAuth, BasicAuth, IngressMTLS, EgressMTLS, OIDC, APIKey, WAF, Cache, CORSHeaders/CORSMap, Context, BundleValidator, ErrorReturn).

**`version2.VirtualServerConfig`**: Top-level struct with HTTP-level directives (Maps, LimitReqZones, CacheZones) and a single Server block.

**`version2.Location`**: Per-route struct with all policy fields (Allow, Deny, LimitReqs, JWTAuth, Cache, CORSEnabled, AddHeaders).

**`version1.IngressNginxConfig`**: Top-level Ingress struct with multiple Server blocks plus Maps, CORSHeaders, LimitReqZones.

**`ConfigParams`** (`config_params.go`): ~125 fields for tunable NGINX params. Flow: defaults -> ConfigMap -> Ingress annotations.

### CRD Struct Pattern

```go
// +kubebuilder:resource:shortName=pol
// +kubebuilder:subresource:status
// +kubebuilder:storageversion
type Policy struct {
    metav1.TypeMeta   `json:",inline"`
    metav1.ObjectMeta `json:"metadata"`
    Spec              PolicySpec   `json:"spec"`
    Status            PolicyStatus `json:"status"`
}
```

- Types: PascalCase singular. Spec/Status: `<CRD>Spec`, `<CRD>Status`. Lists: `<CRD>List`.
- Short names: `vs`, `vsr`, `ts`, `gc`, `pol`. API group: `k8s.nginx.org/v1`.

### Kubebuilder Markers

| Marker | Purpose |
| --- | --- |
| `+kubebuilder:validation:Required` | Field must be present |
| `+kubebuilder:validation:Optional` | Field is optional |
| `+kubebuilder:validation:Pattern=` `` `regex` `` | Regex validation |
| `+kubebuilder:validation:Minimum=N` | Numeric minimum |
| `+kubebuilder:default=value` | Default value |
| `+kubebuilder:validation:XValidation:rule="CEL"` | Cross-field CEL validation |

### Error Handling

- **Warnings**: `map[runtime.Object][]string` in `internal/configs/warnings.go`
- **validationResults**: `isError bool` + `warnings []string`. When `isError = true`, policy dispatcher returns `ErrorReturn: {Code: 500}`
- **Validation errors**: Kubernetes `field.ErrorList` from `k8s.io/apimachinery/pkg/util/validation/field`

More agent context in nginx/kubernetes-ingress

11 other files this repository gives its agents.