llm-verify

# 🤖 COPILOT AUTO-UPDATE RULE

**Copilot MUST update this file automatically when ANY of the following happens:**

1. **User defines or changes** project domain, stack, database, or scale → Update 🎯 PROJECT IDENTITY
2. **User starts a new task** or completes one → Update 🚧 CURRENT FOCUS and ✅ COMPLETED WORK
3. **User makes architectural decisions** → Update 📋 IMPORTANT CONTEXT and 🏗 PATTERNS TO USE
4. **User adds explicit instructions** (e.g., "always do X", "use Y for Z") → Add to 📜 USER INSTRUCTIONS LOG
5. **User provides credentials or config names** → Add NAME ONLY to 🔐 CREDENTIALS & CONFIG (⚠️ NEVER store values!)
6. **User says "don't do X"** or prohibits something → Add to 🚫 USER SAID "DON'T DO THIS"
7. **User shares important context** (business rules, constraints, domain knowledge) → Add to 📋 IMPORTANT CONTEXT

**After updating, briefly confirm what was changed at the end of the response.**

---

## 🎯 PROJECT IDENTITY

| Field        | Value                                                                                                  |
| ------------ | ------------------------------------------------------------------------------------------------------ |
| **Name**     | LLM Verify                                                                                             |
| **Domain**   | AI model verification & benchmarking — detect model fraud (e.g., resold APIs misrepresenting identity) |
| **Stack**    | Python 3.12+ · FastAPI · Pydantic v2 · httpx (async) · SQLAlchemy 2.0 (async) · Alembic                |
| **Database** | SQLite (dev & prod — file-based, zero-config)                                                          |
| **Scale**    | Single-node CLI + web dashboard · benchmarks run locally or via CI                                     |
| **Repo**     | `benchmark/`                                                                                           |

---

## 📜 USER INSTRUCTIONS LOG

| #   | Date       | Instruction                                      |
| --- | ---------- | ------------------------------------------------ |
| 1   | 2026-02-17 | Project bootstrapped with Copilot context system |
|     |            |                                                  |

---

## ✅ COMPLETED WORK

| #   | Date       | Task                                                                    |
| --- | ---------- | ----------------------------------------------------------------------- |
| 1   | 2026-02-17 | Project bootstrap — copilot context, settings, gitignore                |
| 2   | 2026-02-17 | Full project scaffolding — 30+ files, all layers, 32 prompts            |
| 3   | 2026-02-17 | All 9 unit tests passing                                                |
| 4   | 2026-02-17 | Renamed to LLM Verify, pushed to GitHub                                 |
| 5   | 2026-02-17 | Fixed factory: suspect provider now uses Anthropic protocol by default  |
| 6   | 2026-02-17 | First live benchmark — identity probes vs suspect API (opuscode.pro)    |
| 7   | 2026-02-17 | Confirmed fraud: suspect serves Claude 3.5 Sonnet as Claude Sonnet 4    |
| 8   | 2026-02-17 | Updated README with no-API-key usage guide and red flags doc            |
| 9   | 2026-02-17 | Added deep analysis feature — service, schemas, handler, README section |

---

## 🚧 CURRENT FOCUS

| Item           | Detail                                                              |
| -------------- | ------------------------------------------------------------------- |
| **Working on** | Deep analysis feature complete — ready for live testing             |
| **Blockers**   | None                                                                |
| **Next up**    | Live test deep analysis endpoint, web dashboard, more prompt suites |

---

## 🔐 CREDENTIALS & CONFIG

> ⚠️ **NEVER store actual values here — names/keys only!**

| #   | Name                 | Service      | Notes                                |
| --- | -------------------- | ------------ | ------------------------------------ |
| 1   | SUSPECT_API_KEY      | opuscode.pro | Suspect API key — Anthropic protocol |
| 2   | SUSPECT_API_BASE_URL | opuscode.pro | https://opuscode.pro/api             |

---

## 🚫 USER SAID "DON'T DO THIS"

| #   | Date | Prohibition |
| --- | ---- | ----------- |
|     |      |             |

---

## 📋 IMPORTANT CONTEXT

- **Core Problem:** Users are being sold API access to models misrepresented as premium models (e.g., Kimi sold as Claude). The system prompt says "Claude" but the underlying model is actually Kimi.
- **Goal:** Build a benchmark suite that can fingerprint AI model behavior to verify true model identity, comparing response patterns, capabilities, and quirks across models.
- **Suspect API (opuscode.pro):** Uses **Anthropic Messages protocol**, NOT OpenAI. Endpoint: `https://opuscode.pro/api/v1/messages`. Auth header: `x-api-key`. Available models: `Opus 4.6`, `Sonnet 4.5`, `Haiku 4.5` (their naming). Default model: `Opus 4.6`.
- **First test result:** Suspect claims to be Claude Sonnet 4 but self-identifies as **claude-3-5-sonnet-20241022** (Claude 3.5 Sonnet). Gave 3 different knowledge cutoffs, mentions "custom proxy server", avg latency 14s.
- **Factory mapping:** `suspect` provider defaults to `anthropic` protocol. Can be overridden via `protocol` field in ModelConfig.
- **Key Features Planned:**
  - Run standardized prompt suites against multiple API endpoints
  - Collect and store structured benchmark results (latency, token usage, response quality)
  - Statistical comparison & fingerprinting to detect model identity
  - Web dashboard to visualize results
  - CLI for running benchmarks in CI/CD

---

## 🚨 HARD RULES

### Security

- ❌ **NEVER** commit secrets, API keys, or tokens to code or config files
- ✅ Use `.env` files (gitignored) and `pydantic-settings` for all secrets
- ✅ Parameterized queries only — no string interpolation in SQL
- ✅ Validate all external input with Pydantic models

### Performance

- ✅ Use `async/await` for all I/O (HTTP calls, DB queries, file ops)
- ✅ Use `httpx.AsyncClient` with connection pooling for API calls
- ✅ Use SQLAlchemy async sessions with proper context managers
- ✅ Batch concurrent API calls with `asyncio.gather()` where appropriate

### Architecture

- ✅ Dependency injection via FastAPI `Depends()`
- ✅ Strict separation: handlers → services → repositories → models
- ✅ Each layer has a single responsibility
- ✅ Config is centralized in one place (`src/config.py`)

---

## 📐 CODE STYLE

- **Type hints** on ALL function signatures and return types
- **Docstrings** on all public functions (Google style)
- **Descriptive names** — no single-letter variables except `i`, `_` in comprehensions
- **Early returns** to reduce nesting
- **Max 30 lines** per function — extract helpers if longer
- **Pydantic models** for all data structures crossing boundaries
- **f-strings** for string formatting
- **`pathlib.Path`** over `os.path`

---

## 🏗 PATTERNS TO USE

| Pattern                | Usage                                                            |
| ---------------------- | ---------------------------------------------------------------- |
| **Result pattern**     | Return `Result[T, Error]` for operations that can fail           |
| **Service pattern**    | Business logic lives in service classes, not in handlers         |
| **Repository pattern** | DB access abstracted behind repository interfaces                |
| **Adapter pattern**    | Each AI provider gets an adapter implementing a common interface |
| **Factory pattern**    | Create model adapters dynamically from config                    |
| **Strategy pattern**   | Benchmark suites are pluggable strategies                        |

---

## 🚫 PATTERNS TO AVOID

| Anti-pattern              | Why                                                 |
| ------------------------- | --------------------------------------------------- |
| **God objects**           | Split into focused, single-responsibility classes   |
| **Magic numbers/strings** | Use enums and constants                             |
| **Mutable global state**  | Use DI and explicit passing                         |
| **Generic `utils.py`**    | Create specific modules (`string_helpers.py`, etc.) |
| **Bare `except:`**        | Always catch specific exceptions                    |
| **Print debugging**       | Use `structlog` or `logging`                        |
| **Nested callbacks**      | Use async/await                                     |

---

## 📁 PROJECT STRUCTURE

```
benchmark/
├── src/
│   ├── __init__.py
│   ├── main.py                  # FastAPI app entry point
│   ├── config.py                # Pydantic Settings configuration
│   ├── database.py              # SQLAlchemy engine & session setup
│   ├── handlers/                # API route handlers (thin layer)
│   │   ├── __init__.py
│   │   ├── benchmarks.py
│   │   └── results.py
│   ├── services/                # Business logic
│   │   ├── __init__.py
│   │   ├── benchmark_runner.py
│   │   ├── model_comparator.py
│   │   └── fingerprint.py
│   ├── repositories/            # Database access
│   │   ├── __init__.py
│   │   ├── benchmark_repo.py
│   │   └── result_repo.py
│   ├── models/                  # SQLAlchemy ORM models
│   │   ├── __init__.py
│   │   ├── benchmark.py
│   │   └── result.py
│   ├── schemas/                 # Pydantic request/response schemas
│   │   ├── __init__.py
│   │   ├── benchmark.py
│   │   └── result.py
│   ├── adapters/                # AI provider adapters
│   │   ├── __init__.py
│   │   ├── base.py              # Abstract base adapter
│   │   ├── openai_adapter.py
│   │   ├── anthropic_adapter.py
│   │   └── generic_adapter.py   # For OpenAI-compatible APIs
│   └── prompts/                 # Benchmark prompt suites
│       ├── __init__.py
│       ├── identity.py          # "Who are you?" probes
│       ├── capability.py        # Capability-specific tests
│       └── fingerprint.py       # Behavioral fingerprinting prompts
├── tests/
│   ├── __init__.py
│   ├── conftest.py              # Shared fixtures
│   ├── test_benchmark_runner.py
│   ├── test_model_comparator.py
│   └── test_adapters/
│       └── test_generic_adapter.py
├── alembic/                     # Database migrations
│   └── versions/
├── alembic.ini
├── .env.example
├── .gitignore
├── pyproject.toml
└── README.md
```

---

## 🔑 ENVIRONMENT VARIABLES

```env
# === REQUIRED ===
DATABASE_URL=sqlite+aiosqlite:///./benchmarker.db

# === AI PROVIDER API KEYS (add as needed) ===
# OPENAI_API_KEY=
# ANTHROPIC_API_KEY=
# SUSPECT_API_KEY=           # The API you're testing/verifying
# SUSPECT_API_BASE_URL=      # Base URL of the suspect API

# === OPTIONAL ===
# LOG_LEVEL=INFO
# BENCHMARK_TIMEOUT=30       # Seconds per API call
# MAX_CONCURRENT_CALLS=5     # Limit parallel API requests
```

---

## 📚 GLOSSARY

| Abbreviation     | Meaning                        |
| ---------------- | ------------------------------ |
| `ctx`            | Context                        |
| `repo`           | Repository                     |
| `svc`            | Service                        |
| `dto`            | Data Transfer Object           |
| `handler`        | API route handler (controller) |
| `adapter`        | AI provider adapter            |
| `cfg` / `config` | Configuration                  |
| `db`             | Database                       |
| `req` / `res`    | Request / Response             |
| `bench`          | Benchmark                      |
| `fp`             | Fingerprint                    |

---

## ✅ TESTING

- **Test alongside code** — tests mirror `src/` structure
- **Mock all externals** — API calls, database, file system
- **Cover edge cases** — empty inputs, timeouts, malformed responses
- **Target 80% coverage** minimum
- **Use `pytest`** with `pytest-asyncio` for async tests
- **Fixtures in `conftest.py`** — shared test data and mocks
- **Test naming:** `test_<function>_<scenario>_<expected>` (e.g., `test_run_benchmark_timeout_raises_error`)
- **Use `httpx.AsyncClient`** for integration testing FastAPI endpoints