CLAUDE.md

This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.

Project Overview

langsmith-cli is a context-efficient CLI tool for LangSmith that serves as both a standalone developer tool and a Claude Code plugin. It replaces heavy MCP servers with lightweight, on-demand skills while providing 100% feature parity plus superior features.

To understand how to use this tool, read the skill definition at skills/langsmith/SKILL.md and its reference files under skills/langsmith/references/. The skill teaches agents how to use the CLI efficiently (cache-first workflow, mandatory flags, field pruning, etc.).

Key Design Philosophy:

• Lazy Loading: Heavy imports (langsmith, rich) only load when commands execute
• Context Efficiency: Field pruning (--fields) reduces token usage by ~90%
• Dual UX: Rich tables for humans, strict JSON for agents
• Type Safety: Strict SDK contracts, no stringly-typed logic (in progress)

Development Commands

Environment Setup

# Install dependencies
uv sync

# Install pre-commit hooks
uv run pre-commit install

Testing

# Run all tests (parallel by default with -n auto)
uv run pytest

# Run with coverage (serial is faster for coverage)
uv run pytest --cov=src --cov-report=term-missing -n 0

# Run specific test file
uv run pytest tests/test_runs.py -v

# Run specific test
uv run pytest tests/test_runs.py::test_runs_list -v

# Run serially (disable parallelization)
uv run pytest -n 0

# Run smoke tests (requires LANGSMITH_API_KEY)
# Smoke tests use in-process CLI invocation and session fixtures for speed
export LANGSMITH_API_KEY="lsv2_..."
uv run pytest tests/test_smoke.py -v

# Run E2E tests (requires LANGSMITH_API_KEY)
uv run pytest tests/test_e2e.py -v

Code Quality

# Linting and formatting (auto-fix)
uv run ruff check --fix
uv run ruff format

# Type checking
uv run pyright

# Run all pre-commit hooks manually
uv run pre-commit run --all-files

Running the CLI

# Run locally during development
uv run langsmith-cli --help
uv run langsmith-cli runs list --project default

# Run with JSON output (agent mode)
uv run langsmith-cli --json runs list --limit 5

# Verbosity control (following industry standards: pip, Black, etc.)
uv run langsmith-cli runs list               # Default: INFO (progress + warnings)
uv run langsmith-cli -q runs list            # Quiet: WARNING (warnings only)
uv run langsmith-cli -qq runs list           # Silent: ERROR (errors only)
uv run langsmith-cli -v runs list            # Debug: DEBUG (debug + progress + warnings)
uv run langsmith-cli -vv runs list           # Trace: TRACE (ultra-verbose)

# JSON mode with clean stdout/stderr separation
uv run langsmith-cli --json runs list        # Progress to stderr, JSON to stdout
uv run langsmith-cli --json -qq runs list    # Silent mode, clean JSON only
uv run langsmith-cli --json runs list 2>/dev/null  # Suppress diagnostics

Authentication

# Set up API key (creates/updates .env file)
uv run langsmith-cli auth login

# Or manually create .env from template
cp .env.example .env
# Edit .env and add your LANGSMITH_API_KEY

Architecture

Command Structure

The CLI uses Click with a modular command group architecture:

main.py (entry point)
├── @cli (root group with --json flag)
├── auth (group)
│   └── login
├── projects (group)
│   ├── list
│   ├── get
│   ├── create
│   ├── update
│   └── delete
├── runs (group)
│   ├── list
│   ├── get
│   ├── get-latest
│   ├── stats
│   ├── open
│   ├── watch
│   ├── search
│   ├── sample
│   ├── analyze
│   ├── tags
│   ├── metadata-keys
│   ├── fields
│   ├── describe
│   ├── view-file
│   ├── usage
│   ├── export
│   └── cache (group)
│       ├── download
│       ├── list
│       └── clear
├── datasets (group)
│   ├── list
│   ├── get
│   ├── create
│   ├── delete
│   └── push
├── examples (group)
│   ├── list
│   ├── get
│   ├── create
│   ├── update
│   ├── delete
│   └── from-run
├── prompts (group)
│   ├── list
│   ├── get
│   ├── pull
│   ├── push
│   ├── create
│   ├── delete
│   └── commits
├── feedback (group)
│   ├── list
│   ├── get
│   ├── create
│   └── delete
├── annotation-queues (group)
│   ├── list
│   ├── get
│   ├── create
│   ├── update
│   └── delete
├── experiments (group)
│   └── results
└── self (group)
    ├── detect
    └── update

Key Design Patterns

1. Direct Imports for Strong Types

# ✅ GOOD - Direct imports of langsmith types for strong typing
import langsmith
from langsmith.schemas import Dataset, TracerSessionResult, Run

def resolve_dataset(client: langsmith.Client, name: str) -> Dataset:
    ...

# ❌ BAD - using `object` or `Any` as type (loses all type safety)
def resolve_dataset(client: object, name: str) -> object: ...

Imports and CLI startup performance — read this carefully, there is real tension here.

There are two competing forces:

• Type safety wants top-level imports. TYPE_CHECKING and from __future__ import annotations defer name resolution and lose IDE/pyright tooling on those names; quoted forward refs are easy to typo.
• Epic #117 (startup latency) wants lazy imports in modules that are cold paths — command modules that aren't reached on a typical invocation, or that pull in heavy optional deps (httpx, yaml, network adapters). Every extra top-level from rich.table import Table runs on every CLI startup, including --help.

The rule that resolves the conflict:

1. Default to top-level imports for light modules already on the startup path (click, pydantic, the CLI's own helpers). utils.py does not import langsmith or rich, so neither is paid for at startup; don't assume a heavy SDK is "already loaded".
2. Use lazy (function-scope) imports in command modules that (a) are themselves only loaded by their specific command and (b) pull in deps that utils.py does not already drag in (e.g. httpx, yaml, large third-party SDKs). runs/pricing_cmd.py is the canonical example — it's only loaded when the user runs runs pricing, and httpx/yaml are not on the global hot path. Keep its import httpx / import yaml inside the function body.
3. For type annotations only, prefer the real import over TYPE_CHECKING when feasible. Use TYPE_CHECKING (paired with from __future__ import annotations) only when the type comes from a module you are deliberately keeping out of import-time — and document why in a comment near the if TYPE_CHECKING: block (e.g. # Lazy: pricing command is a cold path; keep langsmith.schemas off the startup graph here.).
4. langsmith.Client stays off the startup path. Importing it costs most of a second, and import langsmith alone is lazy. Command modules annotate with it under TYPE_CHECKING (as projects.py and datasets.py do) and get the instance from get_or_create_client. A command module that is expensive to import as a whole is registered in LAZY_SUBCOMMANDS in main.py instead of with add_command. tests/test_startup_imports.py fails if startup or --help loads langsmith.client, httpx, requests, duckdb, pyarrow, boto3 or the archive command.

Concrete checklist when adding a new command:

• Does the module need only types that utils.py already imports? → Top-level import them.
• Does the module need a heavy dep no other command uses (httpx, pandas, yaml)? → Lazy-import it inside the function, with a one-line comment naming the startup-latency tradeoff.
• Are you introducing from __future__ import annotations purely to satisfy a TYPE_CHECKING block? → Add the comment explaining the cold-path tradeoff, or just use the real import.

2. Dual Output Pattern (JSON vs Rich Tables)

# All commands check ctx.obj.get("json") flag
if ctx.obj.get("json"):
    import json
    click.echo(json.dumps(data, default=str))
else:
    from rich.table import Table
    # ... render table

3. Context Efficiency Pattern (Field Pruning)

# runs get command supports --fields to reduce token usage
# Instead of returning full 20KB run object, return only requested fields
@runs.command("get")
@click.option("--fields", help="Comma-separated field names")
def get_run(ctx, run_id, fields):
    run = client.read_run(run_id)
    if fields:
        # Prune to only requested fields
        field_list = fields.split(",")
        data = {k: v for k, v in run_dict.items() if k in field_list}

4. Logging and Verbosity Pattern (stdout/stderr Separation)

# All commands use CLILogger for diagnostic output with proper stream separation
@runs.command("list")
def list_runs(ctx, output_format, count, output, ...):
    # Get logger from context
    logger = ctx.obj["logger"]

    # Determine if output is machine-readable (use stderr for diagnostics)
    is_machine_readable = (
        ctx.obj.get("json") or
        output_format in ["csv", "yaml"] or
        count or
        output
    )
    logger.use_stderr = is_machine_readable

    # Use logger methods for all diagnostic output
    logger.debug("API call: POST /runs/query")  # -v: DEBUG level
    logger.info("Fetching 100 runs...")          # Default: INFO level
    logger.warning("Warning: Failed to fetch...")  # Always shown (unless -qq)
    logger.error("Error reading file...")        # Always shown
    logger.success("Wrote N items to file")      # INFO level

    # Data output continues using appropriate stream
    if ctx.obj.get("json"):
        click.echo(json_dumps(data))  # stdout
    else:
        logger.data(table, mode="table")  # stdout

Verbosity Levels (following pip, Black, and Python CLI standards):

• Default (no flags): INFO level (20) - progress + warnings + errors
• -q: WARNING level (30) - warnings + errors only, no progress
• -qq: ERROR level (40) - only errors (cleanest for scripts)
• -v: DEBUG level (10) - debug details + info + warnings + errors
• -vv: TRACE level (5) - ultra-verbose (HTTP, timing, raw data)

Stream Separation:

• stdout: Data output (JSON, CSV, YAML, tables, count results)
• stderr: Diagnostic messages (progress, warnings, errors) in machine-readable modes
• Table mode exception: Diagnostics stay on stdout (mixing is OK for humans)

File Organization

src/langsmith_cli/
├── __init__.py
├── main.py              # Entry point, CLI group registration, global error handler
├── cli_logging.py       # CLILogger for verbosity control and stream separation
├── config.py            # Credentials file management
├── field_analysis.py    # Field discovery and statistics (runs fields/describe)
├── filters.py           # FQL filter builders and time parsing
├── utils.py             # Shared helpers (output formatting, project resolution, etc.)
└── commands/            # Modular command implementations
    ├── auth.py          # Authentication (login)
    ├── projects.py      # Project management
    ├── runs.py          # Runs/traces (largest module)
    ├── datasets.py      # Dataset operations
    ├── examples.py      # Dataset examples
    ├── prompts.py       # Prompt management
    └── self_cmd.py      # Self-inspection (detect, update)

tests/
├── conftest.py          # Pytest fixtures (CliRunner, model factories)
├── test_main.py         # Root CLI + global error handler tests
├── test_logging.py      # CLILogger tests
├── test_auth.py         # Auth command tests
├── test_projects.py     # Projects command tests
├── test_runs.py         # Runs command tests
├── test_runs_list.py    # Runs list command tests (filters, JSON, formats)
├── test_runs_get.py     # Runs get/get-latest/open tests
├── test_runs_roots.py   # Runs --roots flag tests
├── test_runs_sample.py  # Runs sample command tests
├── test_datasets.py     # Datasets command tests
├── test_examples.py     # Examples command tests
├── test_prompts.py      # Prompts command tests
├── test_self.py         # Self detect/update command tests
├── test_utils.py        # Utility function tests
├── test_smoke.py        # Smoke tests (requires API key)
└── test_e2e.py          # End-to-end tests (requires API key)

skills/langsmith/
└── SKILL.md             # Claude Code skill definition

docs/
├── PRD.md               # Product requirements
├── COMMANDS_DESIGN.md   # Detailed command specifications
└── dev/                 # Development documentation
    ├── SESSION_DIRECTIVES.md  # User preferences and tooling choices
    ├── TYPE_SAFETY_GUIDE.md   # Type safety guidelines
    ├── TESTING_PERFORMANCE.md # Test performance optimization
    └── MCP_PARITY.md          # MCP compatibility mapping

Code Quality Principles

Rule 1: Always Use SDK's Pydantic Models

Never create custom response models - always reuse the LangSmith SDK's Pydantic models.

• ✅ Use langsmith.schemas.Dataset, Example, Run, Prompt, TracerSessionResult
• ✅ For JSON output, use .model_dump(include={...}, mode="json") to select fields
• ✅ Access model fields directly: project.name, dataset.id
• ❌ Don't use getattr(obj, "field", default) - the fields are guaranteed to exist
• ❌ Don't create duplicate models in a separate models.py file

Why: SDK models are type-safe, validated, and the source of truth. Creating duplicates causes maintenance burden and drift.

Rule 2: Always Use the LangSmith SDK

Use the LangSmith SDK for all functionality - only use REST API if SDK doesn't provide the capability.

• ✅ Use langsmith.Client() methods
• ✅ Explore SDK first: dir(client), inspect source code, read SDK docs
• ❌ Don't make raw HTTP requests if SDK provides the method

Why: SDK handles auth, retry logic, rate limiting, and provides type-safe interfaces.

Rule 3: Use Proper Exception Types

Never match errors by string - always use SDK's exception types from langsmith.utils.

• ✅ from langsmith.utils import LangSmithConflictError, LangSmithNotFoundError
• ✅ except LangSmithConflictError: for handling conflicts
• ❌ if "already exists" in str(e): - string matching is brittle

Why: Exception types are semantic contracts that won't break with message changes.

Type Safety Guidelines

For comprehensive type safety guidelines, see docs/dev/TYPE_SAFETY_GUIDE.md

Philosophy: Zero Tolerance for Weak Types

Strong types catch bugs at development time, not runtime. Weak types defer errors to production.

Golden Rules:

1. ❌ Never use Any without a documented reason
2. ❌ Never use bare list or dict - always parameterize: list[T], dict[K, V]
3. ✅ Always use SDK Pydantic models when available (Run, Dataset, Example, Prompt, TracerSessionResult)
4. ✅ Always handle None explicitly with T | None
5. ✅ Always use direct attribute access over getattr()
6. ✅ Always run pyright before committing (must be 0 errors)
7. ✅ Use Python 3.12+ syntax: list[T] not List[T], dict[K, V] not Dict[K, V], T | None not Optional[T]

Common Type Patterns

Parameterize Collections:

# ❌ WRONG
def process_data(items: list) -> dict:
    results: List = []
    return {}

# ✅ CORRECT
from langsmith.schemas import Run

def process_runs(runs: list[Run]) -> dict[str, list[str]]:
    results: list[str] = []
    return {"names": results}

Use SDK Models:

# ❌ WRONG
def format_run(run: Any) -> dict:
    return {"name": getattr(run, "name", "Unknown")}

# ✅ CORRECT
from langsmith.schemas import Run

def format_run(run: Run) -> dict[str, str]:
    return {"name": run.name}  # Type checker guarantees .name exists

Generic Functions:

# ❌ WRONG
from typing import Callable, Any

def sort_items(items: list, key_func: Callable[[Any], Any]) -> list:
    return sorted(items, key=key_func)

# ✅ CORRECT
from typing import TypeVar, Callable

T = TypeVar('T')
K = TypeVar('K')

def sort_items(items: list[T], key_func: Callable[[T], K]) -> list[T]:
    return sorted(items, key=key_func)

Handle Optional Values:

# ❌ WRONG
def get_error(run: Any) -> str:
    return run.error  # What if error is None?

# ✅ CORRECT
from langsmith.schemas import Run

def get_error(run: Run) -> str | None:
    return run.error  # Type checker knows this can be None

def get_error_message(run: Run) -> str:
    return run.error or "No error"  # Handle None explicitly

When Is Any Acceptable?

Only in these rare cases:

1. JSON data being serialized/deserialized - dict[str, Any] for JSON output where values can be str, int, bool, datetime, nested dicts, etc.
2. Click Context object (Click's obj is Any - extract and narrow immediately)

For everything else, use strong types:

• Generic functions: Use TypeVar instead of Any

  # ❌ WRONG
  def sort_items(items: list, key_func: Callable[[Any], Any]) -> list:
      ...
  
  # ✅ CORRECT
  T = TypeVar('T')
  K = TypeVar('K')
  def sort_items(items: list[T], key_func: Callable[[T], K]) -> list[T]:
      ...

• Console objects: Use Protocol or actual type, not Any

  # ❌ WRONG
  def print_msg(console: Any) -> None:
      ...
  
  # ✅ CORRECT (Protocol)
  class ConsoleProtocol(Protocol):
      def print(self, *args: Any, **kwargs: Any) -> None: ...
  
  def print_msg(console: ConsoleProtocol) -> None:
      ...
  
  # ✅ CORRECT (Actual type)
  from rich.console import Console
  def print_msg(console: Console) -> None:
      ...

Document Any when you must use it:

def output_json(data: list[dict[str, Any]]) -> None:
    """Output JSON data.

    Args:
        data: JSON data (Any is acceptable - values can be str, int, bool, nested dicts, etc.)
    """

Verification

Always verify type safety with:

# Type checking (zero errors is the goal)
uv run pyright

# Tests still pass
uv run pytest

# Linting
uv run ruff check .

Testing Patterns

Current Test Approach

Tests use Click's CliRunner with unittest.mock patches:

def test_runs_list(runner):
    with patch("langsmith.Client") as MockClient:
        mock_client = MockClient.return_value
        mock_client.list_runs.return_value = [mock_run]
        result = runner.invoke(cli, ["runs", "list"])
        assert result.exit_code == 0

Known Improvement Opportunity: Tests currently use patch() with string paths. Per project standards (docs/AGENTS.md), should migrate to patch.object() for better refactor-safety.

Test Data: Always Use Pydantic Models, Never MagicMock

Critical Rule: Always create test data using real LangSmith Pydantic model instances, not MagicMock objects.

❌ BAD - Using MagicMock for test data:

def test_datasets_list(runner):
    with patch("langsmith.Client") as MockClient:
        mock_client = MockClient.return_value
        d = MagicMock()
        d.name = "test-dataset"  # This doesn't validate schema
        d.model_dump.return_value = {...}  # Manually mocking model_dump
        mock_client.list_datasets.return_value = iter([d])

✅ GOOD - Using real Pydantic models:

from langsmith.schemas import Dataset
from uuid import UUID
from datetime import datetime, timezone

def test_datasets_list(runner):
    with patch("langsmith.Client") as MockClient:
        mock_client = MockClient.return_value
        # Create real Dataset instance with validation
        d = Dataset(
            name="test-dataset",
            description="Test",
            data_type="kv",
            id=UUID("ae99b6fa-a6db-4f1c-8868-bc6764f4c29e"),
            created_at=datetime(2024, 7, 3, tzinfo=timezone.utc),
            modified_at=datetime(2024, 7, 3, tzinfo=timezone.utc),
            example_count=10,
        )
        mock_client.list_datasets.return_value = iter([d])

Why:

• Real models validate field types and required fields
• model_dump() works correctly without mocking
• Catches schema mismatches before production
• Tests are more realistic and reveal integration issues
• Pydantic field validators run, catching bad data early

Helper Pattern: Create reusable factory functions in tests/conftest.py:

def create_dataset(name="test", example_count=10) -> Dataset:
    return Dataset(
        name=name,
        data_type="kv",
        id=UUID("ae99b6fa-a6db-4f1c-8868-bc6764f4c29e"),
        created_at=datetime(2024, 7, 3, tzinfo=timezone.utc),
        modified_at=datetime(2024, 7, 3, tzinfo=timezone.utc),
        example_count=example_count,
    )

Test Coverage Requirements

• All new commands must have unit tests
• Critical paths need integration tests
• E2E tests validate full SDK interaction (require API key)
• Aim for 100% coverage of new code

Handling Mixed stdout/stderr in Tests (CliRunner)

Click's CliRunner mixes stdout and stderr. When commands use logger.info() before JSON output, the --json test output may contain non-JSON lines. Use the shared parse_json_output() helper from conftest:

from conftest import parse_json_output

def test_json_output(runner, tmp_path):
    result = runner.invoke(cli, ["--json", "runs", "export", str(tmp_path), ...])
    data = parse_json_output(result.output)  # Finds last JSON line
    assert data["status"] == "success"

Anti-pattern: Don't use json.loads(result.output) directly when commands emit progress messages — it will fail with JSONDecodeError.

SDK Exception Types in Tests

Always use real SDK exception types in test side_effects, never bare Exception:

# ❌ BAD - bare Exception won't be caught by specific handlers
mock_client.delete_example.side_effect = Exception("Not found")

# ✅ GOOD - matches the actual exception hierarchy
from langsmith.utils import LangSmithNotFoundError
mock_client.delete_example.side_effect = LangSmithNotFoundError("Not found")

Reusable Patterns Across Commands

Name-or-ID Resolution (resolve_project(), resolve_dataset())

When a command accepts a name or ID argument, use the existing resolve helpers:

• projects.py: resolve_project(client, name_or_id, include_stats=True)
• datasets.py: resolve_dataset(client, name_or_id)

These helpers follow a consistent pattern:

1. Auto-detect UUIDs via _looks_like_uuid() to save an API call
2. Fall back from name to ID lookup
3. Catch (LangSmithNotFoundError, LangSmithError, ValueError) for friendly errors
4. Raise click.ClickException on not-found

from langsmith_cli.commands.projects import resolve_project

project = resolve_project(client, name_or_id, include_stats=True)

Shared Filter Building (utils.py: build_runs_list_filter())

Never manually reconstruct status/tag/time filters. Use the canonical helper:

combined_filter, error_filter = build_runs_list_filter(
    filter_=filter_, status=status, tag=tag, since=since, last=last
)

Shared Multi-Project Fetching (utils.py: fetch_from_projects())

Never write manual project iteration loops. Use fetch_from_projects():

def _fetch_runs(c, proj, **kw):
    if proj is not None:
        return c.list_runs(project_name=proj, **kw)
    return c.list_runs(**kw)

result = fetch_from_projects(client, projects, _fetch_runs, project_query=pq, ...)
raise_if_all_failed_with_suggestions(result, client, pq, logger, "runs")

Destructive Operations Pattern

For delete/destructive commands, follow this pattern:

1. Require --confirm flag (skip with click.confirm(abort=True))
2. Resolve entity first (verify it exists), then perform the operation
3. Use specific SDK exceptions (LangSmithNotFoundError), never broad except Exception
4. In JSON mode, return {"status": "success", ...} on success

Filename Sanitization

When writing user-controlled filenames, sanitize with regex:

import re
safe = re.sub(r'[<>:"/\\|?*\x00-\x1f]', "_", name)
safe = safe.strip(". ")

Reuse-First: Search Before Writing

Before writing new logic, always search for existing helpers in utils.py and commands/runs.py. Duplicating logic that already exists creates maintenance burden and drift.

Checklist before writing code:

1. Search utils.py for existing helpers (filter builders, output formatters, project resolution)
2. Search commands/runs.py for existing helpers (tag/metadata filtering, model extraction, grouping)
3. If the logic is reusable, add it to utils.py — not inline in the command function
4. If adding a new CLI option that exists on another command, check how that command implements it

Key shared helpers:

• build_tag_fql_filters(tags) — FQL has(tags, ...) clauses
• filter_runs_by_tags(runs, tags) — Client-side AND-logic tag filtering
• build_runs_list_filter(...) — Canonical filter builder for runs list
• build_time_fql_filters(since, last, before) — Time range FQL
• fetch_from_projects(...) — Multi-project iteration with error handling
• resolve_project_filters(...) — Project name/pattern/regex resolution
• parse_fields_option(fields) — Parse comma-separated field strings

Common Anti-Patterns to Avoid

1. hasattr() on Pydantic models → Use p.field is not None instead
2. Broad except Exception in loops → Catch specific SDK exceptions
3. Manual filter construction → Use build_runs_list_filter()
4. Manual project iteration → Use fetch_from_projects()
5. Duplicated utility logic → Extract to helpers (e.g., normalize_split())
6. --flag default=True is_flag=True → No-op; use --flag/--no-flag pair
7. ThreadPoolExecutor for local file I/O → Sequential is faster (no GIL benefit)
8. TYPE_CHECKING / __future__ annotations workaround → Import types directly; langsmith is already loaded at startup
9. client: object or client: Any → Use client: langsmith.Client for strong types
10. Inline tag FQL building → Use build_tag_fql_filters() from utils.py
11. Manual tag filtering on cached runs → Use filter_runs_by_tags() from utils.py

Engineering Standards (from docs/AGENTS.md)

1. Type Safety (Zero Tolerance for Weak Types)

• ❌ Prohibit: getattr(), hasattr(), stringly-typed logic
• ❌ NEVER use # type: ignore comments - Fix the underlying type issue instead
• ✅ Require: Direct attribute access, strict SDK contracts, Enums for logic
• ✅ Write safe code with proper type guards and None checks
• ✅ Always use LangSmith SDK's Pydantic models directly (Dataset, Example, Run, Prompt, TracerSessionResult)
• ✅ For context efficiency, use .model_dump(include={...}, mode="json") to select specific fields
• ✅ Never create duplicate response models - reuse SDK models with field selection
• Current Status: Fully implemented - all commands use SDK Pydantic models with type-safe attribute access

2. Performance (Keep CLI Fast)

• ❌ Don't add new top-level imports of heavy libraries to modules that don't already need them
• ❌ Don't introduce new heavy dependencies without measuring CLI startup impact
• ✅ langsmith and rich are already loaded at startup (via utils.py) — use them directly for strong types
• ✅ If adding a new module that doesn't need langsmith at import time, keep heavy imports inside functions
• ✅ Always prefer strong types over lazy loading workarounds — don't sacrifice type safety for startup time
• Current Status: langsmith/rich load at startup (~300ms). Acceptable. Don't make it worse.

3. Architecture (Pure Logic vs View)

• Logic Layer: Returns Pydantic models/typed objects, never prints
• View Layer: Handles Rich tables or JSON output
• Context Safety: All commands support --json flag
• Current Status: Mixed - some logic calls console.print() directly (opportunity for refactor)

4. Error Handling

• ❌ NEVER silently swallow errors with bare except: pass or except Exception: pass
• ❌ Discourage broad Exception catching without logging
• ❌ Forbidden: Matching errors by string
• ✅ Use specific SDK exceptions: LangSmithConflictError, LangSmithNotFoundError, LangSmithAuthError, etc.
• ✅ Import exceptions from langsmith.utils
• ✅ Choose appropriate error handling based on severity:
  • Errors: Use raise or click.ClickException() for fatal issues that should stop execution
  • Stack traces: Use console.print_exception() or logger.exception() for debugging unexpected failures
  • Warnings: Use console.print("[yellow]Warning:...") for recoverable issues users should know about
  • Debug messages: Use logger.debug() or console.print("[dim]...") for diagnostic information
• ✅ When catching broad exceptions, always log what went wrong before continuing
• Current Status: Fully implemented - all commands use proper exception types and log failures

5. Context Efficiency (Plugin Standard)

• Default JSON output should be "sparse"
• No full trace blobs (20kb+) unless requested
• Skill maintenance: Update SKILL.md when CLI arguments change
• Current Status: Field pruning implemented; SKILL.md up-to-date

Feature Parity with LangSmith MCP

The CLI provides 100% feature parity with the official LangSmith MCP server. See docs/dev/MCP_PARITY.md for detailed command mappings and parameter coverage.

Claude Code Plugin Integration

Skill Definition

The plugin is defined in skills/langsmith/SKILL.md which teaches Claude how to use the CLI efficiently:

• MANDATORY for agents: Always pass --json as the FIRST argument to every langsmith-cli command (except auth login and runs watch). Without it, the CLI outputs Rich tables that are unparseable and waste tokens.
• Use --fields to reduce context size (~90% token reduction)
• Use --status error for quick debugging
• Keep --limit small (default 10)
• Use --before / --since / --last for time windows (no raw FQL needed)
• Use runs usage --from-cache for cost/token analysis (NOT inline Python scripts)
• Use --group-by, --breakdown, --interval, --metadata for aggregation
• Prefer CLI commands over inline scripts for analysis — the CLI has built-in commands for cost/token aggregation, time distribution, and metadata filtering (see "Cost & Token Analysis Recipes" in SKILL.md). Only fall back to scripts for genuinely unsupported analysis.

Plugin Manifests

The repository contains two manifest files:

• .claude-plugin/marketplace.json - Registers the repository as an installable marketplace
• .claude-plugin/plugin.json - Defines the plugin itself

Installation

Method 1: Using Terminal Commands

# Add the marketplace
claude plugin marketplace add gigaverse-app/langsmith-cli

# Install the plugin
claude plugin install langsmith-cli@langsmith-cli

Method 2: Using Claude Code Interactive UI

Inside Claude Code, run /plugin (without arguments) to open the interactive plugin manager:

1. Navigate to the "Marketplaces" tab
2. Add marketplace: gigaverse-app/langsmith-cli
3. Navigate to the "Discover" tab
4. Install the langsmith-cli plugin

Local Development

# Add local marketplace
claude plugin marketplace add /home/aviadr1/projects/langsmith-cli

# Install the plugin
claude plugin install langsmith-cli@langsmith-cli

Pydantic Model Usage Pattern

Always Use SDK's Pydantic Models

The LangSmith SDK provides Pydantic models for all entities:

• langsmith.schemas.TracerSessionResult - Projects/Sessions
• langsmith.schemas.Dataset - Datasets
• langsmith.schemas.Example - Dataset examples
• langsmith.schemas.Run - Runs/traces
• langsmith.schemas.Prompt - Prompts

Pattern:

# ✅ GOOD - Use SDK models with focused field selection
def list_items(ctx, limit):
    client = langsmith.Client()
    items_list = list(client.list_items(limit=limit))

    if ctx.obj.get("json"):
        # Select only needed fields for context efficiency
        data = [
            item.model_dump(
                include={"id", "name", "created_at"},
                mode="json",
            )
            for item in items_list
        ]
        click.echo(json.dumps(data, default=str))
        return

    # For table display, access fields directly
    for item in items_list:
        table.add_row(item.name, str(item.id))

# ❌ BAD - Don't use getattr() or create custom models
def list_items_bad(ctx, limit):
    items = list(client.list_items(limit=limit))
    data = [
        {
            "name": getattr(i, "name", "Unknown"),  # Don't do this
            "id": str(getattr(i, "id", "")),  # Don't do this
        }
        for i in items
    ]

Benefits:

1. Type safety - IDE autocomplete and type checking work
2. Context efficiency - Only return fields you need
3. No duplicate models - SDK models are the source of truth
4. Validation - Pydantic validates data automatically

Common Patterns for New Commands

Adding a New Command

1. Create command file in src/langsmith_cli/commands/
2. Define Click group and commands:

import click
from rich.console import Console

console = Console()

@click.group()
def mycommand():
    """Description of command group."""
    pass

@mycommand.command("list")
@click.pass_context
def list_items(ctx):
    """List items."""
    # Lazy import SDK
    import langsmith
    client = langsmith.Client()

    items = client.list_items()

    # Handle JSON vs Rich output
    if ctx.obj.get("json"):
        import json
        click.echo(json.dumps([i.dict() for i in items], default=str))
    else:
        from rich.table import Table
        table = Table(title="Items")
        table.add_column("Name")
        for item in items:
            table.add_row(item.name)
        console.print(table)

3. Register in main.py:

from langsmith_cli.commands.mycommand import mycommand
cli.add_command(mycommand)

4. Write tests in tests/test_mycommand.py:

from langsmith_cli.main import cli
from unittest.mock import patch

def test_mycommand_list(runner):
    with patch("langsmith.Client") as MockClient:
        mock_client = MockClient.return_value
        mock_client.list_items.return_value = []
        result = runner.invoke(cli, ["mycommand", "list"])
        assert result.exit_code == 0

5. Update SKILL.md with new command usage

Known Issues & Improvement Opportunities

Type Safety

• ✅ FIXED: Removed all getattr() usage - now uses direct attribute access
• ✅ FIXED: Removed string-based error matching - now uses proper SDK exception types
• Pyright reports 4 type warnings (non-critical but fixable)

Architecture

• Some logic mixed with view code (console.print() calls in command functions)
• Could benefit from separating data fetching from rendering
• Mock usage could be upgraded to patch.object instead of patch

Security

• .mcp.json contains hardcoded API key (should use environment variable)

Dependencies

Runtime:

• click >=8.3.1 - CLI framework
• langsmith >=0.6.2 - LangSmith SDK
• pydantic >=2.12.5 - Data validation
• python-dotenv >=1.2.1 - .env file support
• rich >=14.2.0 - Terminal formatting

Development:

• pytest >=9.0.2 - Testing framework
• ruff >=0.14.11 - Linting/formatting
• pyright >=1.1.408 - Type checking
• pre-commit >=4.5.1 - Git hooks

Python Version: >=3.12

Releasing

ALWAYS Use the Release Script

Never manually bump versions or create tags/releases. Use scripts/release.sh which handles everything:

# Patch bump (0.4.0 → 0.4.1) — default
./scripts/release.sh

# Minor bump (0.4.0 → 0.5.0)
./scripts/release.sh minor

# Major bump (0.4.0 → 1.0.0)
./scripts/release.sh major

# Explicit version
./scripts/release.sh 0.5.0

# Skip tests for docs-only releases
./scripts/release.sh --skip-tests

# Auto-confirm (no prompts)
./scripts/release.sh -y

What the Release Script Does

1. Bumps version in all 4 files simultaneously:
   • pyproject.toml
   • .claude-plugin/plugin.json
   • .claude-plugin/marketplace.json (both metadata.version and plugins[0].version)
   • uv.lock
2. Runs quality checks: ruff lint, ruff format, pyright
3. Runs tests (unless --skip-tests)
4. Creates git commit and annotated tag
5. Pushes to remote (triggers CI → PyPI publish → GitHub release)

Version Files — Never Edit Manually

These files contain version strings that must stay in sync:

• pyproject.toml — PyPI package version
• .claude-plugin/plugin.json — Plugin version for Claude Code
• .claude-plugin/marketplace.json — Marketplace listing version (2 locations)
• uv.lock — Auto-updated by the script

Editing any of these manually creates version drift. The release script is the single source of truth for version bumps.

When to Release

• Code changes: Always use ./scripts/release.sh (runs tests by default)
• Docs/skill-only changes: Use ./scripts/release.sh --skip-tests (still bumps version)
• After updating SKILL.md: A release is needed for plugin users to get the updated skill

CI/CD Pipeline (Automated)

On tag push (v*), GitHub Actions automatically:

1. Runs full test suite
2. Builds wheel and sdist
3. Publishes to PyPI (Trusted Publishing, no tokens)
4. Creates GitHub release with artifacts

Git Workflow

Per docs/dev/SESSION_DIRECTIVES.md:

• Use uv for all dependency management
• Commit often with clear messages
• Use feature branches and PRs for larger features
• Ensure git status is clean before commits
• Run pre-commit hooks before committing
• Never use git commit --amend or force push (per user's global CLAUDE.md)