liter-llm

# liter-llm

> Universal LLM API client for 174 providers with native bindings for 14 languages.

## About

liter-llm provides unified access to 174 LLM providers through a single API.
Built in Rust with native bindings for Python, TypeScript, Go, Java, Ruby, PHP,
C#, Elixir, WebAssembly, and C/FFI. Provider routing is automatic based on
model name prefixes (e.g. "openai/gpt-4o" routes to OpenAI).

## Organization

Part of the xberg.io ecosystem:
- **xberg.io** — Organization: https://xberg.io
- **Xberg** — Document intelligence for 91+ formats: https://github.com/xberg-io/xberg
  - Xberg docs: https://docs.xberg.io
- **html-to-markdown** — HTML to Markdown converter: https://github.com/xberg-io/html-to-markdown
- **tree-sitter-language-pack** — Multi-language parsing: https://github.com/xberg-io/tree-sitter-language-pack
- **liter-llm** — Universal LLM API client (this project): https://github.com/xberg-io/liter-llm

## Links

- Website: https://xberg.io
- Documentation: https://docs.liter-llm.xberg.io
- GitHub: https://github.com/xberg-io/liter-llm
- Discord: https://discord.gg/xt9WY3GnKR
- Contact: naaman@xberg.io

## Capabilities

### Chat Completions
- `chat(request)` — Single-turn and multi-turn chat completions
- `chat_stream(request)` — Streaming chat completions (SSE)
- Tool calling with JSON schema validation and parallel tool execution
- System messages, developer messages, multi-turn conversation history

### Embeddings
- `embed(request)` — Generate text embeddings
- Batch embedding of multiple texts in a single request
- Multiple embedding models and providers supported

### Reranking
- `rerank(request)` — Rerank documents by relevance to a query

### Image Generation
- `image_generate(request)` — Generate images from text prompts

### Audio
- `speech(request)` — Text-to-speech audio generation
- `transcribe(request)` — Speech-to-text transcription

### Content Moderation
- `moderate(request)` — Classify content for policy violations

### Search
- `search(request)` — Web/document search across 12 providers (Brave, Tavily, Google PSE, etc.)

### OCR
- `ocr(request)` — Document OCR with Markdown output across 4 providers (Mistral, Azure, etc.)

### Model Discovery
- `list_models()` — List available models for the configured provider

### File Operations
- `create_file(request)` — Upload a file
- `retrieve_file(file_id)` — Get file metadata
- `delete_file(file_id)` — Delete a file
- `list_files(query)` — List uploaded files
- `file_content(file_id)` — Download file content

### Batch Operations
- `create_batch(request)` — Create a batch processing job
- `retrieve_batch(batch_id)` — Get batch status
- `list_batches(query)` — List batches
- `cancel_batch(batch_id)` — Cancel a batch

### Responses API
- `create_response(request)` — Create a response
- `retrieve_response(id)` — Retrieve a response
- `cancel_response(id)` — Cancel a response

### Configuration
- TOML configuration file (`liter-llm.toml`) with auto-discovery
- Custom provider loading from config files
- API key, base URL, timeouts, retries, custom headers
- Provider auto-detection via model prefix (e.g. "openai/gpt-4o")
- model_hint for pre-resolving provider at construction time
- Cache configuration (max entries, TTL)
- Budget tracking (global limit, per-model limits, enforcement)
- Hook system for request/response/error lifecycle events
- Custom provider registration
- OpenDAL cache backends (Redis, S3, GCS, filesystem, 40+ via Apache OpenDAL)
- Cooldown / circuit breaker after transient errors
- Per-model rate limiting (RPM/TPM)
- Background health checks
- Per-request cost tracking
- OpenTelemetry tracing spans

## Supported Languages

| Language | Package | Install |
|----------|---------|---------|
| Rust | crates.io/crates/liter-llm | `cargo add liter-llm` |
| Python | pypi.org/project/liter-llm | `pip install liter-llm` |
| TypeScript/Node.js | npmjs.com/package/@xberg-io/liter-llm | `pnpm add @xberg-io/liter-llm` |
| Go | github.com/xberg-io/liter-llm/packages/go/v2 | `go get github.com/xberg-io/liter-llm/packages/go/v2` |
| Java | io.xberg.literllm:liter-llm (Maven) | `implementation("io.xberg.literllm:liter-llm:1.9.0-rc.1")` |
| C# / .NET | NuGet: LiterLlm | `dotnet add package LiterLlm` |
| Ruby | RubyGems: liter_llm | `gem install liter_llm` |
| Elixir | Hex: liter_llm | `{:liter_llm, "~> 1.7"}` in mix.exs |
| PHP | Packagist: xberg-io/liter-llm | `composer require xberg-io/liter-llm` |
| WebAssembly | npmjs.com/package/@xberg-io/liter-llm-wasm | `pnpm add @xberg-io/liter-llm-wasm` |
| C / FFI | Build from source | `cargo build --release -p liter-llm-ffi` |

## CLI Installation

Homebrew:
  brew trust xberg-io/tap
  brew install xberg-io/tap/liter-llm

Cargo:
  cargo install liter-llm-cli

Docker:
  docker pull ghcr.io/xberg-io/liter-llm

## Proxy Server

OpenAI-compatible API gateway with 22 endpoints. Drop-in replacement for litellm proxy.

Docker quickstart (35MB Chainguard image):
  docker run -p 4000:4000 -e LITER_LLM_MASTER_KEY=sk-key ghcr.io/xberg-io/liter-llm

CLI:
  liter-llm api --config liter-llm-proxy.toml

Features: model routing (174 providers), virtual API keys, rate limiting (RPM/TPM), cost tracking, budget enforcement, response caching, SSE streaming, OpenAPI 3.1 at /openapi.json.

Config format: TOML with ${ENV_VAR} interpolation. Auto-discovers liter-llm-proxy.toml.

Example config:
  [server]
  host = "0.0.0.0"
  port = 4000

  [auth]
  master_key = "${LITER_LLM_MASTER_KEY}"

  [[virtual_keys]]
  key = "sk-team-frontend"
  models = ["openai/*", "anthropic/*"]
  rpm = 60
  budget = 50.0

  [[providers]]
  name = "openai"
  api_key = "${OPENAI_API_KEY}"

## MCP Server

Model Context Protocol server with 22 tools matching the REST API. Supports stdio and HTTP transports.

CLI:
  liter-llm mcp --transport stdio
  liter-llm mcp --transport http --port 3001

Tools: chat completions, streaming, embeddings, image generation, speech, transcription, moderation, search, OCR, reranking, file operations, batch operations, responses, model listing.

## Supported Providers (174)

OpenAI, Anthropic, Google Gemini, Google Vertex AI, AWS Bedrock, Azure OpenAI,
Groq, Mistral, Cohere, Together AI, Fireworks AI, Perplexity, DeepSeek,
Cerebras, Cloudflare Workers AI, Ollama, Voyage AI, Jina AI, Hugging Face,
Anyscale, Replicate, AI21, NLP Cloud, SambaNova, xAI, and 100+ more.

Provider routing is automatic. Use model prefixes to switch providers:
- `openai/gpt-4o` → OpenAI
- `anthropic/claude-sonnet-4-20250514` → Anthropic
- `groq/llama3-70b` → Groq
- `google/gemini-2.0-flash` → Google AI

## Usage Examples

### Python — Basic Chat

```python
import asyncio
import os
from liter_llm import LlmClient

async def main() -> None:
    client = LlmClient(api_key=os.environ["OPENAI_API_KEY"])
    response = await client.chat(
        model="openai/gpt-4o",
        messages=[{"role": "user", "content": "Hello!"}],
    )
    print(response.choices[0].message.content)

asyncio.run(main())
```

### TypeScript — Streaming

```typescript
import { LlmClient } from "@xberg-io/liter-llm";

const client = new LlmClient({ apiKey: process.env.OPENAI_API_KEY! });
const chunks = await client.chatStream({
  model: "openai/gpt-4o",
  messages: [{ role: "user", content: "Tell me a story" }],
});

for (const chunk of chunks) {
  process.stdout.write(chunk.choices[0]?.delta?.content ?? "");
}
console.log();
```

### Python — Tool Calling

```python
import asyncio
import os
from liter_llm import LlmClient

async def main() -> None:
    client = LlmClient(api_key=os.environ["OPENAI_API_KEY"])

    tools = [
        {
            "type": "function",
            "function": {
                "name": "get_weather",
                "description": "Get the current weather for a location",
                "parameters": {
                    "type": "object",
                    "properties": {
                        "location": {"type": "string", "description": "City name"},
                    },
                    "required": ["location"],
                },
            },
        }
    ]

    response = await client.chat(
        model="openai/gpt-4o",
        messages=[{"role": "user", "content": "What is the weather in Berlin?"}],
        tools=tools,
    )

    choice = response.choices[0]
    if choice.message.tool_calls:
        for call in choice.message.tool_calls:
            print(f"Tool: {call.function.name}, Args: {call.function.arguments}")

asyncio.run(main())
```

### Go — Embeddings

```go
package main

import (
	"context"
	"fmt"
	"os"

	llm "github.com/xberg-io/liter-llm/packages/go/v2"
)

func main() {
	client := llm.NewClient(llm.WithAPIKey(os.Getenv("OPENAI_API_KEY")))
	resp, err := client.Embed(context.Background(), &llm.EmbeddingRequest{
		Model: "openai/text-embedding-3-small",
		Input: llm.NewEmbeddingInputMultiple([]string{"The quick brown fox jumps over the lazy dog"}),
	})
	if err != nil {
		panic(err)
	}
	fmt.Printf("Dimensions: %d\n", len(resp.Data[0].Embedding))
	fmt.Printf("First 5 values: %v\n", resp.Data[0].Embedding[:5])
}
```