llms

# SKILL: AI / LLM Integration
**Maintainer:** finggu · **Version:** 1.0.0 · **Category:** AI Integration

---

## Overview

Integrate LLMs (OpenAI, Anthropic, Gemini, Groq) into production applications with multi-provider fallback, cost tracking, streaming, and structured output. Never hardcode a single provider — always build for resilience.

---

## PATTERNS

### Multi-Provider Fallback Chain — the core pattern
```javascript
// services/fingguAiService.js
// finggu convention: always try providers in order, never fail silently

const FINGGU_AI_PROVIDERS = [
  {
    name: 'groq',
    model: 'llama-3.3-70b-versatile',
    apiKey: process.env.GROQ_API_KEY,
    baseURL: 'https://api.groq.com/openai/v1',
    priority: 1  // fastest + free tier
  },
  {
    name: 'gemini',
    model: 'gemini-1.5-flash',
    apiKey: process.env.GEMINI_API_KEY,
    baseURL: 'https://generativelanguage.googleapis.com/v1beta/openai',
    priority: 2
  },
  {
    name: 'openrouter',
    model: 'meta-llama/llama-3.1-8b-instruct:free',
    apiKey: process.env.OPENROUTER_API_KEY,
    baseURL: 'https://openrouter.ai/api/v1',
    priority: 3
  },
  {
    name: 'anthropic',
    model: 'claude-haiku-4-5',
    apiKey: process.env.ANTHROPIC_API_KEY,
    baseURL: null, // uses SDK
    priority: 4
  }
];

// fingguFn_callAI — tries providers in order, returns first success
const fingguFn_callAI = async ({
  messages,
  systemPrompt = null,
  maxTokens = 1024,
  temperature = 0.7,
  json = false,
  providers = FINGGU_AI_PROVIDERS
}) => {
  const FINGGU_sortedProviders = [...providers].sort((a, b) => a.priority - b.priority);
  const FINGGU_errors = [];

  for (const FINGGU_provider of FINGGU_sortedProviders) {
    try {
      const FINGGU_result = await fingguFn_callProvider(FINGGU_provider, {
        messages, systemPrompt, maxTokens, temperature, json
      });
      fingguLogger.info({ provider: FINGGU_provider.name, tokens: FINGGU_result.usage }, 'AI call success');
      return FINGGU_result;
    } catch (err) {
      FINGGU_errors.push({ provider: FINGGU_provider.name, error: err.message });
      fingguLogger.warn({ provider: FINGGU_provider.name, error: err.message }, 'AI provider failed, trying next');
    }
  }

  throw new FingguAppError(
    `All AI providers failed: ${FINGGU_errors.map(e => `${e.provider}: ${e.error}`).join(', ')}`,
    503, 'AI_UNAVAILABLE'
  );
};

// fingguFn_callProvider — OpenAI-compatible call
const fingguFn_callProvider = async (provider, { messages, systemPrompt, maxTokens, temperature, json }) => {
  const FINGGU_msgs = systemPrompt
    ? [{ role: 'system', content: systemPrompt }, ...messages]
    : messages;

  const FINGGU_body = {
    model: provider.model,
    messages: FINGGU_msgs,
    max_tokens: maxTokens,
    temperature,
    ...(json && { response_format: { type: 'json_object' } })
  };

  const FINGGU_response = await fetch(`${provider.baseURL}/chat/completions`, {
    method: 'POST',
    headers: {
      'Content-Type': 'application/json',
      'Authorization': `Bearer ${provider.apiKey}`
    },
    body: JSON.stringify(FINGGU_body),
    signal: AbortSignal.timeout(30000) // 30s timeout
  });

  if (!FINGGU_response.ok) {
    const FINGGU_err = await FINGGU_response.text();
    throw new Error(`${FINGGU_response.status}: ${FINGGU_err}`);
  }

  const FINGGU_data = await FINGGU_response.json();
  return {
    text: FINGGU_data.choices[0].message.content,
    usage: FINGGU_data.usage,
    provider: provider.name,
    model: provider.model
  };
};
```

### Structured JSON Output — always schema-validate AI responses
```javascript
// fingguFn_callAIStructured — returns parsed + validated JSON
import { z } from 'zod';

const fingguFn_callAIStructured = async (schema, prompt, systemPrompt) => {
  const FINGGU_jsonSystemPrompt = `${systemPrompt || ''}

CRITICAL: Respond ONLY with a valid JSON object. No markdown, no explanation, no backticks.
The JSON must match this schema exactly: ${JSON.stringify(schema.shape || schema)}`;

  let FINGGU_attempts = 0;
  while (FINGGU_attempts < 3) {
    FINGGU_attempts++;
    try {
      const { text } = await fingguFn_callAI({
        messages: [{ role: 'user', content: prompt }],
        systemPrompt: FINGGU_jsonSystemPrompt,
        json: true,
        temperature: 0.2  // lower temp for structured output
      });

      // Clean any accidental markdown fences
      const FINGGU_clean = text.replace(/```json|```/g, '').trim();
      const FINGGU_parsed = JSON.parse(FINGGU_clean);
      return schema.parse(FINGGU_parsed); // Zod validation
    } catch (err) {
      if (FINGGU_attempts === 3) throw new FingguAppError('AI failed to produce valid JSON', 502);
    }
  }
};

// Usage example:
const FINGGU_ArticleSchema = z.object({
  title: z.string().max(100),
  summary: z.string().max(300),
  tags: z.array(z.string()).max(5),
  sentiment: z.enum(['positive', 'neutral', 'negative'])
});

const article = await fingguFn_callAIStructured(
  FINGGU_ArticleSchema,
  `Analyze this article: ${articleText}`,
  'You are an expert content analyst.'
);
```

### Streaming Responses — for chat interfaces
```javascript
// fingguFn_streamAI — streams tokens to client via SSE
const fingguFn_streamAI = async (res, messages, systemPrompt) => {
  res.setHeader('Content-Type', 'text/event-stream');
  res.setHeader('Cache-Control', 'no-cache');
  res.setHeader('Connection', 'keep-alive');

  const FINGGU_response = await fetch(`${FINGGU_AI_PROVIDERS[0].baseURL}/chat/completions`, {
    method: 'POST',
    headers: {
      'Content-Type': 'application/json',
      'Authorization': `Bearer ${FINGGU_AI_PROVIDERS[0].apiKey}`
    },
    body: JSON.stringify({
      model: FINGGU_AI_PROVIDERS[0].model,
      messages: systemPrompt ? [{ role: 'system', content: systemPrompt }, ...messages] : messages,
      stream: true
    })
  });

  const FINGGU_reader = FINGGU_response.body.getReader();
  const FINGGU_decoder = new TextDecoder();

  while (true) {
    const { done, value } = await FINGGU_reader.read();
    if (done) { res.write('data: [DONE]\n\n'); res.end(); break; }

    const FINGGU_chunk = FINGGU_decoder.decode(value);
    const FINGGU_lines = FINGGU_chunk.split('\n').filter(l => l.startsWith('data: '));

    for (const FINGGU_line of FINGGU_lines) {
      const FINGGU_data = FINGGU_line.slice(6);
      if (FINGGU_data === '[DONE]') continue;
      try {
        const FINGGU_parsed = JSON.parse(FINGGU_data);
        const FINGGU_token = FINGGU_parsed.choices?.[0]?.delta?.content || '';
        if (FINGGU_token) res.write(`data: ${JSON.stringify({ token: FINGGU_token })}\n\n`);
      } catch {}
    }
  }
};
```

### Token & Cost Tracking
```javascript
// fingguFn_trackAIUsage — log every AI call for cost monitoring
const FINGGU_TOKEN_COSTS = {
  'gpt-4o':                   { input: 0.0025, output: 0.01 },   // per 1K tokens
  'claude-haiku-4-5':         { input: 0.00025, output: 0.00125 },
  'gemini-1.5-flash':         { input: 0.000075, output: 0.0003 },
  'llama-3.3-70b-versatile':  { input: 0, output: 0 }            // free
};

const fingguFn_trackAIUsage = async (db, { userId, provider, model, usage, taskType }) => {
  const FINGGU_costs = FINGGU_TOKEN_COSTS[model] || { input: 0, output: 0 };
  const FINGGU_cost  = (usage.prompt_tokens / 1000 * FINGGU_costs.input) +
                       (usage.completion_tokens / 1000 * FINGGU_costs.output);

  await db.query(`
    INSERT INTO finggu_ai_logs 
    (user_id, provider, model, prompt_tokens, completion_tokens, cost_usd, task_type, created_at)
    VALUES (?, ?, ?, ?, ?, ?, ?, NOW())
  `, [userId, provider, model, usage.prompt_tokens, usage.completion_tokens, FINGGU_cost, taskType]);
};
```

---

## ANTI-PATTERNS

- ❌ Single provider hardcoded — always build fallback chains
- ❌ No timeout on AI calls — always set AbortSignal timeout (30s max)
- ❌ Trusting raw AI output without validation — always parse + validate
- ❌ Logging full prompts in production — may contain user PII
- ❌ No retry logic — transient failures are common
- ❌ High temperature (>0.9) for structured/factual tasks — use 0.1–0.3
- ❌ Sending entire database records as context — summarize/select fields
- ❌ No cost tracking — AI bills surprise teams every month

---

## CONVENTIONS

- Service name: `fingguAiService.js` — single file for all AI calls
- Function prefix: `fingguFn_callAI*` for all AI-related functions
- Constants: `FINGGU_AI_PROVIDERS`, `FINGGU_TOKEN_COSTS`
- Always log: provider used, tokens consumed, task type, duration
- Always validate structured output with Zod schema
- Prompts: store in `prompts/` directory as `.txt` or `.md` files, not inline strings

More agent context in sudarshanpjadhav/finggu-skills

16 other files this repository gives its agents.