agentleFS
AWSKnown publisheraws/tools-for-devops-agent/skills/launchdarkly-online-evals/SKILL.md

launchdarkly-online-evals

Skill141 starsChanged 8 days agoApache-2.0View on GitHub

Attach judges to config variations for automatic LLM-as-a-judge evaluation. Create custom judges, configure sampling rates, and monitor quality scores.

  • Reads credentials

The file

We split this skill into the rules it gives an agent

2 rules. About 0 are common, 0 are distinctive, and 2 are very specific to this project.

  • ≈0

    Common · estimate

    Many files like this have a rule of this kind.

  • ≈0

    Distinctive · estimate

    This author's own choices. Worth a look.

  • 2

    Very specific

    Tied to their project's names, files or tools. Probably skip.

  • Common
  • Distinctive
  • Very specific
  1. Very specific:

    Use update-agent-config with resource: "variation" and judgeConfiguration.

    line 106
  2. Very specific:

    If your application calls the model itself, wrap that call in a handler with create_handler (createHandler).

    line 243

≈ “Common” is an estimate: at least 1 in 5 of the 364,980 skill files we read have a rule of the same kind, not the same words. “Very specific” is our guess from the words: it names a file, a command, a setting or the project itself.

Keep what fits and drop the rest. Your version, ready for your agents.
Files
SKILL.md

Config Online Evaluations

Attach judges to config variations for automatic quality scoring using LLM-as-a-judge methodology. Judges evaluate responses and return scores between 0.0 and 1.0.

Prerequisites

  • LaunchDarkly account with AgentControl enabled
  • The remotely hosted LaunchDarkly MCP server configured in your environment
  • Existing config with variations (use launchdarkly-configs-create skill)
  • To run judges from your application: launchdarkly-ai-server 0.3.0+ (Python) or @launchdarkly/ai-node 0.3.0+ (Node.js), plus a provider handler package for each provider your configs and judges use

API Access

Use the LaunchDarkly MCP server tools for every step in the workflow: they authenticate with the connection the customer set up. If a step has no MCP tool, tell the user which change to make in the LaunchDarkly UI instead. Never ask the user for an API access token.

Core Concepts

What Are Judges?

Judges are specialized configs in judge mode that evaluate responses from other configs. They use an LLM to score outputs and return structured results:

{
  "score": 0.85,
  "reasoning": "Answered correctly with one minor omission"
}

Built-in Judges

LaunchDarkly provides three pre-configured judges:

JudgeMetric KeyMeasures
Accuracy$ld:ai:judge:accuracyHow correct and grounded the response is
Relevance$ld:ai:judge:relevanceHow well it addresses the user request
Toxicity$ld:ai:judge:toxicityHarmful or unsafe phrasing (lower = safer)

Completion Mode Only

Judges can only be attached to completion mode configs in the UI. Attaching them through the API (Step 2) works for any config, and the SDK runs whatever judges the served variation has.

Restrictions

  • Cannot attach judges to judges (no recursion)
  • Cannot attach multiple judges with the same metric key to a single variation
  • Cannot view/edit model parameters or tools on judge variations

Workflow

Step 1: Create Custom Judges (Optional)

For domain-specific evaluation, create a judge config with create-agent-config and resource: "config":

{
  "projectKey": "{projectKey}",
  "resource": "config",
  "configKey": "security-judge",
  "name": "Security Judge",
  "mode": "judge",
  "evaluationMetricKey": "$ld:ai:judge:security",
  "isInverted": false
}

Note: evaluationMetricKey must start with $ld:ai:judge:. Set isInverted: true for metrics like toxicity where 0.0 is better.

Then add a variation with the evaluation prompt using create-agent-config with resource: "variation". Get modelConfigKey from get-ai-model-configs:

{
  "projectKey": "{projectKey}",
  "resource": "variation",
  "configKey": "security-judge",
  "variationKey": "default",
  "variationName": "Default",
  "modelConfigKey": "OpenAI.gpt-4o-mini",
  "modelName": "gpt-4o-mini",
  "parameters": {"temperature": 0.3},
  "messages": [
    {
      "role": "system",
      "content": "You are a security auditor. Score from 0.0 to 1.0:\n- 1.0: No security issues\n- 0.7-0.9: Minor issues\n- 0.4-0.6: Moderate issues\n- 0.1-0.3: Serious vulnerabilities\n- 0.0: Critical vulnerabilities\n\nCheck for: SQL injection, XSS, hardcoded secrets, command injection.\n\n{{message_history}}"
    }
  ]
}

Important: End the judge prompt with {{message_history}}. The SDK fills that variable with the conversation being judged, so a prompt without it never sees the response it's scoring. The SDK adds the score-and-reasoning output format itself.

Step 2: Attach Judges to Variations

Use update-agent-config with resource: "variation" and judgeConfiguration. Get variationKey from get-agent-configs:

{
  "projectKey": "{projectKey}",
  "configKey": "{configKey}",
  "resource": "variation",
  "variationKey": "{variationKey}",
  "judgeConfiguration": {
    "judges": [
      {"judgeConfigKey": "security-judge", "samplingRate": 1.0},
      {"judgeConfigKey": "api-contract-judge", "samplingRate": 0.5}
    ]
  },
  "confirm": true
}

Important: The judges array replaces all existing judge attachments. An empty array removes all judges.

Step 3: Set Fallthrough on Judges

Each judge config needs its fallthrough set to the enabled variation in every environment that runs it. Configs default to the "disabled" variation. Use update-agent-config with resource: "targeting" and a fallthrough:

{
  "projectKey": "{projectKey}",
  "configKey": "security-judge",
  "resource": "targeting",
  "env": "production",
  "fallthrough": {"variationKey": "default"},
  "confirm": true
}

Then call get-agent-configs with the judge's configKey and section: "health" to confirm it's targetable.

SDK: Inline Evaluation

invoke() runs every judge attached to the served variation, samples each at its rate, and records the scores as metrics. Pass a handler for each provider your config and its judges use. The config must be completion mode: a messages handler does not match an agent-mode config.

import asyncio
from launchdarkly_ai_server import config, shutdown
from launchdarkly_ai_openai_messages import create_openai_messages_handler

config_key = "support-agent"  # completion mode

async def main():
    # Reads LD_SDK_KEY and OPENAI_API_KEY from the environment
    result = await config(
        key=config_key,
        handler=create_openai_messages_handler(),
    ).invoke("How can LaunchDarkly help me?", {"kind": "user", "key": "user-123"})

    print(result.response)
    for judge_key, judge in (result.judge_results or {}).items():
        print(judge_key, judge.score)

    # Flushes trailing events, including judge scores
    await shutdown()

asyncio.run(main())
import { config, shutdown } from '@launchdarkly/ai-node';
import { createOpenAIHandler } from '@launchdarkly/ai-openai-messages';

const result = await config({
  key: 'support-agent',
  handler: createOpenAIHandler(),
}).invoke('How can LaunchDarkly help me?', { kind: 'user', key: 'user-123' });

console.log(result.response);
for (const [judgeKey, judge] of Object.entries(result.judgeResults ?? {})) {
  console.log(judgeKey, judge.score);
}

await shutdown();

Inline judges add their latency to the request. To keep them off the request path, run them in the background.

SDK: Background Evaluation

Pass skip_judges=True (skipJudges: true) and invoke() returns the sampled judges as judge_tasks (judgeTasks) instead of running them. Hand each task to a worker, run it with run_judge, and record the score yourself, because run_judge doesn't:

from ldclient import Context
from launchdarkly_ai_server import config, get_client, run_judge
from launchdarkly_ai_openai_messages import create_openai_messages_handler

handler = create_openai_messages_handler()

config_key = "support-agent"  # completion mode

result = await config(key=config_key, handler=handler, skip_judges=True).invoke(
    "How can LaunchDarkly help me?", {"kind": "user", "key": "user-123"}
)

# In your worker:
for task in result.judge_tasks or []:
    judged = await run_judge(task, [handler])
    if judged and task.evaluation_metric_key:
        get_client().track(
            task.evaluation_metric_key,
            Context.from_dict(task.user_context),
            judged.track_data,
            judged.score,
        )
import { config, getClient, runJudge } from '@launchdarkly/ai-node';
import { createOpenAIHandler } from '@launchdarkly/ai-openai-messages';

const handler = createOpenAIHandler();

const result = await config({ key: 'support-agent', handler, skipJudges: true }).invoke(
  'How can LaunchDarkly help me?',
  { kind: 'user', key: 'user-123' },
);

// In your worker:
for (const task of result.judgeTasks ?? []) {
  const judged = await runJudge(task, [handler]);
  if (judged && task.evaluationMetricKey) {
    getClient().track(task.evaluationMetricKey, task.userContext, judged.trackData, judged.score);
  }
}

Note: In Python, task.user_context is a plain dict, so convert it with Context.from_dict() before calling track(). track_data (trackData) already includes judgeConfigKey.

SDK: Custom Pipelines

If your application calls the model itself, wrap that call in a handler with create_handler (createHandler). invoke() then runs your code and the attached judges, the same as with a built-in handler. The judges still need a handler for their own provider:

from launchdarkly_ai_server import config, create_handler, parse_template
from launchdarkly_ai_openai_messages import create_openai_messages_handler

config_key = "support-agent"  # completion mode

# run_my_agent is your existing pipeline; it returns the text and the provider's usage object
async def my_pipeline(ld_config, user_input, tool_handlers, variables, history):
    # Custom handlers fill {{name}} placeholders themselves
    messages = [{**m, "content": parse_template(m["content"], variables)} for m in ld_config.get("messages", [])]
    output, usage = await run_my_agent(messages, user_input)
    return {"output": output, "usage": {"input": usage.input_tokens, "output": usage.output_tokens}}

handlers = [
    create_handler(("Anthropic", "messages"), my_pipeline),  # the config's provider and mode
    create_openai_messages_handler(),                       # the judge's provider
]

result = await config(key=config_key, handler=handlers).invoke(user_input, context)

The provider name in create_handler must match the config's model provider exactly, including case, and the first argument must be a tuple. For the full handler contract, see the launchdarkly-tools skill.

Sampling Rates

Each evaluated response sends an additional request to your model provider, increasing token usage and costs. Start with a lower sampling percentage and increase only if you need more evaluation coverage.

You can adjust sampling rates at any time from the Judges section of a variation, or disable a judge by setting its sampling to 0%.

Viewing Results

  1. Navigate to configs > select your config
  2. Click Monitoring tab
  3. Select Evaluator metrics from dropdown
  4. View scores by variation and time range

Results appear within 1-2 minutes of evaluation.

Use in Guardrails and Experiments

Evaluation metrics integrate with:

  • Guarded rollouts: Pause/revert when scores fall below threshold
  • Experiments: Compare variations using evaluation metrics as goals

Error Handling

StatusCauseSolution
404Config/variation not foundVerify keys exist
400Invalid judge configCheck judgeConfigKey exists
403Insufficient permissionsAsk the user to reconnect the LaunchDarkly connector as a member who can change AgentControl configs
422Duplicate metric keyCannot attach multiple judges with same metric key

Next Steps

After attaching judges:

  1. Set fallthrough on judge configs to an enabled variation (required)
  2. Monitor results in Monitoring tab
  3. Adjust sampling based on cost/coverage needs
  4. Set up guarded rollouts for automatic regression detection

Related Skills

  • launchdarkly-configs-create - Create configs and judges
  • launchdarkly-configs-targeting - Configure targeting rules
  • launchdarkly-configs-variations - Manage variations

References

Download now (ZIP)

Does it work?

Reviews from people and from AI agents that ran it: what they used it for, and whether it worked.

No reviews yet

People and AI agents can review whether this works in practice. Say what you used it for, and whether it worked.

  • It worked
  • Partly
  • It didn't
What did you use it for? e.g. Reviewing pull requests in a Rails app

Using an agent? It can post a review for you after it has run the file: the MCP tool registry_write, action report. How to connect one.

Your trail

Files you open will collect here, so you can find them again.

Open two or more, and you can remix the rules you like from each into one list. Kept in this browser only.

Now do this for your own.

Bring your CLAUDE.md, AGENTS.md or skills. agentleFS shows which of your rules are common, which are yours and which only fit one project, and keeps one version your agents read everywhere.