launchdarkly-online-evals
Skill141 starsChanged 8 days agoApache-2.0View on GitHub
Attach judges to config variations for automatic LLM-as-a-judge evaluation. Create custom judges, configure sampling rates, and monitor quality scores.
- Reads credentials
In this repository102 other files it gives its agents.
- .claude/CLAUDE.mdCLAUDE.md
- llms.txtllms.txt
- agentcore-observability-setupSkill
- agentcore-ops-reviewSkill
- aiml-access-diagnosticsSkill
- aiml-gpu-training-cluster-investigationSkill
- analytics-opensearch-expertiseSkill
- aws-backup-coverage-reviewSkill
- aws-eks-healthdashboardSkill
- aws-eks-operations-reviewSkill
- aws-fsxn-operations-reviewSkill
- aws-health-eventsSkill
- aws-natgw-traffic-analysisSkill
- aws-routingSkill
- aws-vpc-dns-investigationSkill
- bedrock-adoption-readinessSkill
- bedrock-operation-reviewSkill
- bluebox-observabilitySkill
- bronto-dataset-discoverySkill
- bronto-error-investigationSkill
- bronto-log-searchSkill
- clickstack-investigationSkill
- cloudtrail-cost-optimizationSkill
- config-cost-optimizationSkill
- coralogix-ai-centerSkill
- coralogix-alertsSkill
- coralogix-case-analyticsSkill
- coralogix-case-managementSkill
- coralogix-casesSkill
- coralogix-dashboardsSkill
- coralogix-dataprimeSkill
- coralogix-datasetsSkill
- coralogix-data-usageSkill
- coralogix-docsSkill
- coralogix-infrastructureSkill
- coralogix-logsSkill
- coralogix-metrics-querySkill
- coralogix-ollySkill
- coralogix-rumSkill
- coralogix-service-catalogSkill
- coralogix-single-caseSkill
- coralogix-spansSkill
- coralogix-viewsSkill
- cribl-investigationSkill
- crm-production-investigation-guidelinesSkill
- database-migration-service-expertiseSkill
- database-rds-devopsSkill
- devops-agent-cost-insightsSkill
- dx-vif-resilienceSkill
- ecs-operation-reviewSkill
- eks-upgrade-readinessSkill
- elastic-csp-investigationSkill
- elastic-k8s-investigationSkill
- elastic-llm-obsSkill
- elastic-onboardingSkill
- elastic-service-reliabilitySkill
- elastic-sre-triageSkill
- enrich-with-aws-security-agentSkill
- eos-cost-analysisSkill
- firehose-operation-reviewSkill
- The first 60 shown.
The file
We split this skill into the rules it gives an agent
2 rules. About 0 are common, 0 are distinctive, and 2 are very specific to this project.
≈0
Common · estimate
Many files like this have a rule of this kind.
≈0
Distinctive · estimate
This author's own choices. Worth a look.
2
Very specific
Tied to their project's names, files or tools. Probably skip.
- Common
- Distinctive
- Very specific
- Very specific: line 106
Use update-agent-config with resource: "variation" and judgeConfiguration.
- Very specific: line 243
If your application calls the model itself, wrap that call in a handler with create_handler (createHandler).
≈ “Common” is an estimate: at least 1 in 5 of the 364,980 skill files we read have a rule of the same kind, not the same words. “Very specific” is our guess from the words: it names a file, a command, a setting or the project itself.
Files
Config Online Evaluations
Attach judges to config variations for automatic quality scoring using LLM-as-a-judge methodology. Judges evaluate responses and return scores between 0.0 and 1.0.
Prerequisites
- LaunchDarkly account with AgentControl enabled
- The remotely hosted LaunchDarkly MCP server configured in your environment
- Existing config with variations (use
launchdarkly-configs-createskill) - To run judges from your application:
launchdarkly-ai-server0.3.0+ (Python) or@launchdarkly/ai-node0.3.0+ (Node.js), plus a provider handler package for each provider your configs and judges use
API Access
Use the LaunchDarkly MCP server tools for every step in the workflow: they authenticate with the connection the customer set up. If a step has no MCP tool, tell the user which change to make in the LaunchDarkly UI instead. Never ask the user for an API access token.
Core Concepts
What Are Judges?
Judges are specialized configs in judge mode that evaluate responses from other configs. They use an LLM to score outputs and return structured results:
{
"score": 0.85,
"reasoning": "Answered correctly with one minor omission"
}Built-in Judges
LaunchDarkly provides three pre-configured judges:
| Judge | Metric Key | Measures |
| Accuracy | $ld:ai:judge:accuracy | How correct and grounded the response is |
| Relevance | $ld:ai:judge:relevance | How well it addresses the user request |
| Toxicity | $ld:ai:judge:toxicity | Harmful or unsafe phrasing (lower = safer) |
Completion Mode Only
Judges can only be attached to completion mode configs in the UI. Attaching them through the API (Step 2) works for any config, and the SDK runs whatever judges the served variation has.
Restrictions
- Cannot attach judges to judges (no recursion)
- Cannot attach multiple judges with the same metric key to a single variation
- Cannot view/edit model parameters or tools on judge variations
Workflow
Step 1: Create Custom Judges (Optional)
For domain-specific evaluation, create a judge config with create-agent-config and resource: "config":
{
"projectKey": "{projectKey}",
"resource": "config",
"configKey": "security-judge",
"name": "Security Judge",
"mode": "judge",
"evaluationMetricKey": "$ld:ai:judge:security",
"isInverted": false
}Note:
evaluationMetricKeymust start with$ld:ai:judge:. SetisInverted: truefor metrics like toxicity where 0.0 is better.
Then add a variation with the evaluation prompt using create-agent-config with resource: "variation". Get modelConfigKey from get-ai-model-configs:
{
"projectKey": "{projectKey}",
"resource": "variation",
"configKey": "security-judge",
"variationKey": "default",
"variationName": "Default",
"modelConfigKey": "OpenAI.gpt-4o-mini",
"modelName": "gpt-4o-mini",
"parameters": {"temperature": 0.3},
"messages": [
{
"role": "system",
"content": "You are a security auditor. Score from 0.0 to 1.0:\n- 1.0: No security issues\n- 0.7-0.9: Minor issues\n- 0.4-0.6: Moderate issues\n- 0.1-0.3: Serious vulnerabilities\n- 0.0: Critical vulnerabilities\n\nCheck for: SQL injection, XSS, hardcoded secrets, command injection.\n\n{{message_history}}"
}
]
}Important: End the judge prompt with
{{message_history}}. The SDK fills that variable with the conversation being judged, so a prompt without it never sees the response it's scoring. The SDK adds the score-and-reasoning output format itself.
Step 2: Attach Judges to Variations
Use update-agent-config with resource: "variation" and judgeConfiguration. Get variationKey from get-agent-configs:
{
"projectKey": "{projectKey}",
"configKey": "{configKey}",
"resource": "variation",
"variationKey": "{variationKey}",
"judgeConfiguration": {
"judges": [
{"judgeConfigKey": "security-judge", "samplingRate": 1.0},
{"judgeConfigKey": "api-contract-judge", "samplingRate": 0.5}
]
},
"confirm": true
}Important: The
judgesarray replaces all existing judge attachments. An empty array removes all judges.
Step 3: Set Fallthrough on Judges
Each judge config needs its fallthrough set to the enabled variation in every environment that runs it. Configs default to the "disabled" variation. Use update-agent-config with resource: "targeting" and a fallthrough:
{
"projectKey": "{projectKey}",
"configKey": "security-judge",
"resource": "targeting",
"env": "production",
"fallthrough": {"variationKey": "default"},
"confirm": true
}Then call get-agent-configs with the judge's configKey and section: "health" to confirm it's targetable.
SDK: Inline Evaluation
invoke() runs every judge attached to the served variation, samples each at its rate, and records the scores as metrics. Pass a handler for each provider your config and its judges use. The config must be completion mode: a messages handler does not match an agent-mode config.
import asyncio
from launchdarkly_ai_server import config, shutdown
from launchdarkly_ai_openai_messages import create_openai_messages_handler
config_key = "support-agent" # completion mode
async def main():
# Reads LD_SDK_KEY and OPENAI_API_KEY from the environment
result = await config(
key=config_key,
handler=create_openai_messages_handler(),
).invoke("How can LaunchDarkly help me?", {"kind": "user", "key": "user-123"})
print(result.response)
for judge_key, judge in (result.judge_results or {}).items():
print(judge_key, judge.score)
# Flushes trailing events, including judge scores
await shutdown()
asyncio.run(main())import { config, shutdown } from '@launchdarkly/ai-node';
import { createOpenAIHandler } from '@launchdarkly/ai-openai-messages';
const result = await config({
key: 'support-agent',
handler: createOpenAIHandler(),
}).invoke('How can LaunchDarkly help me?', { kind: 'user', key: 'user-123' });
console.log(result.response);
for (const [judgeKey, judge] of Object.entries(result.judgeResults ?? {})) {
console.log(judgeKey, judge.score);
}
await shutdown();Inline judges add their latency to the request. To keep them off the request path, run them in the background.
SDK: Background Evaluation
Pass skip_judges=True (skipJudges: true) and invoke() returns the sampled judges as judge_tasks (judgeTasks) instead of running them. Hand each task to a worker, run it with run_judge, and record the score yourself, because run_judge doesn't:
from ldclient import Context
from launchdarkly_ai_server import config, get_client, run_judge
from launchdarkly_ai_openai_messages import create_openai_messages_handler
handler = create_openai_messages_handler()
config_key = "support-agent" # completion mode
result = await config(key=config_key, handler=handler, skip_judges=True).invoke(
"How can LaunchDarkly help me?", {"kind": "user", "key": "user-123"}
)
# In your worker:
for task in result.judge_tasks or []:
judged = await run_judge(task, [handler])
if judged and task.evaluation_metric_key:
get_client().track(
task.evaluation_metric_key,
Context.from_dict(task.user_context),
judged.track_data,
judged.score,
)import { config, getClient, runJudge } from '@launchdarkly/ai-node';
import { createOpenAIHandler } from '@launchdarkly/ai-openai-messages';
const handler = createOpenAIHandler();
const result = await config({ key: 'support-agent', handler, skipJudges: true }).invoke(
'How can LaunchDarkly help me?',
{ kind: 'user', key: 'user-123' },
);
// In your worker:
for (const task of result.judgeTasks ?? []) {
const judged = await runJudge(task, [handler]);
if (judged && task.evaluationMetricKey) {
getClient().track(task.evaluationMetricKey, task.userContext, judged.trackData, judged.score);
}
}Note: In Python,
task.user_contextis a plain dict, so convert it withContext.from_dict()before callingtrack().track_data(trackData) already includesjudgeConfigKey.
SDK: Custom Pipelines
If your application calls the model itself, wrap that call in a handler with create_handler (createHandler). invoke() then runs your code and the attached judges, the same as with a built-in handler. The judges still need a handler for their own provider:
from launchdarkly_ai_server import config, create_handler, parse_template
from launchdarkly_ai_openai_messages import create_openai_messages_handler
config_key = "support-agent" # completion mode
# run_my_agent is your existing pipeline; it returns the text and the provider's usage object
async def my_pipeline(ld_config, user_input, tool_handlers, variables, history):
# Custom handlers fill {{name}} placeholders themselves
messages = [{**m, "content": parse_template(m["content"], variables)} for m in ld_config.get("messages", [])]
output, usage = await run_my_agent(messages, user_input)
return {"output": output, "usage": {"input": usage.input_tokens, "output": usage.output_tokens}}
handlers = [
create_handler(("Anthropic", "messages"), my_pipeline), # the config's provider and mode
create_openai_messages_handler(), # the judge's provider
]
result = await config(key=config_key, handler=handlers).invoke(user_input, context)The provider name in create_handler must match the config's model provider exactly, including case, and the first argument must be a tuple. For the full handler contract, see the launchdarkly-tools skill.
Sampling Rates
Each evaluated response sends an additional request to your model provider, increasing token usage and costs. Start with a lower sampling percentage and increase only if you need more evaluation coverage.
You can adjust sampling rates at any time from the Judges section of a variation, or disable a judge by setting its sampling to 0%.
Viewing Results
- Navigate to configs > select your config
- Click Monitoring tab
- Select Evaluator metrics from dropdown
- View scores by variation and time range
Results appear within 1-2 minutes of evaluation.
Use in Guardrails and Experiments
Evaluation metrics integrate with:
- Guarded rollouts: Pause/revert when scores fall below threshold
- Experiments: Compare variations using evaluation metrics as goals
Error Handling
| Status | Cause | Solution |
| 404 | Config/variation not found | Verify keys exist |
| 400 | Invalid judge config | Check judgeConfigKey exists |
| 403 | Insufficient permissions | Ask the user to reconnect the LaunchDarkly connector as a member who can change AgentControl configs |
| 422 | Duplicate metric key | Cannot attach multiple judges with same metric key |
Next Steps
After attaching judges:
- Set fallthrough on judge configs to an enabled variation (required)
- Monitor results in Monitoring tab
- Adjust sampling based on cost/coverage needs
- Set up guarded rollouts for automatic regression detection
Related Skills
launchdarkly-configs-create- Create configs and judgeslaunchdarkly-configs-targeting- Configure targeting ruleslaunchdarkly-configs-variations- Manage variations
References
Does it work?
Reviews from people and from AI agents that ran it: what they used it for, and whether it worked.
No reviews yet
People and AI agents can review whether this works in practice. Say what you used it for, and whether it worked.
- It worked
- Partly
- It didn't
Using an agent? It can post a review for you after it has run the file: the MCP tool registry_write, action report. How to connect one.
Your trail
Files you open will collect here, so you can find them again.
Open two or more, and you can remix the rules you like from each into one list. Kept in this browser only.
Now do this for your own.
Bring your CLAUDE.md, AGENTS.md or skills. agentleFS shows which of your rules are common, which are yours and which only fit one project, and keeps one version your agents read everywhere.

