AwesomeJev.github.io / site

# Awesome Jev

Independent community project, not affiliated with TypeSafe AI.
Canonical website: https://awesomejev.github.io/
Curated snapshot: 2026-10-18. Findings are author-reported; no model experiments reproduced here.

# Jev guide: decisions, confidence and evaluation

Jev is a useful case study in models that return decisions for software. This independent guide separates documented interfaces, provider claims, and empirical evaluation. It does not reproduce model experiments.

## What is Jev?

Jev is TypeSafe’s System One model. A request supplies state and typed questions; the response contains bounded answers and probabilities that application code can use for classification, routing, or review. [TypeSafe introduction](https://docs.typesafe.ai/introduction)

TypeSafe describes its training as reinforcement learning for calibrated decisions, or RLCD. That description is a provider claim, not enough information to reconstruct Jev’s architecture or training procedure. [TypeSafe launch article](https://typesafe.ai/blog/introducing-system-one-models-and-jev)

## How does a typed decision differ from an LLM’s structured output?

A typed decision interface specifies the permitted alternatives before inference. Generative LLMs can also use constrained decoding to produce structured outputs. An interface alone does not establish which model is more accurate, calibrated, or efficient. [XGrammar paper](https://arxiv.org/abs/2411.15100)

A valid answer can still select the wrong option. Keep schema validity separate from semantic correctness, and test ambiguous instructions, adversarial inputs, and option-order changes on your own workload. [Jev 1.13 limitations](https://docs.typesafe.ai/model-jaggedness/jev-1.13)

## What do Choice, Score, and Noul return?

Choice selects from caller-defined options and returns the selected option, a probability for every option, and confidence. Use it for a bounded category or action. [Choice documentation](https://docs.typesafe.ai/primitives/choice)

Score evaluates ordered, descriptive levels. Its score is the probability-weighted mean of level indices, accompanied by the distribution and confidence; it is not an exact measurement extracted from the input. [Score documentation](https://docs.typesafe.ai/primitives/score)

Noul returns the probability that a yes/no proposition is true. It has no separate confidence field. A value near the middle signals uncertainty, not necessarily a medium degree of the property. [Noul documentation](https://docs.typesafe.ai/primitives/noul)

Keep ordinal label usage, a Score mean, returned confidence, and probability calibration separate. In the tested models and datasets, the v2 ordinal-scale study reports concentrated final choices despite broad average candidate probabilities; finer scales increased compression, and accuracy gains did not necessarily restore utilization. Where a final label is selected, compare its frequency with reference-label frequencies; do not assume every level should occur equally often. Treat this as an additional diagnostic, not a calibration test. The findings are author-reported in the abstract, not our reproduction or evidence that Jev’s Score API has the same failure. [Ordinal-scale utilization study, v2](https://arxiv.org/abs/2609.38827v2)

## Is confidence the probability that an answer is correct?

Not automatically. TypeSafe derives confidence from the returned distribution. For Choice, four options with a highest probability of 0.70 produce confidence 0.60. Score uses a different formula. Calibration concerns observed frequencies across predictions, not a guarantee about one answer. [Confidence documentation](https://docs.typesafe.ai/confidence)

## How should I evaluate decisions and calibration?

Separate prediction quality from probability quality. Alongside accuracy or a task-appropriate error measure, inspect reliability plots and proper scoring rules such as Brier score and log loss. Evaluate on held-out data; do not tune thresholds on the final test set. [Calibration research](https://proceedings.mlr.press/v70/guo17a.html) · [Proper scoring rules](https://sites.stat.washington.edu/people/raftery/Research/PDF/Gneiting2007jasa.pdf)

For abstention or escalation, report coverage and error among accepted cases, and test whether the fallback actually corrects mistakes. Freeze model versions and prompts; measure end-to-end latency, failures, and costs separately. This is an evaluation plan, not a result reported for Jev. [SelectiveNet](https://proceedings.mlr.press/v97/geifman19a.html) · [Model versions](https://docs.typesafe.ai/models)

AudioJev uses order calibration to mean probability stability under equivalent candidate permutations. Align candidates by meaning before comparing reordered distributions, then assess accuracy and correctness calibration separately. Its released model card describes one seed, reports original-order scores and discloses audio-source overlap; those results should not be substituted for the paper’s three-seed averages. [AudioJev release scope](https://huggingface.co/shlv/AudioJev)

## Can a calibrated judge still be biased?

Keep probability calibration, agreement with human rankings and verdict stability separate. The CALM study finds position and rewrite sensitivity in its tested typed-decision judges despite favorable calibration or consistency. Its labels and rewrites are inherited from benchmarks; changes in clarity and other unintended factors limit a causal reading of tone effects. [Typed decision models as judges, v1](https://arxiv.org/html/2610.18122v1#Sx1)

PRISM uses counterfactual emphasis probes and human labels to fit a correction to overall judge scores. Its reported ranking-agreement gains on SummEval and HANNA are a different target from matching probabilities to observed correctness. Rubric-free does not mean label-free, and the extra probes have a cost. This PRISM method is separate from the similarly named Prism-1 option-readout architecture. [PRISM original abstract, v1](https://arxiv.org/abs/2610.18389v1) · [PRISM limits, v1](https://arxiv.org/html/2610.18389v1#Sx1) · [Prism-1, v1](https://arxiv.org/abs/2610.22066v1)

Our proposed evaluation reports ranking agreement, probability reliability and perturbation sensitivity in separate columns. Verify that a supposedly meaning-preserving rewrite retains the task’s label, reserve held-out groups for learned corrections, and count probe calls in the evaluation budget. Repeated identical requests estimate variability; they do not establish correctness.

## How should I compare decision models with trained classifiers and LLMs?

The v2 decision-gate benchmark reports that a fine-tuned DeBERTa-v3-large had the highest observed accuracy on all but one labeled benchmark. That result concerns access to task labels. For your comparison, separate settings with and without task-specific training labels; record data exposure, development/test splits, output formats and failures before comparing accuracy, selective coverage or cost. [Decision-gate benchmark, v2](https://arxiv.org/abs/2610.00346v2)

LLM-as-Jev v3 studies frozen Qwen readouts, optional KL-anchored fine-tuning and budgeted reasoning. To distinguish these effects, we recommend a frozen comparator with the same backbone, explicit option sets and splits, and separately recorded reasoning budgets. The reported comparisons concern two Qwen backbones and an independent framework; they do not reveal hosted Jev’s training or architecture. [LLM-as-Jev, v3](https://arxiv.org/abs/2610.02076v3)

HakemBench v1 states that most Turkish reference labels are AI-derived, not human-verified, and that earlier test results influenced the lab’s own model on flagged tracks. Report label provenance, known prior test exposure and per-track metric components. Translation probes should remain separate from independent language validation. These are proposed comparison checks; this collection has not reproduced the studies. [HakemBench, v1](https://arxiv.org/abs/2610.02293v1)

Question grouping is part of the inference protocol. Perplexity Decider v1.2 lets questions within a request attend to one another; its fast path isolates separate requests. The card reports that grouping related attributes can help while grouping competing retrieval candidates can hurt. By contrast, tasksource-decider-nano describes tree attention that isolates options while sharing the state. Record the grouping rule alongside scores; a multi-question API does not imply independent answers. [Perplexity v1.2: question grouping](https://huggingface.co/perplexity-ai/pplx-decider-v1.2/raw/225c913ddc8761a8264f04f64548e1f97e443eb3/README.md) · [tasksource: tree attention](https://huggingface.co/tasksource/tasksource-decider-nano/raw/46e88e08cc1a3122a348ce134f18865e93f8e1c8/README.md)

## What does a fast benchmark establish about deployment?

Identify whether the evidence comes from measured equipment, simulation, replay or an operating service. The HVAC study uses public equipment measurements, but few fault days and feature-dependent diagnoses limit transfer. HydroJEV uses simulated windows with the same hydraulic model generating both events and the twin. Its interval margins and multiplicity-corrected tests are distinct; one transfer endpoint does not pass the adjusted test. Neither study establishes a general field-performance guarantee. [HVAC study limits, v1](https://arxiv.org/html/2610.09937v1#S5.SS5) · [HydroJEV simulation limits, v1](https://arxiv.org/html/2610.02048v1#S4.SS4) · [Transfer endpoints](https://arxiv.org/html/2610.02048v1#Sx18)

Keep successful-request latency separate from observation, feature construction, retry waits, queueing and downstream execution. The wireless study excludes pacing and retry waits and reuses an external orchestration study to show that faster interpretation need not improve completed service outcomes. The traffic-classification study does not time packet observation or feature extraction. Our proposed deployment comparison records every stage, deadline misses and final utility under the same workload. [Wireless timing boundaries, v1](https://arxiv.org/html/2610.04345v1#S4.SS1) · [External orchestration evidence](https://arxiv.org/html/2610.04345v1#S4.SS4) · [Traffic-classification scope, v1](https://arxiv.org/html/2610.00376v1#S5)

Output validity and paraphrase stability are not sufficient tests of control behavior. The 5G testbed study changes only the question while retaining state and options, exposing a fine-tuned checkpoint that keeps returning its training answer. Its RAN plane is not closed, there is no real radio, and risk statements depend on the calibration protocol. Test the expected change in action as well as invariance, and evaluate the complete controller rather than inferring reliability from a typed response. [Changed-question controls, v1](https://arxiv.org/html/2609.33689v1#S8) · [Testbed and certification limits](https://arxiv.org/html/2609.33689v1#S9)

## Can I reuse a confidence gate across languages or domains?

Before reusing a confidence gate in a new language or domain, build a target-specific test set with explicit label definitions and annotation provenance; translated or model-generated labels are not independent human validation. Compare the unchanged gate with a threshold tuned only on separate target-domain validation data. Report accepted-case error and coverage per group, including high-confidence errors under independently checked, answer-preserving context additions.

The social-science audit reports task-dependent confidence failures using benchmark labels; JevOut’s human checks cover only sampled successful perturbations. Neither abstract establishes multilingual transfer. Language-specific label checks and threshold-transfer comparisons are our proposed protocol, not reproduced findings or population-wide guarantees. [Social-science annotation audit, v2](https://arxiv.org/abs/2609.24574v2) · [JevOut, v2](https://arxiv.org/abs/2609.30243v2)

Implementation documentation supplies additional transfer limits. FRIDA-Decisions recommends short Russian–English states and uses Russian instruction suffixes. Jev-AR reports synthetic Arabic training without native-speaker validation and says confidence fitted to its built-in routes is approximate on custom lists. These are documented operating conditions, not evidence that a confidence gate transfers; we did not run either implementation. [FRIDA-Decisions limits](https://github.com/ai-forever/FRIDA-Decisions) · [Jev-AR limitations](https://github.com/atmaneayoubdev/jev-ar)

The Indonesian regional-language study supplies a concrete separation: expected calibration error did not increase while error-ranking ability degraded. Its translation comparison uses human Indonesian translations; classifier comparisons depend on label budgets, and cascades on the chosen fallback and routing fraction. Evaluate those factors separately before transferring a threshold. The preregistration and release claims remain author-reported here; no prediction files were audited. [Indonesian regional-language transfer study, v1](https://arxiv.org/abs/2610.14112v1)

The candidate-coverage benchmark compares source-frozen and target-local thresholds at equal label budgets. A threshold can fail after a task change even when its source calibration constraint was met. Separate artificial omissions, natural retrieval misses and out-of-scope inputs; the latter two need their own denominators. Our calibration and abstention note connects these tests with interface coherence, random-escalation controls and total cascade cost. [Candidate coverage and policy transfer, v2](https://arxiv.org/abs/2610.03387v2) · [Calibration and abstention research note](https://awesomejev.github.io/topics/calibration-and-abstention/)

Language and task changes must be separated. The stance study compares English single-text records with Chinese conversations, while moderation studies vary policies, labeled precedents and answer menus. These are useful task observations, not isolated language-transfer experiments. The language-task note also distinguishes extra-question cost from quality and frozen backbones from supervised calibration. [Stance study, v1](https://arxiv.org/abs/2610.11901v1) · [Language-task evaluation note](https://awesomejev.github.io/topics/language-task-evaluation/)

## Is a fallback option the same as a confidence gate?

A named Other or Defer option is an answer the model must select according to its definition. A confidence gate is application code deciding whether to act on a returned answer or escalate it. TypeSafe’s routing example illustrates the latter. Define which interface you are testing before comparing reliability. [Fallback definitions, v1](https://arxiv.org/html/2610.19856v1#S2.SS0.SSS0.Px2) · [TypeSafe confidence routing](https://docs.typesafe.ai/patterns/confidence-routing)

The Paired Fallback Robustness test changes the correct output from a regular option to fallback while holding the input and option positions fixed. It also tests equivalent fallback definitions. On the tested open models, correct recognition and even a supplied correct category do not guarantee correct reassignment. This is a controlled diagnosis on text datasets, not an established production repair. [Paired reassignment and definition tests, v1](https://arxiv.org/html/2610.19856v1#S3.SS1) · [Study limitations, v1](https://arxiv.org/html/2610.19856v1#Sx1)

Our suggested evaluation reports regular-choice accuracy, correct fallback selection, unnecessary fallback and disagreement between equivalent definitions separately. Test the application’s confidence gate on held-out cases and include escalated calls in its cost. Order stability and stable relative odds are additional properties: Prism-1 explicitly identifies none-of-the-above rules as a poor fit for independent option scoring. [Prism-1 scope and limitations, v1](https://arxiv.org/html/2610.22066v1#S7)

## How should I test decisions over tables and cited evidence?

Separate schema validity, stability under content-preserving changes, and semantic correctness against reference labels. For tables, pair original inputs with alternative layouts or row/column reorderings only when meaning is preserved. Keep values, relationships and arithmetic fixed, then compare choices and probability distributions. A stable answer can still be wrong; low confidence alone does not identify an error. [Tabular decision preprint, v1](https://arxiv.org/abs/2610.19536v1)

For citations, keep the claim and arithmetic fixed while swapping same-number passages that support a different financial role. Include equivalent valid evidence as positive controls, and measure false acceptance of mismatches and false rejection of valid support separately. Inspect paired disagreements rather than counting every changed answer as wrong. These constructed cases are diagnostic probes, not estimates of error rates in real traffic. [Same-number citation preprint, v1](https://arxiv.org/abs/2610.08675v1)

## How should I test tool arguments and option-channel risks?

Separate schema validity, whether cited evidence supports a proposed argument, and whether the argument follows from the request, history and schema defaults. Test correct defaults or inferred values without explicit references, alongside incorrect values paired with visible references. Reference availability is not derivation correctness. If a model writes references from the full history, hide the candidate from it and verify its output; generated references are not ground truth. [Tool-argument verification preprint, v1](https://arxiv.org/abs/2610.18497v1)

Define who can alter request text, tool outputs, option names and policy definitions. Test each channel your threat model lets an attacker control, including confidence on incorrect decisions. The cited guardrail experiments concern open-weight typed-decision models, not a demonstrated compromise of hosted Jev. This is a proposed evaluation protocol, not a reproduced result. [Option-channel guardrail preprint, v1](https://arxiv.org/abs/2610.12292v1)

Separate evidence of label sensitivity from its mechanism. In the open implementations studied by Labels Override Definitions, changing label/definition rendering while holding weights fixed changed the bias. For a system you can inspect, record the rendered request and compare conflicting labels with definition-only or neutral-label variants. This is a proposed diagnostic; it neither identifies hosted Jev’s internal mechanism nor establishes a universal defense. [Labels Override Definitions, v1](https://arxiv.org/abs/2610.02586v1)

## What does an attack success rate measure?

Define the outcome and denominator before calling an attack successful. Probability shifts, action changes, attacker-target selections and downstream harm are different outcomes. Same Attack measures pass-after-attack only among malicious cases each judge flagged when clean. Its separate Jev gate-bypass measure requires a less restrictive action with returned confidence for the severity Score ≥0.7; review can qualify, while low-confidence pass need not. [Same Attack metric definitions, v1](https://arxiv.org/html/2610.17820v1#S4.SS5)

Compare attacks with matched unchanged requests using the same decision rule. JevAdvBench distinguishes a changed most-likely Score level from a changed expected score. For adaptive search, separate optimization calls from validation: Decision Hijacking freezes candidates, then uses five fresh calls and requires at least three target selections for case-level success. This finite-repeat rule does not guarantee later success. [JevAdvBench typed metrics, v1](https://arxiv.org/html/2609.31142v1#S1.p4.1) · [Decision Hijacking validation, v1](https://arxiv.org/html/2609.28613v1#S4.SS4)

Decision Hijacking reports 51 target selections in 2,550 calls for one condition, but 11 majority-success cases out of 510: the denominators answer different questions. Same Attack reports 75% raw severity changes on repeated Jev moderation calls but only 1% routed-action changes. These are author-reported examples, not reproduced results or a ranking. Check answer preservation and delivery separately; equal token counts alone do not prove that an edit was removed. [Call-level and case-level counts, v1](https://arxiv.org/html/2609.28613v1#A1.SS4.p2.1) · [Raw versus routed instability, v1](https://arxiv.org/html/2610.17820v1#A2) · [Delivery inference limits, v1](https://arxiv.org/html/2609.31142v1#S3.SS3.p3.1)

## Are the SDK and independent models released Jev weights?

The official SDK is a client for TypeSafe’s hosted API, not a Jev checkpoint. Independent projects such as Laya publish their own model artifacts. Similar question types or API compatibility do not establish identical architecture, behavior, or accuracy. [Official Python SDK](https://github.com/typesafe-ai/typesafe-sdk-python) · [Laya model card](https://huggingface.co/convaiinnovations/laya)

Check each release’s model card and license separately. For example, Bespoke Nimble 9B v3 declares CC BY-NC 4.0, which restricts commercial use. Publicly downloadable artifacts should not all be labeled unrestricted open-source models. [Bespoke Nimble v3 model card](https://huggingface.co/bespokelabs/Bespoke-Nimble-9B-v3)

API shape does not determine the inference path. Span-02 documents its own runtime and optional generated reasoning, with index settings that differ from normal serving. Lychee documents rejection rather than truncation for oversized requests. Record engine, hardware, reasoning mode, confidence definition and input limits with a comparison; these are publisher descriptions, not deployments tested by this collection. [Span-02 runtime and index settings](https://huggingface.co/respanai/span-02/raw/63cbdd774391f1b7d9bc3829179efe996d0191ce/README.md) · [Lychee model-card limits](https://huggingface.co/LycheeAI/Lychee-27B/raw/f3f95d18a3b1ef3c483c2a5d7e8e5272a2af31e2/README.md)

Input support and calibration can differ by modality within one checkpoint. Liquid d1-omni-600M accepts images or audio per request, limits speech to 30 seconds, and caps text at 896 tokens for image requests. Its fitted temperatures apply to text, not images or audio. The release uses custom inference code and the LFM Open License v1.0; inspect that license separately from the base model or other d1 releases. [Liquid d1-omni-600M: limits and calibration](https://huggingface.co/LiquidAI/d1-omni-600M/raw/02b55d7076f15129e59ab3f94783f32c4b088674/README.md) · [LFM Open License v1.0](https://huggingface.co/LiquidAI/d1-omni-600M/raw/02b55d7076f15129e59ab3f94783f32c4b088674/LICENSE)

## Does an open RLCD implementation reproduce Jev's training?

A shared RLCD name or compatible API does not establish identical training. OpenJev-RLCD explicitly describes an independent implementation and makes no claim about TypeSafe's internal algorithm. It trains a reasoning model's probability readout after sampled rationales; that experimental design is distinct from establishing how the hosted Jev service was trained. [OpenJev-RLCD scope, v1](https://arxiv.org/html/2609.38850v1#S7)

The two-stage recipe first trains the answer readout with a proper score, then reinforces rationales with that score and an anchor to the first-stage model. Its experiments use one 1.7B backbone, three seeds and short training runs; the second stage helps one main task and is neutral on the other. Keep the scored distribution, generated-rationale budget, fitting split and temperature-scaled baselines explicit. A scoring-rule argument is not by itself a deployment calibration guarantee. [Two-stage method and setup, v1](https://arxiv.org/html/2609.38850v1#S3) · [Experimental setup](https://arxiv.org/html/2609.38850v1#S4)

Distillation poses a different question: what can a smaller model learn from teacher judgments? The student-guided routing paper uses a fixed 60-label taxonomy and reports preliminary teacher agreement, with human-reviewed and full-baseline tests still pending. The search-distillation study trains compact evaluators offline; search supplies lookahead and teacher value differs across tasks. Compare label budgets and downstream outcomes, and separate offline labeling cost from runtime inference. [Student-guided routing: pending evaluation, v1](https://arxiv.org/html/2610.02516v1#S7) · [Search distillation: component roles, v1](https://arxiv.org/html/2610.09188v1#S7) · [Labeling cost](https://arxiv.org/html/2610.09188v1#S8)

## Where should I look for research and benchmark evidence?

Start with versioned empirical papers and their available code or data. The broad Jev benchmark study is one entry point; its findings remain author-reported, rather than independently reproduced by this collection. [Jev benchmark paper, v1](https://arxiv.org/abs/2609.37647v1) · [Accompanying benchmark code](https://github.com/AppliedMachineLearning-Lab/jev-benchmarking)

TypeSafe WorkflowEvals publishes workflows and model-generated reference labels; agreement with those references differs from human-label accuracy. The community Decision Index also contains private tests, so public rankings do not make the entire evaluation independently reproducible. The reviewed 0.3.2 snapshot uses a Jev 1.14 lab reference, while its frozen request payloads still name Jev 1.13. Record the model that produced a result separately from the model named in archived inputs. [WorkflowEvals](https://github.com/typesafe-ai/WorkflowEvals) · [Decision Index methodology snapshot](https://huggingface.co/spaces/multimodalart/jev-decision-index/blob/0ddf6758073eb77ae449a3541f57286d7f39d93f/data/methodology.json)

Model cards may describe different evaluation conditions within one table. KnowLine discloses checkpoint selection using public-suite scores and task/source overlap in training. Amos separates zero-shot results from a setting that fits question priors on the benchmark’s train split. Keep those conditions separate, and do not treat publisher-run public results as the private-inclusive Decision Index score or as our reproduction. [KnowLine selection and disclosures](https://huggingface.co/PelaAI/KnowLine-35B-A3B-Gen1/raw/e800968d38761c190476da5c69c6a5087a3c9b29/README.md) · [Amos evaluation conditions](https://huggingface.co/mukaddamzaid/amos-1-large/raw/d67e051c763dc88a1e3662a5003c1e1ea5ffaa74/README.md)

## What can I reproduce from a public research artifact?

Name what you want to reproduce: tables or plots from scored summaries, metrics from released per-example answers, fresh predictions, or a trained checkpoint. Map each output to its required inputs, versions and hashes. Report whether materials were only inspected, commands were executed, or outputs were matched; these are different claims.

Visual Jev’s pinned reproduction guide describes regenerating specified tables and two plots from scored JSON without a GPU. Other figures need omitted per-example predictions and intervention analyses. Fresh predictions require upstream corpora, backbone weights and a GPU; source images are not redistributed. [Visual Jev reproduction guide, pinned commit](https://github.com/guanxuyu-sv/Visual-Jev/blob/a392eaedfa41adae348f9767b55c0f54b3a13b57/REPRODUCE.md)

The observed ordinal-scale artifact documents metric recomputation from released answers. Fresh inference is separate: source text, weights and post-trained checkpoints are omitted, and missing historical data revisions can prevent a reconstruction matching recorded hashes. Its opaque file revision is not an immutable Git commit. Preserve mismatches rather than silently substituting current data. [Ordinal-scale study, v2](https://arxiv.org/abs/2609.38827v2) · [Observed anonymous README, opaque file revision](https://anonymous.4open.science/api/repo/jev_ordinal_scale_bia/file/README.md?v=3ed72968)

The radiology paper’s v1 availability statement says analysis scripts and aggregate results are being prepared for release. Third-party dataset links do not supply those missing author outputs. Our checks concern documentation and selected file inventories; we have not executed these reproduction routes or independently reproduced their results. [Radiology availability statement, v1](https://arxiv.org/html/2609.27607v1#Sx1.p1)

For control loops, a replay of isolated model latency does not establish deadline attainment. The network-control SoK separates request queues, execution times, action feasibility and responsibility for service checks; its examples use one bounded event model. Our proposed deployment check records arrival patterns and the complete path from request to verified outcome, alongside misses and infeasible actions. It does not substitute a synthetic replay for production evidence. [Semantic decision engines in network control, v1](https://arxiv.org/abs/2610.06425v1)

Check that abstract, manuscript and released code describe the same cohort and comparator. Jev-IDS v1 does not: PDF page 2 and the cited code describe a 300-flow GPT-5.6 Luna pilot, while page 7 describes 2,000 flows and Gemini. We retain the paper with this explicit caveat and link the pilot revision; no combined performance claim follows from the two configurations. [Jev-IDS v1: PDF page 2](https://arxiv.org/pdf/2610.01079v1#page=2) · [PDF page 7](https://arxiv.org/pdf/2610.01079v1#page=7) · [Paper-cited pilot revision](https://github.com/jev-sec/jev-ids/tree/286a310fea249a7b4a5a8c45377049e914f8b9c9)

# Curated resources

## Prism-1: Position- and Rival-Independent Scoring for System One Models

Papers · Teng et al. · 2026-10-15 · v1

An option-isolated readout targets order invariance and stable relative odds when candidates change; it requires fine-tuning, limits within-model comparison, and fits independently judged options better than menu-dependent fallback rules.

Primary source: https://arxiv.org/abs/2610.22066v1

## Qwen-Audio-Jev: A Unified Interface for Audio Decisions

Papers · Fang et al. · 2026-10-15 · v1

An independent Jev-style decision head on Qwen-Audio-3.1-ASR-Flash-Next gives Choice, Noul and Score readouts over audio, reporting 84.06% on MMSU, a 6.50% speaker-verification equal error rate and about 20× faster inference than its own base model.

Primary source: https://arxiv.org/abs/2610.21614v1

## JevAgentHarness: Benchmarking, Deploying and Training Decision Models That Share Control with LLMs

Papers · Zhu et al. · 2026-10-15 · v1

Studies fitted confidence thresholds for sharing agent control with decision models.

Primary source: https://arxiv.org/abs/2610.21412v1

## Jev-LiteMAS: Separating Decisions from Generation in Multi-Agent Workflows

Papers · Wang, Zhou, Lou & Li · 2026-10-15 · v1

Routes expert handoffs and screens artifacts before generative re-review; lower generative-token use depends on the workload, with fixed expert pools and savings that shrink as more artifacts need repair.

Primary source: https://arxiv.org/abs/2610.21411v1

## System1-Arena: Which Decisions Can Be Left to System 1?

Papers · Chen et al. · 2026-10-15 · v1

A benchmark spanning matched requests and closed-loop tasks. Experiments have not been reproduced here.

Primary source: https://arxiv.org/abs/2610.21244v1

## Jev for Scientific Decisions: Evaluating Semantic Choices and Their Consequences

Papers · Deng, Fan, Zhang & Xie · 2026-10-15 · v3

In ten prepared scientific cases, the authors separate semantic selections, computed quantities and final labels; comparison-model errors can be hidden by unchanged final labels.

Primary source: https://arxiv.org/abs/2609.24965v3

## Prune the Summary, Not the History: Source-Judged Compaction for Long-Horizon Agents

Papers · Min & Chen · 2026-10-14 · v1

Uses Jev to prune summary entries lacking source support; reported gains concern success in all three runs on two benchmarks, while useful constraints can be discarded and sourced errors retained.

Primary source: https://arxiv.org/abs/2610.20927v1

## From Detection to Decision: Calibration and Deferral for Prompt-Injection Detection

Papers · Lan, Kaul, Pandurangan & Sanyal · 2026-10-14 · v1

On 47,611 held-out prompt-injection examples, Jev ranks second in AUROC after an LLM judge, its native probabilities do not consistently beat post-hoc calibrated classifiers, and a 1% false-negative target forces over 91.8% escalation in every gate-reviewer system.

Primary source: https://arxiv.org/abs/2610.20907v1

## Verdict: Encode Once, Decide Many

Papers · Chouhan & Perry · 2026-10-14 · v1

Encodes shared state once for multiple typed questions; invariance requires identical retained state tokens, while teacher-derived labels, adaptive testing and unstable BoolQ training constrain the reported gains.

Primary source: https://arxiv.org/abs/2610.20778v1

## Lost in the Fallback: How Reliably Do Typed Decision Models Use Fallback Options?

Papers · Xie et al. · 2026-10-14 · v1

Paired tests on three text datasets separate category recognition from executing fallback rules; tested open models can disagree under equivalent definitions even when supplied the correct category, and the study diagnoses failures without establishing a fix.

Primary source: https://arxiv.org/abs/2610.19856v1

## Calibration and Risk-Budgeted Decisions with System One Models

Papers · Messoudi & Mansar · 2026-10-14 · v1

Jev and other System One models make binary allow/block or retain/discard decisions without human deferral; across six moderation, safety and reranking datasets, naive threshold selection exceeds the target risk in about half the settings, against 7.8% for Learn Then Test.

Primary source: https://arxiv.org/abs/2610.19811v1

## Selective Reasoning for Typed Decision Models: When to Start, Read, and Stop

Papers · Zheng & Chen · 2026-10-14 · v1

Gates reasoning start, candidate-answer reading and generation stopping; evidence is limited to a frozen Qwen backbone and three task families.

Primary source: https://arxiv.org/abs/2610.19644v1

## Are AI Decision Models Biased? A Case Study of Jev

Papers · Imperatrice, Kameneva, Greenberg & Romano · 2026-10-14 · v1

In 41,500 responses Jev's selected answers look largely unbiased, but its judgments shift with irrelevant numbers and option order, and several bias manipulations move its probabilities even when the answer stays the same; on 30 adapted conjunction tasks it violates the conjunction rule in 54% of 15,000 responses.

Primary source: https://arxiv.org/abs/2610.19619v1

## Typed Decision Models on Tabular Data

Papers · Zhang, Ye & Zha · 2026-10-14 · v1

Tests classification and verification under content-preserving table rewrites across three task families; confidence can flag instability without proving correctness.

Primary source: https://arxiv.org/abs/2610.19536v1

## SecJev: Bringing Security Expertise to System One Decision Models

Papers · Chen et al. · 2026-10-14 · v2

Adapts decision scorers to 14 security tasks; source-label prediction and displayed-policy execution are distinct targets, task difficulty is uneven, and benchmark accuracy does not establish end-to-end attack prevention.

Primary source: https://arxiv.org/abs/2610.03073v2

## More Choices, Fewer Decisions: Ordinal-Scale Bias in JEV-like Direct-Decision Models

Papers · Gao et al. · 2026-10-14 · v2

Audits ordinal label underuse across selected decision models; finer scales increase compression, while accuracy, final-label utilization and averaged candidate probabilities remain distinct measures.

Primary source: https://arxiv.org/abs/2609.38827v2

## DecisionRobust: Benchmarking Robustness for JEV Style Decision Models Across Text and Vision

Papers · Lan et al. · 2026-10-13 · v1

Text and vision attacks with label-validity checks and held-out evaluation models.

Primary source: https://arxiv.org/abs/2610.19118v1

## BLADE: Auditing Information Use for Failure Attribution in LLM Multi-Agent Systems

Papers · Guo, Long & Zhang · 2026-10-13 · v1

Uses Jev to audit information use at agent handoffs and localize failures; comparisons share one decision model on five benchmark subsets, and attribution depends on the completeness of retained traces.

Primary source: https://arxiv.org/abs/2610.18780v1

## Jeva: System One Models from First Principles

Papers · Li & Wang · 2026-10-13 · v1

A decision-model architecture exploring null handling, option permutations, scaling, and caching. Its design is Jeva’s, not Jev’s disclosed internals.

Primary source: https://arxiv.org/abs/2610.18773v1

## VideoJev: System One Model for Video Understanding and Interaction

Papers · Zhu et al. · 2026-10-13 · v1

Structured video decisions with temporal spans and candidate scores; cached and joint inference reuse visual computation.

Primary source: https://arxiv.org/abs/2610.18615v1

## Nudge the Judge: Uncovering Program Decision Hijacking in Real-World Jev Applications

Papers · Luo et al. · 2026-10-13 · v1

JevProbe traces attacker-controlled context into the decisions that 228 popular Jev applications act on; in 2,273 curated cases (at most ten per app, flips selected first, so not an unbiased sample) 81.5% flip, 98.6% of flips with potential downstream security impact, and existing defenses offer limited protection.

Primary source: https://arxiv.org/abs/2610.18612v1

## Jev-ToM: Evaluating Theory of Mind in Typed Decision Models

Papers · Chung et al. · 2026-10-13 · v1

On 11,117 English Theory of Mind questions from nine benchmarks, Jev reaches 73.2% source-balanced accuracy and ranks first or second in all six capability groups among twelve typed and generative models; confidence-based selection is benchmark-dependent, and supplied reasons repair or overturn answers depending on how they are attached.

Primary source: https://arxiv.org/abs/2610.18610v1

## Reading Right, Acting Wrong: Diagnosing Decision Failures of System One Models in Agent Loops

Papers · Li, Krishnan & Padman · 2026-10-13 · v1

Pairs factual probes with action choices to diagnose decision failures.

Primary source: https://arxiv.org/abs/2610.18579v1

## How Reliable Are System-One Decisions in Medicine? A Comparative Study of Jev, Jev-like Models, and LLMs

Papers · Du et al. · 2026-10-13 · v1

Across 15 medical datasets (8,742 instances), Jev rivals leading models on text-driven tasks, is weaker on numerical ones and is the most stable under question-format changes; most models miss absent correct answers. A benchmark study, not clinical validation.

Primary source: https://arxiv.org/abs/2610.18529v1

## Judges Follow the Reference: The Derivation Gap in Verifying Tool-Call Arguments

Papers · Ma et al. · 2026-10-13 · v1

Tool-argument judges including Jev reject some correct derived values without visible references; model-generated references are evaluated on three benchmarks.

Primary source: https://arxiv.org/abs/2610.18497v1

## Recalibrating Many Questions with Few Labels: Minimax Rates and Identification Limits

Papers · Li & Yang · 2026-10-13 · v1

Derives excess-Brier risk bounds for pooling recalibration across questions under shared-distribution and monotonicity assumptions; fixed-effects and empirical-Bayes results have different targets, and risk lower bounds do not directly become calibration-error lower bounds.

Primary source: https://arxiv.org/abs/2610.18488v1

## PRISM: Calibrating Black-Box Judges with Counterfactual Probes

Papers · Yamamoto & Kuroki · 2026-10-13 · v1

Learns a score correction from counterfactual probes and human labels; improved ranking agreement on two text tasks is distinct from probability calibration, and probe cost and cross-task transfer remain relevant limits.

Primary source: https://arxiv.org/abs/2610.18389v1

## Consider, Then Choose: Order-Invariant Decisions in System One Models

Papers · Huang, Teng, Deng & An · 2026-10-13 · v1

Runoff scores options individually, then balances finalist positions; order invariance assumes deterministic responses and a unique ranking, and does not resolve option-name bias or guarantee more accurate decisions.

Primary source: https://arxiv.org/abs/2610.18233v1

## Cheap, Consistent, Calibrated, but Biased? Typed Decision Models as Judges

Papers · Lukassen, Weisser & Silbersdorff · 2026-10-13 · v1

CALM pairwise challenges expose position and rewrite sensitivity despite consistency and calibration; inherited reference labels and rewrites that change more than tone limit causal interpretation and transfer to other judging tasks.

Primary source: https://arxiv.org/abs/2610.18122v1

## Same Attack, Different Judge: A First Controlled Adversarial Comparison of Jev, LLM Judges, and Purpose-Built Safety Classifiers

Papers · Li, Huang, Zhang & Liu · 2026-10-13 · v1

A controlled adversarial comparison of Jev, LLM judges, and purpose-built safety classifiers.

Primary source: https://arxiv.org/abs/2610.17820v1

## LLM-as-Jev: LLMs Are Already Jev-Style Decision Models -- When and How to Fine-Tune Them

Papers · Li & Wagle · 2026-10-13 · v3

Compares frozen Qwen decision readouts with optional KL-anchored fine-tuning and budgeted reasoning; reported gains depend on backbone size and routing workload, not disclosure of hosted Jev internals.

Primary source: https://arxiv.org/abs/2610.02076v3

## Decision Models vs. LLMs as Rubric Judges: Cheaper, Faster, and Wrong in the Same Places

Papers · Rao & Callison-Burch · 2026-10-13 · v3

Compares rubric judges and examines correlated errors that can limit cascade improvements.

Primary source: https://arxiv.org/abs/2609.29769v3

## Releasing the Inner Jev: A Simple Recipe for Open Decision Models

Papers · Geuter, Cheng & Alvarez-Melis · 2026-10-12 · v1

Answer-letter readout and supervised LoRA baselines; evaluation includes model-generated labels.

Primary source: https://arxiv.org/abs/2610.17205v1

## FunDM: Cheap & Label-Free Decision Models from Frozen Language Models

Papers · Panchal · 2026-10-12 · v1

Distills teacher probabilities into small heads using fewer layers of a frozen Qwen3-8B; label-free means no correct-answer training labels, and efficiency measurements cover one GPU and engine workload rather than production serving.

Primary source: https://arxiv.org/abs/2610.17175v1

## Does a Hosted Decision Model Respect Irrelevant Alternatives? A Pre-Registered Test of Option-Set Dependence in Jev

Papers · Rajput, Chauhan & Kosaraju · 2026-10-12 · v1

Adding an irrelevant option shifts jev-1.13.0’s gold-versus-rival log-odds, more so as options grow, so its probabilities are relative to the option list; independent yes/no scoring removes this but is less accurate (pre-registered, 299,463 calls, three datasets).

Primary source: https://arxiv.org/abs/2610.16655v1

## GraphDecisionBench: Benchmarking System One Models on Graph Tasks

Papers · Yang, Zhang & Zhao · 2026-10-12 · v3

Compares graph decisions across fourteen model-interface configurations; local adjacency, graph-text evidence use and sequential solution quality are separate capabilities.

Primary source: https://arxiv.org/abs/2610.06354v3

## General Decision Models: Benchmarking and Insights Beyond Jev

Papers · Duan et al. · 2026-10-12 · v2

Bilingual decision evaluation and reliability when local decisions compose into larger systems.

Primary source: https://arxiv.org/abs/2610.03935v2

## Suggested Is Not Superseded: A Pre-Registered Evaluation of a System-One Model as a Memory-Retirement Gate

Papers · Chen & Le · 2026-10-11 · v1

Jev memory-retirement evaluation separates confirmed updates from suggestions; mixed LLM/human labels, shared errors and selection on a small development set constrain cascade claims.

Primary source: https://arxiv.org/abs/2610.15710v1

## Poisoned into Action: Context Attacks on Typed Decision Workflows

Papers · Wu, Chang & Yan · 2026-10-11 · v1

Traces context poisoning through retrieval, typed choices and persisted execution while measuring retained clean utility; two synthetic development workflows and nonsignificant prespecified contrasts make the findings exploratory.

Primary source: https://arxiv.org/abs/2610.15537v1

## SanSi: A Looped Typed Decision Model for System 1.5 Thinking

Papers · Gan, Lee & Kang · 2026-10-10 · v2

Looped hidden-state computation for typed decisions; additional loops trade compute for accuracy.

Primary source: https://arxiv.org/abs/2610.07730v2

## JevCascade: Readout Stability and Inference-Time Compute Allocation for Prefill-Only Decision Models

Papers · Li & Chen · 2026-10-10 · v2

Candidate-menu interventions and confidence cascades; ranking stability does not establish probability calibration.

Primary source: https://arxiv.org/abs/2610.07716v2

## Koa-decision: Fast and Consistent Structured Decision Making with Generative LLMs

Papers · Dai et al. · 2026-10-10 · v2

Uses atomic label tokens and supervised fine-tuning for single-step constrained decisions; latency follows a specified sequential serving protocol, adaptation differs across baselines, and the production intent data cannot be released.

Primary source: https://arxiv.org/abs/2609.36115v2

## Jev in Medicine: A Benchmark Evaluation

Papers · Madrid-García & Merino-Barbancho · 2026-10-10 · v4

Preliminary medical-benchmark evaluation of Jev 1.13 reports task-dependent accuracy, calibration and weak unanswerable-question detection; it does not establish clinical deployment validity.

Primary source: https://arxiv.org/abs/2609.34024v4

## Same Scores, Different Decisions: Evaluating JEV and Language Models for Legal Document Understanding

Papers · Zhang et al. · 2026-10-10 · v2

ContractNLI request variations can change which answers are wrong while aggregate accuracy stays unchanged; stable answers may preserve errors. This benchmark does not validate real legal deployment.

Primary source: https://arxiv.org/abs/2609.27678v2

## DICE: Tilting the Odds through Interface Attackson System One Model

Papers · Dong, Lu, Guo & Wang · 2026-10-09 · v1

Tests bounded substitutions at state-extraction, question-generation and output-interpretation adapters while leaving authentic Jev responses intact; three synthetic pilots locate decision-integrity failures without estimating deployed attack prevalence.

Primary source: https://arxiv.org/abs/2610.14404v1

## Does System One Transfer? Calibrated Typed-Decision Models, Label Budgets, and Translation for Indonesian Regional Languages

Papers · Fadli, Rusdah, Utami & Amana · 2026-10-09 · v1

Tests Jev and multilingual Laya on Indonesian regional-language tasks; stable expected calibration error can coexist with worse error ranking, while human translation, label budgets and cascade choices change comparisons.

Primary source: https://arxiv.org/abs/2610.14112v1

## When Faster Decisions Do Not Make Better Agents: An Empirical Study of Jev in Agent Workflows

Papers · Ling, Yu, Chen & Fang · 2026-10-09 · v1

Jev cut decision cost and latency for five agents in ALFWorld and AndroidWorld (4,500 runs), but two mobile agents lost more success, extra reasoning and interactions absorbed the savings, and cost per success rose for one agent.

Primary source: https://arxiv.org/abs/2610.13092v1

## TypedBench: A Benchmark for Calibration, Framing Sensitivity, and Cost in System One Decision Models

Papers · Sharma, Duncan, Caron & Vollmer · 2026-10-09 · v2

The hosted model is wording-sensitive and underconfident, and under asymmetric costs acting on its probabilities can cost more than taking its top answer; synthetic policy-labelled suites also report paraphrase ranges and calibration against a finite-sample noise floor.

Primary source: https://arxiv.org/abs/2610.11392v2

## JEV-as-a-Judge: Accept When Confident, Escalate When Unsure

Papers · Li, Miao, Krishnan & Padman · 2026-10-09 · v5

Studies confidence-based escalation from Jev to a reasoning judge, including limits under distribution shift.

Primary source: https://arxiv.org/abs/2609.26550v5

## One Word Opens the Gate: The Option-Channel Attack on Typed Decision Models as Agent Guardrails

Papers · Azizi, Baghaei Potraghloo & Pedram · 2026-10-08 · v1

Tests option-channel attacks on seven open-weight models; reversed decisions can remain confident. The reported experiments do not test hosted Jev.

Primary source: https://arxiv.org/abs/2610.12292v1

## Specialized Decision Models vs. General-Purpose LLMs: Benchmarking Jev Across Knowledge, Reasoning, and Multilingual Tasks

Papers · Li et al. · 2026-10-08 · v1

Knowledge, reasoning and multilingual multiple-choice evaluation of Jev; reported rankings and mathematical weaknesses apply to the selected models and benchmarks.

Primary source: https://arxiv.org/abs/2610.11978v1

## Can Decision Models Understand Stance? Evaluating Jev Against General-Purpose LLMs

Papers · Li et al. · 2026-10-08 · v1

Compares Jev on English single-text stance and Chinese conversational stance. Support/opposition errors grow in the conversational task, but language, dataset and input structure change together; differing batching and prompt formats also limit a general language-capability ranking.

Primary source: https://arxiv.org/abs/2610.11901v1

## MetaEncoder: Exploring the Limit of Bi-Encoders for Multimodal System One Decision Making with Natural Language Interface

Papers · Cheng et al. · 2026-10-08 · v1

Bi-encoder approach to multimodal decisions and retrieval through natural-language interfaces.

Primary source: https://arxiv.org/abs/2610.11316v1

## JevOut: Natural Context Can Flip Decision Models

Papers · Xu et al. · 2026-10-08 · v2

Optimizes context additions around unchanged questions and options in four decision systems; results concern initially correct decisions, and human validity checks cover a sampled subset of successful attacks.

Primary source: https://arxiv.org/abs/2609.30243v2

## Type-Safe Is Not Error-Free: Typed Decision Models Follow the Option Name, Not the Definition Bound to It

Papers · Sun, Xu, Shi & Yang · 2026-10-08 · v3

Tests conflicts between option names and definitions despite structurally valid outputs.

Primary source: https://arxiv.org/abs/2609.26758v3

## Where Can a Decision Model Diagnose HVAC Faults? Reasoning Demand, Physical Representation, and Robustness Under Shift

Papers · Jung · 2026-10-07 · v1

Tests fault diagnosis across equipment, representations and operating shifts. Physical features enable selected diagnoses, while context-dependent faults remain difficult; shared-day training conditions, small fault-day counts and dataset-specific option sets limit general claims of robustness.

Primary source: https://arxiv.org/abs/2610.09937v1

## From Probabilities to Decisions: Search and Multi-Teacher Distillation with Jev

Papers · Sadoun, Sharif & Banad · 2026-10-06 · v1

Distills Jev and LLM judgments into compact search evaluators and rerankers; matched-label-budget gains in chess do not generalize to every tested game. The deployed bot also uses Stockfish, opening books and tablebases, and API wall time differs from GPU labeling time.

Primary source: https://arxiv.org/abs/2610.09188v1

## Same-Number Citation Swaps: Stress-Testing Jev as a Financial Evidence Judge

Papers · Xu et al. · 2026-10-06 · v1

Same-number citation swaps probe financial-evidence verification; rejecting wrong roles trades off with accepting equivalent support in a constructed test.

Primary source: https://arxiv.org/abs/2610.08675v1

## Token-Efficient Multi-Agent Collaboration via System One-Guided Computational Division of Labor

Papers · Zhou et al. · 2026-10-06 · v1

Separates bounded multi-agent coordination from LLM reasoning through a controller–reader loop; reported GPT-4o-token and end-to-end latency reductions concern complete framework comparisons on seven benchmarks.

Primary source: https://arxiv.org/abs/2610.08155v1

## Benchmarking System One Models in Online Moderation

Papers · Mazzoni & Failla · 2026-10-06 · v1

Separates written rules, retrieved precedents and candidate restrictions across five moderation benchmarks. Retrieval can improve discrimination or merely change flagging rates; using labeled evaluation examples as precedents and representing media through generated captions limits deployment claims.

Primary source: https://arxiv.org/abs/2610.07953v1

## JEV versus LLMs: Accuracy, Cost and Calibration on Seven Political Science Replications

Papers · Denney & DiGiuseppe · 2026-10-06 · v2

Studies accuracy, cost, and calibration in seven political-science replication tasks.

Primary source: https://arxiv.org/abs/2610.06625v2

## Benchmarking candidate coverage and rejection policy transfer in typed decision models

Papers · Lu & Wu · 2026-10-06 · v2

Separates artificial omissions, retrieval misses and out-of-scope queries under matched calibration budgets; a source-task threshold can reject covered target inputs or miss absent answers. Candidate membership, label ambiguity and small natural-miss samples limit transfer claims.

Primary source: https://arxiv.org/abs/2610.03387v2

## Benchmarking System One decision models against trained classifiers and language models for automated decision gates

Papers · Rafe & Das · 2026-10-06 · v2

A common-request harness compares typed decision models, generative readouts and trained classifiers; accuracy, selective coverage and out-of-scope acceptance depend on label access and task.

Primary source: https://arxiv.org/abs/2610.00346v2

## SharedKV-BT: Node-Local Typed Decisions for Behavior-Tree Agents

Papers · Wake & Wagle · 2026-10-05 · v1

Combines shared-prefix candidate scoring with behavior-tree node interfaces and external execution checks. The experiments separate scoring speed, stage restrictions and postcondition authority; predefined trees, simulated tasks and order-sensitive choices limit broader closed-loop reliability claims.

Primary source: https://arxiv.org/abs/2610.07327v1

## ufakzeka-karar: An Open Turkish Typed-Decision Model with Order-Invariant Option Scoring

Papers · Teke · 2026-10-05 · v1

An open 182M-parameter Turkish model scores each option blind to the others, so answers ignore option order, and adds a “not sure” signal; its 7th-of-16 HakemBench rank is not blind, since later runs trained after full-test results were read.

Primary source: https://arxiv.org/abs/2610.06744v1

## SoK: Semantic Decision Engines in Network Control Loops

Papers · Li et al. · 2026-10-05 · v1

Surveys semantic engines in network control loops; separates valid decisions, action feasibility, service checks and deadline attainment, with bounded event-model tests rather than production validation.

Primary source: https://arxiv.org/abs/2610.06425v1

## Probability Contracts: Accuracy, Coherence, and Decisions Across LLM Interfaces

Papers · Chen & Li · 2026-10-05 · v2

Known-posterior finite worlds separate probability accuracy, interface coherence and decision loss. Averaging can improve expected Brier score without improving a policy at every deferral cost; valid-report coverage and each comparison's reference policy remain essential.

Primary source: https://arxiv.org/abs/2609.37470v2

## Hidden Risks of Jev: An Empirical Study of Security, Privacy, and Dual Use

Papers · Wang et al. · 2026-10-04 · v1

Tests the official Jev API and the local NanoJev for prompt injection, data-poisoning backdoors, membership, attribute and knowledge inference, and dual use in four detection tasks; the authors report Jev remains vulnerable to the examined threats.

Primary source: https://arxiv.org/abs/2610.04985v1

## System One Models for Wireless Decision-Making:Applications and Performance Evaluation

Papers · Rahimi et al. · 2026-10-03 · v1

Compares bounded wireless action selection with LLMs and task-specific baselines. Lower successful-request latency trades off against decision quality; a separately reported orchestration study shows that faster interpretation need not improve end-to-end completion. Local and hosted timing boundaries differ.

Primary source: https://arxiv.org/abs/2610.04345v1

## Calibrated to Whom? Persona and Language Effects on Cultural Values in JEV

Papers · Asseri & Asseri · 2026-10-03 · v2

English/Arabic persona-conditioned Jev survey audit; the effects concern simulated respondents in one model, not measurements of real populations.

Primary source: https://arxiv.org/abs/2609.36399v2

## Evaluating System One Models for Agent Security Decisions: Reliability, Calibration, and Selective Automation

Papers · Liu · 2026-10-03 · v3

Examines reliability, group-specific confident errors, and selective automation in security decisions.

Primary source: https://arxiv.org/abs/2609.33401v3

## To Jev or Not? Evaluating the Accuracy and Efficiency of Structured Decision Models for Hate-Speech Moderation

Papers · Paschalides, Pallis & Dikaiakos · 2026-10-02 · v1

HATEDECIDE tests definitions and decomposed judgments across four English datasets. More explicit criteria change predictions without reliably improving annotation agreement; component outputs are not verified explanations, and diagnostic-set cost comparisons do not establish production moderation quality.

Primary source: https://arxiv.org/abs/2610.03324v1

## Batched Speech Decisions Without Decoding: Single-Token Supervision Lets a Frozen LLM Hear Beyond the Transcript

Papers · Jin et al. · 2026-10-02 · v1

DuplexJev connects ASR hidden states to a frozen LLM for batched answer-token readout; connector-specific speech tasks and eight-GPU timing do not establish single-device or live-agent latency.

Primary source: https://arxiv.org/abs/2610.02638v1

## Labels Override Definitions in Jev-Style Typed Decision Models

Papers · Azizi, Baghaei Potraghloo & Pedram · 2026-10-01 · v1

Studies option-label bias in open-weight decision models; prompt-rendering changes with fixed weights implicate the label/definition interface, without testing hosted Jev.

Primary source: https://arxiv.org/abs/2610.02586v1

## Student-Guided Teacher Distillation for Efficient LLM Task Routing: Positioning Against Jev-Style System-1 Classifiers

Papers · Wu et al. · 2026-10-01 · v1

Preliminary distillation for a fixed 60-label task taxonomy uses a student shortlist to reduce teacher scoring. Teacher agreement is not human accuracy; complete held-out controls remain pending, and the best checkpoint differs from the version used for serving measurements. No Jev head-to-head result is established.

Primary source: https://arxiv.org/abs/2610.02516v1

## HakemBench: A Turkish Benchmark of Typed Decisions

Papers · Teke · 2026-10-01 · v1

Turkish typed-decision evaluation reports decision quality, Brier-based probability quality and risk–coverage alongside input probes; most labels are AI-derived, and the lab’s model used feedback from earlier test results.

Primary source: https://arxiv.org/abs/2610.02293v1

## Fast Models, Slow Evidence: A Paired and Self-Audited Evaluation of System-1 Decision Models for LLM Agent Harnesses

Papers · Li · 2026-10-01 · v1

Paired Laya/Jev evaluation of 11 agent decision points includes a self-audit: pre-screen cost, end-to-end quality, threshold leakage and unnatural channel content change deployment conclusions. Public English-dominant labels and zero-shot configurations limit generalization.

Primary source: https://arxiv.org/abs/2610.02267v1

## HydroJEV: A one-second, training-free screen for cyber-attack and fault attribution in water distribution networks

Papers · Mu et al. · 2026-10-01 · v1

Simulated water-network triage combines Jev, rule agreement and an LLM reviewer. Fewer reviewer calls coexist with accepted attacks, event-selection constraints and a transfer endpoint that misses multiplicity correction; reported timing and non-inferiority margins do not establish field reliability.

Primary source: https://arxiv.org/abs/2610.02048v1

## Prune First, Decide Fast: Scalable Semantic Query Processing with JEVDB

Papers · Wang, Yan, Zhao & Liu · 2026-10-01 · v1

Combines semantic SQL with relational pruning, semantic conditions and an optional decision-to-LLM cascade. Lossless pruning requires sound conditions and key extraction; empirically learned filters may lose recall. Imported SemBench baselines and separately amortized index preparation qualify efficiency comparisons.

Primary source: https://arxiv.org/abs/2610.02046v1

## AudioJev: Direct Audio Decisions with Order-Calibrated Probabilities

Papers · Lv et al. · 2026-10-01 · v1

Learns audio-conditioned candidate probabilities with aligned option-permutation training; order stability and empirical correctness calibration remain different properties.

Primary source: https://arxiv.org/abs/2610.01293v1

## Jev-IDS: System One Models for Network Intrusion Detection

Papers · Severo, Quincozes & Dias · 2026-10-01 · v1

Studies typed intrusion decisions, but v1 mixes configurations: its abstract and pinned code describe a 300-flow GPT-5.6 Luna pilot, while its body describes 2,000 flows and Gemini. These cohorts and comparators must remain separate; benchmark-withheld attack types are not operational zero-day validation.

Primary source: https://arxiv.org/abs/2610.01079v1

## Beyond Answer Confidence: A Controlled Audit of Self-Knowledge in a Black-Box Decision Model

Papers · Shankaranarayana, Runje & Jannink · 2026-10-01 · v1

Controls information and surface cues to distinguish Jev answer confidence from self-knowledge; black-box behavior does not establish access to an internal knowledge state.

Primary source: https://arxiv.org/abs/2610.01006v1

## Jev-as-a-Judge for RAG Claim Verification: Confidence-Gated Cascading Between a Non-Generative Verifier and a Frontier LLM

Papers · Do Rosario & Atef · 2026-09-30 · v1

Paired verification of 495 RAG claims finds different false-verification rates despite similar aggregate scores. One threshold is cross-fitted, but design choices and matched random-escalation controls remain exploratory; simulated savings do not establish equivalence or a better production gate.

Primary source: https://arxiv.org/abs/2610.20634v1

## AnyJev Technical Report

Papers · Zhang et al. · 2026-09-30 · v1

Training-free decision readout with label-prior and option-order correction.

Primary source: https://arxiv.org/abs/2610.00831v1

## JevSpawn: Adaptive Agentic Inference through Compositional Action Spaces

Papers · Su & Huang · 2026-09-30 · v1

Composes finite action fields and explores alternatives using interaction feedback.

Primary source: https://arxiv.org/abs/2610.00437v1

## OmniMed-Jev: Calibrating LVLM Confidence for Trustworthy Medical Multimodal Decisions via System One

Papers · Tang & Chen · 2026-09-30 · v1

Medical candidate-conditioned decisions are compared with a matched generative baseline; calibration gains do not isolate interface effects from training differences or establish clinical readiness.

Primary source: https://arxiv.org/abs/2610.00381v1

## A First Glance at Jev for Network Traffic Classification: Accuracy, Processing Time, and Cost

Papers · Xu & Mei · 2026-09-30 · v1

Classifies ten application labels from early packet features over 26 later collection weeks. Labeled examples improve Jev but the tested tree ensembles remain more accurate; unequal supervision and service configurations prevent attributing the gap to architecture or intrinsic label efficiency.

Primary source: https://arxiv.org/abs/2610.00376v1

## When the Right Answer Is Missing: An Arithmetic-Dependent Rejection Bottleneck in Jev

Papers · Zhong, Li & Lai · 2026-09-30 · v1

Paired numerical menus expose strong answer-present selection but weak none-of-the-above rejection. Separate development thresholds recover much of the tested rejection performance, while Boolean verification behaves differently; this diagnoses an interface-dependent failure, not hidden model mechanisms or universal threshold transfer.

Primary source: https://arxiv.org/abs/2609.39496v1

## Bongard: Training Machine Intuition

Papers · Ding, Jin & Ji · 2026-09-30 · v1

Shared evidence encoding and candidate-probability readout, trained with representation and action-outcome feedback.

Primary source: https://arxiv.org/abs/2609.39111v1

## OpenJev-RLCD: A Working RLCD Implementation

Papers · Gao & Wang · 2026-09-30 · v1

An independent two-stage RLCD recipe calibrates a reasoning model's answer readout, then reinforces rationales with a proper-score reward. Three-seed results on a 1.7B model show task-dependent gains; the implementation explicitly makes no claim to recover TypeSafe's internal training algorithm.

Primary source: https://arxiv.org/abs/2609.38850v1

## Evaluating and Benchmarking the System One Model Jev

Papers · Deußer, Sparrenberg & Sifa · 2026-09-29 · v1

An early benchmark study of Jev. Read the task selection and evaluation setup alongside its reported results.

Primary source: https://arxiv.org/abs/2609.37647v1

## Chinese-Jev: Bringing System One Model to Chinese-Language Tasks

Papers · Wang et al. · 2026-09-29 · v1

A Chinese typed-decision model and proposed benchmark; domain-specific training calls for independent validation.

Primary source: https://arxiv.org/abs/2609.36965v1

## Dyad: Extending Large Language Models with Native Typed Decision-Making

Papers · Zhan et al. · 2026-09-28 · v1

Adds an environment-conditioned action encoder to an LLM for typed actions; frozen-LLM adaptation and joint reinforcement learning are separate settings, and the Jev latency comparison uses previously published API measurements.

Primary source: https://arxiv.org/abs/2609.36116v1

## Mnemon: Raw Records, Fast Judgments, Slow Thoughts

Papers · Wang · 2026-09-28 · v1

Keeps raw memory records, separating Jev relevance judgments from LLM search planning and answer generation.

Primary source: https://arxiv.org/abs/2609.36059v1

## Jev thinks “I don't know”, but doesn't say it: Introducing Sys1Cal-v1 Dataset for Probability Calibration

Papers · Porcedda · 2026-09-28 · v1

A controlled calibration study using constructed probabilities. Interpretations of latent uncertainty remain hypotheses.

Primary source: https://arxiv.org/abs/2609.35342v1

## Decide, Don't Generate: Competitive Dimensional ABSA with Jev's Typed Decisions

Papers · Zhang et al. · 2026-09-28 · v1

Uses frozen Jev decisions plus 488 coefficients fitted to labels for multilingual sentiment regression and extraction. Supervised calibration and span selection drive the gains; reconstructed aggregate rankings, reused development data and incomplete overlap removal qualify the comparisons.

Primary source: https://arxiv.org/abs/2609.35293v1

## JEV as a Judge for Agent Trace Security: An Empirical Comparison with Generative LLM Judges

Papers · Wang & Gao · 2026-09-28 · v1

Compares Jev and generative judges on retrospective agent-trace risk classification; dataset rankings vary, and successful-call latency and valid-judgment cost must be read with coverage.

Primary source: https://arxiv.org/abs/2609.34862v1

## Emo-Jev: Probabilistic Reasoning for Emotion Classification with Jev

Papers · Zhang & Yu · 2026-09-27 · v1

Adds atomic-question decomposition or multiple judgment paths for eight text-classification benchmarks. Neither extension improves average macro-F1 over direct Jev in the reported setup; extra questions increase cost, and fixed aggregation rules do not establish calibrated uncertainty.

Primary source: https://arxiv.org/abs/2610.08829v1

## Laya as a Typed Probabilistic Assessor: An Independent Reproduction and a Preregistered Study of Calibration and Selective Escalation

Papers · Nandakishore · 2026-09-27 · v1

A pinned Laya checkpoint is underconfident on synthetic teacher labels. Temperature fitting improves probability metrics, but the frozen calibrator-selection rule and escalation gate fail held-out targets; reported Jev comparisons are indirect, and the initial reproduction predates preregistration.

Primary source: https://arxiv.org/abs/2609.33843v1

## Type-Safe Decision Frameworks for Agentic 5G Control: A Theory-Driven Testbed Characterization of Where They Can Be Applied

Papers · Kourtis & Xilouris · 2026-09-27 · v1

Maps typed frameworks to timing, label-budget and decision-schema conditions on a 5G testbed. Changed-question probes expose failures hidden by output validity and paraphrase stability; simulator labels, open RAN-loop scope and conditional risk assumptions constrain deployment conclusions.

Primary source: https://arxiv.org/abs/2609.33689v1

## Beyond Calibration: Do a Typed-Decision Model's Probabilities Obey the Probability Axioms?

Papers · Li, He & Li · 2026-09-27 · v1

Tests coherence between logically related probabilities, separately from accuracy and calibration.

Primary source: https://arxiv.org/abs/2609.33209v1

## PACT: Pairwise-Anchored Calibrated Tuning for Single-Token Typed Decisions

Papers · Lin · 2026-09-26 · v1

Uses certified contrastive pairs, permutation consistency and contextual temperature fitting to improve decision stability and ordinal error; a 324-item synthetic holdout, one model scale and certificate dependence limit generalization.

Primary source: https://arxiv.org/abs/2609.35865v1

## Typed Decision Models: An Early Evidence Audit and Evaluation Checklist

Papers · Tang & Zheng · 2026-09-26 · v1

An evidence audit and a checklist for evaluating typed decision models.

Primary source: https://arxiv.org/abs/2609.32160v1

## JevAdvBench: A Benchmark and Black-Box Attacks for Reinforcement Learning for Calibrated Decisions Models

Papers · Hu et al. · 2026-09-25 · v1

A black-box adversarial benchmark. A changed decision is not necessarily an incorrect decision.

Primary source: https://arxiv.org/abs/2609.31142v1

## LAVOIR: Teaching a Single-Pass Decision Encoder When and What to Ask with Amortized Value of Information

Papers · Yilmaz, Tasdemir & Gozay · 2026-09-25 · v1

Estimates whether asking for missing information would improve a typed decision.

Primary source: https://arxiv.org/abs/2609.30706v1

## Jev-Mobile: Jev as an Executor for Mobile GUI Agents

Papers · Zhang · 2026-09-24 · v1

Uses VLM local-goal planning with Jev action selection over Android accessibility-tree candidates; reported time and API-cost savings are conditional on successful trajectories and the tested controller configurations.

Primary source: https://arxiv.org/abs/2609.30186v1

## Just Ask Jev: Reinforcement Learning for Calibrated Decisions as a Zero-Shot Detector of AI Alignment Failures

Papers · Guo et al. · 2026-09-24 · v1

Benchmarks Jev alignment-failure detection while separating question wording from input context; benchmark-scorer labels dominate, and context gains often depend on fields encoding the label.

Primary source: https://arxiv.org/abs/2609.29429v1

## From Text Decisions to Pixels: An Study of Jev-Style Visual Choice Model

Papers · Zhou, Yang & Zhao · 2026-09-24 · v1

A Jev-style visual-choice study separating adaptation gains, candidate-readout validity, and calibration limits.

Primary source: https://arxiv.org/abs/2609.29283v1

## Decision Hijacking: Prompt Injection Attacks on Jev's Typed Probabilistic Decisions

Papers · Wu & Lim · 2026-09-23 · v1

Audits Jev injection on reconstructed InjecAgent cases, separating probability shifts from attacker-target selections; optimization results are checked on fresh calls, within this benchmark setting.

Primary source: https://arxiv.org/abs/2609.28613v1

## NumericJev: Jev-like LLM Numerical Decoding with Multiway Decision Trees

Papers · Ye, Liu & Jiang · 2026-09-23 · v1

Uses repeated Choice calls to refine a known numerical interval without retraining. Resolution is not accuracy or calibrated uncertainty, and early wrong branches are irreversible. Arithmetic variants share 64 base families; historical-index recall and later controller demonstrations are separate evidence.

Primary source: https://arxiv.org/abs/2609.28587v1

## Can Jev Judge Radiology Reports? Evaluating a System One Model for Clinical Factuality

Papers · Huang et al. · 2026-09-23 · v1

Jev checks support between generated and reference radiology reports in both directions; expert agreement depends on report size and error definitions and does not establish prospective clinical effectiveness.

Primary source: https://arxiv.org/abs/2609.27607v1

## KITE: Scaling Jev Population Experiments with Sparse Flagship Calibration

Papers · Li · 2026-09-23 · v1

Jev state tables and sparse flagship-model anchors support population simulations with shared human–model discrepancy estimates; larger simulations do not add human evidence or establish prospective intervention effects.

Primary source: https://arxiv.org/abs/2609.27535v1

## JEV-Star: Fast, Low-Cost StarCraft II Control with Language-Model Planning

Papers · Ma, Zhao, Zeng & Zhao · 2026-09-23 · v1

Combines Jev action selection with persistent language-model planning for StarCraft II; limited full-game evidence and simultaneous interface changes prevent isolating the planning contribution.

Primary source: https://arxiv.org/abs/2609.27331v1

## Evaluating Decision Models for Text Annotation in Computational Social Science

Papers · Ibrahim & Zaki · 2026-09-23 · v2

Evaluates zero-shot social-science annotation and confidence-routed LLM cascades; lower costs coexist with task-dependent errors and overconfidence, so confidence cannot substitute for label validation.

Primary source: https://arxiv.org/abs/2609.24574v2

## REFLEX with Jev for Efficient Selective Control in LLM Agents

Papers · Wu & Lim · 2026-09-22 · v1

Confidence-gated Jev control with LLM fallback; gains depend on action alternatives and baseline routing.

Primary source: https://arxiv.org/abs/2609.26532v1

## Visual Jev: Accurate and Efficient Decisions from Shared Visual Context

Papers · Yu & Yao · 2026-09-22 · v1

An independent visual decision architecture shares image context across batched questions using an LM-head readout; warm amortized speedups trade higher peak memory, while accuracy gains concentrate on trained task families.

Primary source: https://arxiv.org/abs/2609.25845v1

## Jev-Mem: System-One-Controlled Agentic Memory for Efficient AI Agents

Papers · Jiang, Li & Li · 2026-09-21 · v1

Studies Jev as a controller for agent memory. Reported outcomes concern the complete memory workflow.

Primary source: https://arxiv.org/abs/2609.23986v1

## this-that-model-1.0: A typed decision model that decides in 30 ms, for a millionth of a cent

Papers · Cheng, Dai & Sun · 2026-09-20 · v1

An open typed-decision model using hidden-state readout. The abstract reports a small external comparison.

Primary source: https://arxiv.org/abs/2609.23886v1

## OneJev1.5-27B

Models · OmniJev · Apache-2.0

27B full fine-tune for text, image and video decisions, served through qev. Training includes public benchmark training splits; reported scores cover the public suite, and Jev comparison numbers are imported rather than a matched multimodal rerun.

Primary source: https://huggingface.co/OmniJev/OneJev1.5-27B

## d1-omni-600M

Models · Liquid AI · LFM Open License v1.0

587M encoder for text with images or up to 30 seconds of English speech; one request cannot mix both media. Only text uses fitted temperatures. Image requests cap text at 896 tokens; custom code and the LFM Open License apply.

Primary source: https://huggingface.co/LiquidAI/d1-omni-600M

## Torchcast Decision 27B

Models · Torchcast AI · CC BY-NC 4.0

Merged LoRA decision model with a CUDA serving package; weights are noncommercial, serving code Apache-2.0. Text-only, checkpoint-selected results use Decision Index 0.2.1; long states and more than 26 options change the inference path.

Primary source: https://huggingface.co/torchcast-ai/torchcast-decision-27b

## tasksource-decider-nano

Models · tasksource · Apache-2.0

150M joint cross-encoder with shared-state tree attention and choice/noul outputs. Rejects pairs above 8,192 tokens; training includes public benchmark training splits. Reported speed and public-suite scores use the documented A10 setup.

Primary source: https://huggingface.co/tasksource/tasksource-decider-nano

## Saracura PT-BR 4B

Models · Felhen · Apache-2.0

Brazilian Portuguese Kev adapter and pointer head. Source-matched tests and two-fold human-set calibration do not establish general Portuguese performance; private email-tuned results belong to an unreleased model. Use the documented 2,048-token state setting for comparisons.

Primary source: https://huggingface.co/felhen-ai/saracura-ptbr-4b

## Perplexity Decider v1.2 27B

Models · Perplexity · Apache-2.0

27B text/image model with joint question readouts: questions can affect one another. Public-suite training exposure and grouping affect comparisons. Anonymous Hub metadata and a weight HEAD were accessible on 18 Oct; the card still instructs private access.

Primary source: https://huggingface.co/perplexity-ai/pplx-decider-v1.2

## Amos 1 large

Models · mukaddamzaid · Apache-2.0

English typed decisions with image inputs; distinguish zero-shot results from train-split prior fitting, and public-only Decision Index results from the full evaluation.

Primary source: https://huggingface.co/mukaddamzaid/amos-1-large

## KnowLine-35B-A3B-Gen1

Models · PelaAI · Apache-2.0 weights; MIT server

English/Chinese MoE decision checkpoint; public-suite checkpoint selection, disclosed task/source overlap and self-run calibration ranges constrain comparisons with independent held-out evaluations.

Primary source: https://huggingface.co/PelaAI/KnowLine-35B-A3B-Gen1

## Lychee-27B

Models · LycheeAI · CC BY-NC 4.0

Noncommercial Qwen-derived typed-decision model with publisher-reported public-suite results; requests exceeding context or GPU-memory limits are documented as rejected rather than truncated.

Primary source: https://huggingface.co/LycheeAI/Lychee-27B

## Span-02

Models · Respan · Apache-2.0

Multimodal decision model with a dedicated Blackwell Rust/CUDA runtime and optional reasoning; distinguish default serving from deterministic index settings and output-specific confidence semantics.

Primary source: https://huggingface.co/respanai/span-02

## AudioJev checkpoint

Models · shlv · Qwen2.5-Omni-3B derivative · Qwen Research

One released AudioJev training seed with a Qwen Research license; original-order scores disclose audio-source overlap and are distinct from the paper’s three-seed averages.

Primary source: https://huggingface.co/shlv/AudioJev

## ufakzeka-karar

Models · ufak AI · Apache-2.0

Turkish typed-decision checkpoint with shipped temperature scaling. Its card flags test-informed training and worsening support-task calibration; benchmark results are not blind.

Primary source: https://huggingface.co/ufakai/ufakzeka-karar

## Jeva-4B

Models · Jeva authors · license not declared

Author checkpoint with backbone, adapter and decision-head files. The inspected card declares no model license; weights have not been loaded or tested by this project.

Primary source: https://huggingface.co/LIQIIIII/Jeva-4B

## Laya

Models · convaiinnovations · Apache-2.0

An encoder-based decision family with English, multilingual, and workflow-specific checkpoints.

Primary source: https://huggingface.co/convaiinnovations/laya

## Kev-4B

Models · Jared Palmer · Apache-2.0

A Qwen3.5-4B-Base LoRA adapter and pointer head for decisions over documents and typed questions.

Primary source: https://huggingface.co/jaredpalmer/kev-4b

## Bespoke Nimble 9B v3

Models · Bespoke Labs · CC BY-NC 4.0

A Qwen3.5-9B LoRA adapter returning candidate probabilities. Released under a noncommercial license.

Primary source: https://huggingface.co/bespokelabs/Bespoke-Nimble-9B-v3

## Intern-Decision-4B

Models · InternLM · Apache-2.0

A decision-tuned Qwen checkpoint with a structured inference wrapper and image support.

Primary source: https://huggingface.co/internlm/Intern-Decision-4B

## GLiNER2.5-Decide

Models · Fastino · Apache-2.0

A 340M English classifier with runtime label sets and single-label or multi-label decisions.

Primary source: https://huggingface.co/fastino/GLiNER2.5-Decide

## d3

Models · vLLM Semantic Router · Apache-2.0

A text, image, and video decision model with Choice, Yes/No, and Score outputs.

Primary source: https://huggingface.co/vllm-sr/d3

## Clef

Models · Cloudflare · Apache-2.0

A multimodal model with a joint schema head, returning probabilities over allowed answers.

Primary source: https://huggingface.co/Cloudflare/clef

## Abide

Tools · coldteadotai · rule checks · MIT

Jev checks edits and turns against compiled project rules and requests repairs; missing credentials or network let edits pass unchecked, and replay findings depend on the chosen rubric.

Primary source: https://github.com/coldteadotai/abide

## jev-fabric

Tools · fabric-runtime · process orchestration · MIT

Native process orchestration with bounded logs, deadlines and explicit typed decisions; commands run with caller privileges, exit status needs outcome checks, and selected policy proofs do not verify the whole runtime.

Primary source: https://github.com/fabric-runtime/jev-fabric

## atoma

Tools · atoma-run · agent workflows · AGPL-3.0-only

Multi-agent planning, execution and evidence review with Jev routing; shared agent/skill learning can expose document-derived material across organizations, and tenant isolation is not independently audited.

Primary source: https://github.com/atoma-run/atoma

## circuit

Tools · Barneyjm · training code · Apache-2.0

LoRA and pointer-head training, evaluation and serving for text, image and audio decisions; publisher tests cover selected task sets and do not establish universal calibration or option-order invariance.

Primary source: https://github.com/Barneyjm/circuit

## cua-s1.js

Tools · ai-ecoverse · browser port · MIT

Browser ports of Cua form and next-action models; output-parity checks do not establish generalization, and the small form model can confidently skip unfamiliar fields.

Primary source: https://github.com/ai-ecoverse/cua-s1.js

## jev-omni.js

Tools · ai-ecoverse · WebGPU port · Apache-2.0

Work-in-progress WebGPU port for Jev-Omni text and image decisions; audio/video are unfinished, model downloads are large, and reported parity is limited to selected fixtures and hardware.

Primary source: https://github.com/ai-ecoverse/jev-omni.js

## SearchJev training pipeline

Tools · EvoScientist · training code · Apache-2.0

Search-agent decision training with soft-label and Brier losses, reliability weights and per-question-type temperature fitting; the documented pipeline and split controls were not reproduced.

Primary source: https://github.com/EvoScientist/SearchJev

## mJev

Tools · SoMarkAI · multimodal runtime · Apache-2.0

Shared-context candidate scoring with HF/vLLM routes; normalized option probabilities are not calibrated confidence, and the reported cache speedup trades higher memory in one controlled video setup.

Primary source: https://github.com/SoMarkAI/mJev

## FRIDA-Decisions

Tools · SberAI · independent runtime · MIT code

Packed Russian–English encoder runtime for typed questions and ranking; state truncation, Russian instruction suffixes and margin-versus-probability semantics remain explicit.

Primary source: https://github.com/ai-forever/FRIDA-Decisions

## OpenThai-SystemOne

Tools · iapp-technology · Thai/English pipeline

Thai/English text decision-model training and serving pipeline with a slot readout; the documented recipe does not establish independently validated calibration or interface equivalence.

Primary source: https://github.com/iapp-technology/openthai-systemone

## Quyet runtime

Tools · ncchinh · runtime · Apache-2.0

Runtime for Quyet decision models with checkpoint-dependent English/Vietnamese coverage; it documents ten-option limits and state truncation, without evaluated adversarial robustness.

Primary source: https://github.com/ncchinh/quyet

## JevLang

Tools · TimMikeladze · policy and replay

TypeScript/Python policy layer separates typed model answers from routing and action dispatch, with documented journals and replay; a trace does not establish truthful inputs or calibrated thresholds.

Primary source: https://github.com/TimMikeladze/JevLang

## sysone decision-learning library

Tools · sgaseretto · decision learning

Training, calibration and interpretation tools for several text, image and protein backends; documented compatibility checks and small experiments do not validate every backend or workload.

Primary source: https://github.com/sgaseretto/sysonelib

## protoc-gen-jev

Tools · Buf · experimental generator · MIT

Generates typed Go, TypeScript and Python clients from annotated Protobuf schemas. Generator and runtime releases must be matched; backend behavior was not tested here.

Primary source: https://github.com/bufbuild/protoc-gen-jev

## TypeSafe Python SDK

Tools · Python · MIT

The official sync and async client for typed Jev questions and answers.

Primary source: https://github.com/typesafe-ai/typesafe-sdk-python

## TypeSafe JavaScript SDK

Tools · TypeScript / JavaScript · MIT

The official JavaScript client with answer types inferred from your questions.

Primary source: https://github.com/typesafe-ai/typesafe-sdk-js

## System One Adapter

Tools · Python · MIT

Compare LLMs through the same question interface, with probability outputs and request diagnostics.

Primary source: https://github.com/typesafe-ai/system-one-adapter-python

## TypeSafe Agent Skills

Tools · Coding agents · MIT

Official guidance for coding agents building with the System One API.

Primary source: https://github.com/typesafe-ai/skills

## TypeSafe Public Examples

Tools · Cookbooks & demos · MIT

Source code accompanying official documentation, demos, and cookbooks.

Primary source: https://github.com/typesafe-ai/typesafe-public-examples

## TypeSafe for n8n

Tools · Workflow integration · MIT

TypeSafe-maintained nodes for adding typed decisions to n8n workflows.

Primary source: https://github.com/typesafe-ai/n8n-nodes-typesafe-ai

## evalsafe-invoice-processing

Datasets · TypeSafe · WorkflowEvals

Official dataset for the invoice-processing workflow evaluation.

Primary source: https://huggingface.co/datasets/typesafeAI/evalsafe-invoice-processing

## evalsafe-customer-service

Datasets · TypeSafe · WorkflowEvals

Official dataset for the customer-service workflow evaluation.

Primary source: https://huggingface.co/datasets/typesafeAI/evalsafe-customer-service

## evalsafe-security-incidents

Datasets · TypeSafe · WorkflowEvals

Official dataset for the security-incident workflow evaluation.

Primary source: https://huggingface.co/datasets/typesafeAI/evalsafe-security-incidents

## evalsafe-agent-trace-observability

Datasets · TypeSafe · WorkflowEvals

Official evaluation dataset for the agent-trace observability workflow.

Primary source: https://huggingface.co/datasets/typesafeAI/evalsafe-agent-trace-observability

## Bekko System One Dataset v0

Datasets · hotchpotch · Training mixture

A community training mixture for System One-style decision models. Read the card for its composition and usage conditions.

Primary source: https://huggingface.co/datasets/hotchpotch/bekko-system-one-dataset-v0

## Bespoke Nimble 9B v3 — Decision Index runs

Datasets · Bespoke Labs · Evaluation archive

An archive of Decision Index evaluation runs for Bespoke Nimble 9B v3.

Primary source: https://huggingface.co/datasets/bespokelabs/bespoke-nimble-9b-v3-decision-index

## Browser decision-vision benchmark

Evaluations · ai-ecoverse · browser comparison

Mixed visual/text comparison of browser decision models using native formats; ambiguous GUI labels, shared hardware and different browsers limit broad ranking and latency claims.

Primary source: https://github.com/ai-ecoverse/decision-vision-bench

## English Jev RAG Benchmark

Evaluations · emretheus · RAG evaluation

English RAG study separates frozen-pool reranking from frozen-context answer generation; gating analyses are exploratory, and free-tier gateway pacing is not isolated model latency.

Primary source: https://github.com/emretheus/jev-rag-benchmark

## Decision models as guardrails

Evaluations · raxIT Labs · partial public split

Common policy benchmark compares decision models and managed guardrails with fixed scoring; an unpublished test slice prevents exact full-board reproduction from the public clone.

Primary source: https://github.com/raxITlabs/decision-models-as-guardrails

## 113-class complaint classification benchmark

Evaluations · Ariño de la Rubia · recorded-output analysis

Compares zero-shot, retrieval and trained classifiers using released predictions. Numeral tracing and analysis replay differ from retraining or new model calls.

Primary source: https://github.com/earino/zero-shot-complaint-benchmark

## Jev RAG Chinese Benchmark

Evaluations · Community evaluation · 100 queries · single run

Chinese reranking comparison with fixed candidate pools and a pinned OpenRouter-served Jev version. Small single-run results and concurrent latency remain workload-specific.

Primary source: https://github.com/vicfei/jev-rag-zh-benchmark

## JevAdvBench analysis and evaluation code

Evaluations · MIT code · CC BY-NC 4.0 data

Recorded-response analysis and new API evaluations use identical-input reruns as a control. Most labels are model-derived; this project has not executed the analyses.

Primary source: https://github.com/JevAdvBench/JevAdvBench

## Reading Right, Acting Wrong — diagnostic dashboard

Evaluations · Li, Krishnan & Padman · author report

Interactive factual-probe and action-decision diagnostics. The current report covers 22 systems, distinct from the fixed paper’s 18; it is not a verified code or data release.

Primary source: https://yubol-bobo.github.io/System_one_model_evaluation/

## Jev Decision Index

Evaluations · Hugging Face · community

A community index of decision models and evaluations. Full rankings include private tests.

Primary source: https://huggingface.co/spaces/multimodalart/jev-decision-index

## Workflow Evals

Evaluations · Interactive evaluation explorer

Inspect four structured workflows. Scores measure agreement with model-generated references.

Primary source: https://evals.typesafe.ai/

## WorkflowEvals — Code

Evaluations · Python · Apache-2.0

The official evaluation runner, workflow definitions, scoring, and plotting tools.

Primary source: https://github.com/typesafe-ai/WorkflowEvals

## WorkflowEvals — Datasets

Evaluations · Hugging Face · four datasets

Published cases, model-generated references, and recorded runs for inspecting reproducibility.

Primary source: https://huggingface.co/collections/typesafeAI/workflowevals

## Artificial Analysis Decision Index

Evaluations · Artificial Analysis · Methodology v1.0

External evaluation of classification, tool calling, knowledge and reasoning. The suite includes private questions.

Primary source: https://artificialanalysis.ai/evaluations/artificial-analysis-decision-index

## Entagl: Jev vs Gemini workflow evaluation

Evaluations · Entagl Research · 2026-09-23

Author-run labeled decisions and workflow replays. Private customer cases limit public reproducibility.

Primary source: https://www.entagl.com/blog/typesafe-jev-benchmark-ai-decision-models

## jev-benchmarking

Evaluations · AppliedMachineLearning-Lab · paper code · MIT

Code accompanying Deußer et al.'s 37-dataset study.

Primary source: https://github.com/AppliedMachineLearning-Lab/jev-benchmarking

## jev-bench

Evaluations · PavelRavvich · benchmark code

Classification, raw predictions, calibration, latency, cost, and cascades. Check sampling and confidence definitions before comparing results.

Primary source: https://github.com/PavelRavvich/jev-bench

## AY Automate benchmark

Evaluations · AY Automate · benchmark write-up

Routing and injection-detection experiments with linked code and raw calls. Reported by the benchmark author.

Primary source: https://www.ayautomate.com/blog/jev-vs-llm-benchmark

## A Survey on Jev: Prior Research, Adaptation Costs, and Evidence of System One Models

Guides · Chen et al. · community survey v0.5

Survey manuscript connecting routing, label-probability readout and adaptation costs. The README abstract was inspected; this listing is not a full-manuscript review or empirical replication.

Primary source: https://github.com/evan-py-chen/jev-survey

## Jev Cookbook (Chinese)

Guides · Datawhale · CC BY-NC-SA 4.0

Community Chinese course of eleven notebook chapters, from the three primitives and design patterns to 18 walkthroughs of official cookbook recipes, a smart-home experiment, a 231-question evaluation, agent integration and RLCD fine-tuning of the open Laya model; offline examples need no API key. Noncommercial license; documentation translations are unofficial.

Primary source: https://github.com/datawhalechina/jev-cookbook

## Introducing System One Models & Jev

Guides · TypeSafe · September 15, 2026

The launch article. A primary source for the provider’s design claims and their stated caveats.

Primary source: https://typesafe.ai/blog/introducing-system-one-models-and-jev

## Jev: For the builders AND dreamers

Guides · TypeSafe · 2026-10-14 · jev-1.14.0

Release notes for jev-1.14.0 under jev-preview: opt-in header flags for images (also on jev-1.13.0) and long context, Box for 1.14 only, and the deprecation-notice plan. Its intelligence chart is provider-run and, by its own account, likely biased toward TypeSafe’s models.

Primary source: https://typesafe.ai/blog/jev-1-14

## Primitives: Choice, Score, Noul

Guides · API concepts

Choose between categorical decisions, ordered ratings, and yes/no probabilities.

Primary source: https://docs.typesafe.ai/primitives

## Confidence

Guides · Probability & routing

How confidence is derived from a distribution, and how it differs from probability.

Primary source: https://docs.typesafe.ai/confidence

## Jev 1.13 Jaggedness

Guides · Limitations · reviewed October 2, 2026

Documented limitations: numeric precision, long irrelevant context, adversarial inputs, and option-order sensitivity.

Primary source: https://docs.typesafe.ai/model-jaggedness/jev-1.13

## Guardrails for LLMs

Guides · Official cookbook

Combine narrow risk questions with application rules for passing, reviewing, or blocking content.

Primary source: https://docs.typesafe.ai/cookbooks/llm_guardrails

## Double-checking Citations

Guides · Official cookbook

Check whether a citation’s source context supports a claim.

Primary source: https://docs.typesafe.ai/cookbooks/citation_check

## Hierarchical Classification

Guides · Official cookbook

Use Choice probabilities to explore deep classification trees.

Primary source: https://docs.typesafe.ai/cookbooks/hierarchical_classification

## SDE cascade

Guides · TypeSafe · Official cookbook

Check inexpensive extraction with per-field Jev questions before escalation; the walkthrough uses cached and illustrative outputs.

Primary source: https://docs.typesafe.ai/cookbooks/sde_cascade

## Structure recovery

Guides · TypeSafe · Official cookbook

Restore Markdown structure through line-continuation and block-type decisions while code retains the original wording.

Primary source: https://docs.typesafe.ai/cookbooks/autoformat

## Confidence-gated routing

Guides · TypeSafe · Official pattern

Route uncertain cases to another system or a person.

Primary source: https://docs.typesafe.ai/patterns/confidence-routing

## Composite scoring

Guides · TypeSafe · Official pattern

Combine separate judgments with weights controlled by code.

Primary source: https://docs.typesafe.ai/patterns/composite-scoring

## RAG passage classification

Guides · TypeSafe · Official cookbook

Filter retrieved context: score each passage, then decide in code which ones reach the answering model.

Primary source: https://docs.typesafe.ai/cookbooks/classifying_rag_passages

## Pre-parsed value extraction

Guides · TypeSafe · Official cookbook

Choose among bounded candidates found by patterns, then normalize the verbatim value in code.

Primary source: https://docs.typesafe.ai/cookbooks/pre_parsed_value_extraction_cookbook

## Date extraction

Guides · TypeSafe · Official cookbook

Ask for the date parts a document names; resolve, calculate and validate them in code.

Primary source: https://docs.typesafe.ai/cookbooks/date_extraction_cookbook

## Strictly Proper Scoring Rules, Prediction, and Estimation

Foundations · Gneiting & Raftery · JASA 2007 · Author PDF

A foundation for evaluating and incentivizing honest probabilistic predictions.

Primary source: https://sites.stat.washington.edu/people/raftery/Research/PDF/Gneiting2007jasa.pdf

## On Calibration of Modern Neural Networks

Foundations · Guo et al. · ICML 2017

A practical reference for calibration and temperature scaling.

Primary source: https://proceedings.mlr.press/v70/guo17a.html

## SelectiveNet

Foundations · Geifman & El-Yaniv · ICML 2019

Studies prediction with a reject option and the trade-off between coverage and risk.

Primary source: https://proceedings.mlr.press/v97/geifman19a.html

## Language Models (Mostly) Know What They Know

Foundations · Kadavath et al. · 2022

An investigation of language models’ self-evaluation and uncertainty.

Primary source: https://arxiv.org/abs/2207.05221

## LMQL — Prompting Is Programming

Foundations · Beurer-Kellner et al. · PLDI 2023

A programming-language approach to constrained language-model interaction.

Primary source: https://www.sri.inf.ethz.ch/publications/beurerkellner2023prompting

## XGrammar

Foundations · Dong et al. · MLSys 2025

Efficient grammar-constrained generation: a useful comparison point for typed outputs.

Primary source: https://arxiv.org/abs/2411.15100

## DSPy

Foundations · Khattab et al. · ICLR 2024

Declarative language-model programs and pipeline optimization.

Primary source: https://arxiv.org/abs/2310.03714

## SGLang

Foundations · Zheng et al. · NeurIPS 2024

Efficient execution of structured language-model programs.

Primary source: https://papers.nips.cc/paper_files/paper/2024/file/724be4472168f31ba1c9ac630f15dec8-Paper-Conference.pdf

# Research topics

## Jev moderation, stance and sentiment evaluation

Canonical page: https://awesomejev.github.io/topics/language-task-evaluation/
Updated: 2026-10-18

A strong language-task benchmark score does not by itself establish policy adherence or transfer to another language. Compare annotation rules, input context, candidate menus, supervision and inference budgets before reusing the result.

Five fixed-version abstracts, selected methods/results/limitations and one implementation README were inspected. Findings remain author-reported; no prediction records or experiments were reproduced.

### Policy adherence or label agreement?

HATEDECIDE finds that supplying definitions and decomposing a moderation decision do not reliably improve agreement with existing dataset labels. A prediction can follow the supplied definition yet disagree with the original annotation policy. Component answers make the aggregation rule inspectable, but they are not verified explanations without component-level reference labels. Evaluate the actual policy and both false-positive and false-negative costs; English diagnostic examples do not establish everyday or multilingual moderation performance. [HATEDECIDE: limitations, v1](https://arxiv.org/html/2610.03324v1#S5.SS1)

### What information does retrieval add?

The online-moderation study separates written rules, labeled precedents and restrictions on the answer menu. Its main policy contrast adds precedents while keeping the complete menu fixed. For OLID and HateXplain, however, precedents come from the cleaned evaluation population with the target excluded; this differs from using only a frozen training collection. Media inputs are generated textual captions, not a test of native Jev vision. Report these information sources and candidate restrictions alongside the score. [Information conditions, v1](https://arxiv.org/html/2610.07953v1#S3.SS2) · [Tasks and media representations](https://arxiv.org/html/2610.07953v1#S3.SS1)

### What changes across languages?

The stance study compares English text–target pairs with Chinese speaker-tagged conversations. Language, dataset and conversational structure therefore change together. Jev has more difficulty with support/opposition in the conversational benchmark, but this does not isolate a language effect or prove an internal mechanism. Some LLMs also receive batches while Jev receives single records. Our suggested transfer test holds task structure and information access fixed and checks target-language annotation definitions independently. [Stance evaluation setup, v1](https://arxiv.org/html/2610.11901v1#S4) · [Conversation-length analysis](https://arxiv.org/html/2610.11901v1#S5.SS4)

### Do more questions improve the result?

Emo-Jev compares direct classification with atomic-question composition and multiple predefined judgment paths. Both extensions have lower average macro-F1 than direct Jev in the reported eight-dataset setup, with task-specific gains and losses. Extra paths add dependent stages and cost; some retry usage is missing. HATEDECIDE likewise finds no consistent benefit from decomposition. Compare the direct baseline, the full added-question budget and held-out outcomes rather than assuming that more intermediate answers provide better reasoning or calibrated probabilities. [Emo-Jev results, v1](https://arxiv.org/html/2610.08829v1#S4.SS3) · [Efficiency accounting](https://arxiv.org/html/2610.08829v1#S4.SS5) · [HATEDECIDE decomposition](https://arxiv.org/html/2610.03324v1#S5)

### What does a frozen backbone still learn?

The DimABSA system uses frozen Jev with 488 coefficients fitted to task labels for regression and extraction. Its strong regression aggregate relies on supervised calibration; the competition ranks individual corpora, not the reconstructed pooled aggregate. Development scoring does not nest the full feature/threshold selection process, and some text overlap remains. The public code includes fitted parameters and metric summaries, while text-bearing prediction caches are withheld. Separate backbone tuning, supervised postprocessing, selection and artifact availability when comparing results. [DimABSA protocol, v1](https://arxiv.org/html/2609.35293v1#A2) · [Implementation and release limits](https://github.com/ZhangYiqun018/jev-dimabsa)

## Jev calibration, abstention and confidence routing

Canonical page: https://awesomejev.github.io/topics/calibration-and-abstention/
Updated: 2026-10-18

Better-calibrated probabilities do not guarantee a reliable abstention gate or a cheaper cascade. Evaluate the whole policy on held-out target data, separate missing-answer populations, and compare confidence routing with random escalation at the same budget.

Six fixed-version abstracts, selected manuscript sections and release documentation were inspected. Findings remain author-reported; no model experiment, prediction replay or proof verification was performed.

### Which quantity should improve?

Probability accuracy, interface coherence, accepted-case error and decision cost answer different questions. Probability Contracts uses worlds with known posteriors to separate them: averaging two reports can improve expected Brier score without improving a decision at every deferral cost. Its guarantee compares averaging with random interface selection, not with each interface individually. Report validity and coverage too; an interface that produces no usable probabilities has undefined probability accuracy. [Probability Contracts, v2](https://arxiv.org/html/2609.37470v2#S7.SS2) · [Study limits](https://arxiv.org/html/2609.37470v2#S8)

### Why is the correct answer missing?

An artificially removed answer, a retrieval miss and an out-of-scope query are different populations. The candidate-coverage benchmark reports them separately; at ten retrieval candidates, its natural-miss result has only five supporting queries. The arithmetic rejection study adds another distinction: selecting None in a menu differs from verifying each candidate with a Boolean question. Thresholds tuned on separate development problems improve its tested rejection tradeoff, but do not establish a universal threshold. [Coverage sources and denominators, v2](https://arxiv.org/html/2610.03387v2#S3.SS2) · [Arithmetic rejection and threshold rules, v1](https://arxiv.org/html/2609.39496v1#S5)

### Will a threshold survive a new task?

A source-task calibration constraint is not a target-task risk guarantee. The coverage study holds calibration budgets equal and finds that transferred thresholds can over-reject covered inputs or stop detecting missing answers. The Laya reproduction separately finds that a better probability metric does not ensure the frozen gate meets its held-out error target. Its labels come from a synthetic teacher, and its initial reproduction is retrospective. Compare source-frozen and target-recalibrated policies on disjoint target data, with label budgets stated. [Fixed-budget transfer, v2](https://arxiv.org/html/2610.03387v2#S5.SS4) · [Laya study limitations, v1](https://arxiv.org/html/2609.33843v1#S11)

### Does confidence improve routing?

A cascade should be compared with the cheap model alone, the fallback alone, and random escalation at a matched budget. In the 495-claim RAG study, similar aggregate scores conceal different false-verification rates. One threshold is cross-fitted, while its matched-random control uses in-sample operating points; neither nonsignificance nor that control establishes equivalence. Our proposed stronger comparison freezes the entire routing procedure and random-control budget before testing on new claims. [Cross-fitted threshold, v1](https://arxiv.org/html/2610.20634v1#S9.SS2) · [Matched random escalation](https://arxiv.org/html/2610.20634v1#S9.SS5) · [Design limitations](https://arxiv.org/html/2610.20634v1#S12)

### What belongs in total cost?

Count the first-stage decision for every item, then all escalated calls, retries and routing overhead. Report cost per attempted input, coverage and end-to-end quality together. Fast Models, Slow Evidence corrected omitted pre-screen cost, confusion between gate accuracy and final quality, and in-sample threshold optimism. The RAG cascade is an offline replay whose price calculation excludes second-hop overhead, retries and queueing; it has no controlled latency result. These studies do not establish current prices or universal production savings. [Self-audited agent evaluation, v1](https://arxiv.org/abs/2610.02267v1) · [Replay and cost limits](https://arxiv.org/html/2610.20634v1#S12)

### What do the artifacts support?

The candidate-coverage repository separates recorded-output scoring from source-text reconstruction and fresh inference. sys1-eval documents raw-response reanalysis; the RAG repository requires upstream access to the gold labels. Laya’s ancillary inventory lists predictions, metric code and manifests, but also withheld collection/training source. A listed file or publisher checksum is not an independently reproduced result. We inspected the documentation and inventories, without running these procedures or auditing their prediction records. [Candidate-coverage code](https://github.com/luckykevvv/Decision_Model_Benchmark) · [Agent evaluation code](https://github.com/David-DL-Space/sys1-eval) · [RAG verification code](https://github.com/adorosario/jev-rag-claim-verification) · [Laya ancillary inventory](https://arxiv.org/src/2609.33843v1/anc/README.txt)

## Option-label bias in Jev-style decision models

Canonical page: https://awesomejev.github.io/topics/option-label-bias/
Updated: 2026-10-18

A valid typed answer can follow an option’s familiar name instead of its assigned definition. The observed failure and its cause are separate questions: evidence from inspected open implementations should not be treated as an explanation of hosted Jev’s internals.

Independent research note based on versioned abstracts and the collection’s scoped reviews. Findings are author-reported; no attack or model experiment was run here.

### Which studies answer which question?

Sun and colleagues test conflicts between option names and definitions. Azizi and colleagues’ Labels Override Definitions examines how open implementations render those fields, including interventions that hold weights fixed. Their later One Word Opens the Gate evaluates attacker-controlled option channels in guardrail settings. These are distinct papers with different experimental questions; sharing a failure theme does not make their results interchangeable. [Type-Safe Is Not Error-Free, v3](https://arxiv.org/abs/2609.26758v3) · [Labels Override Definitions, v1](https://arxiv.org/abs/2610.02586v1) · [One Word Opens the Gate, v1](https://arxiv.org/abs/2610.12292v1)

### Why inspect the rendered request?

The fixed-weight rendering intervention in Labels Override Definitions provides evidence about the tested interface implementation. That is a narrower claim than saying every constrained decision head has the same defect. Before comparing systems, identify whether the option key, label and definition all reach the model, and whether the adapter rewrites or drops any of them. [Rendering intervention: original abstract](https://arxiv.org/abs/2610.02586v1)

### Does order stability fix label bias?

Runoff scores options individually before comparing balanced finalist listings. Its order-invariance argument assumes deterministic responses and a unique canonical ranking; the paper explicitly leaves option-name bias outside its target. Verdict-MQ describes architectural order/grouping invariance when retained state tokens are identical after truncation. These are conditional claims about changing a request’s arrangement, not proof of semantic correctness, calibrated confidence or resistance to malicious option names. [Runoff limitations and assumptions, v1](https://arxiv.org/html/2610.18233v1#Sx1.p1) · [Runoff invariance assumptions, v1](https://arxiv.org/html/2610.18233v1#A5.SS0.SSS0.Px2.p1) · [Verdict original abstract, v1](https://arxiv.org/abs/2610.20778v1)

### What changes when an option is added?

Prism-1 isolates option readouts to preserve order invariance and relative odds among original options when the candidate set changes, with state and question held fixed. This does not keep every normalized probability unchanged: an added score changes the softmax denominator. Its paper requires fine-tuning and limits the guarantee to numerical precision; independently judged options fit better than menu-dependent rules such as none of the above. Injected text in the shared state remains outside this protection. Evaluate menu-dependent fallback behavior separately, using the paired-rule tests rather than treating stable rankings as correct abstention. [Prism-1 limitations, v1](https://arxiv.org/html/2610.22066v1#S7) · [Choice-axiom tradeoffs, v1](https://arxiv.org/html/2610.22066v1#S5.SS6) · [Paired fallback test, v1](https://arxiv.org/html/2610.19856v1#S3.SS1)

### A diagnostic to use in your own evaluation

Our suggested comparison fixes the state, task rule, model version and scoring rule. Prepare equivalent requests with informative labels, neutral identifiers and definition-only options where the interface permits them. Add a separate conflicting-label condition, with expected answers checked before querying. Compare answer changes, errors and confidence on errors; preserve the actual rendered inputs. Use a held-out set to test any proposed repair. Test option permutations and identical-request repeats separately from label changes. Match the retained input after truncation when testing an invariance claim.

### What would remain unproven?

An adapter fix on one open implementation is not a universal injection defense. The guardrail study concerns open-weight models and does not demonstrate a compromise of hosted Jev. Keep the attacker’s permitted input channels explicit, and distinguish a changed distribution, a wrong choice and a downstream action. The diagnostic above is a proposed protocol, not a reproduced finding. [Guardrail study and tested scope](https://arxiv.org/abs/2610.12292v1)

## AudioJev: audio decisions, order calibration and release limits

Canonical page: https://awesomejev.github.io/topics/audiojev/
Updated: 2026-10-17

AudioJev is an independent audio decision model that scores supplied alternatives from a waveform and question. Its order-calibration objective addresses sensitivity to candidate order; it does not establish that a returned probability equals the empirical chance of being correct.

Based on the fixed-v1 abstract, public model card and inference-library overview. No weights, audio or inference code were executed or downloaded.

### What does the model return?

The paper describes training paired candidate orders and aligning their distributions by candidate identity. Inference scores the supplied candidates without an order ensemble. The separate inference library exposes Choice and Noul requests, plus an experimental Score interface, and can share an audio encoding across questions. This is a closed set: your application still has to supply appropriate alternatives and a fallback when needed. [AudioJev paper, v1](https://arxiv.org/abs/2610.01293v1) · [Inference-library documentation](https://github.com/SihanLv/AudioJev-Inference)

### How does DuplexJev differ?

DuplexJev connects ASR-encoder states to a frozen language model, whereas AudioJev describes a shared full-parameter model with an option-order objective. The speech paper reports different connector configurations and timing on an eight-GPU node. These are different training and serving arrangements, not a matched speed comparison. For a useful comparison, hold audio tasks, candidate sets, batching and hardware fixed, and separate audio encoding, question scoring and end-to-end latency. [DuplexJev, v1](https://arxiv.org/abs/2610.02638v1) · [AudioJev, v1](https://arxiv.org/abs/2610.01293v1)

### Which results belong to the released checkpoint?

Keep the paper’s three-seed averages separate from the model card’s single-seed, original-order results. The card also discloses audio-source overlap in part of the hidden MMAU evaluation. Neither a hidden evaluation split nor a fixed final checkpoint automatically establishes independence from all fitting or development sources. [Checkpoint and evaluation scope](https://huggingface.co/shlv/AudioJev)

### How should I evaluate it?

For your own test, define a task-specific answer set and verify labels independently. Compare unchanged audio under meaning-preserving candidate permutations, align the returned probabilities by candidate identity, and measure both accuracy and order sensitivity. Assess correctness calibration separately, on held-out labeled examples. If a downstream action depends on a threshold, report the accepted fraction and error rate together. These are proposed checks, not results produced by this collection.

### What is available, and under which terms?

The library is presented as Apache-2.0 code; the checkpoint declares the separate Qwen Research license. Review the model’s own terms before use. The release does not establish open-ended generation, visual performance or live turn-taking readiness. Follow the linked runtime documentation for hardware and deployment requirements; a documented command is not a verified deployment by this project. [Model card and license declaration](https://huggingface.co/shlv/AudioJev) · [Runtime and deployment entry point](https://github.com/SihanLv/AudioJev-Inference)

## GraphDecisionBench: evaluating Jev on graph decisions

Canonical page: https://awesomejev.github.io/topics/graph-decision-benchmarks/
Updated: 2026-10-17

GraphDecisionBench evaluates different graph capabilities rather than producing one universal model ranking. Correctly recognizing an edge does not establish structural reasoning, and completing a feasible construction does not establish a high-quality solution.

This note uses the v3 abstract and the repository’s overview and availability statements. It is not a full-paper review or benchmark reproduction.

### What are the three evaluation questions?

The benchmark separates structural graph queries, the use of graph-plus-text information, and decisions that construct a solution over multiple steps. These comparisons diagnose different behavior. Fourteen model-interface configurations appear across the experiments; that count should not be relabeled as fourteen independent models or as identical coverage of every task. [GraphDecisionBench paper, v3](https://arxiv.org/abs/2610.06354v3)

### What must be matched before comparing scores?

The repository treats each task and input condition separately and documents unsupported interfaces. Its structural probes use small supplied graphs and constructed answer quotas, which do not represent natural label frequencies in the source networks. When assessing a comparison, inspect the task predicate, candidate set, scoring rule and adapter before drawing a broader capability conclusion. [Task contracts and input conditions](https://github.com/VictorYXL/GraphDecisionBench)

### A useful control for sequential decisions

Our suggested evaluation compares a model-driven selector with the proposal rule alone, using the same candidate generator, constraints and stopping conditions. Measure feasibility and final objective quality separately. Preserve unsuccessful trajectories and their costs. If a heuristic narrows the choices, any improvement may reflect both the proposals and the selector; an identical proposal-only control helps separate their contributions.

### Which version and artifacts are covered?

The paper was first submitted on October 5, 2026; this note uses v3 dated October 12. The repository offers code and aggregate tables but states that per-query and per-trajectory outputs are not distributed. Regenerating a chart from summaries would therefore be a different operation from checking individual predictions or rerunning the models. No such operation was performed here. [Versioned paper and submission history](https://arxiv.org/abs/2610.06354v3) · [Public release scope](https://github.com/VictorYXL/GraphDecisionBench)