3AM Marketer
AI

AI Decision Models: Latency, APIs, and Architecture

AI decision models replace autoregressive generation with single-pass scoring over fixed schemas, dropping median latency to 38.8 ms with zero output fees.

38.8 ms
Clef-flash38.8 ms
Clef209.3 ms
Jev524.1 ms
Median Latency, from the entry's own table. 6 sources, checked 10 Oct 2026. Full table

AI decision models evaluate structured choices in a single forward pass instead of generating autoregressive text, cutting median evaluation latencies to as low as 38.8 ms while charging zero output token fees. By outputting direct probabilities over discrete schemas rather than streaming conversational strings, these architectures eliminate decoding overhead for high-throughput routing.23

What is an AI decision model?

An AI decision model processes both a prompt and candidate options simultaneously in a single forward pass, calculating normalized probability distributions across discrete targets without an autoregressive decoding loop. Cloudflare, Amazon, and OpenAI have developed systems sharing this design, which avoids conversational JSON text generation and downstream parsing regexes.13

Cloudflare company logo.
HaeB, CC BY-SA 4.0, via Wikimedia Commons

Hosted APIs: Jev versus OpenAI Decisions

TypeSafe AI released its Jev AI model on September 15, 2026, opening general signups on September 20, pausing them two days later, and restoring them on September 27. Within a week of release, over 100,000 users joined the Discord server, around 2,000 GitHub projects used the model, and approximately 13 percent of paid teams on Vercel's AI Gateway adopted it within 24 hours. On October 1, TypeSafe AI said it had onboarded a quarter of the Fortune 500.15

Both Jev and OpenAI Decisions support three question formats: yes/no, choices, or scores. Jev evaluates up to 255 candidate options in a single POST /v1/systemone request at 4.2 cents per million input tokens with zero output token fees. OpenAI's Decisions API charges 10 cents per million input tokens with zero output token fees, using the gpt-6-luna decision model to add multimodal image support over Jev's text-only 32k context interface.234

  • Jev4.2
  • OpenAI Decisions10
Input token pricing across commercial decision models, in cents per million tokens, October 2026. (¢ per 1M tokens)4

Benchmark accuracy and latency across Clef and Jev

Cloudflare open-sourced Clef and Clef-flash on Hugging Face under an Apache 2.0 license, running on Workers AI with a 64k context window and a vision encoder. Clef relies on a Qwen backbone with a prefill-only forward pass to score schema choices non-autoregressively. In functional evaluations, Clef handled website fetching, rendering, and classification in 2.2 seconds compared to 4.7 seconds for gpt-oss-120b.2

Cloudflare blog post announcing the open-source Clef and Clef-flash decision models.
Screenshot of blog.cloudflare.com, captured 2026-10-10
Inference latency and accuracy figures across hosted decision models.2
ModelMedian Latencyp95 LatencyBFCL ExactAPI-Bank Acc
Clef-flash38.8 ms122.4 ms98.7693.11
More on Clef-flash
BFCL Exact
98.76
API-Bank Acc
93.11
Clef209.3 ms238.6 ms98.4791.93
More on Clef
BFCL Exact
98.47
API-Bank Acc
91.93
Jev524.1 ms536.0 ms95.7588.19
More on Jev
BFCL Exact
95.75
API-Bank Acc
88.19
  • Clef-flash98.76
  • Clef98.47
  • Jev95.75
BFCL Exact benchmark scores across hosted decision models, October 2026.2
  • Clef-flash38.8ms
  • Clef209.3ms
  • Jev524.1ms
Median inference latency across hosted decision models, in milliseconds, October 2026.2

Clef-flash achieved 97.73 on Home appliances case exact evaluations and 90.93 macro-F1 on BANKING77, while Clef reached 97.43 macro-F1 on CLINC150+OOS and 69.19 on ToolRet nDCG@10. Across specialized enterprise tests, Clef scored 64.7 on invoice processing, 76.3 on customer service, and 62.9 on security incidents. Jev outperformed both Cloudflare models on the When2Call routing test at 80.97 accuracy.2

Local deployment with Strands, Laya, and Kev

Teams running self-hosted AI agents can deploy open-weight checkpoints to avoid external API calls. Strands Decider 2B replaces the LM head of Qwen3.5-2B with a pointer head containing just over 1 million parameters and fine-tunes the torso with a rank-16 LoRA adapter, recording a median latency of around 115 ms on an Nvidia RTX 3090 and around 153 ms on an M3 MacBook. It ranked 3rd of 33 in the 2B class on JevBench's public set, and 1st of 30 when excluding models just over 2B.6

Laya by Convai Innovations provides Apache 2.0 checkpoints between 322 million and 421 million parameters using ModernBERT-large and mmBERT-base. A single decision processes in 39.5 ms on its base English checkpoint and 32.8 ms on its multilingual checkpoint on an Nvidia Tesla T4 GPU, reaching throughputs of 103 to 332 decisions per second under batching. Jared Palmer's Kev project provides Qwen-based tiers at Kev-0.8B, Kev-4B, Kev-9B, and Kev-27B at 51 gigabytes.3

For front-door classification and agent tool calls, routing decisions should move from autoregressive LLMs to single-pass models whenever candidate sets are bounded. Edge setups with sub-50 millisecond budgets can start with Clef-flash or a local checkpoint such as Strands Decider 2B, the fastest options in the figures above.236