Decision models vs LLMs: where typed outputs beat tokens in production

On 15 September 2026, TypeSafe AI launched Jev at $0.042 per million input tokens, with output tokens free, as a model that returns typed probabilities instead of generated text, according to TypeSafe AI.

That launch sharpened a useful distinction. A decision model takes unstructured state and returns typed probabilistic answers for a fixed schema, such as yes/no, choice or score, instead of writing free text, according to TypeSafe AI and Simon Willison.

The public record supports a narrow verdict. Decision models fit fixed-schema automation because they avoid token-by-token output, parsing and type repair, while token LLMs still fit open-ended drafting, explanation and tasks with broad or shifting answer spaces, according to TypeSafe AI, Convai Innovations and Rizzo Flow.

The hard part is evidence quality. The strongest public numbers here are for latency, pricing and output shape. The weakest part is a neutral, apples-to-apples benchmark that measures decision models against token LLMs on the same fixed-schema task for accuracy, calibration, latency and cost in one harness. No source in the public record provides that complete package.

What a decision model is, in plain terms?

The practical definition is simple. A decision model consumes text or structured state and emits typed values that software can use directly, rather than prose that another system must parse, according to TypeSafe AI.

Willison describes the three common question types in Jev as yes/no, choice and score, which maps closely to the categories that Convai Innovations calls noul, choice and score for Laya.

That distinction matters because it changes the software contract. A token LLM can often be prompted to emit JSON, but it still generates strings sequentially and can produce refusals, extra commentary or invalid formatting, while Rizzo Flow says its system reads logits for allowed answers only and turns them into schema-validated JSON without sampling.

TypeSafe AI says this design is aimed at “AI-powered workflows / smart if-statements”, not chat. Willison makes the same point in plainer language: decision models are useful for classification-style work such as spam detection, labelling, prioritisation and reranking.

Convai Innovations says Laya returns typed answers with calibrated probabilities in a single forward pass and “never generates text”, while Rizzo Flow says it is “the open, local take on Jev” and reproduces the interface pattern with open weights and local hardware. The family resemblance is clear even though the implementations differ.

The limit is equally clear. TypeSafe AI says existing LLMs remain suited to chatbots, copilots and coding agents, and its own launch post positions Jev as a tool for bounded decisions rather than general text generation.

The closest public benchmark is useful, but it is not one neutral harness

The request for a side-by-side benchmark is reasonable, but the sources here do not give one clean, neutral test that measures decision models against token LLMs on the same fixed-schema task for accuracy, latency, calibration and cost. What they do give is a public comparison assembled from mixed provenance: competitor-led quality tables, third-party latency measurements cited by a competitor, and vendor pricing.

Sources: Convai Innovations model card, Laya README and TypeSafe AI launch post.

The result is still useful. The public numbers point in one direction on latency and operational shape: fixed-schema decision models are built to answer many bounded questions for one state without a decode loop. They do not settle the broader claim that decision models beat LLMs in general, because the neutral benchmark that would prove that is still missing from this source set.

Why fixed schemas cut latency and parsing failures?

The main speed gain is architectural, not cosmetic. TypeSafe AI says Jev uses a parallel sampler and gives up string generation entirely, while Convai Innovations says Laya returns typed answers in a single forward pass.

Convai Innovations reports 32.8 ms for one question on Laya-multilingual, 40.1 ms for five, 72.3 ms for 10 and 337 ms for 50 on a Tesla T4. Those figures matter because they show sublinear growth as the number of questions rises, which is exactly the pattern workflow systems want.

Willison notes that Jev can accept many questions for one state and evaluate them in parallel, so “sending many questions should take a similar time to sending just one”. That is the same production story in one sentence.

Rizzo Flow explains the mechanism in more detail. The state is prefilled once, each question branches from the same cache, every question becomes multiple choice, and the system reads only the logits of the allowed answer letters. Its README says that means no decoding loop, no output parsing and no type errors by construction.

The local timings are slower than the encoder-based systems, but they point the same way. Rizzo Flow says one short decision takes about 50 ms on an RTX 5060 Ti at 8-bit, 21 questions on a 2,000-token state take about 1 second, and the project’s Snake demo delivered about 150 ms per decision round trip with 0 generated tokens.

TypeSafe AI says Jev responds in 70-500 ms and claims 40x-200x speedups over frontier LLMs on “System One shaped queries”. That is a vendor claim, and the launch post itself says its published evals were generally run from laptops on the US West Coast. It is still consistent with the other public sources: when the job is fixed-schema, text decoding is wasted work.

The operational effect is easy to underrate. Once output is typed, software no longer needs prompt wrappers, JSON repair, refusal handling or regex clean-up for the final answer. That does not remove model error, but it does remove a class of integration error that teams often misread as intelligence.

What the public calibration evidence does and does not show?

Confidence is where decision models make their strongest conceptual claim. TypeSafe AI says Jev “always communicates confidence and uncertainty with every output” and that higher confidence means higher accuracy.

Convai Innovations says Laya is trained with reinforcement learning against strictly proper scoring rules, so reporting honest probabilities is the only way to maximise reward. The argument is not just that the model should emit a score, but that training should punish dishonest confidence.

The strongest public numbers in the public record are not from a neutral benchmark. Convai Innovations says expected calibration error was 0.081 for Laya and 0.246 for Jev, lower is better, after temperature scaling. The same source says Jev assigned zero probability to the true label on 16% of DAIR Emotion examples. Those are concrete figures, but they come from a competitor comparison.

Rizzo Flow takes the opposite tack and says its probabilities are uncalibrated unless users calibrate them on their own data, and that it makes “no claim of matching Jev or SemIf in quality”. That caveat is worth more than marketing copy because it draws a line between typed outputs and calibrated ones. The first is easy to implement. The second is hard.

Willison argues that the confidence story still leaves a black-box problem. A floating-point score can be easier to automate around than prose, but it can also hide the reasons behind a judgment. If a model flags spam, churn risk or applicant quality, calibration alone does not tell you which features drove that call or whether the signal is biased.

That is the right level of scepticism. Public calibration figures suggest decision models are at least trying to solve the problem directly. They do not prove that a model is fair, accountable or safe in a sensitive workflow.

Why LLMs still keep the wider job description?

The case for token LLMs is not that they are cleaner for fixed-schema work. It is that they can do much more than fixed-schema work. TypeSafe AI says LLMs remain suited to human-in-the-loop tasks such as chatbots, copilots and coding agents, while decision models are aimed at bounded workflow steps.

Fu, Chen, Jia and Sharan report that transformers in in-context linear regression share a convergence rate similar to iterative Newton’s method, converge exponentially faster than gradient descent and can learn on ill-conditioned data where gradient descent struggles. That does not make them ideal workflow routers, but it helps explain why token transformers remain unusually flexible learners.

Hanna, Liu and Variengien report that GPT-2 small uses a circuit for a greater-than task and that its final multilayer perceptrons boost the probability of valid end years. Again, the result is narrow, but it shows that autoregressive models can acquire general procedures from pre-training rather than task-specific retraining.

There is also a public sign that bounded decision models have their own weak spots. Convai Innovations says Jev leads on Banking77, with 0.870 on 72 labels versus 0.425 for Laya on 77 labels at default settings. That is still a decision-model comparison, not Jev versus a token LLM, but it shows a real limitation: once the label space is large and semantically crowded, not every typed-output system performs well.

Willison gives a good rule of thumb. Decision models are good for classification-style calls and reranking. If the output itself must be language, code or a novel explanation, you are back in LLM territory because the output is the work product, not just a control signal.

Why deployment cost and memory still matter after the model choice?

The cost case splits in two: hosted APIs and self-hosted weights. TypeSafe AI lists Jev at $0.042 per million input tokens and says output is “FREE (too cheap to meter)”. The same launch post says frontier LLM input pricing runs from $0.20 to $10 per million tokens and output tokens cost about 5x more than input.

Open models expose a different set of trade-offs. Convai Innovations lists Laya’s English and typed-decisions checkpoints at 421M parameters and its multilingual checkpoint at 322M parameters. Our calculation from the published Hugging Face weights says 0.4 billion parameters need about 0.8 GB of memory at 16-bit precision, about 0.4 GB at 8-bit and about 0.2 GB at 4-bit for weights alone, while the published safetensors files total 2.3 GB on disk.

Source: Convai Innovations model card, Rizzo Flow README, and our calculation from the published Laya weight files.

The less glamorous cost is model reload time. Laya’s README says a cold checkpoint rebuild takes a 7.4-second median reload on CPU and 10.3 seconds on a T4 when traffic alternates languages with only one model kept hot. The same source says preloading removes that and brings per-request latency down to 32.8 ms on GPU or 193-464 ms on CPU.

Rizzo Flow says the model loads in about 10 seconds, its default 4B Q8_0 file is 4.4 GB, and the smaller 1.7B option is about 2x faster but “much less accurate”. Those are mundane deployment details, but they often decide whether a team can run a model locally or close to data.

Who benefits from each public claim?

TypeSafe AI is selling early access to Jev, so its claims about speed, pricing and “similar levels of intelligence” are launch claims from the vendor. The post is still useful because it states the product’s definition, pricing model and intended scope clearly.

Convai Innovations publishes open weights and detailed timings for Laya, which gives outsiders more to inspect than a hosted API alone. It also compares Laya directly with Jev, which makes some of its most interesting quality claims inherently competitive.

Rizzo Flow benefits from demand for local decision systems, but its README is careful about caveats. It says the project is independent, does not reproduce Jev’s proprietary architecture or RLCD training, and does not claim matching quality. That makes it a good source for implementation detail and a weak source for broad quality claims, which is a sensible split.

Simon Willison is not selling a model in this context. His post is useful because it explains the category, lists concrete use cases such as reranking, and pushes back on the black-box and bias issues that vendors tend to downplay.

The NeurIPS 2024 paper by Fu, Chen, Jia and Sharan, the NeurIPS 2023 paper by Hanna, Liu and Variengien and the arXiv paper by Kong, Liu, Li, Yogatama and Ver Steeg are not product launches. They are useful for understanding what transformers and diffusion models can represent internally, not for proving any startup’s pricing or latency.

What we could not verify?

No source in the public record provides a single neutral benchmark that measures a decision model and a token LLM on the same fixed-schema task across accuracy, latency, calibration and cost with a shared harness. The closest public reconstruction mixes vendor claims, competitor comparisons and third-party timings cited second-hand.

We could not verify the draft’s claim that Rizzo Flow is “far faster than frontier chat models on the same problem shape” from an apples-to-apples source. Rizzo Flow’s README gives its own timings, and TypeSafe AI gives broad frontier-LLM timings, but the public record does not contain a controlled comparison on the same task and harness.

We also could not verify Jev’s internal architecture or RLCD training method beyond what TypeSafe AI says in its launch post. In the public record, there is no technical paper, model card or reproducible benchmark package from TypeSafe that would let outsiders inspect the mechanism in detail.

Rizzo Flow says its probabilities are uncalibrated unless users calibrate them on their own data. the public record does not include a broad benchmark that compares Rizzo Flow with Jev or Laya on common enterprise tasks, so its role here is as an implementation example, not a settled quality baseline.

Willison warns that bias should be front and centre for systems that reduce judgments to scores. no public source here includes subgroup fairness audits for hiring, moderation or triage. The next useful artefact is a neutral harness with shared schemas, calibration metrics and published latency setup; the public record does not have that yet.

Sources

• https://typesafe.ai/blog/introducing-system-one-models-and-jev
• https://huggingface.co/convaiinnovations/laya
• https://github.com/NandhaKishorM/laya
• https://github.com/Rizzo-AI-Academy/rizzo-flow
• https://simonwillison.net/2026/Sep/21/jev/
• https://laya.convaiinnovations.com/
• https://arxiv.org/abs/2310.17086
• https://arxiv.org/abs/2305.00586
• https://arxiv.org/abs/2310.07972