Jev, from TypeSafe AI, has drawn enthusiastic attention because it captures a real need: in agents and software workflows, there are a great many nodes where the model must make a structured decision rather than generate a passage of text. Most decision problems can be abstracted into (or converted or reduced to) a multiple-choice question with a single answer: given the context, pick the most suitable one from a finite set of candidates, and supply a score that downstream steps can use.

General-purpose large models already have considerable judgment ability on their own. So a natural question arises: without training a specialized model from scratch, can we add a simple readout layer on top of an off-the-shelf LLM and turn it into a practical decision model?

The usual approach is to ask the LLM to emit the choice according to a JSON schema. But for a single-choice decision, generating a complete JSON token by token and then parsing and validating it brings considerable overhead, and most of those tokens are just the JSON structure itself. With a simple prompt protocol, we can dispense with JSON output and sharply reduce the number of autoregressive generation steps. One basic protocol is: present the question along with candidates labeled A, B, C, and so on, and require the model to reply with a single label; add U to mean "cannot answer."

This already counts as a basic decision model. But it only gives us a single result — we get no probability distribution or ranking from the model, and therefore no way to do any further processing.

One direct idea is to read, at the decision position, the next-token logits corresponding to each candidate label. This also lets us programmatically restrict the permitted output tokens to the set of candidate labels, thereby avoiding invalid output. We then apply softmax over just these scores, yielding a distribution over the valid options plus U that sums to 1. We call this protocol together with its readout logic V0, that is, the baseline. The "layer" here is a readout layer in the functional sense; it is not an added neural network layer that needs training.

It bears emphasizing: softmax only guarantees that the distribution is mathematically normalized; it performs no probability calibration of its own. Whether the scores are trustworthy depends entirely on the underlying model.

We tested V0 using a local Gemma 4 12B with thinking off, on 20-candidate development questions from Banking77. Across two sets totaling 153 questions, V0 got 107 right (69.9%); the correct answer was among the top two in the first round for 123 questions (80.4%). This shows that even when the top-ranked choice is wrong, the ranking the model gives through its logits still contains useful information. We also found that merely swapping the order of the candidates can change the model's choice. This shows the model does not satisfy invariance under option permutation; conversely, improving permutation invariance could well improve the model's performance.

Our first idea was ensembling. Asking the same question several times with different option orders, then mapping the probabilities back to the original options and fusing them, offers a chance to cancel out part of the positional bias. In our experiments this approach did improve accuracy or log loss, and AnyJev independently made this point earlier. But the approach is uneconomical: for a 20-option question, even with heavily pruned cyclic permutations, it still takes 20 inference passes. That forfeits the decision model's advantages of economy and low latency, and the compute cost approaches simply letting the base model think.

We also tried a cheaper shortcut: first measure the model's "prior preference" over labels and positions using blank, information-free questions, or questions isomorphic to the real protocol but with no option content, then subtract that from the logits of the real questions. The results were not good. The preference measured with a genuinely blank prompt is not the same thing as the preference inside a decision question, and subtracting it directly actually lowered accuracy; the partial correction from the isomorphic content-free prompt helped on individual metrics but produced no stable gain in accuracy. This suggests to us that the order and label bias is not a constant independent of question content that can be uniformly subtracted away, but is instead highly semantically dependent. Therefore, solving this problem at the root without sacrificing performance requires handling it during training, or at least during post-training, exactly as TypeSafe says in its manifesto. My conjecture is that a simple LoRA on domain data (for example, Banking questions with shuffled option orders) is unlikely to yield generalizable gains.

So we tried a lighter-weight ensembling idea: since the first round often ranks the correct answer within the top two, let those two run a "final round." This is V1: first run V0; when first-round confidence falls below a fixed threshold, take the two highest-scoring valid options and have the model read the original question again, choosing once more between just those two original labels. The second round is not told which option ranked first in round one, nor is it answering an additional yes/no question. It amounts to extracting from the same model a second judgment, focused on two candidates. Strictly speaking, we do not simply average the two rounds' probabilities; rather, we use the second round's choice to adjudicate the low-confidence questions.

On the 153 questions above, V1 got 113 right (73.9%), with 55 questions (35.9%) going to the final round. On a further 462 held-out questions from the Banking77 development set, using a threshold fixed in advance, the number correct rose from the first round's 338/462 (73.2%) to 347/462 (75.1%), invoking the second round on only 133 questions (28.8%). This gives a clear accuracy-versus-compute tradeoff. That said, "roughly 30% triggering" is only the result for this set of Banking questions; on our synthetic arithmetic questions, achieving a comparable gain required a markedly higher rate of final-round invocation.

The confidence score that decides whether a question goes to the final round passes through a small numerical mapping. What it changes is the score scale that the threshold corresponds to; it does not change the first round's option ranking, and on its own it cannot improve V0's accuracy.

As for the inference implementation, the second round can reuse the KV cache for the question portion from the first round, needing only to process the short appended question and then read the candidate label scores once more. In our local experiments, the p50 of the added latency for the second round was about 0.22 seconds. It is not free, but it is far lighter than regenerating a full response.

In this way we obtain a Simple Decision Model that does not alter the base model's weights: V0 completes a decision through the protocol and a single readout, and V1 appends a Top-2 adjudication when confidence is low. The experiments we have so far show that it brings reproducible gains on the limited datasets we tested, validated on a held-out set. This of course falls far short of reproducing Jev, but these experiments still give us two important hints:

  1. Base LLMs that have not undergone probability-calibration post-training exhibit clear probability calibration problems (we also regard invariance under option permutation as a probability calibration problem in the broad sense); we conjecture that solving this problem can improve their decision performance.
  2. The Top-2 final round we propose is essentially a simple, lightweight ensembling method. That it delivers stable and fairly meaningful gains suggests that developing simple, lightweight ensembling methods may be an effective short- to medium-term path. Methods of this kind need to be designed carefully in conjunction with the prompt protocol and the model architecture, and they leave a great deal of engineering room for cache optimization and parallelization.

In the course of these experiments, we used GPT-6 to generate a series of nonsense probability datasets. For example: toss an ideal coin 100 times — which interval will the number of heads fall into? Options A through G each correspond to an interval. The true probability for questions of this kind can be computed exactly, so they can serve as a training set with basic probabilistic properties. We have not yet done post-training. Whether very limited post-training on these nonsense datasets can improve a base LLM's probability calibration without sacrificing performance is the next interesting question.