Make the Model Choose, Not Write: Selection Beats Generation for Structured Decisions

Hand a small language model a five-option question and it can answer roughly 59% of CommonsenseQA correctly without writing a single word. It reads the question, reads the options, and emits one letter. One forward pass. No prose, no hedging, no room to wander into a sentence that sounds confident but has no evidence behind it.

Now ask a much larger model a different question: “Why was this change made?” Watch it hand you a tidy, plausible reason that appears nowhere in the pull request.

Both models are doing the same thing — predicting the next token — but the second one has far more ways to be wrong. This article is about an architectural choice a lot of teams skip past: do you want the model to generate an answer, or to choose from a list you wrote?

The 11-pass problem

Say you want a typed answer. Ask for JSON and the model has to build it one token at a time: brace, key, colon, value. Eleven tokens means eleven forward passes through the network. Every one of those passes is a chance for the output to drift, and every one costs latency and money.

Now suppose the real question was “which of these five options is correct?” You don’t need a sentence. You need a choice. And a choice can be made in a single pass: run the prompt once, look at the probability distribution the model produces at the final position, restrict it to the tokens that represent your options, and take the highest.

That’s the whole trick. It’s called constrained decoding — you mask the vocabulary so only your options can be emitted — and it turns an open-ended writing task into a multiple-choice test the model takes in one shot.

What you get for free

• A bounded failure mode. The model can still be wrong, but only in ways you enumerated. It cannot invent a sixth category.
• A full ranking, not just a winner. One pass gives you the score for every option at once, so “second best” is free information.
• An auditable artifact. You can log the candidate list, the scores, and the threshold that was applied. That log is reproducible.
• Something you can unit test. A selection layer over a fixed candidate set can be tested with fixtures. A generation layer can only be eyeballed.

The two-question pattern

Here is a concrete example worth stealing. changelog-bot pulls the “why” out of a pull request and puts it into a CHANGELOG. The tempting approach is to hand the whole PR body to a model and ask why the change was made.

The team behind it deliberately didn’t do that, because the model would summarize, paraphrase, or simply infer a reason that was never written down. Instead they treated WHY extraction as a selection problem:

1. Preprocess the PR text to find candidate sentences — anything near markers like “why,” “because,” “so that,” “to avoid,” “motivation,” “context.”
2. Score each candidate against two separate questions: “Does this candidate explicitly state why the change was made?” and “Does it state a reason that applies to this change?”
3. Require both scores to clear a threshold.
4. Ship the winning candidate verbatim — the original sentence, not a model rewrite.

Step 4 is the one people skip. The model’s job is to decide which sentence, not to compose one. That single rule eliminates the most common failure in AI-generated changelogs: a beautiful reason no human ever wrote.

Why two questions and not one

Because “this is a reason” and “this is the reason for this change” are different claims. A PR might contain the sentence “The README was updated because the previous explanation was confusing.” That is obviously a reason — for the README change. Attach it to the cache-invalidation fix and your changelog is now confidently wrong.

Most teams discover this only after shipping. The fix is cheap if you design for it up front: separate relevance from evidence, and score both.

Your candidate list is the spec

Selection feels safer than generation, and it is — but it moves the hard part rather than removing it. If your candidate set doesn’t contain the truth, the model will confidently pick the best of the wrong things. You get the same bad answer you’d get from asking a human to choose from a bad list.

So treat the candidate set like an API contract. Two rules help:

• Keep candidates short and homogeneous. Sentences, not paragraphs. Options of wildly different lengths make scores hard to compare.
• Test the empty case. Write a handful of examples where no candidate is correct and confirm your thresholds actually reject them.

Always give it a “none of the above”

The most valuable option in any selection task is the one that means “nothing here is good enough.” Without it the model always answers, because the argmax of a probability distribution always exists. That is how a reason that reads well and means nothing ends up in your release notes.

Two reasonable policies:

• Drop it silently for low-stakes output — a missing “why” line is fine.
• Route to a human for anything a customer reads or a system acts on. Log the candidate set and the scores next to the request, so the reviewer sees the options the machine considered and the ones it rejected.

Then measure how often fallback fires. If it never fires across a few hundred real inputs, your threshold isn’t filtering anything — it’s decoration.

Do not read the probability as confidence

This is where people get into trouble, and it deserves hard numbers.

In the CommonsenseQA experiment, a small model run through constrained decoding scored about 59% accuracy (725 of 1,221). When predictions were grouped into confidence bins, the top bin — scores between 0.9 and 1.0 — held 809 of those 1,221 predictions. Its average stated confidence was 0.9855. Its actual accuracy was 0.7009.

Read that again: when the model was roughly “98.5% sure,” it was right 70% of the time. The 0.8–0.9 bin was worse — average confidence 0.8555 against accuracy 0.4711. A model that is not calibrated gives you a number that ranks options better than it describes the world.

What to do with the score instead

• Use it to rank, not to decide. Within a single run, the ordering of options is far more trustworthy than the absolute number attached to them.
• Calibrate on your own labels. Take 100–200 hand-labeled examples. Bin them by score. Compare average score to average accuracy per bin. If the two diverge by more than a few points, you need a mapping, not a mindset.
• Re-calibrate after any fine-tune. The same experiment showed a small fine-tune moving accuracy from 0.5938 to 0.6241 — a modest gain that also shifts the score distribution. Your old thresholds are now guesses.
• Watch for genuinely ambiguous inputs. Asked where you’d most likely find a bat, the model put 0.9978 on “cave,” as if “baseball game” and “attic” weren’t equally defensible. High confidence on a question with no single right answer isn’t a model bug. It’s an acceptance-criteria bug.

When generation is still the right tool

Selection is not a universal replacement. Use this rule:

• Can you enumerate the plausible outputs before the call? If yes, select. Classification, routing, triage, review gates, “is this claim supported by the source?”, “which of these tickets is the duplicate?” — all selection.
• Is the answer genuinely open-ended? Drafting a paragraph, explaining a concept, summarizing a document — generate.
• Is a wrong answer recoverable? If not, put a selection step downstream of the generation. “Does this draft contain a claim that isn’t in the source material? yes / no / unclear” is a selection task even when the producer is a generator.

That last point is the practical takeaway for agent builders. You rarely get to remove free-form generation from a pipeline. You can almost always attach a constrained gate at the end of it — and a gate that can only say yes, no, or unclear is much harder to talk past than an instruction that says “do not hallucinate.”

A migration path that fits in a sprint

1. Find your regexes. Every place you parse model output to extract an enum is already a selection task with a wasteful front end. Replace the “answer with one of these words” prompt and the regex with masking over the option tokens.
2. Convert your yes/no prompts. “Is this a duplicate?” becomes a two-option selection with a score you can threshold and log.
3. Add the none option and count it. Track fallback rate per task for a week. Anything at 0% or 100% deserves a look.
4. Calibrate only what gates action. You don’t need to calibrate everything — just the thresholds a human or another system acts on.

The deeper point is that a language model asked to write will always fill gaps, because filling gaps is what it was trained to do. Asking it to choose is how you keep the gaps yours to fill.

Key Takeaways

• Constrained selection turns a multi-pass generation task into a single forward pass and hands you a full ranking of your options for free.
• Bounded failure is the real prize: the model can only be wrong in ways you enumerated.
• Split evidence from relevance — “this is a reason” and “this is the reason for this change” are two questions, and both need thresholds.
• Ship the original candidate text. Let the model choose the sentence; don’t let it compose one.
• Always include a “none of the above” path, and measure how often it fires.
• Treat the output score as a ranking device, not a probability: in one measured run, the 0.9–1.0 bin averaged 0.9855 confidence and 0.7009 accuracy.
• Calibrate on 100–200 labeled examples, and re-calibrate after any fine-tune.
• Keep generation for open-ended writing, but put a selection gate at the end of any pipeline where a wrong answer is expensive.