tl;dr: Instead of generating tokens autoregressively, you can build a fast "decision model" by running a single forward pass on an LLM (e.g., Qwen3-1.7B) and masking logits to only allowed answer tokens (A-E), picking the highest probability. Benchmarked on CommonsenseQA, this yields ~59% accuracy (62% after finetuning), but the raw probabilities are badly miscalibrated—overconfident in the 0.9+ range. Temperature scaling (T≈3.8) flattens the distribution so confidence scores actually match empirical accuracy.