The Feedback Bench

On the first rung nobody closes the loop: you get one shot, so every bit of intelligence you can offer has to ride in with the input. That’s all “prompt engineering” is — not incantations, just the four honest levers: who the modelA program whose behavior was tuned from examples rather than written by hand — a big pile of adjustable numbers. should be (a system promptStanding instructions given to a model before the conversation starts — who to be, what rules to follow.), who it’s talking to, what shape the answer must take, and an example worth imitating.

The grid above holds everything else fixed — same model, same seedThe number that picks the starting static — same seed, same prompt, same settings → the exact same image. — so each toggle shows a causal delta, not luck. Watch the bare question come back as a jargon lecture, then watch constraints snap it to two clean sentences. The output was always in the model; the prompt decides which output you meet.

Before going near loops you should see the dice. A language model doesn’t emit words — it emits logitsThe raw, unnormalized scores a model emits before they're turned into probabilities., one score for every tokenThe chunks a language model actually reads — usually word pieces, not whole words. it knows, squashed by softmaxTurns a list of raw scores into percentages that are positive and sum to 1 — bigger scores get a bigger share. into a probability list. Your reply is one draw from that list, and the sampler’s knobs — temperatureA knob that reshapes the model's word-choice odds — low keeps only safe favorites, high lets long-shot words into the draw., top-k, top-pA sampling rule that keeps just enough of the likeliest words to cover a set share of the odds, and drops the rest. — reshape it before the draw.

The distributions above are real GPT-2. Try arithmetic: even “two plus two equals” isn’t a sure thing to a small model — its favorite next word is one. Variance isn’t a bug you prompt away; it’s the raw material every loop is built to tame. Cold sampling gives you repeatable mediocrity; hot sampling gives you occasional brilliance you’ll need feedback to catch.

The second rung: you close the loop. Draft, judge, feed the judgment back, revise — run it in the player above and the resemblance to station 02 is not a metaphor. A feedback loop with a model inside is gradientThe direction of steepest improvement: for every adjustable number, which way (and how hard) to nudge it. descent where you are the loss function: your critique is the gradient, each revision is a step, and the whole machine converges only if the signal you feed it actually points downhill.

The simulator strips that to the studs — no LLM, just drafts that improve when revised and a rule for when to stop. Give the stop signal to an external judge who sees true quality and the curves converge every time. Give it to the worker’s own noisy self-estimate and drag the noise up: trials start quitting below the bar while sincerely believing they cleared it. Keep that picture — it’s the whole next section.

The third rung hands the model the loop itself: a goal, a tool, and the right to decide its own next step. That’s all an agentA model given tools and a goal, allowed to act, observe what happened, and decide its own next step. is. Agentic engineering is loss-function design — the model is now estimating its own gradient, and everything that makes agents powerful or fragile falls out of how good that estimate is.

Scrub the player’s transcripts again and watch the middle rung carefully: the critique loop found the spec’s ambiguity — should a bare ‘45’ parse as seconds or raise an error? — reasoned about it honestly, and chose wrong, because reflection can tell you where the spec is silent, but it cannot tell you which reasonable answer reality picked. The agent didn’t out-think it. It ran the tests, read expected 45, got ValueError, and stopped guessing. Ground truth beat introspection in two moves — that’s why the operator’s craft is mostly building loops whose feedback carries real signal: tests, compilers, rubrics, users. (This site is itself the exhibit: every station was built with an agent in exactly this kind of loop, with me and a build server as the loss function.)