The twenty things a production AI engineer is expected to know

The job moved one layer out

The post opens by renaming the skill. The work is no longer writing a clever prompt; it's building the machinery around the model so it behaves.

Two terms carry this. The first is engineering the scaffolding around the model rather than the words you send it (the harness — the context you load, the tools you expose, the loop that runs). The second is managing what fits in the model's limited reading window (context engineering — deciding what to write to a file, what to pull in, what to summarise away).

If you've been building with agents for a while, treat that part as the setup, not the lesson. The real content starts when the list turns to what breaks.

Most of the list is failure-handling

Strip out the half about serving models on your own hardware (caches, batching, quantization) and what's left is a catalogue of ways an agent fails in production.

The model returns JSON that doesn't parse, so you need a step that catches it and asks for a fix instead of crashing (a structured-output repair loop). The model invents a tool call that doesn't exist, so tool definitions need argument-checking and a defined fallback. An agent gets stuck swapping the same change back and forth, so it needs a hard cap on cycles and tools (a termination condition — the loop must be told when to stop and ask a human). And quality slips without anyone noticing unless you're watching the run: token counts, latency, errors, and drift over time (observability — being able to see what the agent actually did, after the fact).

This is the unglamorous 80% of the work, and the post is right that it's where reliability is won or lost.

The keeper: it was already learned for distributed systems

The last four lines are why this piece is worth the reading time. The author claims each agent-reliability pattern is an old idea wearing new clothes, and maps four directly.

A safety stop that halts a loop after repeated failures is a *circuit breaker*. Limiting how much damage one tool can do is a *bulkhead* (the ship-compartment idea: seal one flooded section so the whole hull doesn't sink). A multi-step agent that can undo its earlier steps when a later one fails is a saga pattern (each step ships with its own rollback). And a tool an agent looks up and calls is *service discovery*, which today means MCP.

The point isn't the trivia. It's that the reliability engineering agents need is a field that already exists, with fifty years of hard-won answers, and you don't have to invent it from the model outward.

Where it overreaches

Read as a curriculum, the list is misleading in one specific way: it's three different jobs printed as one. Roughly half of it (KV cache eviction, paged attention, continuous batching, INT4 versus AWQ versus GPTQ) only matters if you run the model server yourself. On a hosted API those knobs aren't even exposed to you, so that half is unactionable for most builders. A second slice (fine-tuning, distillation) is a machine-learning job most app builders never touch.

The genuinely portable part — the repair loops, the budgets, the evals, the observability, the safety boundaries — is maybe a third of the list, and the post gives no priority order. A beginner sees a wall instead of a path.

The distributed-systems mapping is also a little lossy. A saga assumes every step is reversible, but an agent that already sent the email or posted the tweet has nothing to roll back.

The same idea in three dialects

The mapping gets sharper if you run it through a domain you already know. In finance, a circuit breaker is a trading kill-switch or a fund's stop-loss. Argument validation on a tool call is the maker-checker (four-eyes) control before a payment goes out. Idempotency is the rule that a double-submitted instruction doesn't book the trade twice.

In software, the same controls are code review and capped retries. In AI, quality-checking the output is LLM-as-a-judge.

It's one discipline — don't let a single action do unbounded damage, and check work before it lands — speaking three dialects. That's why it generalises: nobody learns AI reliability from scratch, they translate controls they've already watched operate somewhere else.

Vocabulary

• harness engineering — building the scaffolding around a model (context, tools, loop), not the prompt
• context engineering — managing what fits in the model's limited reading window each step
• structured-output repair loop — catch malformed model output and ask it to fix, don't crash
• termination condition — a hard cap that forces a looping agent to stop or ask a human
• observability — seeing what an agent actually did: tokens, latency, errors, drift
• bulkhead — isolate parts so one failure can't sink the whole system (ship compartments)
• saga pattern — a multi-step process where each step carries its own rollback
• MCP — a standard way an agent discovers and calls the tools available to it

If you're building — what to watch for

• Give every agent loop a hard stop: a cap on cycles and tool calls, plus a rule for when to hand back to a human. A loop with no termination condition is a bill and a mess waiting to happen.
• Cap the blast radius of each tool before you widen what the agent can reach. Validate arguments, define the fallback for an invalid call, and keep destructive actions behind a different boundary from read-only ones.
• Treat rollback as part of the step, not an afterthought, and be honest about the steps that have none. Anything that sends, posts, or pays is irreversible, so it needs a check before it lands rather than an undo after.
• Quality regressions are silent by default. If nothing makes noise when output gets worse, you find out from your readers, not your logs. Keep a small golden set of outputs you judged genuinely good and check each new run against it.
• Watch cost and latency per feature. Token counts, run time and error rates are the cheapest observability you can add, and the thing most builders skip until the invoice arrives.

Reading it critically

• It prints three jobs as one: app-builder on a hosted API, model-serving infrastructure (the KV-cache, quantization and batching half), and machine learning (fine-tuning). On a hosted API the serving half isn't even exposed to you, so verify before treating it as a to-learn list.
• No sequence or priority. Twenty flat items with no 'learn these five first' reads as a wall to a beginner, not a path.
• The distributed-systems analogy is lossy: a saga assumes reversible steps, but a sent email or a posted tweet can't be rolled back, so the rollback-chain mapping oversells.
• 'You already learned these for distributed systems' assumes the reader did. Coming from finance or operations, the reliability patterns are the new part and the AI is not, which is the opposite of the post's framing.