Field notes
Approaches to Agentic Engineering
What I've learned shipping production agents for startups and enterprises across many industries.
How to read the city
Agent carrying a task out
Agent bringing a result back
The Orchestrator assigns all work
People sign off on risky actions
Live activity
Patterns
Start with the simplest shape that works. Most production systems are workflows with a few agentic steps, not free-roaming agents.
Workflows
LLMs and tools run through predefined code paths. Predictable, cheap, and easy to debug.
Best for: Well-understood, repeatable tasks
Agents
The LLM decides its own steps and tool calls, looping until the goal is met.
Best for: Open-ended tasks where the path isn't known up front
Find the balance
Every system sits somewhere between plain code and pure model judgment. Drag the slider to see what you trade as you move right.
Fixed, wired workflows
connectors · scheduled jobs · hard-coded rules
Agentic judgment
non-deterministic · takes over steps as trust grows
40% agentic
Blueprint (the sweet spot)
Deterministic steps wrap a few agentic ones. Most production systems I ship live here.
What makes AI agentic: the loop
A chatbot answers once. An agent decides what to do next, acts with tools, reads the results, and keeps going until the goal is met. Every agent, however fancy, is this loop.
What non-deterministic means in practice
An agent chooses its own path, so the same request can play out differently every time. Press the button and watch three runs of one task.
Same task, same model, same tools: “Summarize this customer's last three tickets and recommend a next step.”
Run 1
- thinking…
Run 2
- thinking…
Run 3
- thinking…
Running…Each run took a different path through the tools. That's what non-deterministic means in practice: you don't test an agent once, you measure its pass rate over many runs and add checks where paths go wrong.
Six building blocks
Prompt chaining
Each LLM call works on the previous output.
Routing
Classify the input, send it to a specialist.
Parallelization
Run independent subtasks at once, then merge.
Orchestrator and workers
A planner hands pieces to specialist agents.
Evaluator and optimizer
One model drafts, another critiques, repeat.
Autonomous agent
Tools in a loop, steered by feedback from the environment.
Ways to organize many agents
Choose by how much control you need versus how much the system must flex.
Centralized
One orchestrator assigns every task. Predictable and easy to trace.
Decentralized
Peers coordinate directly. Flexible and resilient, harder to debug.
Hierarchical
Managers guide teams of workers. Balances control with scale.
Federated
Separate systems cooperate over shared protocols like MCP or A2A.
Deterministic walls
Wrap agentic steps in plain code. The walls contain the unreliability; the agent only does the parts that need judgment.
- 1Load context and check out a clean workspace
- 2Understand the task and plan
- 3Do the work
- 4Validate: types, lint, schema, business rules
- 5Run the relevant tests
- 6Fix failures (two rounds max, then escalate)
- 7Package the result for human review
Never let the model do math
Even 99% accuracy is too low for numbers that matter. Have the model write the calculation and run it in a sandbox.
- 1
LLM extracts the inputs
Quantities, rates, dates pulled from documents
- 2
LLM writes the calculation
As short, readable code instead of mental math
- 3
Sandbox runs it
Isolated container, deterministic result
- 4
Rules check the output
Verified against limits before anyone relies on it
Tools over prompts
If the agent needs a fact or an action, give it a tool (MCP works well) instead of pasting the answer into the prompt. Tools stay current; pasted context goes stale.
Typed handoffs
Wherever an agent hands off to code or another agent, use a schema (Pydantic, BAML, JSON schema). Free text between steps is where pipelines quietly break.
Curate the toolbox
Don't hand an agent hundreds of tools. Pick the dozen or so that fit this task; too many choices drowns the context and degrades decisions.
Human at the irreversible step
Let agents do reversible work freely and pause for a person only before sending, deploying, deleting, or paying.
Challenges
Where agent projects stall. Most of these are math, context, and feedback problems, not model problems.
Errors compound
Each step that can fail multiplies. Long chains need checks between steps, or fewer steps.
End-to-end success when every step is 95% reliable.
Capability isn't reliability
Agents are non-deterministic, so one good run proves very little.
Succeeds at least once in 3 tries
Succeeds all 3 times
Same agent, 75% accurate per run. Demos measure the first number; production needs the second.
The doorman fallacy
A hotel replaces its doorman with an automatic door, because the job is 'opening the door'. Then regulars feel unwelcome, security slips, and the lobby feels cheaper. Agents fall into the same trap.
- 1Map the whole job, not the task. Shadow the people doing it; the job description is the tip of the iceberg.
- 2Name the invisible value. Trust, judgment, relationships, and noticing what is off rarely show up in metrics until they vanish.
- 3Automate the task, redeploy the human. Let the agent take the repetitive part and free people for the parts only they can do.
Thin or scattered context
Most bad answers aren't model failures; the agent simply didn't have the facts. They live in tickets, docs, code, logs, and people's heads.
Context overload
The opposite failure: dumping everything into the window. Noise degrades reasoning, raises cost, and hides the one fact that mattered.
Silent success
Agents report done when they aren't. Without traces and checks you find out when a customer does.
Drift
The same prompt behaves differently across runs, model versions, and providers. What passed last month may not pass today.
Approaches
The habits that get agents from impressive demo to something a business can depend on.
Automate intelligence, keep human judgment
Split the work by what a mistake costs, not by what the model can technically do.
Let agents handle
- ✓Extracting fields from documents
- ✓Searching and matching records
- ✓Normalizing and comparing data
- ✓Checklist and policy verification
- ✓Drafting routine documents
- ✓Tracking status and deadlines
Keep people in charge of
- ●High-value or irreversible decisions
- ●Interpreting intent and ambiguity
- ●Relationship-driven negotiations
- ●Budget and priority trade-offs
- ●Exceptions and escalations
- ●Final quality sign-off
Route by confidence
Have the agent score its own confidence, then let the score decide who acts.
Confidence 90%+
Auto-execute
Agent acts, logs it, and samples are spot-checked.
Confidence 70–90%
Propose
Agent drafts the action; a person approves with one click.
Confidence <70%
Hand off
Route to a human with the agent's notes attached.
Thresholds are a starting point. Tune them per action and by what a mistake costs, and raise autonomy only when evals back it up.
Evals are the spec
You can't unit test your way to a reliable agent, but you can measure it. Build the eval set before the feature.
The agent testing pyramid
Lots of cheap deterministic tests at the bottom, a few expensive end-to-end runs at the top.
End-to-end simulations and human review
~5% of tests
Full task, binary business outcome.
LLM-as-judge benchmarks
~15% of tests
Run 3 to 5 trials, track the pass rate, not the exact output.
Record-and-replay integration tests
~30% of tests
Capture real model calls, replay them, assert the tool sequence.
Deterministic unit tests
~50% of tests
Business rules, math, schemas. Milliseconds, every commit.
Three kinds of graders
Use the cheapest grader that can tell right from wrong.
Code graders
Exact match, regex, tool-call and rule checks. Fast and objective.
Best for: Math, rules, schemas
Model graders
LLM-as-judge with a rubric, run three times with majority vote.
Best for: Quality, tone, completeness
Human graders
150 to 250 labeled examples to calibrate the model judge.
Best for: Edge cases and ambiguity
Roll out behind eval gates
Each stage has to pass its gate before more people see the agent.
Shadow
Agent runs on real traffic; nobody sees its output.
Gate: Success rate above target
Canary
5 to 10% of traffic, rollback ready.
Gate: No quality regression
Staged
Expand by team or region, watch for drift.
Gate: Accuracy and recall hold
Full
Everyone, with ongoing 2 to 5% sampling.
Gate: Failures become new evals
AI coding agents: the new way teams build
Stripe's "Minions" are unattended coding agents: an engineer describes a task, and a finished pull request comes back for review. The model matters less than the developer infrastructure around it.
1,300+
agent-written PRs merged per week
<10s
to spin up an isolated devbox
~15
tools curated per task, from ~500
2
CI rounds max before a human steps in
The five-layer pipeline
Any team can adopt this shape with off-the-shelf coding agents, containers, and a few MCP tools.
- 1
Trigger
Chat, CLI, or an automated signal like a flaky test.
- 2
Sandbox
A clean, disposable environment with no production access.
- 3
Blueprint run
Deterministic steps wrapped around agentic ones.
- 4
Quality gates
Lint, types, tests: same bar as human code.
- 5
Human review
A templated PR; a person always approves the merge.
A reference stack
The layers I reach for on most agent builds. Swap any vendor; keep the layers.
1Experience
2Orchestration
3Tools
4Knowledge
5Models
6Trust
Where it's heading
Each new capability lets agents take on longer, messier work on their own. Plan for the horizon to keep moving.
Long context and memory
The whole codebase, contract set, or customer history stays in view, so agents stop forgetting what they read an hour ago.
Computer use
Agents work through any app's screen, not just the ones with an API.
Learning from feedback
Corrections become memory and evals, so the same mistake is made once.
Cheap decision models
Fast typed judgments make it affordable to put a guardrail on every step.
Illustrative, based on METR's finding that the length of tasks agents can finish has been doubling roughly every seven months. As that horizon grows, the seesaw shifts toward agentic work, but the gates, approvals, and evidence have to move with it.
Evals before features
Build a small golden set of real tasks first and score it on every change. Evals are the spec for non-deterministic software.
Trace everything
Log every prompt, tool call, and result (Langfuse or similar) so you can replay the exact run instead of guessing.
Route across models
Strong models plan and judge, faster ones do routine steps, and providers stay swappable.
Grow autonomy with evidence
Start as a workflow with tight gates. Loosen them behind feature flags as the numbers earn it.
Rich context: ontologies and knowledge graphs
An agent can only reason about what it can see. Giving it connected, trustworthy context is the single biggest quality lever I know.
The facts an agent needs are usually spread across a dozen systems that each describe the same things (a customer, a product, a release) in their own words. An ontology is a shared vocabulary for those things and how they relate; a knowledge graph is that vocabulary filled in with real data.
Give an agent the graph instead of a pile of documents and it stops guessing at connections. It can follow a question from symptom to cause to owner, and show its working. Netflix describes this well in its write-up on ontology-driven observability; here is the simplified version for any team.
Many sources, one answer
Each source answers a different question. The agent queries them live, checks the knowledge graph for what has been learned before, and reconciles everything into one answer it can back up.
Metrics
What moved, where, and for whom?
Activity
Which systems or customers are involved?
Changes
What changed just before?
Logs & tickets
Which errors or complaints line up?
Chat
Has anyone discussed this, and who can help?
Docs & code
What is the policy, or the fix?
Agent
asks each source, then reconciles
Knowledge graph
queried over the shared ontology
One synthesized answer
ranked hypotheses, each with its evidence
The knowledge flywheel
Each rotation encodes knowledge, and richer structure makes the next rotation smarter.
Observe
Harvest chat, tickets, logs, metrics, docs, and code changes.
Enrich
Agents map each signal onto the ontology. Each one reads what earlier agents wrote.
Infer
Correlation plus an LLM find patterns invisible in the raw data.
Adapt
Known patterns become rules, alerts, and agent playbooks.
Flywheel effect: the causal chain from one incident becomes a known pattern for the next.
From data to wisdom
One event, climbing the ladder. Agents do their best work near the top.
Data
"Timeout at 14:32"
Information
The checkout service timed out right after release 2.3.
Knowledge
Release → timeout → alert → abandoned carts (0.87 confidence)
Wisdom
Checkout releases cause 40% of serious incidents, so add a canary gate.
Iteration encodes knowledge
The ontology grows from a schema into memory. Patterns become queryable facts.
"Checkout had an incident."
"Checkout incidents follow payment-library upgrades; retry logic is the root cause 40% of the time."
Agents can now answer: "Which services keep breaking after the same kind of change?"
Getting started without a big platform
- 1Name your core entities (customer, product, document, person, event) and the few relationships between them. One page is enough.
- 2Pick two or three sources and map their IDs onto those entities. Reconciling identity is most of the value.
- 3Expose the graph to agents as a tool, for example an MCP server, so they query it on demand.
- 4Write agent outcomes back into the graph, and grow the ontology only when a real question needs it.
Human in the Loop
My channel on agentic AI in the real world: what works in production, what breaks, and where people still matter.
Watch on YouTube →Building agents for your team?
I help companies design, ship, and evaluate production agentic systems.
Get in Touch