Observability and Debugging AI Agents
Every agent in this series prints or returns a final result, with no real, structured record of the actual steps taken to get there. This part adds that — genuinely necessary once an agent runs in real, actual production against real requests.
Why agent observability is a real, distinct problem from normal logging
Normal application logging: a real, predictable sequence of
operations, genuinely easy to trace
Agent logging: the actual, real sequence of steps VARIES between
runs, per the AI Fundamentals series' own sampling coverage — the
same real request can genuinely take a different real path through
the loop each timeThis directly extends the same honest, real challenge the AI Projects series flagged for testing AI features — genuine non-determinism means a static test passing once doesn't guarantee the same real behavior on the next actual run, making real, structured logging of what ACTUALLY happened each time genuinely more important than for ordinary code.
A real, structured per-step trace logger
import time
import json
class AgentTrace:
def __init__(self, task_id: str):
self.task_id = task_id
self.steps = []
def log_step(self, iteration: int, action: str, details: dict) -> None:
self.steps.append({
"iteration": iteration,
"action": action,
"details": details,
"timestamp": time.time(),
})
def save(self) -> None:
AgentTraceLog.objects.create(task_id=self.task_id, trace=json.dumps(self.steps))def run_agent_with_tracing(request: str, task_id: str) -> str:
trace = AgentTrace(task_id)
messages = [{"role": "user", "content": request}]
for iteration in range(5):
response = client.messages.create(model="claude-sonnet-5", max_tokens=1024, tools=tools, messages=messages)
trace.log_step(iteration, "model_response", {"stop_reason": response.stop_reason})
if response.stop_reason != "tool_use":
trace.log_step(iteration, "final_answer", {"text": response.content[0].text})
trace.save()
return response.content[0].text
tool_call = next(b for b in response.content if b.type == "tool_use")
result = tool_functions[tool_call.name](**tool_call.input)
trace.log_step(iteration, "tool_call", {"tool": tool_call.name, "input": tool_call.input, "result": str(result)})
messages.append({"role": "assistant", "content": response.content})
messages.append({"role": "user", "content": [{"type": "tool_result", "tool_use_id": tool_call.id, "content": str(result)}]})
trace.save()
return "Exceeded expected steps."This real, per-step trace is precisely what makes part 18's error-recovery behavior and part 17's planning decisions genuinely reviewable after the fact — without it, understanding WHY a real agent took a specific, unexpected real path requires guessing; with it, the exact real sequence of reasoning, actions, and results is directly inspectable.
A real, practical trace review, concretely
def print_trace(task_id: str) -> None:
trace = AgentTraceLog.objects.get(task_id=task_id)
for step in json.loads(trace.trace):
if step["action"] == "tool_call":
print(f"[{step['iteration']}] Called {step['details']['tool']}({step['details']['input']}) → {step['details']['result']}")
elif step["action"] == "final_answer":
print(f"[{step['iteration']}] Final: {step['details']['text']}")Real, example trace output:
[0] Called check_inventory({'product_id': 'eth-light-roast'}) → {'stock': 14}
[1] Called get_supplier_lead_time({'supplier_id': 'addis-imports'}) → {'lead_time_days': 12}
[2] Called create_reorder_request({'product_id': 'eth-light-roast', 'quantity': 86}) → {'status': 'created'}
[3] Final: I checked inventory (14 units, below threshold)...This is genuinely part 6's execution trace, described narratively there, now a real, structured, permanently recorded artifact — directly usable for debugging an unexpected real result, or for part 21's mistake-pattern analysis across many real, actual runs.
Aggregate metrics worth tracking across many real runs
Real, practical metrics: average iterations per task, which tools
get called most often, real error rate per tool, tasks that hit
the max_iterations fallbackThis directly connects to the LLM & Advanced AI series' own evaluation coverage — genuine, aggregate patterns across many real traces (not just one) are what reveal systemic issues, like a specific real tool failing unusually often, or a genuinely common request type consistently hitting the iteration ceiling.
The real, honest limit: tracing shows what happened, not why it was correct
A real, complete trace tells you exactly what the agent did — it
doesn't automatically tell you whether that was the genuinely
RIGHT thing to do; that judgment still needs part 19 of the LLM &
Advanced AI series' own evaluation techniques applied on topNext: security risks in AI agents — real, honest coverage of prompt injection and other genuine attack surfaces unique to agentic systems.