Once you have many agents delegating work, debugging becomes the main cost. You need to answer basic questions fast: Which agent made the call? Which tools did it touch? What did it see? What did it cost?

This is now a common pain point in “agent mesh” style systems. With one agent, you can log everything. With 20 agents delegating to each other, you need structure.

This article covers how to build observability into a multi-agent system. Not just logging. Traceability, controllability, and auditability. The three things that actually matter when things break.

What goes wrong in multi-agent systems

Single-agent apps are straightforward. One agent. One request. One response. You log the request, log the response, done.

Multi-agent systems are different. Agents delegate to other agents. Agents call tools. Tools return data that goes into other agents. Failures cascade. Root causes hide.

Delegation chains hide root causes

Agent A asks Agent B to do something. Agent B asks Agent C. Agent C calls a tool. The tool fails. Agent C returns an error. Agent B doesn’t handle it. Agent A gets a partial result.

Where did it break? You don’t know. The error message says “Agent A failed.” But Agent A didn’t fail. Agent C’s tool call failed. Agent B didn’t handle it. Agent A just got bad data.

Without tracing, you’re debugging blind. You see the symptom. You don’t see the cause.

Partial failures look like success

An agent completes a task. It returns success. But one tool call failed. The agent worked around it. The result is wrong, but the agent doesn’t know.

You see “Task completed successfully” in your logs. The user sees wrong data. You don’t know why. The agent thought it succeeded. It didn’t.

This happens when agents don’t validate tool outputs. Or when error handling is too permissive. Or when agents make assumptions about tool behavior.

One agent’s bad context contaminates others

Agent A retrieves bad data from a tool. It passes that data to Agent B. Agent B uses it. Agent C gets bad data from Agent B. The contamination spreads.

By the time you notice, three agents have bad context. You don’t know which one introduced it. You don’t know which tool returned bad data. You don’t know when it happened.

You need to track data lineage. Where did this data come from? Which tool returned it? Which agent processed it? When did it enter the system?

The cost of not knowing

When you can’t answer “what happened,” you spend time debugging. You read logs. You guess. You add more logging. You read more logs. You guess again.

This is expensive. Developer time. System downtime. User frustration.

The solution isn’t more logging. It’s structured observability. Traceability, controllability, auditability.

The 3 pillars: traceability, controllability, auditability

Observability has three parts. Each solves a different problem.

Traceability: end-to-end spans across agents + tools

Traceability answers “what happened.” You can follow a request from start to finish. You see every agent that touched it. Every tool that was called. Every decision that was made.

This is distributed tracing. OpenTelemetry spans. Correlation IDs. Parent-child relationships.

A trace shows:

• Which agent started the task
• Which agents it delegated to
• Which tools each agent called
• How long each step took
• What errors occurred

With traces, you can see the full path. You can find the root cause. You can understand the flow.

Controllability: budgets, rate limits, kill switches

Controllability answers “can I stop it.” You can set limits. You can enforce budgets. You can kill runaway tasks.

This is about preventing runaway costs. Token budgets. Time budgets. Tool call budgets. Rate limits.

A budget system tracks:

• How many tokens have been used
• How long the task has been running
• How many tool calls have been made
• Whether limits have been exceeded

When a limit is exceeded, the system stops. It doesn’t keep running. It doesn’t waste resources.

Auditability: “who said what” + “what sources were used”

Auditability answers “who did what and why.” You can see which agent made which decision. You can see which tools provided which data. You can see the reasoning chain.

This is about accountability. Compliance. Debugging. Understanding decisions.

An audit log shows:

• Which agent made each call
• What prompt was used
• What tool was called
• What data was returned
• What decision was made
• What sources were cited

With audit logs, you can answer “why did the agent do that.” You can see the reasoning. You can verify citations. You can check for hallucinations.

A practical tracing model

Tracing needs structure. Correlation IDs. Event streams. Metadata.

Correlation IDs: trace_id, span_id, task_id

Every request gets three IDs:

• trace_id: The entire request, from start to finish. All agents, all tools, all spans share the same trace_id.
• span_id: One operation. One agent step. One tool call. Each span has a unique span_id and a parent_span_id.
• task_id: The logical task. One user request might create multiple tasks. Each task has a task_id.

interface TraceContext {
  trace_id: string;      // Same for entire request
  span_id: string;       // Unique per operation
  parent_span_id?: string; // Links to parent span
  task_id: string;       // Logical task identifier
}

When Agent A delegates to Agent B, Agent B gets the same trace_id. Agent B creates a new span_id. Agent B sets parent_span_id to Agent A’s span_id.

This creates a tree. You can see the delegation chain. You can see which spans are siblings. You can reconstruct the flow.

A standard event stream

Every agent emits events. Standard events. Structured data.

AgentStarted: An agent begins processing a task.

{
  event: "AgentStarted",
  trace_id: "trace-123",
  span_id: "span-456",
  task_id: "task-789",
  agent_id: "agent-a",
  timestamp: 1234567890,
  metadata: {
    model: "gpt-4",
    prompt_version: "v2.1",
    tenant_id: "tenant-abc"
  }
}

ToolCalled: An agent calls a tool.

{
  event: "ToolCalled",
  trace_id: "trace-123",
  span_id: "span-457",
  parent_span_id: "span-456",
  task_id: "task-789",
  agent_id: "agent-a",
  tool_name: "database_query",
  tool_input: { query: "SELECT * FROM users" },
  timestamp: 1234567891
}

ToolResult: A tool returns a result.

{
  event: "ToolResult",
  trace_id: "trace-123",
  span_id: "span-457",
  task_id: "task-789",
  agent_id: "agent-a",
  tool_name: "database_query",
  tool_output: { rows: [...] },
  latency_ms: 45,
  success: true,
  timestamp: 1234567936
}

AgentDelegated: An agent delegates to another agent.

{
  event: "AgentDelegated",
  trace_id: "trace-123",
  span_id: "span-456",
  task_id: "task-789",
  source_agent: "agent-a",
  target_agent: "agent-b",
  delegation_context: { task: "process_order" },
  timestamp: 1234567940
}

AgentCompleted: An agent finishes processing.

{
  event: "AgentCompleted",
  trace_id: "trace-123",
  span_id: "span-456",
  task_id: "task-789",
  agent_id: "agent-a",
  result: { status: "success", data: {...} },
  latency_ms: 150,
  token_usage: { input: 500, output: 200 },
  timestamp: 1234568000
}

Attach metadata: model, prompt version, tool name, tenant, latency, token usage

Every event includes metadata. Not just what happened. Context.

• Model: Which model was used? gpt-4? claude-3? This affects cost and behavior.
• Prompt version: Which prompt version? v1.0? v2.1? This affects behavior.
• Tool name: Which tool was called? This affects debugging.
• Tenant: Which tenant? This affects isolation and billing.
• Latency: How long did it take? This affects performance.
• Token usage: How many tokens? This affects cost.

This metadata makes traces useful. You can filter by model. You can find expensive operations. You can debug tenant-specific issues.

Budgets that actually work

Budgets prevent runaway costs. But they need to be enforced. Not just tracked.

Token budget per task (not per request)

Set token budgets per task. Not per request. A task might involve multiple agents. Multiple tool calls. Multiple LLM calls.

Track tokens across the entire task. When the task exceeds its budget, stop. Don’t let it continue.

interface TaskBudget {
  task_id: string;
  max_tokens: number;
  spent_tokens: number;
  max_time_ms: number;
  spent_time_ms: number;
  max_tool_calls: number;
  spent_tool_calls: number;
}

When an agent makes an LLM call, increment spent_tokens. When spent_tokens exceeds max_tokens, reject the call. Return an error. Don’t process it.

Time budget per delegation chain

Set time budgets per delegation chain. A task might delegate to multiple agents. Each delegation adds latency. The total time matters.

Track time from task start. When the task exceeds its time budget, stop. Don’t let it continue.

This prevents tasks from running forever. It prevents cascading delays. It enforces SLAs.

Tool budget (max tool calls / tool classes)

Set tool call budgets. Limit how many times an agent can call tools. Or limit calls to specific tool classes.

This prevents agents from calling tools in loops. It prevents expensive tool calls from running repeatedly. It enforces resource limits.

interface ToolBudget {
  max_calls_per_task: number;
  max_calls_per_tool_class: Record<string, number>;
  spent_calls: number;
  spent_calls_by_class: Record<string, number>;
}

When an agent calls a tool, check the budget. If the budget is exceeded, reject the call. Return an error.

”Stop conditions” for loops and flapping

Agents can get stuck in loops. They call the same tool repeatedly. They delegate to the same agent repeatedly. They never make progress.

Add stop conditions:

• Max iterations: If an agent makes the same call N times, stop.
• Max delegation depth: If a task delegates more than N levels deep, stop.
• Seen-task detection: If a task has been seen before, stop.

interface StopConditions {
  max_iterations: number;
  max_delegation_depth: number;
  seen_tasks: Set<string>;
}

When an agent tries to make a call, check stop conditions. If any condition is met, stop. Return an error. Don’t continue.

Quality controls inside the mesh

Agents make mistakes. They hallucinate. They make wrong decisions. You need quality controls.

A “Verifier agent” pattern (don’t trust the main agent)

Don’t trust the main agent’s output. Verify it. Use a verifier agent.

The verifier agent checks:

• Is the output in the correct format?
• Does it match the schema?
• Are required citations present?
• Is the reasoning sound?

If verification fails, reject the output. Don’t use it. Return an error.

This adds latency. But it prevents bad outputs from propagating. It catches errors early.

Output contracts + schema checks

Define output contracts. Schemas. Required fields. Required formats.

When an agent produces output, validate it against the contract. If it doesn’t match, reject it. Don’t use it.

interface OutputContract {
  schema: JSONSchema;
  required_fields: string[];
  required_citations: boolean;
  max_length?: number;
}

This catches format errors. It catches missing fields. It enforces structure.

Citation/grounding signals for tool outputs

Require citations. When an agent uses tool output, require a citation. Track which tool provided which data.

This enables auditability. You can see where data came from. You can verify sources. You can check for hallucinations.

interface Citation {
  tool_name: string;
  tool_call_id: string;
  data_excerpt: string;
  timestamp: number;
}

When an agent uses tool output, attach a citation. Store it in the trace. Include it in the audit log.

Security + safety signals

Multi-agent systems have security risks. Prompt injection. Data leakage. Unauthorized tool access.

Tool output sanitization

Sanitize tool outputs before they enter agent context. Strip control characters. Validate formats. Escape special sequences.

This prevents prompt injection through tool outputs. It prevents control character attacks. It enforces data hygiene.

function sanitizeToolOutput(output: string): string {
  // Remove control characters (except newlines and tabs)
  let sanitized = output.replace(/[\x00-\x08\x0B-\x0C\x0E-\x1F\x7F]/g, '');
  
  // Limit length
  if (sanitized.length > MAX_TOOL_OUTPUT_LENGTH) {
    sanitized = sanitized.substring(0, MAX_TOOL_OUTPUT_LENGTH) + '... [truncated]';
  }
  
  // Escape JSON special characters if needed
  // (depends on how output is used)
  
  return sanitized;
}

This is a first line of defense. It doesn’t solve everything. But it prevents common attacks.

Prompt injection detection points

Detect prompt injection attempts. Check tool outputs for injection patterns. Check user inputs. Check retrieved text.

When injection is detected, log it. Reject the input. Don’t process it.

function detectPromptInjection(text: string): boolean {
  const injectionPatterns = [
    /ignore previous instructions/i,
    /system:.*override/i,
    /\[INST\].*\[/INST\]/i,
    // Add more patterns
  ];
  
  return injectionPatterns.some(pattern => pattern.test(text));
}

This is heuristic. It won’t catch everything. But it catches common patterns.

Allowlist tools per agent role (least privilege)

Don’t give every agent access to every tool. Use allowlists. Each agent role gets a specific set of tools.

This enforces least privilege. It reduces attack surface. It prevents unauthorized tool access.

interface AgentRole {
  role_id: string;
  allowed_tools: string[];
  allowed_tool_classes: string[];
}

When an agent tries to call a tool, check the allowlist. If the tool isn’t allowed, reject the call. Return an error.

A reference implementation outline

Here’s how to build this. Not theory. Code.

OpenTelemetry spans per agent step

Use OpenTelemetry for tracing. Create spans for each agent step. Create spans for each tool call.

import { trace, Span } from '@opentelemetry/api';

class AgentTracer {
  startAgentSpan(agentId: string, taskId: string, traceId: string): Span {
    const tracer = trace.getTracer('agent-mesh');
    const span = tracer.startSpan(`agent.${agentId}`, {
      attributes: {
        'agent.id': agentId,
        'task.id': taskId,
        'trace.id': traceId,
      },
    });
    return span;
  }

  startToolSpan(toolName: string, parentSpan: Span): Span {
    const tracer = trace.getTracer('agent-mesh');
    const span = tracer.startSpan(`tool.${toolName}`, {
      parent: parentSpan,
      attributes: {
        'tool.name': toolName,
      },
    });
    return span;
  }
}

This creates standard OpenTelemetry spans. You can use any OpenTelemetry backend. Jaeger. Tempo. Datadog. New Relic.

Central log store + event indexing

Store all events in a central log store. Index them by trace_id, task_id, agent_id, tool_name.

Use Elasticsearch. Or ClickHouse. Or a time-series database. Something that can handle high volume. Something that can query by multiple fields.

interface EventStore {
  ingest(event: AgentEvent): Promise<void>;
  queryByTraceId(traceId: string): Promise<AgentEvent[]>;
  queryByTaskId(taskId: string): Promise<AgentEvent[]>;
  queryByAgentId(agentId: string, timeRange: TimeRange): Promise<AgentEvent[]>;
}

This enables fast queries. You can find all events for a trace. You can find all events for a task. You can find all events for an agent.

Dashboards: top failure reasons, cost per workflow, tool error rates

Build dashboards. Not just logs. Visualizations.

• Top failure reasons: Which errors occur most often? Which agents fail most? Which tools fail most?
• Cost per workflow: How much does each workflow cost? Which workflows are expensive? Which agents are expensive?
• Tool error rates: Which tools fail most? What are the error rates? What are the latency percentiles?

This enables proactive monitoring. You can see problems before they become critical. You can optimize expensive workflows. You can fix failing tools.

Code samples

Here are working examples. Production-ready patterns.

Sample 1: OpenTelemetry instrumentation wrapper around “agent step execution”

import { trace, Span, context, SpanStatusCode } from '@opentelemetry/api';

interface AgentStep {
  agentId: string;
  taskId: string;
  traceId: string;
  parentSpanId?: string;
  input: unknown;
  model: string;
  promptVersion: string;
}

interface AgentStepResult {
  output: unknown;
  tokenUsage: { input: number; output: number };
  latencyMs: number;
  error?: Error;
}

class InstrumentedAgentExecutor {
  private tracer = trace.getTracer('agent-mesh');

  async executeStep(step: AgentStep): Promise<AgentStepResult> {
    const span = this.tracer.startSpan(`agent.${step.agentId}`, {
      attributes: {
        'agent.id': step.agentId,
        'task.id': step.taskId,
        'trace.id': step.traceId,
        'model': step.model,
        'prompt.version': step.promptVersion,
      },
    });

    const startTime = Date.now();
    let result: AgentStepResult;

    try {
      // Execute agent step (your actual agent logic)
      const output = await this.runAgentStep(step);
      
      const latencyMs = Date.now() - startTime;
      const tokenUsage = this.estimateTokenUsage(step.input, output);

      span.setAttributes({
        'agent.latency_ms': latencyMs,
        'agent.tokens.input': tokenUsage.input,
        'agent.tokens.output': tokenUsage.output,
        'agent.status': 'success',
      });

      result = {
        output,
        tokenUsage,
        latencyMs,
      };

      span.setStatus({ code: SpanStatusCode.OK });
    } catch (error) {
      const latencyMs = Date.now() - startTime;
      
      span.setAttributes({
        'agent.latency_ms': latencyMs,
        'agent.status': 'error',
        'error.message': error.message,
        'error.type': error.constructor.name,
      });

      span.setStatus({
        code: SpanStatusCode.ERROR,
        message: error.message,
      });

      result = {
        output: null,
        tokenUsage: { input: 0, output: 0 },
        latencyMs,
        error: error as Error,
      };
    } finally {
      span.end();
    }

    return result;
  }

  private async runAgentStep(step: AgentStep): Promise<unknown> {
    // Your actual agent execution logic
    // This is a placeholder
    return { result: 'placeholder' };
  }

  private estimateTokenUsage(input: unknown, output: unknown): { input: number; output: number } {
    // Simple estimation (in production, use actual token counting)
    const inputStr = JSON.stringify(input);
    const outputStr = JSON.stringify(output);
    return {
      input: Math.ceil(inputStr.length / 4), // Rough estimate: 4 chars per token
      output: Math.ceil(outputStr.length / 4),
    };
  }
}

This wrapper creates OpenTelemetry spans for every agent step. It tracks latency, token usage, errors. It sets span status. It adds attributes.

Sample 2: A shared AgentEvent schema + logger (structured JSON)

type AgentEventType = 
  | 'AgentStarted'
  | 'ToolCalled'
  | 'ToolResult'
  | 'AgentDelegated'
  | 'AgentCompleted'
  | 'AgentError';

interface BaseAgentEvent {
  event: AgentEventType;
  trace_id: string;
  span_id: string;
  task_id: string;
  timestamp: number;
  agent_id: string;
  metadata: Record<string, unknown>;
}

interface AgentStartedEvent extends BaseAgentEvent {
  event: 'AgentStarted';
  metadata: {
    model: string;
    prompt_version: string;
    tenant_id: string;
  };
}

interface ToolCalledEvent extends BaseAgentEvent {
  event: 'ToolCalled';
  parent_span_id: string;
  tool_name: string;
  tool_input: unknown;
}

interface ToolResultEvent extends BaseAgentEvent {
  event: 'ToolResult';
  tool_name: string;
  tool_output: unknown;
  latency_ms: number;
  success: boolean;
}

interface AgentDelegatedEvent extends BaseAgentEvent {
  event: 'AgentDelegated';
  source_agent: string;
  target_agent: string;
  delegation_context: Record<string, unknown>;
}

interface AgentCompletedEvent extends BaseAgentEvent {
  event: 'AgentCompleted';
  result: unknown;
  latency_ms: number;
  token_usage: { input: number; output: number };
}

type AgentEvent = 
  | AgentStartedEvent
  | ToolCalledEvent
  | ToolResultEvent
  | AgentDelegatedEvent
  | AgentCompletedEvent;

class AgentEventLogger {
  constructor(private eventStore: EventStore) {}

  logAgentStarted(
    traceId: string,
    spanId: string,
    taskId: string,
    agentId: string,
    metadata: { model: string; prompt_version: string; tenant_id: string }
  ): void {
    const event: AgentStartedEvent = {
      event: 'AgentStarted',
      trace_id: traceId,
      span_id: spanId,
      task_id: taskId,
      timestamp: Date.now(),
      agent_id: agentId,
      metadata,
    };
    this.eventStore.ingest(event);
  }

  logToolCalled(
    traceId: string,
    spanId: string,
    parentSpanId: string,
    taskId: string,
    agentId: string,
    toolName: string,
    toolInput: unknown
  ): void {
    const event: ToolCalledEvent = {
      event: 'ToolCalled',
      trace_id: traceId,
      span_id: spanId,
      parent_span_id: parentSpanId,
      task_id: taskId,
      timestamp: Date.now(),
      agent_id: agentId,
      tool_name: toolName,
      tool_input: toolInput,
      metadata: {},
    };
    this.eventStore.ingest(event);
  }

  logToolResult(
    traceId: string,
    spanId: string,
    taskId: string,
    agentId: string,
    toolName: string,
    toolOutput: unknown,
    latencyMs: number,
    success: boolean
  ): void {
    const event: ToolResultEvent = {
      event: 'ToolResult',
      trace_id: traceId,
      span_id: spanId,
      task_id: taskId,
      timestamp: Date.now(),
      agent_id: agentId,
      tool_name: toolName,
      tool_output: toolOutput,
      latency_ms: latencyMs,
      success,
      metadata: {},
    };
    this.eventStore.ingest(event);
  }

  logAgentDelegated(
    traceId: string,
    spanId: string,
    taskId: string,
    sourceAgent: string,
    targetAgent: string,
    delegationContext: Record<string, unknown>
  ): void {
    const event: AgentDelegatedEvent = {
      event: 'AgentDelegated',
      trace_id: traceId,
      span_id: spanId,
      task_id: taskId,
      timestamp: Date.now(),
      agent_id: sourceAgent,
      source_agent: sourceAgent,
      target_agent: targetAgent,
      delegation_context: delegationContext,
      metadata: {},
    };
    this.eventStore.ingest(event);
  }

  logAgentCompleted(
    traceId: string,
    spanId: string,
    taskId: string,
    agentId: string,
    result: unknown,
    latencyMs: number,
    tokenUsage: { input: number; output: number }
  ): void {
    const event: AgentCompletedEvent = {
      event: 'AgentCompleted',
      trace_id: traceId,
      span_id: spanId,
      task_id: taskId,
      timestamp: Date.now(),
      agent_id: agentId,
      result,
      latency_ms: latencyMs,
      token_usage: tokenUsage,
      metadata: {},
    };
    this.eventStore.ingest(event);
  }
}

This logger emits structured events. Every event has the same base structure. Every event includes trace_id, span_id, task_id. This enables correlation. This enables queries.

Sample 3: Budget manager (token/time/tool limits) with enforced stop conditions

interface TaskBudget {
  task_id: string;
  max_tokens: number;
  spent_tokens: number;
  max_time_ms: number;
  spent_time_ms: number;
  start_time: number;
  max_tool_calls: number;
  spent_tool_calls: number;
  max_tool_calls_per_class: Record<string, number>;
  spent_tool_calls_per_class: Record<string, number>;
}

interface StopConditions {
  max_iterations: number;
  max_delegation_depth: number;
  seen_tasks: Set<string>;
}

class BudgetManager {
  private budgets: Map<string, TaskBudget> = new Map();
  private stopConditions: Map<string, StopConditions> = new Map();

  createBudget(
    taskId: string,
    limits: {
      maxTokens?: number;
      maxTimeMs?: number;
      maxToolCalls?: number;
      maxToolCallsPerClass?: Record<string, number>;
    }
  ): void {
    const budget: TaskBudget = {
      task_id: taskId,
      max_tokens: limits.maxTokens ?? Infinity,
      spent_tokens: 0,
      max_time_ms: limits.maxTimeMs ?? Infinity,
      spent_time_ms: 0,
      start_time: Date.now(),
      max_tool_calls: limits.maxToolCalls ?? Infinity,
      spent_tool_calls: 0,
      max_tool_calls_per_class: limits.maxToolCallsPerClass ?? {},
      spent_tool_calls_per_class: {},
    };
    this.budgets.set(taskId, budget);

    const stopConditions: StopConditions = {
      max_iterations: 100,
      max_delegation_depth: 5,
      seen_tasks: new Set(),
    };
    this.stopConditions.set(taskId, stopConditions);
  }

  checkTokenBudget(taskId: string, tokensToSpend: number): void {
    const budget = this.budgets.get(taskId);
    if (!budget) {
      throw new Error(`No budget found for task ${taskId}`);
    }

    if (budget.spent_tokens + tokensToSpend > budget.max_tokens) {
      throw new Error(
        `Token budget exceeded for task ${taskId}: ${budget.spent_tokens + tokensToSpend} > ${budget.max_tokens}`
      );
    }

    budget.spent_tokens += tokensToSpend;
  }

  checkTimeBudget(taskId: string): void {
    const budget = this.budgets.get(taskId);
    if (!budget) {
      throw new Error(`No budget found for task ${taskId}`);
    }

    const elapsed = Date.now() - budget.start_time;
    budget.spent_time_ms = elapsed;

    if (elapsed > budget.max_time_ms) {
      throw new Error(
        `Time budget exceeded for task ${taskId}: ${elapsed}ms > ${budget.max_time_ms}ms`
      );
    }
  }

  checkToolCallBudget(taskId: string, toolName: string, toolClass?: string): void {
    const budget = this.budgets.get(taskId);
    if (!budget) {
      throw new Error(`No budget found for task ${taskId}`);
    }

    // Check total tool calls
    if (budget.spent_tool_calls >= budget.max_tool_calls) {
      throw new Error(
        `Tool call budget exceeded for task ${taskId}: ${budget.spent_tool_calls} >= ${budget.max_tool_calls}`
      );
    }

    // Check per-class tool calls
    if (toolClass && budget.max_tool_calls_per_class[toolClass]) {
      const spent = budget.spent_tool_calls_per_class[toolClass] || 0;
      const max = budget.max_tool_calls_per_class[toolClass];
      if (spent >= max) {
        throw new Error(
          `Tool call budget exceeded for class ${toolClass} in task ${taskId}: ${spent} >= ${max}`
        );
      }
      budget.spent_tool_calls_per_class[toolClass] = spent + 1;
    }

    budget.spent_tool_calls += 1;
  }

  checkStopConditions(
    taskId: string,
    iteration: number,
    delegationDepth: number,
    taskSignature: string
  ): void {
    const conditions = this.stopConditions.get(taskId);
    if (!conditions) {
      return; // No stop conditions set
    }

    // Check max iterations
    if (iteration > conditions.max_iterations) {
      throw new Error(
        `Max iterations exceeded for task ${taskId}: ${iteration} > ${conditions.max_iterations}`
      );
    }

    // Check max delegation depth
    if (delegationDepth > conditions.max_delegation_depth) {
      throw new Error(
        `Max delegation depth exceeded for task ${taskId}: ${delegationDepth} > ${conditions.max_delegation_depth}`
      );
    }

    // Check seen tasks
    if (conditions.seen_tasks.has(taskSignature)) {
      throw new Error(
        `Circular task detected for task ${taskId}: ${taskSignature}`
      );
    }

    conditions.seen_tasks.add(taskSignature);
  }

  getBudget(taskId: string): TaskBudget | undefined {
    return this.budgets.get(taskId);
  }

  cleanup(taskId: string): void {
    this.budgets.delete(taskId);
    this.stopConditions.delete(taskId);
  }
}

This budget manager enforces limits. It throws errors when limits are exceeded. It tracks spending. It prevents runaway costs.

Sample 4: A redaction/sanitization function for tool outputs before they enter context

const MAX_TOOL_OUTPUT_LENGTH = 100000; // 100KB
const MAX_LINES = 1000;

function sanitizeToolOutput(output: unknown): string {
  // Convert to string if needed
  let text: string;
  if (typeof output === 'string') {
    text = output;
  } else if (typeof output === 'object' && output !== null) {
    text = JSON.stringify(output);
  } else {
    text = String(output);
  }

  // Remove control characters (except newlines, tabs, carriage returns)
  text = text.replace(/[\x00-\x08\x0B-\x0C\x0E-\x1F\x7F]/g, '');

  // Limit length
  if (text.length > MAX_TOOL_OUTPUT_LENGTH) {
    text = text.substring(0, MAX_TOOL_OUTPUT_LENGTH) + '\n... [truncated]';
  }

  // Limit lines
  const lines = text.split('\n');
  if (lines.length > MAX_LINES) {
    text = lines.slice(0, MAX_LINES).join('\n') + '\n... [truncated lines]';
  }

  // Remove potential prompt injection patterns (basic heuristic)
  const injectionPatterns = [
    /ignore\s+previous\s+instructions/gi,
    /system:\s*override/gi,
    /\[INST\].*?\[\/INST\]/gi,
    /<\|im_start\|>.*?<\|im_end\|>/gi,
  ];

  for (const pattern of injectionPatterns) {
    if (pattern.test(text)) {
      // Log the detection but don't fail - just remove the pattern
      console.warn('Potential prompt injection pattern detected in tool output');
      text = text.replace(pattern, '[REDACTED]');
    }
  }

  // Normalize whitespace (collapse multiple spaces, but preserve structure)
  text = text.replace(/[ \t]+/g, ' ');
  text = text.replace(/\n{3,}/g, '\n\n');

  return text;
}

function validateToolOutput(output: string): { valid: boolean; errors: string[] } {
  const errors: string[] = [];

  // Check length
  if (output.length > MAX_TOOL_OUTPUT_LENGTH) {
    errors.push(`Output exceeds maximum length: ${output.length} > ${MAX_TOOL_OUTPUT_LENGTH}`);
  }

  // Check for null bytes
  if (output.includes('\x00')) {
    errors.push('Output contains null bytes');
  }

  // Check for excessive nesting (if JSON)
  try {
    const parsed = JSON.parse(output);
    const depth = getObjectDepth(parsed);
    if (depth > 20) {
      errors.push(`JSON nesting depth too high: ${depth} > 20`);
    }
  } catch {
    // Not JSON, that's fine
  }

  return {
    valid: errors.length === 0,
    errors,
  };
}

function getObjectDepth(obj: unknown, currentDepth = 0): number {
  if (typeof obj !== 'object' || obj === null) {
    return currentDepth;
  }

  if (Array.isArray(obj)) {
    return Math.max(...obj.map(item => getObjectDepth(item, currentDepth + 1)), currentDepth);
  }

  return Math.max(
    ...Object.values(obj).map(value => getObjectDepth(value, currentDepth + 1)),
    currentDepth
  );
}

This sanitization function removes dangerous content. It limits size. It detects injection patterns. It normalizes whitespace. It validates structure.

Sample 5: A “verifier” that checks outputs against a JSON schema + required citations

import Ajv from 'ajv';

interface VerificationResult {
  valid: boolean;
  errors: string[];
  missing_citations: string[];
  schema_errors: string[];
}

interface VerifierConfig {
  schema?: JSONSchema;
  required_fields?: string[];
  require_citations?: boolean;
  max_length?: number;
  min_length?: number;
}

class OutputVerifier {
  private ajv: Ajv;

  constructor() {
    this.ajv = new Ajv({ allErrors: true, strict: false });
  }

  verify(
    output: unknown,
    config: VerifierConfig,
    citations?: Array<{ tool_name: string; data_excerpt: string }>
  ): VerificationResult {
    const errors: string[] = [];
    const missing_citations: string[] = [];
    const schema_errors: string[] = [];

    // Convert output to string for length checks
    const outputStr = typeof output === 'string' ? output : JSON.stringify(output);

    // Check length
    if (config.max_length && outputStr.length > config.max_length) {
      errors.push(`Output exceeds maximum length: ${outputStr.length} > ${config.max_length}`);
    }

    if (config.min_length && outputStr.length < config.min_length) {
      errors.push(`Output below minimum length: ${outputStr.length} < ${config.min_length}`);
    }

    // Check required fields (if output is an object)
    if (config.required_fields && typeof output === 'object' && output !== null) {
      const obj = output as Record<string, unknown>;
      for (const field of config.required_fields) {
        if (!(field in obj) || obj[field] === undefined || obj[field] === null) {
          errors.push(`Missing required field: ${field}`);
        }
      }
    }

    // Check schema
    if (config.schema) {
      const validate = this.ajv.compile(config.schema);
      const valid = validate(output);
      if (!valid) {
        schema_errors.push(...(validate.errors || []).map(err => 
          `${err.instancePath}: ${err.message}`
        ));
        errors.push('Schema validation failed');
      }
    }

    // Check citations
    if (config.require_citations) {
      if (!citations || citations.length === 0) {
        missing_citations.push('No citations provided');
        errors.push('Citations are required but none were provided');
      } else {
        // Check that citations are actually used in output
        const outputLower = outputStr.toLowerCase();
        for (const citation of citations) {
          // Simple check: see if citation data appears in output
          const excerptLower = citation.data_excerpt.toLowerCase().substring(0, 50);
          if (!outputLower.includes(excerptLower)) {
            missing_citations.push(`Citation from ${citation.tool_name} not found in output`);
          }
        }
      }
    }

    return {
      valid: errors.length === 0,
      errors,
      missing_citations,
      schema_errors,
    };
  }
}

// Example usage
const verifier = new OutputVerifier();

const outputSchema = {
  type: 'object',
  required: ['status', 'data', 'citations'],
  properties: {
    status: { type: 'string', enum: ['success', 'error'] },
    data: { type: 'object' },
    citations: {
      type: 'array',
      items: {
        type: 'object',
        required: ['tool_name', 'source'],
        properties: {
          tool_name: { type: 'string' },
          source: { type: 'string' },
        },
      },
    },
  },
};

const result = verifier.verify(
  {
    status: 'success',
    data: { result: 'example' },
    citations: [
      { tool_name: 'database', source: 'users table' },
    ],
  },
  {
    schema: outputSchema,
    required_fields: ['status', 'data'],
    require_citations: true,
  },
  [
    { tool_name: 'database', data_excerpt: 'users table' },
  ]
);

if (!result.valid) {
  console.error('Verification failed:', result.errors);
}

This verifier checks outputs against schemas. It validates required fields. It checks citations. It returns detailed error messages.

Close

Observability in multi-agent systems isn’t optional. It’s required. Without it, you’re debugging blind.

Start with tracing

Add OpenTelemetry spans. Emit structured events. Track correlation IDs. This gives you traceability. You can see what happened.

Add budgets early

Set token budgets. Set time budgets. Set tool call budgets. Enforce them. This gives you controllability. You can stop runaway costs.

Log everything

Log agent starts. Log tool calls. Log delegations. Log completions. Include metadata. This gives you auditability. You can answer “who said what.”

Build quality controls

Add verifiers. Add schema checks. Add citation requirements. Sanitize tool outputs. This prevents bad outputs from propagating.

Monitor proactively

Build dashboards. Track failure rates. Track costs. Track latency. This enables proactive monitoring. You can see problems before they become critical.

The goal isn’t perfection. It’s visibility. When things break, you need to know why. Traces, budgets, and audit logs get you there.

Multi-agent systems are complex. They will fail. Plan for it. Handle it. Make it observable.