Time-Travel Debugging

Time-travel debugging lets you replay any agent pipeline execution step-by-step, inspecting the exact state at each stage. By capturing checkpoints at every agent boundary, the system enables precise root cause analysis without needing to reproduce the original conditions.

How It Works

Every pipeline execution automatically creates checkpoints at each stage transition. A checkpoint captures:

• Input data passed to the agent
• Output data returned by the agent
• Agent configuration at the time of execution (model, prompt version, settings)
• Timing information (start time, duration, queue wait time)
• Token usage and cost for the step
• Error details if the step failed

These checkpoints are persisted in the checkpoint store and linked to the pipeline’s run_id, creating a complete audit trail of every execution.

Replay Interface

The replay interface presents a timeline of the pipeline execution with each agent represented as a step. You can:

Step Through Execution

Navigate forward and backward through the pipeline stages using the step controls. At each step, the interface displays:

• The agent that executed at this stage
• The complete input payload the agent received
• The complete output payload the agent produced
• Duration and cost of the step
• Any warnings or errors encountered

Inspect Input and Output

Each checkpoint’s input and output data is displayed in a structured JSON viewer with syntax highlighting, collapsible sections, and search. For large payloads, the viewer supports:

• Diff mode — highlight what changed between input and output
• Path copy — click any field to copy its JSON path
• Raw mode — view the unformatted JSON for copy/paste

Compare Runs Side-by-Side

Select two pipeline runs to view them side-by-side in a comparison layout. This is useful for:

• Understanding why two runs produced different results
• Comparing performance before and after a configuration change
• Identifying which agent in the pipeline introduced a regression
• Validating that a fix resolved the issue without side effects

The comparison view highlights differences in inputs, outputs, timing, and cost at each corresponding step.

Checkpoint Store

Checkpoints are stored using the CheckpointStore abstraction, which supports tenant-scoped access to ensure organizations can only view their own execution data.

from src.orchestrator.checkpoint_store import CheckpointStore
 
# Retrieve all checkpoints for a pipeline run
checkpoints = await checkpoint_store.get_checkpoints(run_id="run_abc123")
 
# Each checkpoint contains the full agent state
for cp in checkpoints:
    print(f"Agent: {cp.agent_id}")
    print(f"Input: {cp.input_data}")
    print(f"Output: {cp.output_data}")
    print(f"Duration: {cp.duration_ms}ms")
    print(f"Cost: ${cp.cost:.4f}")

Checkpoints are retained according to your organization’s retention policy. The default retention period is 30 days, configurable per organization.

Use Cases

API Endpoints

List Checkpoints

Retrieve all checkpoints for a given pipeline run.

GET /api/v1/checkpoints?run_id=run_abc123

{
  "run_id": "run_abc123",
  "pipeline": "data-analysis-v3",
  "status": "completed",
  "checkpoints": [
    {
      "step": 0,
      "agent_id": "vision_analysis_v3",
      "input_data": { "image_url": "https://..." },
      "output_data": { "entities": [...], "confidence": 0.94 },
      "duration_ms": 2340,
      "cost": 0.0082,
      "timestamp": "2026-02-08T14:23:11.204Z"
    },
    {
      "step": 1,
      "agent_id": "style_analysis_v3",
      "input_data": { "entities": [...] },
      "output_data": { "analysis_profile": {...}, "results": [...] },
      "duration_ms": 1870,
      "cost": 0.0124,
      "timestamp": "2026-02-08T14:23:13.544Z"
    }
  ]
}

This endpoint requires a valid JWT and X-Org-ID header for tenant-scoped results.