Cost Tracking

The gateway records the cost of every LLM request in real time. Costs are accumulated in Redis for fast budget enforcement and persisted to MongoDB for audit and analytics. The dashboard provides a cost governance panel for monitoring spend across organizations, models, and time periods.

How cost recording works

After every proxy response completes — both streaming and non-streaming — the gateway runs the following sequence:

1. Calculate actual cost from the provider’s reported token counts (prompt tokens, completion tokens, reasoning tokens, cache tokens)
2. Increment daily cost in Redis (gateway:daily_cost:{org_id}:{date}) for budget enforcement
3. Increment monthly cost in Redis (gateway:monthly_cost:{org_id}:{YYYY-MM}) for monthly budget enforcement
4. Persist the full usage record to MongoDB (gateway_usage collection) for audit
5. Track usage against the API key for per-key cost attribution
6. Emit Prometheus metrics for system-level monitoring

Per-call logging

Every gateway request produces a usage record with the following fields:

Token counting

Pre-request estimation

Before a request is proxied, the gateway estimates input token count using tiktoken BPE encoding. The correct encoding is selected per model family:

The estimator handles both OpenAI and Anthropic message formats, including multi-part content blocks, system prompts, and tool definitions.

If tiktoken fails for any reason, the estimator falls back to a character heuristic (1 token per 4 characters).

Post-request actual counts

After the response completes, actual token counts are extracted from the provider’s response:

For streaming responses, usage data is extracted from the final SSE chunk. OpenAI requires stream_options: {"include_usage": true} to report usage in streaming mode — the gateway adds this automatically.

Per-org aggregation

Cost data is aggregated at the organization level using Redis counters:

Daily cost tracking

• Key: gateway:daily_cost:{org_id}:{YYYY-MM-DD}
• TTL: 48 hours
• Operation: INCRBYFLOAT for precise float accumulation
• Used by: Governance chain (Step 2) for daily budget enforcement

Monthly cost tracking

• Key: gateway:monthly_cost:{org_id}:{YYYY-MM}
• TTL: 35 days
• Operation: INCRBYFLOAT for precise float accumulation
• Used by: Governance chain (Step 2) for monthly budget enforcement

Daily request count

• Key: gateway:daily_requests:{org_id}:{YYYY-MM-DD}
• TTL: 48 hours
• Operation: INCR
• Used by: Analytics and dashboard request volume charts

Budget alerts and caps

The gateway enforces budget limits at three levels:

Per-request cap

Each request’s estimated cost is checked against max_cost_per_request. This prevents a single expensive request (e.g., a large context window with GPT-5) from consuming a disproportionate share of the budget.

Daily budget cap

The sum of daily_cost + estimated_cost is checked against daily_budget. When the cap is reached, all requests for the org are blocked until the next UTC day.

Monthly budget cap

The sum of monthly_cost + estimated_cost is checked against monthly_budget. When the cap is reached, all requests for the org are blocked until the next calendar month.

Custom limits can be configured per organization via the dashboard or the B2B API.

Cost comparison across providers

The gateway records the provider and model for every request, making it straightforward to compare costs across providers for equivalent tasks. The dashboard’s cost governance panel includes:

• Cost by provider: Aggregate spend broken down by OpenAI, Anthropic, Google, Groq, Mistral, xAI, and more
• Cost by model: Per-model spend ranking to identify the most expensive models in use
• Cost trends: Daily and weekly cost charts with trend indicators
• Per-key attribution: Cost breakdown by API key, useful for tracking spend per application or team

Runner session costs

For requests originating from managed runner sessions, the gateway additionally tracks costs at the session level via the CostSLOEngine. This enables:

• Per-session cost limits (separate from org-level budgets)
• Session cost accumulation in Redis for real-time enforcement
• Cost attribution to specific runner executions in the audit log

Dashboard integration

The cost data recorded by the gateway feeds directly into the dashboard’s cost governance panel:

• Real-time spend: Current daily and monthly totals pulled from Redis
• Historical data: Full usage history queried from the MongoDB gateway_usage collection
• Budget utilization: Visual indicators showing percentage of daily and monthly budgets consumed
• Cost alerts: Configurable notifications when spend approaches budget thresholds

Access the cost governance panel in the dashboard under Gateway > Cost Tracking.

Querying cost data

Via the CLI

# Daily cost snapshot
./scripts/analytics costs today
 
# Cost history for the last 7 days
./scripts/analytics costs history 7
 
# Full analytics snapshot
./scripts/analytics snapshot today

Via the B2B API

# Get cost summary for the current org
curl https://api.curate-me.ai/api/v1/admin/gateway/costs/summary \
  -H "Authorization: Bearer $TOKEN" \
  -H "X-Org-ID: $ORG_ID"
 
# Get usage records with filters
curl "https://api.curate-me.ai/api/v1/admin/gateway/usage?provider=openai&limit=50" \
  -H "Authorization: Bearer $TOKEN" \
  -H "X-Org-ID: $ORG_ID"

Backend implementation

Key source files: