Token Logger7 min read
Token usage logging without storing secrets
Build interpretable LLM usage records with clear totals, missing-value handling, deduplication, and transparent cost estimates.
Read Token usage logging without storing secretsToken Logger / Field guide
Usage you can explain, request by request.
Token logging here means recording LLM usage metadata. Keep input, output, and any supported detail categories understandable so an application’s usage history can be reconciled without preserving credentials or conversation content.
Start with the purpose
Keep an accounting record with clear boundaries.
A token usage record connects quantities to an operation, an attempt, a model reference, and a reporting source. Decide which boundary the record represents before summing it. A retry is additional work, while a repeated delivery of the same usage record may be a duplicate that should not increase an operation total.
Keep input and output quantities separate and preserve the source’s definitions. Additional categories can overlap with those totals. A cache-related quantity or another detailed breakdown should not be added as independent usage unless its relationship to the total is understood. Write that relationship down alongside the integration.
Missing usage is different from measured zero usage. A request may end without a complete usage report, or an integration may not expose a detail category. Preserve the distinction in an application accounting record and define how incomplete records are reconciled. That makes uncertainty visible instead of hiding it inside a plausible-looking total.
Cost estimates require their own inputs. Keep the pricing assumption, its effective date, applicable model, unit, and currency separate from measured usage. Explain which cost components the estimate includes. An application estimate can support planning and anomaly review; it should not be presented as a guaranteed provider invoice.
Identifiers can join usage to relevant operational events without joining it to the full prompt or response. Review those joins and access rules together. In this context, a token logger is a usage accounting concept: passwords, API keys, access tokens, and session credentials are not token usage data and do not belong in its records.
Token usage logging describes model consumption. It does not mean collecting authentication tokens, extracting credentials, or maintaining a cryptocurrency token inventory.
Four decisions to make
Put the idea into practice.
Which request boundary?
Separate the user-visible operation from each attempted model call, then state what an aggregate sums.
Which quantities overlap?
Keep reported totals and detailed subsets distinct so a report does not accidentally count the same usage twice.
What is missing?
Represent unavailable usage explicitly and give incomplete records a reconciliation status.
Which price assumption?
Version any estimate inputs separately from observed usage, including unit, effective date, and currency.
A starting checklist
- Record the reporting source and operation boundary.
- Keep input and output quantities separate.
- Document whether detail categories are subsets of totals.
- Distinguish missing values, estimates, and reported zero values.
- Exclude credentials and preserve the assumptions behind estimates.
Continue in the lab.
AI Logging7 min read
AI logging: what to capture around a model request
Trace a model request through timings, retries, tool calls, configuration changes, and outcomes without collecting every message.
Read AI logging: what to capture around a model requestMake your next log a useful one.
Start with the fundamentals, then follow the signal that matters to your work.