api.getLLMTokenUsageForSession() API call in a Code node or an Extension node to fetch the large language model (LLM) total token usage in a conversation or chat session. This way, you can track your LLM provider costs.
The LLM Session Token Counter retrieves the token usage information via LLM provider-specific API calls to return the exact number of tokens the LLM uses. If the information isn’t available by the provider, Cognigy estimates the token usage.
Prerequisite
- You’ve added a node that uses an LLM to your flow, for example, an AI Agent node, LLM Prompt node, Search Extract Output node, or Agent Copilot nodes.
Limitations
- The LLM Session Token Counter doesn’t support embedding models from Aleph Alpha.
Fetching session-wide token usage
- Add a Code node after the node that uses an LLM, for example, after an AI Agent node.
-
Add the following code to the Code node:
tokens variable.
Returned JSON object
Returned JSON object
The following JSON example shows an object returned by
api.getLLMTokenUsageForSession():For testing in the Interaction Panel, you can output the JSON object by adding
api.say() at the end of your Code node.