api.getLLMTokenUsageForSession() API call in a Code Node or an Extension Node to fetch the Large Language Model (LLM) total token usage in a conversation or chat session. This way, you can track your LLM providers costs.
The LLM Session Token Counter retrieves the token usage information via LLM provider-specific API calls to return the exact number of tokens the LLM uses. If the information isn’t available by the provider, Cognigy.AI estimates the token usage.
Prerequisite
- You have added a Node that uses an LLM to your Flow, for example, an AI Agent Node, LLM Prompt Node, Search Extract Output Node, or Agent Copilot Nodes.
Limitations
- The LLM Session Token Counter doesn’t support embedding models from Aleph Alpha.
Fetching Session-Wide Token Usage
- Add a Code Node after the Node that uses an LLM, for example, after an AI Agent Node.
-
Add the following code to the Code Node:
token variable.
Returned JSON Object
Returned JSON Object
The following JSON example shows an object returned by
api.getLLMTokenUsageForSession():For testing in the Interaction Panel, you can output the JSON object by adding
api.say() at the end of your Code Node.