Skip to content

Large Language Model Session Token Counter

Version badge

In Cognigy.AI, you can use the api.getLLMTokenUsageForSession() API call in a Code Node or an Extension to fetch the Large Language Model (LLM) total token usage in a conversation or chat session. This way, you can track your LLM providers costs.

The LLM Session Token Counter retrieves the token usage information via LLM provider-specific API calls to return the exact number of tokens the LLM uses. If the information isn't available by the provider, Cognigy.AI estimates the token usage.

Prerequisite

Limitations

  • The LLM Session Token Counter doesn't support embedding models from Aleph Alpha.

Fetching Session-Wide Token Usage

  1. Add a Code Node after the Node that uses an LLM, for example, after an AI Agent Node.

  2. Add the following code to the Code Node:

    const tokens = api.getLLMTokenUsageForSession();

This Code Node fetches the total token usage in the session and stores it in the token variable.

Returned JSON Object

The following JSON example shows an object returned by api.getLLMTokenUsageForSession(): 

{
    "1c945b38-5dbb-4fcf-9fdd-112bd8177f9c": {
        "IlmDisplayName": "openAI - text- embedding-3-large - 1735891122507",
        "providerType": "openAI",
        "modelType" : "text-embedding-3-large",
        "usage": {
            "inputTokens": 6,
            "outputTokens": 0
        }
    },
    "ce190e1f-eff7-4f6b-9074-9f705375a3d3": {
        "IlmDisplayName": "openAI - gpt-40 - 1735891179896",
        "providerType": "openAI",
        "modelType": "gpt-40",
        "usage": {
            "inputTokens": 441,
            "outputTokens": 106
        }
    }
}

Key Type Description Example
llmReferenceId String The LLM reference ID. ce190e1f-eff7-4f6b-9074-9f705375a3d3
llmDisplayName String The name of the LLM. openAI - gpt-40 - 1735891179896
providerType String The LLM provider. openAI
modelType String The LLM model. gpt-40
usage String The object containing information about the session total token usage. -
inputTokens Number The number of input tokens. 441
outputTokens Number The number output tokens. The output tokens count for embedding models is always 0 since embedding models output embedding vectors instead of tokens. 106

For testing in the Interaction Panel, you can output the JSON object by adding api.say() at the end of your Code Node.

More Information