Skip to content

Large Language Models (LLMs)

Version badge

Large Language Models (LLMs) are specific types of AI models that are designed for generating humanlike text based on the input and context provided. These models are trained on vast amounts of text data, allowing them to learn patterns, syntax, and semantic relationships between words and phrases.

With LLMs, Cognigy AI Agents can understand and respond to user input in a natural way. These models make conversations more engaging by generating relevant and contextually appropriate responses. LLMs also assist in managing dialogues and providing multilingual support, enhancing the overall conversational experience for users.

Supported Models

The table below lists LLMs supported by Cognigy.AI. You can also specify another model that is not presented in the table. To do so, select the Custom Model parameter when creating a model and specify both the model type and name. For more information about adding a custom model for a selected LLM provider, refer to the Add a Model section.

Models/
Cognigy Features		 Intent Sentence Generation AI Enhanced Outputs Lexicon Generation Flow Generation GPT Conversation Node LLM Prompt Node & LLM-powered Answer Extraction Generate Node Output Knowledge Search Sentiment Analysis NLU Embedding Model
Microsoft Azure OpenAI
gpt-3.5-turbo (ChatGPT) + + + + + + + - + -
gpt-3.5-turbo-instruct - - - - - + - - - -
gpt-4 - - - - - + - - - -
text-davinci-003 (Deprecated) + + + + + + + - - -
text-embedding-3-large - - - - - - - - - +
text-embedding-ada-002 - - - - - - - + - -
OpenAI
gpt-3.5-turbo (ChatGPT) + + + + + + + - + -
gpt-3.5-turbo-instruct - - - - - + - - - -
gpt-4 - - - - - + - - - -
text-embedding-3-large - - - - - - - - - +
text-embedding-ada-002 - - - - - - - + - -
Anthropic
claude-3-opus - - - - - + - - - -
claude-3-haiku - - - - - + - - - -
claude-3-sonnet - - - - - + - - - -
claude-v1-100k (Deprecated) - - - - - + - - - -
claude-instant-v1 (Deprecated) - - - - - + - - - -
Google Vertex AI
text-bison-001 (Bard) - - - - - + - - - -
Google Gemini
gemini-1.0-pro - - - - - + - - - -
Aleph Alpha
luminous-extended-control - - - - - + - - - -
luminous-embedding-1281 - - - - - - - + - -

Add a Model

Resolving Connection Errors in Microsoft Azure OpenAI LLMs

In recent releases, we have updated the connection settings to Azure OpenAI LLMs and added new parameters, such as the Resource Name. If you have an older connection (for example, created in the 4.53 release) to Azure OpenAI LLMs, especially Azure OpenAI text-embedding-ada-002 for knowledge search features, you might encounter the following error when an LLM is triggered: Error while performing knowledge search. Remote returned error: Search failed: Could not fetch embeddings due to missing API resource name for Azure OpenAI. To resolve this issue, recreate the LLM and the connection so that both have the latest format.

To add a model to Cognigy.AI, follow these steps:

  1. Open the Cognigy.AI interface.
  2. Go to Build > LLM.
  3. Click +New LLM.
  4. In the New LLM window, select a model from the Model Type list.
  5. Add a unique name and description for your model.

  6. From the Provider list, select an LLM's provider:

    6.1 From the Model list, select a model presented in the list or add a custom model that is not listed. If you select Custom Model, configure the following fields:
    - Model Type — select Chat for models that support the Chat Completions API, Completion for the Completions API, or Embedding for the Embedding API. For mor information, refer to the Azure OpenAI documentation.
    - Model Name — specify an ID of the model that you want to use as a custom. To find model IDs, refer to the Azure OpenAI documentation.
    6.2 Click Save.
    6.3 On the LLM Editor page, go to the Generative AI Connection field.
    6.4 On the right side of the field, click +.
    6.5 In the Connection name, enter a unique name for your connection.
    6.6 From the Connection Type list, select one of the following authorization methods:
    - API Key — add an Azure API Key. This value can be found in the Keys & Endpoint section when examining your resource from the Azure portal. You can use either KEY1 or KEY2.
    - OAuth2 — add credentials for the OAuth 2.0 authorization code flow. OAuth 2.0 offers more control and security than API keys by allowing specific permissions, expiring tokens, and reducing exposure through short-lived tokens instead of constant client secret use. To use this type of connection, fill in the following fields:
        - Client ID — add the Application (client) ID assigned to your app, can be found in the in Azure AI app registration overview.
        - Client Secret — add the application secret created in the Certificates & secrets section of the Azure AI app registration portal.
        - OAuth URL — add the URL to retrieve the access token. The URL should be in the https://<your-domain>.com/as/token.oauth2 format.
        - Scope — add a list of scopes for user permissions, for example, urn:grp:chatgpt.
    6.7 Click Create.
    6.8 Fill in the remaining fields:
    - Resource Name — add a resource name. To find this value, go to the Microsoft Azure home page. Under Azure services, click Azure OpenAI. In the left-side menu, under the Azure AI Services section, select Azure Open AI. Copy the desired resource name from the Name column.
    - Deployment Name — add a deployment name. To find this value, go to the Microsoft Azure home page. Under Azure services, click Azure OpenAI. In the left-side menu, under the Azure AI Services section, select Azure Open AI. Select a resource from the Name column. On the resource page, go to Resource Management > Model deployments. On the Model deployments page, click Manage Deployments. On the Deployments page, copy the desired deployment name from the Deployment name column.
    - Api Version — add an API version. The API version to use for this operation in the YYYY-MM-DD format. Note that the version may have an extended format, for example, YYYY-MM-DD-preview.
    - Custom URL — this parameter is optional. To control the connection between your clusters and the Azure OpenAI provider, you can route connections through dedicated proxy servers, creating an additional layer of security. To do this, specify the URL in the following pattern:https://<resource-name>.openai.azure.com/openai/deployments/<deployment-name>/<model-type>?api-version=<api-verson>. When a Custom URL is added, the Resource Name, Deployment Name, and API Version fields will be ignored.

    6.1 From the Model list, select a model presented in the list or add a custom model that is not listed. If you select Custom Model, configure the following fields:
    - Model Type — select Chat for the https://api.openai.com/v1/chat/completions API, Completion for the https://api.openai.com//v1/completions API, and Embedding for the https://api.openai.com//v1/embeddings API. For more information, refer to the OpenAI Text Generation Models documentation.
    - Model Name — specify a name of the model that you want to use as a custom. To find model names, refer to the Azure OpenAI Service models documentation.
    6.2 Click Save.
    6.3 On the LLM Editor page, go to the Generative AI Connection field.
    6.4 On the right side of the field, click +.
    6.5 Fill in the following fields:
    - Connection name — create a unique name for your connection.
    - apiKey — add an API Key from your OpenAI account. You can find this key in the User settings of your OpenAI account.
    6.6 Click Create.

    6.1 From the Model list, select a model presented in the list or add a custom model that is not listed. If you select Custom Model, configure the following fields:
    - Model Type — select Chat for models that support the Messages API, Completion for the Completions API, or Embedding for the Embeddings API. For mor information, refer to the Anthropic Model Comparison (API format) documentation.
    - Model Name — specify a name of the model that you want to use as a custom. To find model names, refer to the Anthropic documentation.
    6.2 Click Save.
    6.3 Fill in the following fields:
    - Connection name — create a unique name for your connection.
    - apiKey — add an API Key that you generated via Account Settings in Anthropic.
    6.4 Click Create.

    6.1 From the Model list, select a model presented in the list or add a custom model that is not listed. If you select Custom Model, configure the following fields:
    - Model Type — select the Completion type.
    - Model Name — specify a name of the model that you want to use as a custom. To find model names, refer to the Google Vertex AI documentation. Note that, within this connection, Cognigy supports only the text-bison models.
    6.2 Click Save.
    6.3 Fill in the Connection name field by specifying a unique name for your connection.
    6.4 To upload the JSON file with a key for your model, you need to obtain this key. Go to the Google Cloud console and find Vertex AI via the search bar.
    6.5 On the Vertex AI page, click the Enable All Recommended APIs button to activate an API connection, if it is not activated. Ensure that the Vertex AI API is enabled.
    6.6 In the left-side menu, go to the IAM & Admin > Service Accounts.
    6.7 Select Actions and click Manage Keys.
    6.8 On the Keys page, select Add Key and click Create new Key.
    6.9 In the appeared window, select the JSON key type and click Create. The file will be downloaded.
    6.10 In Cognigy, in the New Connection window, click Upload JSON file and upload the file.
    6.11 Click Create.
    6.12 Fill in the remaining fields:
    - Location — add a region for the model. For example, us-central1.
    - API Endpoint — add a service endpoint for the model. For example, us-central1-aiplatform.googleapis.com. Note that the endpoint should be specified without https:// or http://.
    - Publisher — add an owner's name of the model. If not specified, Google will be used by default. This parameter is optional.

    6.1 From the Model list, select a model presented in the list or add a custom model that is not listed. If you select Custom Model, configure the following fields:
    - Model Type — select the Chat type.
    - Model Name — specify a name of the model that you want to use as a custom. To find model names, refer to the Google Gemini documentation. Note that, within this connection, Cognigy supports only the gemini models.
    6.2 Click Save.
    6.3 Fill in the Connection name field by specifying a unique name for your connection.
    6.4 To upload the JSON file with a key for your model, you need to obtain this key. If you have previously used a key for the Google Vertex AI connection, you can also use this key for Google Gemini; proceed to step 6.10 to add the key. If you're setting up the connection for the first time, go to the Google Cloud console and find Vertex AI via the search bar.
    6.5 On the Vertex AI page, click the Enable All Recommended APIs button to activate an API connection, if it is not activated. Ensure that the Vertex AI API is enabled.
    6.6 In the left-side menu, go to the IAM & Admin > Service Accounts.
    6.7 Select Actions and click Manage Keys.
    6.8 On the Keys page, select Add Key and click Create new Key.
    6.9 In the appeared window, select the JSON key type and click Create. The file will be downloaded.
    6.10 In Cognigy, in the New Connection window, click Upload JSON file and upload the file.
    6.11 Click Create.
    6.12 In the Location field, add a region for the model. For example, us-central1.

    6.1 From the Model list, select a model presented in the list or add a custom model that is not listed. If you select Custom Model, configure the following fields:
    - Model Type — select the Completion type.
    - Model Name — specify a name of the model that you want to use as a custom. To find model names, refer to the Aleph Alpha documentation.
    6.2 Click Save.
    6.3 Fill in the following fields:
    - Connection name — create a unique name for your connection.
    - Token — specify a key that you created in your Aleph Alpha account.
    6.4 Click Create.
    6.5 Fill in the remaining field:
    - Custom URL — this parameter is optional. To control the connection between your clusters and the Aleph Alpha provider, you can route connections through dedicated proxy servers, creating an additional layer of security. To do this, specify the base URL. For example, https://api.aleph-alpha.com.

  7. To apply changes, click Save.

  8. To check if the connection was set up, click Test.

When the model is added, you will see it in the list of models.

To apply this model for Cognigy features, go to the settings by clicking Manage LLM Features.

Apply a Model

To apply a model, follow these steps:

  1. Open the Cognigy.AI interface.
  2. In the left-side menu, click Manage > Settings.
  3. In the Generative AI Settings section, activate Enable Generative AI Features. This setting is toggled on by default if you have previously set up the Generative AI credentials.
  4. Navigate to the desired feature and select a model from the list. If there are no models available for the selected feature, the system will automatically select None.
  5. Click Save.

Clone a Model

To create a copy of the existing model, follow these steps:

  1. Go to Build > LLM.
  2. Hover over the existing model and click vertical-ellipsis.
  3. Select Clone from the list.

The model will contain the same settings as the initial one.

Set a Model as Default

Setting a default model ensures a smooth transition when a specific model is removed. It guarantees that there is always a model available to handle compatible use cases, even if the assigned model is removed.

To set a model as the default, follow these steps:

  1. Go to Build > LLM.
  2. Hover over the existing model and click vertical-ellipsis.
  3. Select Make Default from the list.

The setting will be applied for the selected model.

Export a Model as a Package

To reuse a model in other agents, you can package the model.

To package a model, follow these steps:

  1. Go to Build > LLM.
  2. Hover over the existing model and click vertical-ellipsis.
  3. Select Create a package.
  4. Once the package has created, a new task, titled Create package, will be run. To view the task, click task-menu in the upper-right corner.

When the task is completed, the package will be downloaded.

Delete a Model

Note that a default model cannot be deleted. Before deletion, you need to remove the default tag.

To delete a model, follow these steps:

  1. Go to Build > LLM.
  2. Hover over the existing model and click vertical-ellipsis.
  3. Select Delete.
  4. Confirm the deletion. Features relying on this model will stop working if no default model is configured to support those features.

The model will be deleted from the list.

Retry Mechanism

If Cognigy encounters issues while trying to connect with LLM providers that don't return a 200 OK response code, Cognigy will automatically attempt to reconnect up to three times using a retry mechanism. This retry mechanism is designed to improve the reliability and stability of the system when using certain Cognigy Nodes and features. By attempting to reconnect multiple times, it increases the chances of successful communication, reduces disruptions and ensures smoother operations within the system.

The following Nodes and features are affected by the retry mechanism:

The retry mechanism can be customized through the environment variables for the on-premises installations. By adjusting these environment variables, system administrators can fine-tune the retry behavior to suit specific performance requirements and network conditions.

Environment Variable Description Default Value
GENERATIVE_AI_RETRY_OPTIONS_NUMBER_OF_RETRIES Determines the number of retries after the first attempt before failing. 2
GENERATIVE_AI_RETRY_OPTIONS_MIN_TIMEOUT Determines the timeout between retries to avoid rate limiting. 50ms
GENERATIVE_AI_RETRY_OPTIONS_MAX_TIMEOUT Determines the maximum timeout between retries. 50ms
GENERATIVE_AI_RETRY_OPTIONS_FACTOR Determines how long a retry timeout will last. The timeout duration will increase with each subsequent retry attempt. If the factor is greater than 1, the timeout duration will increase exponentially. This means that with each subsequent attempt, the waiting time will not just increase by a constant amount, but rather by multiples. For example, if the initial timeout is 1 second and the factor is set to 2, then the second timeout will be 2 seconds, the third will be 4 seconds, the fourth will be 8 seconds, and so on. 1

More Information

  1. This feature is currently in Beta and may contain issues. Only one type of embedding LLM should be used per project. If you choose to use luminous-embedding-128, you must create a new project. Once you have chosen an embedding model for a project, you cannot switch to a different embedding model; you must use a different project. Failing to do so will result in errors while this feature is in Beta. ↩