- Improved user experience. Automatically switching to a fallback model ensures uninterrupted service, minimizing disruptions when the main model encounters issues.
- Quick recovery. The fallback mechanism allows the system to recover quickly during failures, such as model outages or updates. This approach prevents extended delays and ensures operations resume smoothly without significant downtime.
- Greater flexibility. Fallback models can be deployed in different regions or configured to meet varying resource requirements, providing adaptability. For example, they can serve as regional backups or handle workloads during high-demand periods.
Restrictions
- Creating fallbacks for embedding models isn’t supported.
- You can add only one fallback to a main model.
Create fallback models
Creating a fallback model works the same way as creating a main model. If your project has only one model, you can add another by choosing one from the same LLM provider or another provider with similar capabilities. Some providers, such as Azure OpenAI and Amazon Bedrock, allow you to use the same model across different regions.For example, if you’re using Azure OpenAI
gpt-4o in the West US region as your main model, you can also use gpt-4o as a fallback model in the Sweden Central region.
Add fallback models
To add a fallback model, follow these steps:- In the navigation menu of the project, go to Manage > AI & Voice Providers > Language Models.
- In the Models view, click
in the row of the model to which you want to add a fallback, and select Edit. - Activate the Fallback model toggle.
- Select a model from the list. If no model is available, create a new one.
- Configure the following fields:
- Number of failed requests. Specify the number of failed requests to the main model before the fallback model is triggered. You can specify up to five attempts. The default value is
1. - Duration in minutes of active fallback. Define how long the system should use the fallback model instead of the main model. After the specified time, the fallback will deactivate, and the main model will resume. The duration can be up to 30 minutes. The default value is
30. - Email notifications. Add up to five email addresses to receive notifications when the fallback model is triggered. Enter each email in a separate input field using the format
example@domain.com.
- Number of failed requests. Specify the number of failed requests to the main model before the fallback model is triggered. You can specify up to five attempts. The default value is
- Click Save.