- Scenarios — define personas, missions, and success criteria for testing agents.
- Simulation runs — simulate scenarios to evaluate the performance of agents.
- Simulations — the compilation of multiple simulation runs displaying different agent behaviors in the same scenario. Each simulation run provides unique test outcomes for analysis. Additionally, you can automate testing by scheduling periodic simulations, and ensure continuous validation of your agent’s performance and reliability.
Key benefits
- Realistic end-to-end testing. Simulate full conversations, from greetings to resolutions. Use multiple personas to test how your agent handles different user personas and Scenarios.
- Simulations across multiple interfaces. Run simulations in the Simulator or the Interaction Panel. The Interaction Panel shows the results of the simulations that you start there. To review simulated conversations, use the Conversations Explorer.
- Early issue detection and optimization. Detect gaps, edge cases, and quality issues before deployment to improve reliability and deliver a better user experience.
- Faster, automated QA. Automatically test agents across environments, ensuring high coverage and consistent quality.
- Scheduled simulations. Schedule simulations to automate periodic tests at a specified frequency. Regular testing helps ensure that new deployments work as expected.
Prerequisites
- To start a simulation, you need an LLM that supports the Simulator.
- To generate scenarios automatically, you need to:
- Configure an LLM for design-time features. Select the platform-provided LLM or another model that supports design-time features.
- Make sure that your Project includes an agent or a flow with a Flow Agent Node.
Restrictions
- The following features aren’t supported:
- Voice interactions. The Simulator only supports text-based conversations.
- Multivariate simulations in the Interaction Panel. To run a multivariate simulation, use the Simulator.
- Handovers to human agents. The Interaction Panel will show the message
Agent Handover is not supported in the Interaction Panel, please use the Webchat or another channel.
- Simulations support only handoffs from an agent to a flow that the agent triggers with the Handoff to Flow system tool.
How the simulator works
- Create a scenario. Set the persona, mission, and success criteria. See Scenarios.
- Simulate. Start the simulation and let the agent interact with the persona that you created. Run the simulation multiple times to gather more accurate insights. See Simulations.
- Evaluate. View key metrics on the Simulator dashboard, such as complete success rate, average sentiment, and number of turns. You can then adjust the flow logic, training data, or system prompts based on the results. Rerun simulations to verify improvements.