- Steps: simulate a user input and can include an input text and input data.
- Assertions: check the agent’s responses and different aspects of the flow for each step against a defined value. You can also use the Check for absence option. The assertion then checks that the defined value is missing.
Assertion types
Assertion types
Limitations
- You can run up to 10 playbooks simultaneously. You can change this limit for on-premises installations by specifying the
MAX_CONCURRENT_PLAYBOOK_EXECUTIONSvariable in thevalues.yamlfile. - A playbook can contain up to 50 steps.
Restrictions
-
Playbooks don’t allow dots
(.)in JSON keys. Referencing subkeys likeuser.namecan cause runs to be hidden or snapshots to fail. Instead, validate the full JSON object:
Working with playbooks
- GUI
- CLI
- API
In Advanced > Playbooks, you can create, rename, run, and delete playbooks. You can also export them as a package:
. Then select and run a playbook. A playbook can run only against a flow. You can hover over the assertion checks for more details.To create playbooks from a conversation, follow these steps:
- To create a playbook, click Create playbook.
- To open the playbook editor, select a playbook from the list.
- For other actions, use the row menu: Rename, Export as package, Show runs, Run playbook, and Delete.
- To find a playbook, use Search playbooks.
For each step, enter Input text and Input data to send to the agent. Input data must be a JSON object. To check the agent’s response, click Add assertion. To add a step, click Create step. To reorder steps, drag them by the handle.Run a PlaybookTo run a playbook, select Run playbook from the row menu or from the playbook editor. In the Run playbook dialog, set the following options:
- Snapshot: Select No Snapshot or one of the project’s snapshots.
- Flow: Select the flow to test.
- Locale: Select the locale to test.
- Creator: Me or All.
- Status: All, Passed, or Failed.
- Chat with the agent on the
tab. - Click
in the interaction panel header and select Create playbook or Create playbook with assertions.