How to manage Zeus AI QA Evaluations
Last updated September 17, 2026
Overview
Evals allow you to create and manage evaluation datasets to test and measure the accuracy, response quality, and system behaviors of your AI assistant. By defining real or synthetic customer questions, expected reference outputs, and tool expectations, you can systematically benchmark your AI’s performance across workflows before deploying changes live.
Key features
- Custom Evaluation Datasets: Build evaluation examples comprising customer messages, attachments, ideal reference outputs, and expected tool executions.
- JSON & Multi-Turn Support: Toggle JSON mode and use the + Add Message Type option to construct multi-turn conversational threads with specific role types.
- Granular Example Settings: Associate evaluations with specific customer records via Add customer metadata, select target agent profiles, and assign up to 10 tags for structured test organization.
- Bulk Testing Runs: Execute evaluations in bulk across specific integrations or filtered tag groups to review overall AI assistant performance.
Important: The Bulk testing run features are internal-only.
How to create an evaluation example
- Go to " Zeus AI > Agent > Evals " and click "+Add Example".
- Enter the "Customer Message", or enable the "Use JSON mode (To add follow-up question)" toggle to construct a multi-turn thread. When using JSON mode, click "+ Add Message Type" to append "AI" or "Human" message turns.
- Optionally, click "+ Add Attachment" to upload files.
- Under "Ideal Response", enter the expected output in the "Ideal Response" field.
- Click the "Tools to Call" dropdown to select expected system actions.
- get_relevant_documents: Searches your internal knowledge base and protocols to retrieve factual context for answering customer questions.
- end_conversation: Closes the active chat session once a customer's inquiry is fully resolved.
- escalate_to_human: Transfers the conversation to a support agent when handling complex issues, disputes, or explicit handoff requests.
- get_past_conversations: Fetches the customer's historical chat and ticket logs to leverage context from prior interactions.
- get_session_recording_analysis: Retrieves diagnostic insights and summaries from technical user session logs or screen recordings.
- copilot_draft_message: Generates a suggested response draft for a human agent to review and send during agent-assist workflows.
- Published tools: Users can select their own published tools.
- In the "Example settings" section, configure the metadata:
- Click "Select customer" to choose a specific customer and test how Zeus AI behaves using that user's historical account metadata or traits.
- Click "Select profile" to choose which agent behavior profile applies to the test run:
- A specific profile: Tests the response against persona-specific rules configured in " Zeus AI > Configurations > Profiles " (see Profiles for Zeus AI).
- "Always escalate": Tests or simulates scenarios where AI processing is bypassed entirely to route the conversation directly to a live human agent or support team.
- Click "Select or create tags" to search existing tags or type a name and select "+ Create" (up to 10 tags; lowercase letters, numbers, hyphens, and underscores only).
- Click "Create" to save the example to your evaluation dataset, or click "Run" to execute an immediate test.
How to edit an evaluation example
- Go to " Zeus AI > Agent > Evals " and locate the example on the Examples list.
- Click the ellipsis icon next to the target evaluation entry and select "Edit".
- Apply the desired updates to the customer message, ideal response, tool selections, or example settings.
- Save your changes to update the evaluation dataset.
How to run bulk evaluations
Important: The "Bulk Run" tab is internal-only.
- Go to " Zeus AI > Agent > Evals " and switch to the "Bulk Run" tab.
- (Optional) Select an integration from the "Select integration" dropdown.
- (Optional) Filter test cases using the "Filter by tags" dropdown.
- Click "Run all" to execute batch testing across all selected dataset examples.
How to delete an evaluation example
- Go to " Zeus AI > Agent > Evals " and locate the evaluation example in the list.
- Click the ellipsis icon next to the row and select "Delete".
- Confirm the prompt to remove the example permanently.
Evaluation setup example
The following example tests whether the AI assistant correctly handles a high-tier subscription cancellation query according to refund protocols before escalating to a human agent.
Configuration settings
| Setting | Value |
| Question | "I need to cancel my subscription right now. You guys charged me twice this month." |
| Use JSON Mode | Disabled |
| Ideal Response | "I apologize for the double charge. As a VIP member, I can instantly issue a full refund for the duplicate payment and apply a 20% discount to your next billing cycle. Would you like me to process that refund for you?" |
| Tools to Call | get_relevant_documents, escalate_to_human |
| Customer | Enterprise_VIP_User_401 |
| Profile | US customers |
| Tags | cancellations, billing_dispute |
Execution Outcome
When this evaluation example runs:
- The system compares the AI's generated reply against the Ideal Response to check for empathy, refund messaging, and retention discount offer accuracy.
- It verifies whether get_relevant_documents or escalate_to_human was called based on the customer message context.
- If the generated output matches the expected behavior, the test case passes; otherwise, it flags a regression for review.
FAQs and Troubleshooting
What is the purpose of running evals?
Evals allow you to systematically measure whether updates to your protocols, instructions, or tools improve or degrade your Zeus AI assistant's accuracy. Running evaluation datasets helps prevent regressions before deploying changes to live channels.
When should I use JSON mode?
Enable JSON mode when building test cases that involve multi-turn conversations or follow-up interactions. Standard input accepts a single customer message, whereas JSON mode allows you to structure sequence-based inquiries using "+ Add Message Type" to switch between human and AI responses.
Can I run tests on specific subsets of my dataset?
Yes. You can apply tags to your evaluation examples and use the "Filter by tags" dropdown under the "Bulk Run" tab to execute targeted test runs.
Important: The "Bulk Run" tab is internal-only.
Best practices for evals design
- Cover core and edge cases: Create evaluation examples for your most frequent customer inquiries, as well as complex edge cases requiring tools or escalation.
- Keep ideal responses precise: Define clear reference outputs so system benchmarks can accurately measure semantic similarity and response adherence.
- Use tags strategically: Organize your test dataset by feature area or ticket type using tags to simplify bulk runs after updating specific protocols.
- Re-run evals after protocol edits: Always execute a bulk run across relevant tags whenever you modify agent profiles or protocols to verify output quality.