The Test mode in Zeus AI is an interactive tool within the QA section that allows administrators to simulate real-time interactions with the Zeus AI agent before deploying it to end users. It enables you to validate the agent's behavior using knowledge, tools, protocols, guardrails, tone, automations, customer context, model configuration, and other test-specific settings.

Key features

Real-time simulation: Send messages and receive immediate responses from the Zeus AI agent to evaluate its behavior and answer accuracy.
Customer selection: Select a customer to test responses using customer-specific context.
File attachments: Upload files to test how Zeus AI processes documents or images.
Test mode settings: Customize the prompt, context, profile, guardrails, integration, models, and retrieval settings for the test.
Prompt overrides: Test a custom system prompt or use a templated prompt without changing the production configuration.
Model configuration: Select the vendor, model, and temperature for the agent and guardrails models.
Retrieval settings: Configure the similarity threshold and maximum number of documents retrieved per query.
Rejected messages: Display messages rejected during testing to help troubleshoot guardrail behavior.
Save test prompts: Save a test configuration for future testing.

How to use the test mode

Navigate to " Zeus AI > QA > Test mode ".
Enter your query in the "Enter message..." field (For example, "How can I change my password").
Select a customer to test the response using customer context, or click the attachment icon to add a file.
Click the "Send" button or press "Enter".
Review the response generated by the Zeus AI agent.

💡

Important: To modify the test configuration, click the "Settings" icon and update the relevant settings.If a response is inaccurate or does not meet the desired output, you can add it as an evaluation. See How to add a response to an evaluation.

How to configure the test mode settings

The Test mode settings panel contains all settings available for the current test. The panel is scrollable, with additional sections available further down.

The settings are grouped into the following sections:

Agent Model: Select the model used to generate the agent's response.
Guardrails Model: Select the model used to evaluate guardrails.
Retrieval Settings: Control how documents are retrieved during the test.

To configure the test mode settings, follow these steps:

Navigate to " Zeus AI > QA > Test mode " and click the "Settings" icon.
Select a "Prompt override version" to test based on the chosen configuration.
Alternatively, to set up another test mode, enter a "System prompt" and customize the tool descriptions: "End conversation tool description", "Escalate to human tool description", "Get relevant documents tool description" and "Get past conversations tool description".
Enable "Include account information in context" to include account-specific information in the test (Optional).
Enable "Use templated prompt" to use the structured prompt with company-specific fields instead of the custom "System prompt" (Optional).
Select a "Profile", "Guardrails", and "Integration" to use for the test.
Enable "" to show messages rejected during the test (Optional).Display Rejected Messages
Enable "Override default settings" to use the custom model and retrieval settings for the current test instead of the default test mode configuration (Optional).
Under "Agent Model", select the "Vendor", "Model", and "Temperature" to use for generating the agent's response.
Under "Guardrails Model", select the "Vendor", "Model", and "Temperature" to use for evaluating guardrails.
Under "Retrieval Settings", enter the "Similarity threshold" and "Num docs per query" values to control document retrieval.
Click "Apply" to apply the settings to the current test.
(Optional) Click "Save as Test Prompt" to save the configured settings as a reusable test prompt.

💡

A System Prompt gives Zeus AI custom instructions for your test. Use this field to set its role, tone, and specific rules for how it should behave during testing.

"Override default settings"

Enable "Override default settings" when you want to customize the models and retrieval behavior for the current test. When enabled, the settings in "Agent Model", "Guardrails Model", and "Retrieval Settings" override the default test mode configuration.

"Agent Model"

The "Agent Model" controls the model used to generate the Zeus AI response. Select a "Vendor", "Model", and "Temperature".

"Guardrails Model"

The "Guardrails Model" controls the model used to evaluate guardrails. Select a "Vendor", "Model", and "Temperature".

"Retrieval Settings"

The "Retrieval Settings" control how Zeus AI retrieves documents during the test.

"Similarity threshold": Sets the minimum similarity score required for a document to be retrieved. Higher values make retrieval more selective.
"Num docs per query": Sets the maximum number of documents retrieved for each query.

How to add a response to an evaluation

If a response is inaccurate or does not meet the desired output, you can add it as an evaluation example.

Click the "Add to eval" flask icon below the response.
Select the customer message you want to add to the evaluation.
Click "Create Example".
You’ll be directed to the "Create New Example" page, where you can define the desired response and configure the evaluation example.

💡

For more information, see the Evals documentation.

FAQs & Troubleshooting

What is Prompt override?

Prompt override allows you to test a different system prompt in Test mode without changing the prompt used by the production Zeus AI agent. It is useful when you want to experiment with new instructions, behavior, tone, or rules and see how Zeus AI responds before making changes to the live configuration.

When should I use Prompt override?

Use Prompt override when you want to test how Zeus AI behaves with different instructions before updating the production configuration.

Do Test mode conversations affect production metrics or live customer histories?

No. Test mode operates as a sandbox within the QA module and does not impact live tickets.

Can I test how the Zeus AI agent responds to different customer profiles?

Yes. Select a customer and a "Profile" to test how Zeus AI responds using customer-specific context and profile configuration.

Can I test different AI models?

Yes. The "Agent Model" and "Guardrails Model" sections allow you to select different vendors, models, and temperature settings for testing.

Can I test different retrieval configurations?

Yes. The "Retrieval Settings" section allows you to change the "Similarity threshold" and "Num docs per query" values.

Can I test a different system prompt without changing the production configuration?

Yes. Enter the desired instructions in the "System prompt" field. You can also enable "Use templated prompt" to test the templated prompt with company-specific fields.

Can I customize the tool descriptions used during a test?

Yes. Test mode provides fields for "End conversation tool description", "Escalate to human tool description", "Get relevant documents tool description", and "Get past conversations tool description".

Can I test guardrails?

Yes. In"Guardrails", configure the "Guardrails Model", and enable "Display Rejected Messages" to review rejected messages during testing.

Can I attach files during a test?

Yes. Click the attachment icon in the message composer to upload a file and test how Zeus AI processes it.

What should I do if the Zeus AI assistant's response is inaccurate?

First, review the test-specific settings, such as the "System prompt", "Agent Model", "Guardrails Model", and "Retrieval Settings".

If the issue is related to the agent configuration, adjust the relevant settings under the " Zeus > Agent " menu, such as Knowledge, Tools, Protocols, Guardrails, or Tone.

You can also modify or enhance your data sources under " App Configuration > AI > Data sources ", such as websites, Help Center articles, or tickets used to provide information to the AI.

Once your changes are saved, rerun the test to verify the results.

Why can't I see the Zeus AI agent's response after testing an inquiry?

Check the existing rules in " Zeus AI > Configuration > Profile mapping " to see whether a rule has the target profile set to "Always Escalate".

If the selected customer meets the conditions of that rule, Zeus AI will not generate a response. Instead, Test mode will indicate that the inquiry is being escalated to an agent.

This behavior reflects the configured Profile Mapping rule.

💡

Important: Because Test mode is a simulation, it does not create a ticket or route the inquiry to a live agent.

Was this article helpful?

Table of contents

Related articles

How to manage Zeus AI QA Test mode

Overview