Test AI agents, live agents, and conventional bots

You can use Syntrix to test not just non-deterministic AI agents and live agents, but also conventional bots that don’t make use of LLMs.

Live agents may know customers are synthetic

Realistic behavior is the benchmark for any simulation, but the degree of transparency varies. The visibility of the synthetic customer to your agents is a key difference between simulation types. Our "Live Agent Training" simulation type is designed to support transparent training exercises, meaning agents are generally aware that the customer is synthetic, not human.

Your agents might recognize a simulation because of:

• Configuration choices: Depending on your simulation setup, the customer name may appear as "Anonymous" rather than a generated fictitious name. This can be an indicator to the agent that the conversation is a drill.
• Workspace indicators: Agents may infer the nature of the interaction based on specific metadata or flags visible within their workspace.

For scenarios where the agent must be completely unaware they are interacting with a synthetic customer, you’ll want to use our upcoming Mystery Shopping simulation type (coming soon!). A mystery shopper is a synthetic customer that is indistinguishable from a real customer. In the current release, mystery shoppers aren't supported.

Synthetic customer generation is semi-random

When a simulation begins, a synthetic customer is created using a specific combination of one scenario and one persona. This combination is what directs the LLM as it plays the customer role during the test.

For each conversation, the system uses a semi-random method to combine scenarios with personas. Here’s how it works: The system generates a matrix of all possible combinations based on the simulation's configuration, and then it selects a unique combination at random from that set. To maximize coverage, combinations can’t be repeated until all combinations are exhausted.

Success criteria for agents is scenario-specific or universal

Agent performance criteria is defined within two spots in the simulation and training studio:

• Within the scenario
• Within the scorecard

While both are essential for evaluating performance, they serve distinct, yet complementary purposes.

To ensure your evaluation is both specific and comprehensive, it helps to break them down by scope, purpose, and content.

At a glance: The key differences

Examples

Imagine a customer calling to change a flight.

The scenario's agent goals check the business outcome:

• Was the flight date modified to the requested date?
• Is the price difference of the new ticket less than or equal to $100?
• Was the confirmation email sent to the verified address?

In contrast, the scorecard's agent goals check the agent's behavior:

• Did the agent maintain a professional demeanor throughout the interaction?
• Did the agent communicate in a polite and courteous manner?

Keep this distinction in mind when defining agent goals in your scenarios. When you ensure that agent goals are scenario-specific, you target what’s necessary to succeed in that context.

Stay tuned for a future update that will allow scorecard customization, giving you even more flexibility to track agent behavior across all your scenarios.

Transcript review is optional and efficient

Syntrix brings great news here: When evaluating a report, and in specific the simulated customer conversations, you don’t need to read every transcript. The primary deliverable is the AI analysis or summary, which provides quick, actionable insights.

However, for thorough quality assurance or deeper investigation, you can, indeed, review the full transcripts side-by-side with the AI's findings to gain deeper insights.

Synthetic customer assets are reusable

The true power of the synthetic customer assets (scenarios, personas, etc.) lies in their reusability across different training and evaluation needs. For example, a single persona, such as the "Busy parent," can be paired with various scenarios to test multiple customer goals.

Assume for a moment that you’re setting up simulations for Acme Telco, a fictional telecommunications company. You could apply the "Busy parent" persona to the "Report service outage" scenario to evaluate the call center agent's speed and efficiency. The same "Busy parent" persona could be used for the entirely different "Upgrade data plan" scenario to test the sales agent's product knowledge and upselling technique.

This modularity allows you to efficiently test a wide range of customer interactions.

Starter packs accelerate deployment

To get you up and running quickly, Syntrix includes a number of prebuilt profiles, scenarios and personas created by LivePerson for common cases. Take advantage of these. You'll find them front and center in the UI on relevant pages.