Shipping and scaling AI agents

🌈 Abstract

The article discusses how Sierra AI agents are transforming customer experiences in the real world by helping customers with various tasks like setting up Sonos speakers, refreshing SiriusXM radios, and processing OluKai shoe returns and exchanges. It delves into the key lessons learned during each stage of agent development, from running toward complexity and targeting previously unsolvable challenges to equipping agents with the necessary instruction manuals and accelerating learning exponentially in production.

🙋 Q&A

[01] Scaling AI Agents

1. What are some of the key lessons learned by Sierra in scaling AI agents?

• They have incorporated dozens of refinements and product insights into their Agent Development Life Cycle methodology for building, testing, and optimizing AI agents.
• They have found success in running toward complexity and targeting previously unsolvable challenges at the outset of a partnership.
• They have learned how to equip each AI agent with the instruction manual it needs to be successful and avoid failure.
• They have discovered how to accelerate learning exponentially in production once an agent is ready for launch.
• They have found that every agent must be crafted as a full-fledged product, tailored seamlessly to the unique requirements of the company.

2. How do Sierra agents differ from simple Q&A chatbots?

• Sierra agents are designed to handle more complex customer issues that require customer-specific information, multi-step reasoning, and taking action, rather than just providing answers to simple questions.
• The "magic" of Sierra's AI agents comes from their deeper integrations and "agentic" reasoning, allowing them to fully resolve complex customer issues.

3. What is the Agent Development Life Cycle that Sierra uses?

• It is Sierra's methodology for building, testing, and optimizing AI agents, which involves a collaborative design process with customers, detailed journey specifications, and continuous observation and tuning of agent performance.

[02] Designing and Building AI Agents

1. How does Sierra collaborate with customers to design and build AI agents?

• They start with a 90-minute design workshop to align on the different "journeys" the agent must be able to traverse, corresponding to the goals that end users might have.
• They search for problems that are difficult to solve, such as troubleshooting Sonos systems or refreshing SiriusXM radios, and use their Agent OS platform to compose agents with the necessary skills and access to the required tools and systems.

2. How do Sierra agents balance creativity and predictability?

• They provide agents with detailed journey specifications that serve as an "instruction manual," ensuring that business logic is strictly and deterministically enforced, similar to how employees are given handbooks.
• At the same time, agents are designed to be "creative" in the moments that matter, like processing an order or upgrading a plan.

3. How do Sierra's customers ensure the quality and performance of their agents?

• They use Sierra's Experience Manager, which allows them to track agent performance, manage the agent to ensure it is following its instruction manual, and continuously improve the manual and agent over time.
• The Experience Manager also uses AI models to review conversations for quality and tune review processes before production launch, allowing customers to pinpoint and fix any mistakes the agent makes.

[03] Launching and Scaling AI Agents

1. What happens during the "launch" phase when the AI agent begins having live conversations with customers?

• Customers inevitably make unpredictable requests that the agent was not initially designed to handle, and this is where the real learning begins.
• The process of iteratively observing and tuning the AI agent generates a continuous feedback loop, and agents improve exponentially as they handle more interactions and gather more data and insights.

2. How do Sierra's customers leverage the Experience Manager after the agent is live?

• Conversations are automatically annotated with tags and AI-based supervision, which are used to generate reports, track key performance indicators, and route key insights back to customer experience and product teams.
• The Experience Manager continues to be the hub for conversation review, issue management, and interaction testing, allowing for efficient sampling and review of conversations as the agent scales.

3. What are the universal requirements for Sierra's AI agents, and how are they tailored to each customer?

• Universal requirements include being trustworthy, having sophisticated process knowledge and information, the capability to work with internal systems, and performing with high accuracy and empathy.
• At the same time, each agent must be tailored seamlessly to the unique requirements of the customer, representing their brand, internalizing their processes, knowledge bases, business goals, and risk controls.
• Sierra staffs a dedicated agent engineer and product manager on each deployment to ensure the agent is a full-fledged product tailored to the customer's needs.