Tactics for multi-step AI app experimentation - Parea AI

🌈 Abstract

The article discusses tactics for testing and improving multi-step AI applications, using a sample chatbot application over the AirBnB 10k 2023 dataset as an example. It covers the following key points:

🙋 Q&A

[01] Tactics for Testing and Improving Multi-Step AI Apps

1. Questions related to the content of the section:

• What are the key tactics discussed in the article for testing and improving multi-step AI applications?
• How does the cascading effect of failed sub-steps impact the overall accuracy of a multi-step AI application?
• Why is quality assessment (QA) of every sub-step crucial for improving multi-step AI applications?
• What are the two main ways in which Parea helps with testing and evaluating sub-steps of a multi-step AI application?

Answers:

• The key tactics discussed are:
  • Evaluating and testing every sub-step of the AI application to identify areas for improvement
  • Using production logs or synthetic data with ground truth for reference-based evaluation of sub-steps
  • Caching LLM calls to speed up and save cost when iterating on independent sub-steps
• Assuming 90% accuracy for each step, a 10-step application would have a 60% error rate due to the cascading effects of failed sub-steps.
• Quality assessment of every sub-step is crucial to simplify identifying where to improve the application and minimize the cascading effects of failed sub-steps.
• Parea helps in two ways:
  1. It simplifies instrumenting and testing a step, as well as creating reports on how the components perform, using the trace decorator.
  2. It enables running experiments to measure the performance of the app on a dataset and identify regressions across experiments.

[02] Evaluating Sub-Steps with Reference-Based and Synthetic Data

1. Questions related to the content of the section:

• Why is reference-based evaluation easier and more grounded than reference-free evaluation for testing sub-steps?
• How can production logs be used as test data for evaluating sub-steps?
• When production logs are not available, how can synthetic data be generated for evaluating sub-steps?

Answers:

• Reference-based evaluation is easier and more grounded than reference-free evaluation because it provides ground truth data to verify the output of the sub-steps.
• Production logs can be used as test data for evaluating sub-steps by collecting and storing them along with any corrected sub-step outputs.
• When production logs are not available, synthetic data can be generated for evaluating sub-steps by incorporating the relationship between components into the data generation process. The article provides an example of using Instructor with the OpenAI API to generate keyword queries from the provided question, context, and answer triplets in the AirBnB 10k dataset.

[03] Instrumenting and Evaluating Sub-Steps with Parea

1. Questions related to the content of the section:

• How does Parea's trace decorator help with instrumenting and evaluating sub-steps?
• What information does the Log object in Parea's evaluation functions provide?
• How can Parea's experiments be used to compare the outputs of different runs and identify regressions?

Answers:

• Parea's trace decorator logs inputs, outputs, latency, and executes any specified evaluation functions to score the output of a sub-step, creating traces (hierarchical logs) for instrumentation and evaluation.
• The Log object in Parea's evaluation functions provides access to the output of the sub-step being evaluated and the target (correct) value from the dataset.
• Parea's experiments can be used to measure the performance of the AI application on a dataset and enable identifying regressions across different runs by comparing the outputs side-by-side.

[04] Caching LLM Calls to Speed Up Iteration

1. Questions related to the content of the section:

• Why is caching LLM calls important when iterating on independent sub-steps?
• How can a general cache be implemented to cache LLM calls?
• What abstraction can be introduced over LLM calls to apply the cache decorator?

Answers:

• Caching LLM calls is important when iterating on independent sub-steps to speed up the iteration time and avoid unnecessary cost, as other sub-steps might not have changed.
• A general cache can be implemented by maintaining a dictionary that maps the input parameters of the LLM call to the corresponding output, and checking the cache before making a new LLM call.
• An abstraction can be introduced over the LLM calls to apply the cache decorator, which would handle the caching logic and provide a consistent interface for making LLM calls.