Logging Implicit Human Feedback

🌈 Abstract

The article discusses Foyle, an open-source assistant that aims to simplify operations for software developers by using AI to capture and learn from implicit human feedback. It describes the problem of developers having to manually iterate on commands suggested by AI assistants, and how Foyle's notebook-based interface and structured logging approach can help capture this feedback automatically.

🙋 Q&A

[01] Logging Implicit Human Feedback

1. What is the key premise of Foyle regarding building effective AI assistants?

• Foyle's central premise is that creating a user experience that implicitly captures human feedback is critical to building AIs that effectively assist with operations.

2. How does Foyle's notebook-based interface help capture human feedback?

• Foyle's frontend is VSCode notebooks, where AI-generated commands are rendered as cells. When a user executes a cell, the frontend sends the cell contents and its associated UUID to the Foyle server, allowing the system to compare the AI-generated content with the user's executed commands and capture any corrections or feedback.

3. How does Foyle approach logging and tracing to support the capture of human feedback?

• Foyle instruments the application using structured logs, emitting logs to stdout/stderr and shipping them to a logging backend. It attaches a trace ID to each log entry to group all logs related to a particular request. This allows Foyle to build a trace by grouping log entries by trace ID and rekeying them by the associated cell ID.

4. Why does the author prefer using existing logging libraries over custom observability solutions?

• The author believes that using existing logging libraries that support structured logging is a better approach than requiring instrumentation with a specific SDK, as it maintains the separation of concerns between application instrumentation and telemetry storage/processing.

[02] Visualization and Insights

1. What are the challenges the author faces in visualizing the data collected by Foyle?

• The author notes that the primary way they want to view Foyle's data is to look at the processing associated with a particular cell, which requires post-processing the raw logs. This means that out-of-the-box visualizations from observability tools may not be sufficient to provide the desired insights.

2. How does the author approach the visualization problem?

• The author has built a lightweight progressive web app (PWA) in Go, which allows them to query the logs using SQL and join/process the traces in the web app. This enables them to view the traces in real-time without having to build and deploy a streaming pipeline.

3. What are the author's thoughts on the role of visualization and insights in the context of LLM/AI observability?

• The author suggests that while one might expect LLM/AI-focused tools to excel at visualization, the specific needs of their use case (e.g., viewing the processing associated with a particular cell) require custom post-processing of the raw logs, which may not be well-supported by off-the-shelf tools.