Review Queues

Introduction

Streamline human review of your LLM systems with Freeplay's Review Queues. Organize work assignments, generate actionable insights, and transform observations into meaningful improvements.

Review Queues help you establish systematic, organized human review processes for your LLM systems in Freeplay. Easily select completions for your team to evaluate, assign specific work to team members, and generate comprehensive insight reports to share findings across your organization. Review Queues make it simple to maintain consistent review practices and transform observations into actionable improvements.

Creating Review Queues

From the Reviews tab, you can create a new Review Queue with customized instructions for your team and an optional due date to help manage project timelines effectively.

Building Effective Review Queues

Review Queues serve multiple purposes: addressing issues flagged by evaluations, investigating team-identified concerns, or conducting systematic quality assessments. By creating thoughtfully curated review queues, you gain the ability to dive deep into your data, understand underlying patterns, identify areas for improvement, and drive meaningful product enhancements.

Teams that actively use Review Queues develop a much deeper understanding of their LLM applications, enabling them to implement targeted changes that deliver real improvements to their products.

Adding Data to Review Queues

You can add Traces or Completions to a Review Queue for comprehensive analysis. Choose from two convenient methods:

Perfect for adding multiple items at once:

Filter across specific criteria such as metadata, evaluation scores, or input fields
Select multiple Completions from the filtered results
Add them all to your Review Queue in one action

Managing Review Workflows

Once you've created your Review Queue, you can assign specific Completions to different team members based on their expertise and availability. This ensures the right people are reviewing the most relevant content.

Domain experts and analysts can efficiently work through their assigned Completions, providing detailed feedback and analysis. The interface is designed to streamline the review process while capturing comprehensive insights.

Generating Actionable Insights

Every Review Queue automatically generates a comprehensive insights report, enabling analysts to:

Share findings broadly across the organization
Highlight key takeaways for stakeholders
Identify actionable improvements for the LLM system
Track progress over time

These insights reports transform individual review observations into strategic recommendations that drive meaningful system improvements.