Review queues

Introduction

Reviews help you organize and structure human review of your LLM systems in Freeplay. Select completions for your team to review, assign work to specific team members, and generate insight reports to share your findings. Reviews make it easy to maintain regular review practices and turn observations into actionable improvements.

Creating Review Queues

In the Reviews tab you can create a new Review Queue complete with instructions for your team and an optional due date to help manage timelines.

Curating Review Queues

Add production data to review queues from the Observability tab using filtering and auto evals to identify potential candidates for review. Then assign individual completions to your teammates.

Adding Completions

First, navigate to the Observability tab. Then there are two ways to add Completions to a review queue.

Bulk Add

• Filter for across specific criteria like metadata, eval scores, or input fields.
• Bulk select Completions from the filtered tables.
• Add to Review Queue!

Individual Add

When viewing an individual Completion from anywhere in the app you can add the Completion to a Review Queue from the drop down in the Review section of the Completion view.

Working through Review Queues

Once you've created your review you can assign Completions to different teammates.

Domain experts and analyst can then come in to review the completions assigned to them.

Insights

Each Review Queue produces an insights report so the analysts can share the results of their work broadly and the whole team can glean the key takeaways and go take action to improve their LLM system!