1/24/24 Updates

📣

Recent Updates: Live monitoring, Prompt versioning improvements & Thin SDK Preview

• Live Monitoring of LLM sessions using auto-evals
• Prompt versioning UX enhancements, including commit messages & version titles
• New Thin SDK preview — increased control for developers

See all the details here

Live Monitoring Using Auto-Evals

Gain deeper insights into your production LLM systems by monitoring LLM behavior across all environments in realtime. Once you enable auto-evals for your prompts, Freeplay will start to automatically sample a subset of live sessions and score them using the same Evaluation criteria you use for Test Runs. We've also added new graphs that show cost and latency trends -- including average, 90th percentile, and 99th percentile values.

Want to go deeper? Check out the video below.

Prompt Versioning Updates

We've made a couple key updates to improve the prompt versioning user experience. Now, you can easily add commit-style messages and names to each version of your prompt templates. We've also separated the "save" and "deploy" steps, allowing you to save prompt versions without immediate deployment to an environment.

Another demo video is below.

Preview: Thin SDK provides more control to developers

While our standard SDK is designed to keep things simple, some developers have asked for more control and flexibility to accomplish their goals in production. We’re introducing our new Thin SDK to provide that control. Documentation includes Python examples for now, and we'll be adding Node and JVM examples soon.

This new Thin SDK will sit along side our existing encapsulated SDK. The Thin SDK has its own client so there are no breaking changes to existing SDK integrations.

See more detail about the organizing principles of the Thin SDK here.