Understudy Labs

What is Understudy Labs?

An open-source toolkit to capture traces, evaluate cheaper models against benchmarks, and ship specialist routes you own. Capture traces from LLM production workflows with a single install that deploys within coding agents you already use. Hosted infrastructure is optional. Evaluate the captured traces and set a benchmark for success. Every future model switch meets or exceeds it in A/B testing. Train and fine-tune a new model on prompts and weights you always own. Start locally and scale into cloud-hosted processes as you see success. Deploy a new model only when the held-out eval is beaten. Serve it wherever you want. Production data feeds back into training and compounds performance over time. Test. Learn. Deploy. Repeat.

From the website

Orchestra — AI model routing and optimization

Orchestra evaluates production work, improves the models behind it, and trains specialist models your team owns.