LLM Evaluation - Comet's Opik

[manisnesan]

https://github.com/comet-ml/opik

via Comet ML - LLM eval's in six steps

Why

• Absence Blindness

What

• Systematically test and improve
  Generative AI application applications
• A strong evaluation pipeline is iterative

How

• Human review: review a dozen examples. Catch obvious mistakes and identify what to test for in evaluation pipeline
• unit test: right granular test for each part of the pipeline such as retrieval, generation and post processing
• AB testing: conduct controlled experiments to measure the impact of LLM updates. Combine human and automated feedback to drive better decision
• Fine tuning & Debugging: leverage annotated production data to fine tune models. Production grade data improves models automatically.