#llmops

Promptfoo is a CLI and library designed for evaluating and red-teaming Large Language Model (LLM) applications. It helps developers build secure and reliable AI apps, moving beyond trial-and-error. It offers automated evaluations, vulnerability scanning, side-by-side model comparison, and CI/CD integration. A key feature is its ability to run evaluations 100% locally, ensuring privacy. It is developer-first, flexible, battle-tested, and provides data-driven insights.

Ragas, an open-source Python library developed by vibrantlabsai, is a comprehensive toolkit for evaluating and optimizing Large Language Model (LLM) applications. It aims to replace subjective and time-consuming assessments with data-driven, efficient evaluation workflows by offering objective metrics (both LLM-based and traditional), intelligent test data generation, and actionable insights. Ragas can automatically create diverse test datasets, seamlessly integrates with popular LLM frameworks like LangChain and major observability tools, and facilitates building feedback loops to continuously improve LLM apps using production data.