Compendium of LLM Evaluation methods
In response to the need for a broader understanding of evaluation methods and the dissemination of best practices, I am creating a compendium of evaluation techniques intended to aid developers and scientists. My aim is to maintain a pragmatic approach, ensuring the material is clear and the best practices are easily understandable. Furthermore, I have noticed that many effective evaluation methods are not discussed in typical evaluation or benchmarking papers but are found in other research articles that focus on specific tasks. These articles reveal how LLMs are evaluated for particular tasks, and I plan to share these resources as well to enrich our understanding.
This is an early draft, and I plan to include additional papers to make it more comprehensive. I will also enhance the organization and text to help users navigate this rich landscape, making it easier to find methods that align with business needs or other specific evaluation tasks.
I’ll be grateful for comments and recommendations
