EvalLLM: Workshop on the Evaluation of Generative Models (LLM), RAG, and Challenges

The EvalLLM workshop focuses on the evaluation of generative large language models (LLMs) and their integration into complex processing chains, particularly emphasizing Retrieval Augmented Generation (RAG). This workshop aims to address the significant challenges associated with evaluating LLMs, especially considering the existing benchmarks which predominantly reflect Anglo-centric perspectives. The relevance of this workshop is heightened for the French language and other languages outside of English, as it seeks to engage researchers from both industry and academia on the multifaceted evaluation processes necessary for these models in specialized domains.The structure of the workshop will include an invited presentation alongside selected works resulting from a call for papers. This call invites research that pertains to various aspects of LLM evaluation, such as the performance assessment of complete systems, particularly RAGs, and both foundational and fine-tuned models. The workshop will also explore the creation or adaptation of benchmarks tailored to French and other languages, whether they are well-resourced or under-resourced. Research contributions are welcome on topics including NLP task evaluation (such as translation, summarization, and information extraction), adaptation of evaluation methodologies for generative systems, and the ethical dimensions surrounding bias, privacy, and cultural alignment. Additionally, the workshop will consider performance evaluations regarding computational time, memory use, and energy efficiency, as well as user-oriented evaluations focusing on ergonomics and cognitive aspects. Moreover, the workshop will include discussions on multimodal models that integrate text with images or speech.As part of the EvalLLM2026 workshop, there will be two notable challenges aimed at furthering the understanding of LLMs. The first challenge is a task-based evaluation that entails linking entities within the medical field through a few-shot learning context. Meanwhile, the second challenge focuses on RAG and attribution, offering participants a platform to further explore these critical areas. Overall, the EvalLLM workshop presents a vital opportunity for researchers to contribute to the evolving discourse on the evaluation of generative models and their applications.