University of Twente Student Theses
MARS : An Automatic Evaluation Framework for Cross-lingual RAG
Joosten, Wesley (2025) MARS : An Automatic Evaluation Framework for Cross-lingual RAG.
|
PDF
404kB |
| Abstract: | Recently, there have been significant developments in the area of evaluating RAG (Retrieval-Augmented Generation) systems. Unfortunately, this research is limited mainly to English or monolingual systems. For multilingual RAG systems, evaluation is often limited to overall performance metrics such as accuracy, while multilingual RAG comes with additional unique challenges that are currently underexplored. We introduce MARS (Multilingual (Automatic) Assessment of RAG Systems), building on the developments in monolingual evaluation, especially the ARES (Automatic RAG Evaluation System) framework, for the granular evaluation of multilingual RAG systems. MARS can effectively evaluate existing metrics from the RAG triad in multilingual scenarios, as well as Language Consistency, a newly introduced metric to measure a unique challenge in multilingual RAG. |
| Item Type: | Essay (Master) |
| Faculty: | EEMCS: Electrical Engineering, Mathematics and Computer Science |
| Subject: | 54 computer science |
| Programme: | Computer Science MSc (60300) |
| Link to this item: | https://purl.utwente.nl/essays/105174 |
| Export this item as: | BibTeX EndNote HTML Citation Reference Manager |
Show download statistics for this publication
Show download statistics for this publicationRepository Staff Only: item control page