University of Twente Student Theses
As of Friday, 8 August 2025, the current Student Theses repository is no longer available for thesis uploads. A new Student Theses repository will be available starting Friday, 15 August 2025.
Combining Vision language models and gaze tracking in VR art exhibitions
Limbeek, Tessa (2025) Combining Vision language models and gaze tracking in VR art exhibitions.
|
PDF
987kB |
| Abstract: | In recent years, virtual reality and the integration of conversational agents have become more prevalent in art exhibitions, which has led to the exploration of new methods to enhance user experience. Although eye tracking and conversational agents have been utilised separately until now, their combined application remains under-explored. This study investigates how integrating real-time gaze data with a vision language model (VLM) supports the automatic identification of areas of interest (AOIs) and influences user experience in a VR art gallery. To evaluate AOI identification, we first assessed the baseline of the VLM by manually checking the capacity of the VLM with and without contextual information. Afterwards, we introduced the contextual information and compared the answers between the manually defined AOI agent and the gaze-driven agent. To assess the user experience, a user study with 27 participants assessed enjoyment, engagement, personalisation, collaboration, and gaze awareness through a VR visit, a questionnaire and an interview. The results suggest that the VLM, when provided with gaze data on a image and a basic text prompt, can identify AOIs in most cases with a quantitative success rate of 72% of AOIs correctly identified in 53 coordinates. The system can detect AOIs beyond those predefined in the contextual information, generating more focused and relevant responses. Although no statistically significant differences in user experience were observed between gaze-informed and manually guided agents, the findings suggest that a gaze-based approach could support similarly effective user interactions while reducing the manual effort required to define AOIs. This work contributes to the development of adaptive and scalable systems for personalised experiences in VR art environments. |
| Item Type: | Essay (Bachelor) |
| Faculty: | EEMCS: Electrical Engineering, Mathematics and Computer Science |
| Subject: | 54 computer science |
| Programme: | Computer Science BSc (56964) |
| Link to this item: | https://purl.utwente.nl/essays/107359 |
| Export this item as: | BibTeX EndNote HTML Citation Reference Manager |
Show download statistics for this publication
Show download statistics for this publicationRepository Staff Only: item control page