University of Twente Student Theses
As of Friday, 8 August 2025, the current Student Theses repository is no longer available for thesis uploads. A new Student Theses repository will be available starting Friday, 15 August 2025.
A Comprehensive Evaluation of Post-hoc Calibration Methods Across Modern Vision Architectures and Datasets
Li, Shujian (2025) A Comprehensive Evaluation of Post-hoc Calibration Methods Across Modern Vision Architectures and Datasets.
|
PDF
7MB |
| Abstract: | This study presents a comprehensive evaluation of five post-hoc calibration methods (Temperature Scaling, Isotonic Regression, Histogram Binning, Dirichlet Calibration, and a composite Dirichlet-Isotonic method) across a variety of modern vision architectures (ViT, ResNet, ConvNeXt) and datasets. Our multi-perspective analysis reveals that a one-size-fits-all approach to calibration is insufficient. We demonstrate that simple methods like Temperature Scaling are unreliable and can completely fail on certain architectures, whereas flexible non-parametric and composite methods provide statistically significant improvements in calibration. The composite Dirichlet-Isotonic method consistently proves to be the most effective and robust choice, successfully correcting complex, architecture-specific error patterns. Furthermore, our work uncovers critical trade-offs, such as the potential to improve average calibration error (ECE) at the cost of worst-case error (MCE), highlighting the need for careful assessment. The primary contribution of this work is an evidence-based decision framework that guides practitioners in selecting the optimal calibration method based on their specific model architecture and application requirements, thus contributing to more trustworthy and reliable AI systems. |
| Item Type: | Essay (Bachelor) |
| Faculty: | EEMCS: Electrical Engineering, Mathematics and Computer Science |
| Subject: | 54 computer science |
| Programme: | Business & IT BSc (56066) |
| Link to this item: | https://purl.utwente.nl/essays/107457 |
| Export this item as: | BibTeX EndNote HTML Citation Reference Manager |
Show download statistics for this publication
Show download statistics for this publicationRepository Staff Only: item control page