AImodels in production: the needs for measuring and monitoring from a business aspect, and approaches to ground truth delay

Thumbnail Image
Issue Date
Journal Title
Journal ISSN
Volume Title
In recent years, not only the use of artificial intelligence (AI) models in academic research has increased, but the use of AI in the industry has also increased significantly, which has brought a new set of challenges in terms of performance monitoring. This thesis is divided into two parts; the first part focuses on the business perspective of measuring and monitoring the use of AImodels in production, while the second part focuses onmitigating the ground truth delay problem. In the first part of the thesis, using both academic and non-academic sources, as well as conducting interviews, valuable insights were gained from both academic and industry perspectives regarding the measurements, needs, and possibilities when assessing the performance of an AI model from a business perspective. Examples include the use of KPIs and model validation. In the second part of the thesis, two methods, Confidence Based Performance Estimation (CBPE) and Continuous Re-Evaluation (CRE), were evaluated for mitigating the ground truth delay problem in online model monitoring. The CBPE method, which estimates the performance of a model using class probability estimates, was found to be a more practical approach to online model monitoring compared to the CRE method. This is due to the ease of implementation of CBPE and the fact that CBPE, unlike CRE, does not alter themodel and data. Overall, this thesis provides a comprehensive understanding of the challenges and opportunities in monitoring the performance of AImodels in production and offers insight into possible solutions to mitigate the ground truth delay problem in online model monitoring. Keywords: Model monitoring, Online monitoring, Business impact, Ground truth delay, Verification Latency
Faculteit der Sociale Wetenschappen