AImodels in production: the needs for measuring and monitoring from a business aspect, and approaches to ground truth delay
Keywords
Loading...
Authors
Issue Date
2023-02-17
Language
en
Document type
Journal Title
Journal ISSN
Volume Title
Publisher
Title
ISSN
Volume
Issue
Startpage
Endpage
DOI
Abstract
In recent years, not only the use of artificial intelligence (AI) models in academic research has increased,
but the use of AI in the industry has also increased significantly, which has brought a new set of challenges in
terms of performance monitoring. This thesis is divided into two parts; the first part focuses on the business
perspective of measuring and monitoring the use of AImodels in production, while the second part focuses
onmitigating the ground truth delay problem.
In the first part of the thesis, using both academic and non-academic sources, as well as conducting
interviews, valuable insights were gained from both academic and industry perspectives regarding the
measurements, needs, and possibilities when assessing the performance of an AI model from a business
perspective. Examples include the use of KPIs and model validation. In the second part of the thesis, two
methods, Confidence Based Performance Estimation (CBPE) and Continuous Re-Evaluation (CRE), were
evaluated for mitigating the ground truth delay problem in online model monitoring. The CBPE method,
which estimates the performance of a model using class probability estimates, was found to be a more
practical approach to online model monitoring compared to the CRE method. This is due to the ease of
implementation of CBPE and the fact that CBPE, unlike CRE, does not alter themodel and data.
Overall, this thesis provides a comprehensive understanding of the challenges and opportunities in
monitoring the performance of AImodels in production and offers insight into possible solutions to mitigate
the ground truth delay problem in online model monitoring.
Keywords: Model monitoring, Online monitoring, Business impact, Ground truth delay, Verification
Latency
Description
Citation
Supervisor
Faculty
Faculteit der Sociale Wetenschappen