Expect the unexpected: Implementing mental models as Markov chains to predict unexpected events

Keywords
Loading...
Thumbnail Image
Issue Date
2023-06-12
Language
en
Journal Title
Journal ISSN
Volume Title
Publisher
Abstract
Technological advancements are causing humans to increasingly work together with artificial agents [1]. Explainable AI (XAI) is an important and necessary approach when interacting with artificial agents. Information on the expectations of humans can be used to inform better explanations [2]. These expectations humans have of other agents and the world can be formalised as a mental model. This study investigates the possibility of using approximated mental models for predicting unexpected events. An experiment with a Human-Robot Interaction (HRI) element was used to create a dataset with expected and unexpected events. A nontrivial cooperative card game was used as the experiment task. To approximate the mental model of the human, synthetic data was generated using Monte-Carlo Tree Search (MCTS) agents. Next, Markov Chains (MCs) were constructed for this task and trained on this synthetic data. After the initial results, an additional model trained on human data was developed. This was done to check if human data better approximates human mental models. The model trained on synthetic data performed at the chance level (weighted F1: 0.53) while the model trained on human data had moderately positive results (weighted F1: 0.65). Both models showed highly differing results per participant, with the initial model trained on synthetic data showing the most extreme values. The project is a first exploration into the use of Markov models as a MH representation in a complex HRI task to predict unexpected events. Results show that better approximations of MH do result in overall better performance in predicting unexpected events. However, the performance varied considerably between individual participants for both models. For the model developed in this thesis, a classification threshold hyperparameter was identified and its effect on model performance was explored and found to have a very strong infleunce.
Description
Citation
Supervisor
Faculty
Faculteit der Sociale Wetenschappen