Snorkeling for Beginners: Applying Data Programming to Product Moderation in E-commerce
Keywords
Loading...
Authors
Issue Date
2020-11-18
Language
en
Document type
Journal Title
Journal ISSN
Volume Title
Publisher
Title
ISSN
Volume
Issue
Startpage
Endpage
DOI
Abstract
Machine learning classifiers can help product moderators on an e-commerce
platform by suggesting potentially inappropriate products. To enable rapid reactions
to sudden unforeseen events, it is necessary that such classifiers can be
quickly created and adjusted. For this purpose, we investigate the usefulness
of data programming. Data programming enables domain experts (in our case
product moderators) to rapidly create an improve classifiers. Domain experts
design labeling functions (LFs) that implement weak supervision signals (e.g.
heuristics) which can generate noisy suggestions of a label (e.g. inappropriate)
for single data points. These LFs are combined to train a probabilistic label
model, yielding probabilistic labels for a previously unlabeled training data
set. With these probabilistic labels, a classifier is trained. This classifier can be
steered by adjusting the LFs and thereby the automatic creation of training data.
In a case study, we apply data programming to train classifiers for six di erent
categories of inappropriate products. Our LFs need to be designed and improved
by non-coder domain experts in a short amount of time. We investigate
characteristics of LFs that are based on the domain experts’ domain knowledge.
Our results show that defining a high-coverage set of LFs or individual highaccuracy
LFs for positive cases is challenging for domain experts. Furthermore,
not all LFs have a positive e ect on the label model’s performance. The performance
of the trained classifier reflects the domain expert’s level of experience
with a category.
To improve the classifier in a short amount of time, we investigate the use of
intelligently drawn sets of data points, called inspiration sets. The purpose of
inspiration sets is to inspire the domain experts in improving the LFs. We compare
three types of inspiration sets: Previously unlabeled data, data on which
LFs previously disagreed and data most mistaken by the classifier. Overall, we
show that inspiration sets can trigger substantial performance improvements
in a short amount of time. We provide a detailed observation of how these
adjustments a ect the LFs’ e ect on the label model’s performance. Altogether,
our results indicate that data on which LFs conflicted is most promising as an
inspiration set, but that the usefulness of inspiration sets varies across monitors.
Description
Citation
Faculty
Faculteit der Sociale Wetenschappen