Abstract

Morality plays an important role in culture, identity, and emotion. Recent advances in natural language processing have shown that it is possible to classify moral values expressed in text at scale. Morality classification relies on human annotators to label the moral expressions in text, which provides training data to achieve state-of-the-art performance. However, these annotations are inherently subjective and some of the instances are hard to classify, resulting in noisy annotations due to error or lack of agreement. The presence of noise in training data harms the classifier's ability to accurately recognize moral foundations from text. We propose two metrics to audit the noise of annotations. The first metric is entropy of instance labels, which is a proxy measure of annotator disagreement about how the instance should be labeled. The second metric is the silhouette coefficient of a label assigned by an annotator to an …

Date

November 10, 2022

Authors

Negar Mokhberian, Frederic R Hopp, Bahareh Harandizadeh, Fred Morstatter, Kristina Lerman

Conference

2022 IEEE/ACM International Conference on Advances in Social Networks Analysis and Mining (ASONAM)

Pages

147-154

Publisher

IEEE