Context and Meaning: Navigating Disagreements in NLP Annotation

PROGRAM

Sunday, January 19, 2025

 9:20–11:00 Morning Session
9:20–9:30Introduction
9:30–10:30Invited talk: The Spectrum of Human Label Variation: Reflections from the Last Ten Years in NLP
Barbara Plank
10:30–11:00Coffee break
 11:00–14:00 Pre-lunch Session
11:00–11:20Is a bunch of words enough to detect disagreement in hateful content?
Giulia Rizzi, Paolo Rosso and Elisabetta Fersini
11:20–11:40On Crowdsourcing Task Design for Discourse Relation Annotation
Frances Yung and Vera Demberg
11:40–12:00Sources of Disagreement in Data for LLM Instruction Tuning
Russel Dsouza and Venelin Kovatchev
12:00–12:20CoMeDi Shared Task: Median Judgment Classification & Mean Disagreement Ranking with Ordinal Word-in-Context Judgments
Dominik Schlechtweg, Tejaswi Choppa, Wei Zhao and Michael Roth
12:20–14:00Lunch break
 14:00–16:00 Post-lunch Session
14:00–14:20Deep-change at CoMeDi: the Cross-Entropy Loss is not All You Need
Mikhail Kuklin and Nikolay Arefyev
14:20–14:40Predicting Median, Disagreement and Noise Label in Ordinal Word-in-Context Data
Tejaswi Choppa, Michael Roth and Dominik Schlechtweg
 14:40–15:30 Poster session
14:40–15:30GRASP at CoMeDi Shared Task: Multi-Strategy Modeling of Annotator Behavior in Multi-Lingual Semantic Judgments
David Alfter and Mattias Appelgren
14:40–15:30Funzac at CoMeDi Shared Task: Modeling Annotator Disagreement from Word-In-Context Perspectives
Olufunke O. Sarumi, Charles Welch, Lucie Flek and Jörg Schlötterer
14:40–15:30FuocChuVIP123 at CoMeDi Shared Task: Disagreement Ranking with XLM-Roberta Sentence Embeddings and Deep Neural Regression
Phuoc Duong Huy Chu
14:40–15:30JuniperLiu at CoMeDi Shared Task: Models as Annotators in Lexical Semantics Disagreements
Zhu Liu, Zhen Hu and Ying Liu
14:40–15:30MMLabUIT at CoMeDiShared Task: Text Embedding Techniques versus Generation-Based NLI for Median Judgment Classification
Tai Duc Le and Thin Dang Van
14:40–15:30ABDN-NLP at CoMeDi Shared Task: Predicting the Aggregated Human Judgment via Weighted Few-Shot Prompting
Ying Xuan Loke, Dominik Schlechtweg and Wei Zhao
14:40–15:30Automating Annotation Guideline Improvements using LLMs: A Case Study
Adrien Bibal, Nathaniel Gerlek, Goran Muric, Elizabeth Boschee, Steven C. Fincke, Mike Ross and Steven N. Minton
14:40–15:30Ambiguity and Disagreement in Abstract Meaning Representation
Shira Wein
14:40–15:30The Impact of Annotation Choices on Computational Representations of Semantic and Phonological Distance in Sign Languages
Lisa Loy
14:40–15:30Ambiguity Meets Uncertainty: Investigating Uncertainty Estimation for Word Sense Disambiguation
Zhu Liu and Ying Liu
14:40–15:30Disagreement in Metaphor Annotation of Mexican Spanish Science Tweets
Alec M. Sanchez-Montero, Gemma Bel-Enguix, SERGIO LUIS OJEDA TRUEBA and Gerardo Sierra Martínez
15:30–16:00Coffee break
 16:00–16:10 Closing Session
16:00–16:10Closing