Your Model Is Not Predicting Depression Well And That Is Why: A Case Study of PRIMATE Dataset

Kirill Milintsevich, Kairit Sirts, Gaël Dias


Abstract
This paper addresses the quality of annotations in mental health datasets used for NLP-based depression level estimation from social media texts. While previous research relies on social media-based datasets annotated with binary categories, i.e. depressed or non-depressed, recent datasets such as D2S and PRIMATE aim for nuanced annotations using PHQ-9 symptoms. However, most of these datasets rely on crowd workers without the domain knowledge for annotation. Focusing on the PRIMATE dataset, our study reveals concerns regarding annotation validity, particularly for the lack of interest or pleasure symptom. Through reannotation by a mental health professional, we introduce finer labels and textual spans as evidence, identifying a notable number of false positives. Our refined annotations, to be released under a Data Use Agreement, offer a higher-quality test set for anhedonia detection. This study underscores the necessity of addressing annotation quality issues in mental health datasets, advocating for improved methodologies to enhance NLP model reliability in mental health assessments.
Anthology ID:
2024.clpsych-1.13
Volume:
Proceedings of the 9th Workshop on Computational Linguistics and Clinical Psychology (CLPsych 2024)
Month:
March
Year:
2024
Address:
St. Julians, Malta
Editors:
Andrew Yates, Bart Desmet, Emily Prud’hommeaux, Ayah Zirikly, Steven Bedrick, Sean MacAvaney, Kfir Bar, Molly Ireland, Yaakov Ophir
Venues:
CLPsych | WS
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
166–171
Language:
URL:
https://aclanthology.org/2024.clpsych-1.13
DOI:
Bibkey:
Cite (ACL):
Kirill Milintsevich, Kairit Sirts, and Gaël Dias. 2024. Your Model Is Not Predicting Depression Well And That Is Why: A Case Study of PRIMATE Dataset. In Proceedings of the 9th Workshop on Computational Linguistics and Clinical Psychology (CLPsych 2024), pages 166–171, St. Julians, Malta. Association for Computational Linguistics.
Cite (Informal):
Your Model Is Not Predicting Depression Well And That Is Why: A Case Study of PRIMATE Dataset (Milintsevich et al., CLPsych-WS 2024)
Copy Citation:
PDF:
https://aclanthology.org/2024.clpsych-1.13.pdf
Video:
 https://aclanthology.org/2024.clpsych-1.13.mp4