Annotate the Way You Think: An Incremental Note Generation Framework for the Summarization of Medical Conversations

Longxiang Zhang, Caleb D. Hart, Susanne Burger, Thomas Schaaf


Abstract
The scarcity of public datasets for the summarization of medical conversations has been a limiting factor for advancing NLP research in the healthcare domain, and the structure of the existing data is largely limited to the simple format of conversation-summary pairs. We therefore propose a novel Incremental Note Generation (ING) annotation framework capable of greatly enriching summarization datasets in the healthcare domain and beyond. Our framework is designed to capture the human summarization process via an annotation task by instructing the annotators to first incrementally create a draft note as they accumulate information through a conversation transcript (Generation) and then polish the draft note into a reference note (Rewriting). The annotation results include both the reference note and a comprehensive editing history of the draft note in tabular format. Our pilot study on the task of SOAP note generation showed reasonable consistency between four expert annotators, established a solid baseline for quantitative targets of inter-rater agreement, and demonstrated the ING framework as an improvement over the traditional annotation process for future modeling of summarization.
Anthology ID:
2024.lrec-main.105
Volume:
Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024)
Month:
May
Year:
2024
Address:
Torino, Italia
Editors:
Nicoletta Calzolari, Min-Yen Kan, Veronique Hoste, Alessandro Lenci, Sakriani Sakti, Nianwen Xue
Venues:
LREC | COLING
SIG:
Publisher:
ELRA and ICCL
Note:
Pages:
1173–1186
Language:
URL:
https://aclanthology.org/2024.lrec-main.105
DOI:
Bibkey:
Cite (ACL):
Longxiang Zhang, Caleb D. Hart, Susanne Burger, and Thomas Schaaf. 2024. Annotate the Way You Think: An Incremental Note Generation Framework for the Summarization of Medical Conversations. In Proceedings of the 2024 Joint International Conference on Computational Linguistics, Language Resources and Evaluation (LREC-COLING 2024), pages 1173–1186, Torino, Italia. ELRA and ICCL.
Cite (Informal):
Annotate the Way You Think: An Incremental Note Generation Framework for the Summarization of Medical Conversations (Zhang et al., LREC-COLING 2024)
Copy Citation:
PDF:
https://aclanthology.org/2024.lrec-main.105.pdf