Proceedings of the Workshop on Human Evaluation of NLP Systems (HumEval)

Proceedings of the Workshop on Human Evaluation of NLP Systems (HumEval) Anya Belz editor Shubham Agarwal editor Yvette Graham editor Ehud Reiter editor Anastasia Shimorina editor 2021-04 Association for Computational Linguistics Online text conference publication humeval-2021-human https://aclanthology.org/2021.humeval-1.0/ It’s Commonsense, isn’t it? Demystifying Human Evaluations in Commonsense-Enhanced NLG Systems Miruna-Adriana Clinciu author Dimitra Gkatzia author Saad Mahamood author 2021-04 text Proceedings of the Workshop on Human Evaluation of NLP Systems (HumEval) Anya Belz editor Shubham Agarwal editor Yvette Graham editor Ehud Reiter editor Anastasia Shimorina editor Association for Computational Linguistics Online conference publication clinciu-etal-2021-commonsense https://aclanthology.org/2021.humeval-1.1/ 2021-04 1 12 Estimating Subjective Crowd-Evaluations as an Additional Objective to Improve Natural Language Generation Jakob Nyberg author Maike Paetzel author Ramesh Manuvinakurike author 2021-04 text Proceedings of the Workshop on Human Evaluation of NLP Systems (HumEval) Anya Belz editor Shubham Agarwal editor Yvette Graham editor Ehud Reiter editor Anastasia Shimorina editor Association for Computational Linguistics Online conference publication nyberg-etal-2021-estimating https://aclanthology.org/2021.humeval-1.2/ 2021-04 13 24 Trading Off Diversity and Quality in Natural Language Generation Hugh Zhang author Daniel Duckworth author Daphne Ippolito author Arvind Neelakantan author 2021-04 text Proceedings of the Workshop on Human Evaluation of NLP Systems (HumEval) Anya Belz editor Shubham Agarwal editor Yvette Graham editor Ehud Reiter editor Anastasia Shimorina editor Association for Computational Linguistics Online conference publication zhang-etal-2021-trading https://aclanthology.org/2021.humeval-1.3/ 2021-04 25 33 Towards Document-Level Human MT Evaluation: On the Issues of Annotator Agreement, Effort and Misevaluation Sheila Castilho author 2021-04 text Proceedings of the Workshop on Human Evaluation of NLP Systems (HumEval) Anya Belz editor Shubham Agarwal editor Yvette Graham editor Ehud Reiter editor Anastasia Shimorina editor Association for Computational Linguistics Online conference publication castilho-2021-towards https://aclanthology.org/2021.humeval-1.4/ 2021-04 34 45 Is This Translation Error Critical?: Classification-Based Human and Automatic Machine Translation Evaluation Focusing on Critical Errors Katsuhito Sudoh author Kosuke Takahashi author Satoshi Nakamura author 2021-04 text Proceedings of the Workshop on Human Evaluation of NLP Systems (HumEval) Anya Belz editor Shubham Agarwal editor Yvette Graham editor Ehud Reiter editor Anastasia Shimorina editor Association for Computational Linguistics Online conference publication sudoh-etal-2021-translation https://aclanthology.org/2021.humeval-1.5/ 2021-04 46 55 Towards Objectively Evaluating the Quality of Generated Medical Summaries Francesco Moramarco author Damir Juric author Aleksandar Savkov author Ehud Reiter author 2021-04 text Proceedings of the Workshop on Human Evaluation of NLP Systems (HumEval) Anya Belz editor Shubham Agarwal editor Yvette Graham editor Ehud Reiter editor Anastasia Shimorina editor Association for Computational Linguistics Online conference publication moramarco-etal-2021-towards https://aclanthology.org/2021.humeval-1.6/ 2021-04 56 61 A Preliminary Study on Evaluating Consultation Notes With Post-Editing Francesco Moramarco author Alex Papadopoulos Korfiatis author Aleksandar Savkov author Ehud Reiter author 2021-04 text Proceedings of the Workshop on Human Evaluation of NLP Systems (HumEval) Anya Belz editor Shubham Agarwal editor Yvette Graham editor Ehud Reiter editor Anastasia Shimorina editor Association for Computational Linguistics Online conference publication moramarco-etal-2021-preliminary https://aclanthology.org/2021.humeval-1.7/ 2021-04 62 68 The Great Misalignment Problem in Human Evaluation of NLP Methods Mika Hämäläinen author Khalid Alnajjar author 2021-04 text Proceedings of the Workshop on Human Evaluation of NLP Systems (HumEval) Anya Belz editor Shubham Agarwal editor Yvette Graham editor Ehud Reiter editor Anastasia Shimorina editor Association for Computational Linguistics Online conference publication hamalainen-alnajjar-2021-great https://aclanthology.org/2021.humeval-1.8/ 2021-04 69 74 A View From the Crowd: Evaluation Challenges for Time-Offset Interaction Applications Alberto Chierici author Nizar Habash author 2021-04 text Proceedings of the Workshop on Human Evaluation of NLP Systems (HumEval) Anya Belz editor Shubham Agarwal editor Yvette Graham editor Ehud Reiter editor Anastasia Shimorina editor Association for Computational Linguistics Online conference publication chierici-habash-2021-view https://aclanthology.org/2021.humeval-1.9/ 2021-04 75 85 Reliability of Human Evaluation for Text Summarization: Lessons Learned and Challenges Ahead Neslihan Iskender author Tim Polzehl author Sebastian Möller author 2021-04 text Proceedings of the Workshop on Human Evaluation of NLP Systems (HumEval) Anya Belz editor Shubham Agarwal editor Yvette Graham editor Ehud Reiter editor Anastasia Shimorina editor Association for Computational Linguistics Online conference publication iskender-etal-2021-reliability https://aclanthology.org/2021.humeval-1.10/ 2021-04 86 96 On User Interfaces for Large-Scale Document-Level Human Evaluation of Machine Translation Outputs Roman Grundkiewicz author Marcin Junczys-Dowmunt author Christian Federmann author Tom Kocmi author 2021-04 text Proceedings of the Workshop on Human Evaluation of NLP Systems (HumEval) Anya Belz editor Shubham Agarwal editor Yvette Graham editor Ehud Reiter editor Anastasia Shimorina editor Association for Computational Linguistics Online conference publication grundkiewicz-etal-2021-user https://aclanthology.org/2021.humeval-1.11/ 2021-04 97 106 Eliciting Explicit Knowledge From Domain Experts in Direct Intrinsic Evaluation of Word Embeddings for Specialized Domains Goya van Boven author Jelke Bloem author 2021-04 text Proceedings of the Workshop on Human Evaluation of NLP Systems (HumEval) Anya Belz editor Shubham Agarwal editor Yvette Graham editor Ehud Reiter editor Anastasia Shimorina editor Association for Computational Linguistics Online conference publication van-boven-bloem-2021-eliciting https://aclanthology.org/2021.humeval-1.12/ 2021-04 107 113 Detecting Post-Edited References and Their Effect on Human Evaluation Věra Kloudová author Ondřej Bojar author Martin Popel author 2021-04 text Proceedings of the Workshop on Human Evaluation of NLP Systems (HumEval) Anya Belz editor Shubham Agarwal editor Yvette Graham editor Ehud Reiter editor Anastasia Shimorina editor Association for Computational Linguistics Online conference publication kloudova-etal-2021-detecting https://aclanthology.org/2021.humeval-1.13/ 2021-04 114 119 A Case Study of Efficacy and Challenges in Practical Human-in-Loop Evaluation of NLP Systems Using Checklist Shaily Bhatt author Rahul Jain author Sandipan Dandapat author Sunayana Sitaram author 2021-04 text Proceedings of the Workshop on Human Evaluation of NLP Systems (HumEval) Anya Belz editor Shubham Agarwal editor Yvette Graham editor Ehud Reiter editor Anastasia Shimorina editor Association for Computational Linguistics Online conference publication bhatt-etal-2021-case https://aclanthology.org/2021.humeval-1.14/ 2021-04 120 130 Interrater Disagreement Resolution: A Systematic Procedure to Reach Consensus in Annotation Tasks Yvette Oortwijn author Thijs Ossenkoppele author Arianna Betti author 2021-04 text Proceedings of the Workshop on Human Evaluation of NLP Systems (HumEval) Anya Belz editor Shubham Agarwal editor Yvette Graham editor Ehud Reiter editor Anastasia Shimorina editor Association for Computational Linguistics Online conference publication oortwijn-etal-2021-interrater https://aclanthology.org/2021.humeval-1.15/ 2021-04 131 141