Proceedings of BigScience Episode #5 – Workshop on Challenges & Perspectives in Creating Large Language Models

Proceedings of BigScience Episode #5 – Workshop on Challenges & Perspectives in Creating Large Language Models Angela Fan editor Suzana Ilic editor Thomas Wolf editor Matthias Gallé editor 2022-05 Association for Computational Linguistics virtual+Dublin text conference publication bigscience-2022-bigscience https://aclanthology.org/2022.bigscience-1.0/ Lifelong Pretraining: Continually Adapting Language Models to Emerging Corpora Xisen Jin author Dejiao Zhang author Henghui Zhu author Wei Xiao author Shang-Wen Li author Xiaokai Wei author Andrew Arnold author Xiang Ren author 2022-05 text Proceedings of BigScience Episode #5 – Workshop on Challenges & Perspectives in Creating Large Language Models Angela Fan editor Suzana Ilic editor Thomas Wolf editor Matthias Gallé editor Association for Computational Linguistics virtual+Dublin conference publication jin-etal-2022-lifelong 10.18653/v1/2022.bigscience-1.1 https://aclanthology.org/2022.bigscience-1.1/ 2022-05 1 16 Using ASR-Generated Text for Spoken Language Modeling Nicolas Hervé author Valentin Pelloin author Benoit Favre author Franck Dary author Antoine Laurent author Sylvain Meignier author Laurent Besacier author 2022-05 text Proceedings of BigScience Episode #5 – Workshop on Challenges & Perspectives in Creating Large Language Models Angela Fan editor Suzana Ilic editor Thomas Wolf editor Matthias Gallé editor Association for Computational Linguistics virtual+Dublin conference publication herve-etal-2022-using 10.18653/v1/2022.bigscience-1.2 https://aclanthology.org/2022.bigscience-1.2/ 2022-05 17 25 You reap what you sow: On the Challenges of Bias Evaluation Under Multilingual Settings Zeerak Talat author Aurélie Névéol author Stella Biderman author Miruna Clinciu author Manan Dey author Shayne Longpre author Sasha Luccioni author Maraim Masoud author Margaret Mitchell author Dragomir Radev author Shanya Sharma author Arjun Subramonian author Jaesung Tae author Samson Tan author Deepak Tunuguntla author Oskar Van Der Wal author 2022-05 text Proceedings of BigScience Episode #5 – Workshop on Challenges & Perspectives in Creating Large Language Models Angela Fan editor Suzana Ilic editor Thomas Wolf editor Matthias Gallé editor Association for Computational Linguistics virtual+Dublin conference publication talat-etal-2022-reap 10.18653/v1/2022.bigscience-1.3 https://aclanthology.org/2022.bigscience-1.3/ 2022-05 26 41 Diverse Lottery Tickets Boost Ensemble from a Single Pretrained Model Sosuke Kobayashi author Shun Kiyono author Jun Suzuki author Kentaro Inui author 2022-05 text Proceedings of BigScience Episode #5 – Workshop on Challenges & Perspectives in Creating Large Language Models Angela Fan editor Suzana Ilic editor Thomas Wolf editor Matthias Gallé editor Association for Computational Linguistics virtual+Dublin conference publication kobayashi-etal-2022-diverse 10.18653/v1/2022.bigscience-1.4 https://aclanthology.org/2022.bigscience-1.4/ 2022-05 42 50 UNIREX: A Unified Learning Framework for Language Model Rationale Extraction Aaron Chan author Maziar Sanjabi author Lambert Mathias author Liang Tan author Shaoliang Nie author Xiaochang Peng author Xiang Ren author Hamed Firooz author 2022-05 text Proceedings of BigScience Episode #5 – Workshop on Challenges & Perspectives in Creating Large Language Models Angela Fan editor Suzana Ilic editor Thomas Wolf editor Matthias Gallé editor Association for Computational Linguistics virtual+Dublin conference publication chan-etal-2022-unirex 10.18653/v1/2022.bigscience-1.5 https://aclanthology.org/2022.bigscience-1.5/ 2022-05 51 67 Pipelines for Social Bias Testing of Large Language Models Debora Nozza author Federico Bianchi author Dirk Hovy author 2022-05 text Proceedings of BigScience Episode #5 – Workshop on Challenges & Perspectives in Creating Large Language Models Angela Fan editor Suzana Ilic editor Thomas Wolf editor Matthias Gallé editor Association for Computational Linguistics virtual+Dublin conference publication nozza-etal-2022-pipelines 10.18653/v1/2022.bigscience-1.6 https://aclanthology.org/2022.bigscience-1.6/ 2022-05 68 74 Entities, Dates, and Languages: Zero-Shot on Historical Texts with T0 Francesco De Toni author Christopher Akiki author Javier De La Rosa author Clémentine Fourrier author Enrique Manjavacas author Stefan Schweter author Daniel van Strien author 2022-05 text Proceedings of BigScience Episode #5 – Workshop on Challenges & Perspectives in Creating Large Language Models Angela Fan editor Suzana Ilic editor Thomas Wolf editor Matthias Gallé editor Association for Computational Linguistics virtual+Dublin conference publication de-toni-etal-2022-entities 10.18653/v1/2022.bigscience-1.7 https://aclanthology.org/2022.bigscience-1.7/ 2022-05 75 83 A Holistic Assessment of the Carbon Footprint of Noor, a Very Large Arabic Language Model Imad Lakim author Ebtesam Almazrouei author Ibrahim Abualhaol author Merouane Debbah author Julien Launay author 2022-05 text Proceedings of BigScience Episode #5 – Workshop on Challenges & Perspectives in Creating Large Language Models Angela Fan editor Suzana Ilic editor Thomas Wolf editor Matthias Gallé editor Association for Computational Linguistics virtual+Dublin conference publication lakim-etal-2022-holistic 10.18653/v1/2022.bigscience-1.8 https://aclanthology.org/2022.bigscience-1.8/ 2022-05 84 94 GPT-NeoX-20B: An Open-Source Autoregressive Language Model Sidney Black author Stella Biderman author Eric Hallahan author Quentin Anthony author Leo Gao author Laurence Golding author Horace He author Connor Leahy author Kyle McDonell author Jason Phang author Michael Pieler author Usvsn Sai Prashanth author Shivanshu Purohit author Laria Reynolds author Jonathan Tow author Ben Wang author Samuel Weinbach author 2022-05 text Proceedings of BigScience Episode #5 – Workshop on Challenges & Perspectives in Creating Large Language Models Angela Fan editor Suzana Ilic editor Thomas Wolf editor Matthias Gallé editor Association for Computational Linguistics virtual+Dublin conference publication black-etal-2022-gpt 10.18653/v1/2022.bigscience-1.9 https://aclanthology.org/2022.bigscience-1.9/ 2022-05 95 136 Dataset Debt in Biomedical Language Modeling Jason Fries author Natasha Seelam author Gabriel Altay author Leon Weber author Myungsun Kang author Debajyoti Datta author Ruisi Su author Samuele Garda author Bo Wang author Simon Ott author Matthias Samwald author Wojciech Kusa author 2022-05 text Proceedings of BigScience Episode #5 – Workshop on Challenges & Perspectives in Creating Large Language Models Angela Fan editor Suzana Ilic editor Thomas Wolf editor Matthias Gallé editor Association for Computational Linguistics virtual+Dublin conference publication fries-etal-2022-dataset 10.18653/v1/2022.bigscience-1.10 https://aclanthology.org/2022.bigscience-1.10/ 2022-05 137 145 Emergent Structures and Training Dynamics in Large Language Models Ryan Teehan author Miruna Clinciu author Oleg Serikov author Eliza Szczechla author Natasha Seelam author Shachar Mirkin author Aaron Gokaslan author 2022-05 text Proceedings of BigScience Episode #5 – Workshop on Challenges & Perspectives in Creating Large Language Models Angela Fan editor Suzana Ilic editor Thomas Wolf editor Matthias Gallé editor Association for Computational Linguistics virtual+Dublin conference publication teehan-etal-2022-emergent 10.18653/v1/2022.bigscience-1.11 https://aclanthology.org/2022.bigscience-1.11/ 2022-05 146 159 Foundation Models of Scientific Knowledge for Chemistry: Opportunities, Challenges and Lessons Learned Sameera Horawalavithana author Ellyn Ayton author Shivam Sharma author Scott Howland author Megha Subramanian author Scott Vasquez author Robin Cosbey author Maria Glenski author Svitlana Volkova author 2022-05 text Proceedings of BigScience Episode #5 – Workshop on Challenges & Perspectives in Creating Large Language Models Angela Fan editor Suzana Ilic editor Thomas Wolf editor Matthias Gallé editor Association for Computational Linguistics virtual+Dublin conference publication horawalavithana-etal-2022-foundation 10.18653/v1/2022.bigscience-1.12 https://aclanthology.org/2022.bigscience-1.12/ 2022-05 160 172