Semi-Supervised Lifelong Language Learning

Yingxiu Zhao, Yinhe Zheng, Bowen Yu, Zhiliang Tian, Dongkyu Lee, Jian Sun, Yongbin Li, Nevin L. Zhang


Abstract
Lifelong learning aims to accumulate knowledge and alleviate catastrophic forgetting when learning tasks sequentially. However, existing lifelong language learning methods only focus on the supervised learning setting. Unlabeled data, which can be easily accessed in real-world scenarios, are underexplored. In this paper, we explore a novel setting, semi-supervised lifelong language learning (SSLL), where a model learns sequentially arriving language tasks with both labeled and unlabeled data. We propose an unlabeled data enhanced lifelong learner to explore SSLL. Specially, we dedicate task-specific modules to alleviate catastrophic forgetting and design two modules to exploit unlabeled data: (1) a virtual supervision enhanced task solver is constructed on a teacher-student framework to mine the underlying knowledge from unlabeled data; and (2) a backward augmented learner is built to encourage knowledge transfer from newly arrived unlabeled data to previous tasks. Experimental results on various language tasks demonstrate our model’s effectiveness and superiority over competitive baselines under the new setting SSLL.
Anthology ID:
2022.findings-emnlp.290
Volume:
Findings of the Association for Computational Linguistics: EMNLP 2022
Month:
December
Year:
2022
Address:
Abu Dhabi, United Arab Emirates
Editors:
Yoav Goldberg, Zornitsa Kozareva, Yue Zhang
Venue:
Findings
SIG:
Publisher:
Association for Computational Linguistics
Note:
Pages:
3937–3951
Language:
URL:
https://aclanthology.org/2022.findings-emnlp.290
DOI:
10.18653/v1/2022.findings-emnlp.290
Bibkey:
Cite (ACL):
Yingxiu Zhao, Yinhe Zheng, Bowen Yu, Zhiliang Tian, Dongkyu Lee, Jian Sun, Yongbin Li, and Nevin L. Zhang. 2022. Semi-Supervised Lifelong Language Learning. In Findings of the Association for Computational Linguistics: EMNLP 2022, pages 3937–3951, Abu Dhabi, United Arab Emirates. Association for Computational Linguistics.
Cite (Informal):
Semi-Supervised Lifelong Language Learning (Zhao et al., Findings 2022)
Copy Citation:
PDF:
https://aclanthology.org/2022.findings-emnlp.290.pdf
Dataset:
 2022.findings-emnlp.290.dataset.zip
Software:
 2022.findings-emnlp.290.software.zip