Cross-lingual transfer learning with multilingual masked language models

17^th October 2023, 13:00 Ashton Lecture Theatre
Prof. Mamoru Komachi
Hitotsubashi University and Tokyo Metropolitan University, Japan

Abstract

This talk introduces the innovative realm of Multilingual Masked Language Models (MMLMs) as a crucial tool in the pursuit of effective cross-lingual transfer learning. The discussion aims to elucidate the underlying architecture, functionality, and practical applications of MMLMs in advancing NLP’s multilingual capabilities.

In the first part, I talk about key elements such as the transformer-based architecture, the masking mechanism integral to their operation, and the training process for transfer learning. These components collectively work in tandem to facilitate the model's ability to both understand and generate text in a multitude of languages.

In the second part, I argue how we can enhance few-shot learning, which uses only a small amount of training data for fine-tuning a model, by carefully selecting candidates for MMLMs. The main idea is to select hard examples for annotation to improve the model's performance. I show that zero-shot learning can be used to detect hard examples for cross-lingual transfer.

This talk aims to offer attendees a coherent and comprehensive understanding of MMLMs and their practical applications, including grammatical error correction and named entity recognition.

Biography

Mamoru Komachi is a Professor at Hitotsubashi University and a Research Professor at Tokyo Metropolitan University.
He received his M.Eng. and Ph.D. degrees from Nara Institute of Science and Technology (NAIST) in 2007 and 2010. He was
an Assistant Professor at NAIST before joining TMU. His research interests include semantics, information extraction and educational applications of natural language processing.