The episode begins with hosts Hong True from the University of Toronto and Hong Yuan Mei from TTI Chicago introducing the Daily Arxiv Radiostation show and the Computation and Language category for June 13, 2023. They mention selecting two papers out of 15 submissions. The first paper discussed is 'Morphosyntactic probing of multilingual BERT models' by authors including Noah A. Smith from the University of Washington and Allen Institute for Artificial Intelligence. This paper introduces an extensive dataset with 247 tasks across 42 languages from 10 families, derived from Universal Dependencies treebanks. It probes pre-trained transformer models like BERT and XLM Roberta, finding they learn features that perform strongly on these tasks. The paper applies two methods to locate disambiguating information: a new perturbation method masking context parts and Shapley values. A key finding is that preceding context tends to hold more prediction-relevant information than following context. The second paper is 'Domain-Specific Fast Pre-Training Technique Using Document-Level Metadata (FPDM)' by authors including Niloy Ganguly from IIT Kharagpur. It addresses the high compute requirements of state-of-the-art transformers by proposing FPDM, a compute-efficient framework using document metadata and domain-specific taxonomy supervision. The innovation involves continually pre-training an open domain encoder with sentence-level embeddings for long documents during pre-training, but fine-tuning with token-level embeddings. FPDM outperforms transformer-based baselines in character-level F1 scores and other metrics in customer support, scientific, and legal domains, with negligible performance drop on open domain benchmarks. It reduces pre-training compute by approximately 10%, achieving reductions of 1,000, 4,500, and 500 times compared to MLM and/or NSP in those domains respectively. Code and datasets are available at bit.ly/famicode. The host expresses strong liking for this paper at the end.
