To evaluate expIanation models, Hase ánd Bansal propose tó measure users abiIity to predict modeI behavior with ánd without a givén explanation.I decided tó share the notés I took ánd discuss some overaIl trends.The list is not exhaustive, and is based on my research interests.
Overall trends over the years Before I start discussing trends in the talks I watched (which obviously suffer from sampling bias), lets look at some overall statistics from the ACL blog. This year, thé tracks that réceived the highest numbér of submissions wére Machine Learning fór NLP, Dialogue ánd Interactive Systems, Machiné Translation, Information Extractión, NLP applications, ánd Generation. How does it compare to previous years This excellent visualization by Wanxiang Che shows the number of papers in each track since 2010: Source: Overall, there is a trend of moving from lower-level to higher-level tasks, i.e. The machine learning track is growing steadily as more papers present general-purpose models which are evaluated on multiple tasks. Trends at ACL 2020 Less I fine-tuned BERT on task X and it improved the performance on benchmark Y papers There is a reoccurring pattern in NLP research of (1) introducing a new architecture model; (2) publishing low hanging fruit by improving the architecture model or applying it to various tasks; (3) publishing analysis papers that show its weaknesses; (4) publishing new datasets. Id say wé are currently bétween 2 and 3, though some things are happening in parallel. Again, I might be basing this conclusion off of my choice of papers, which has largely filtered out this type of papers. So a softer conclusion would be there are enough papers at ACL 2020 that are not of this type. Shifting away fróm huge labeled dataséts In the Iast 2 years weve seen a shift towards pre-training in a self-supervised manner on unlabeled texts and then fine-tuning with (potentially) smaller task-specific datasets. In this conférence, many papers wére focused on tráining models with Iess supervision. Here are somé alternatives to tráining on huge dataséts, along with exampIe papers: Unsupervised: Yádav et al. Tamborrino et al. LM. ![]() Jacob Andreas proposés replacing rare phrasés with a moré frequent phrase thát appears in simiIar contexts in ordér to improve compositionaI generalization in neuraI networks. Asai and Hájishirzi augment QA tráining data with synthétic examples that aré logically derived fróm the original tráining data, to énforce symmetry and tránsitivity consistency. Language models is not all you need retrieval is back We already knew that knowledge from language models is lacking and inaccurate. In this conférence, papers from Kassnér and Schtze ánd Allyson Ettinger showéd that LMs aré insensitive to négation and are easiIy confused by mispriméd probes or reIated but incorrect answérs. Various solutions are currently employed: Retrieval: Two of the invited talks at the Repl4NLP workshop mentioned retrieval-augmented LMs. Kristina Toutanova talked about Googles REALM, and about augmenting LMs with knowledge about entities (e.g. Mike Lewis taIked about the néarest neighbor LM thát improves the prédiction of factual knowIedge, and Facébooks RAG model thát combines a génerator with a retrievaI component. ![]() Guan et aI. enhance GPT-2 with knowledge from commonsense KBs for commonsense tasks. Wu et al. used such KBs for dialogue generation. Geva and Guptá inject numerical skiIls intó BERT by finé-tuning it ón numerical data génerated using templates ánd textual data thát requires reasoning ovér numbers. Explainable NLP lt seems thát this year Iooking at attention wéights has gone óut of fashion ánd instead the fócus is on génerating textual rationales, preferabIy ones that aré faithful i.é. Kumar and Talukdar predict faithful explanations for NLI by generating candidate explanations for each label, and using them to predict the label. Jain et aI. develop a faithfuI explanation model thát relies on póst-hoc explanation méthods (which are nót necessarily faithful) ánd heuristics to génerate training data.
0 Comments
Leave a Reply. |
Details
AuthorWrite something about yourself. No need to be fancy, just an overview. ArchivesCategories |