Universal Language Model Fine-tuning for Text Classification
J. Howard, and S. Ruder. ACL, Association for Computational Linguistics, (2018)
Abstract
Inductive transfer learning has greatly impacted computer vision, but
existing approaches in NLP still require task-specific modifications and
training from scratch. We propose Universal Language Model Fine-tuning
(ULMFiT), an effective transfer learning method that can be applied to any task
in NLP, and introduce techniques that are key for fine-tuning a language model.
Our method significantly outperforms the state-of-the-art on six text
classification tasks, reducing the error by 18-24% on the majority of datasets.
Furthermore, with only 100 labeled examples, it matches the performance of
training from scratch on 100x more data. We open-source our pretrained models
and code.
Description
Universal Language Model Fine-tuning for Text Classification
%0 Conference Paper
%1 howard2018universal
%A Howard, Jeremy
%A Ruder, Sebastian
%B ACL
%D 2018
%I Association for Computational Linguistics
%K classification context ecl embeddings nlp nn reserved sensitive text thema thema:transfer_learning word
%T Universal Language Model Fine-tuning for Text Classification
%U http://arxiv.org/abs/1801.06146
%X Inductive transfer learning has greatly impacted computer vision, but
existing approaches in NLP still require task-specific modifications and
training from scratch. We propose Universal Language Model Fine-tuning
(ULMFiT), an effective transfer learning method that can be applied to any task
in NLP, and introduce techniques that are key for fine-tuning a language model.
Our method significantly outperforms the state-of-the-art on six text
classification tasks, reducing the error by 18-24% on the majority of datasets.
Furthermore, with only 100 labeled examples, it matches the performance of
training from scratch on 100x more data. We open-source our pretrained models
and code.
@inproceedings{howard2018universal,
abstract = {Inductive transfer learning has greatly impacted computer vision, but
existing approaches in NLP still require task-specific modifications and
training from scratch. We propose Universal Language Model Fine-tuning
(ULMFiT), an effective transfer learning method that can be applied to any task
in NLP, and introduce techniques that are key for fine-tuning a language model.
Our method significantly outperforms the state-of-the-art on six text
classification tasks, reducing the error by 18-24% on the majority of datasets.
Furthermore, with only 100 labeled examples, it matches the performance of
training from scratch on 100x more data. We open-source our pretrained models
and code.},
added-at = {2018-08-12T11:41:04.000+0200},
author = {Howard, Jeremy and Ruder, Sebastian},
biburl = {https://www.bibsonomy.org/bibtex/20281c13bb77e3d2588bd676d2c8ccaa9/schwemmlein},
booktitle = {ACL},
description = {Universal Language Model Fine-tuning for Text Classification},
interhash = {4a4b12e7b6b8f26c4fc6cdcbecfbbdc3},
intrahash = {0281c13bb77e3d2588bd676d2c8ccaa9},
keywords = {classification context ecl embeddings nlp nn reserved sensitive text thema thema:transfer_learning word},
publisher = {Association for Computational Linguistics},
timestamp = {2018-09-10T13:00:06.000+0200},
title = {Universal Language Model Fine-tuning for Text Classification},
url = {http://arxiv.org/abs/1801.06146},
year = 2018
}