Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Switch to AutoModel API #174

Open
JohnGiorgi opened this issue Aug 20, 2019 · 0 comments
Open

Switch to AutoModel API #174

JohnGiorgi opened this issue Aug 20, 2019 · 0 comments
Assignees
Labels
design enhancement New feature or request feature

Comments

@JohnGiorgi
Copy link
Contributor

JohnGiorgi commented Aug 20, 2019

The PyTorch Transformer library recently added a new AutoModel API, which lets you instantiate one of the many pre-trained transformers that are available (BERT, GPT-2, RoBERTa, etc.).

We should switch from our BERT-specific code to AutoModel. Specifically, this will involve swapping all BertModels, BertConfigs, and BertTokenizers for AutoModel, AutoConfig and AutoTokenizer. In the long run, this will let us seamlessly load pre-trained weights from any of the popular transformer language models.

This should be addressed in two pull requests:

  1. Simply use AutoModel, AutoTokenizer and AutoConfig in place of the BERT specific classes. This won't be enough to use any of the transformers weights as our preprocessing steps are still BERT specific, but it will let us use any of the models that have the same preprocessing steps as BERT (like RoBERTa).
  2. Re-write up our preprocessing steps to make them model agnostic.
@JohnGiorgi JohnGiorgi added enhancement New feature or request feature design labels Aug 20, 2019
@JohnGiorgi JohnGiorgi self-assigned this Aug 20, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
design enhancement New feature or request feature
Projects
None yet
Development

No branches or pull requests

1 participant