How to use PretrainedTransformerTokenizer and PretrainedTransformerIndexer without the PretrainedTransformerEmbedder

[gabeorlanski]

I am using the PretrainedTransformerTokenizer to tokenize my target sequences in a composed seq2seq model. I use the PretrainedTransformerEmbedder as the encoder and the AutoRegressiveDecoder for decoding. In the latter, there are target embeddings that I would like to keep as the basic text embeddings rather than use the PretrainedTransformerEmbedder. However, I also wanted to use the PretrainedTransformerIndexer to get access to the full vocabulary of the pretrained model.

However, from my attempts thus far, it seems that using the PretrainedTransformerIndexer does not initialize the vocabulary with its namespace until after the model has been created. For the normal embeddings, this has the issue of causing one of its dimensions always to be 2 as the vocabulary is empty. Is there an officially supported way to address this?

[epwalsh]

Hey @gabeorlanski, you could call Vocabulary.add_transformer_vocab manually at any point to add the transformer vocab under a specific namespace, or you could instantiate the Vocabulary object using Vocabulary.from_pretrained_transformer (use "type": "from_pretrained_transformer" in your config).

[gabeorlanski]

I tried that, and unfortunately leads to a large number of OOV tokens, could it be the namespaces I am trying to use?

Here are the config snippets:

"dataset_reader": {
     "target_tokenizer": {
            "type": "basic",
            "underlying_tokenizer": {
                "type": "pretrained_transformer",
                "model_name": "prajjwal1/bert-tiny",
                "add_special_tokens": false
            },
            "tokenize_key": true
        },
        "target_indexer": {
            "tokens": {
                "type": "pretrained_transformer",
                "model_name": "prajjwal1/bert-tiny",
                "max_length": 512,
                "namespace": "target_tokens"
            }
        }
},
"vocabulary": {
    "type": "from_pretrained_transformer",
    "model_name": "prajjwal1/bert-tiny",
    "namespace": "target_tokens"
}

Note that the target_tokenizer is a custom tokenizer class that has an underlying AllenNLP tokenizer.

[epwalsh]

OOV tokens in your tokenized target sequences?

[gabeorlanski]

Yes, I do not know how because without the vocab or PretrainedIndexer, it works fine. With a single ID indexer.

[gabeorlanski]

@epwalsh So my fix, not a good one or one that is particularly nice, was to add:

for namespace in vocabulary_.get_namespaces():
    vocabulary_.add_tokens_to_namespace(
        getattr(dataset_reader, 'special_tokens', []),
        namespace
    )

to this spot in the train command and to add

"tokenizer_kwargs": {
    "additional_special_tokens": ["<t>", "</t>", "NULL_TOKEN", "<val>"]
}

to both the indexers and the tokenizers.

Unfortunately, because I am on windows, it is a lot of repetition because I do not have jsonnet. But maybe overall project configs can help that issue.