Extracting spanBERT embeddings

[alicesaunders]

Is it possible to extract word embeddings from a model that I’ve trained myself that can then be used in further anaysis? E.g bias analysis on the embeddings with PCA and the cosine similarity.
thank you!

[epwalsh]

This is certainly possible. Are you just trying to extract the corresponding weights from the embedding layer, or are you trying to get the contextual hidden states of each token within a sentence that you pass through your model?

The former would be pretty easy, you could just pull those weights out the state dictionary.

[alicesaunders]

I was hoping to get the contextual hidden state of particular tokens that could then be plotted to see where they lie in the embedding space. In particular I was hoping to extract gender pair words (such as he, she), and calculate their distance from specific occupation words (such as lawyer, hairdresser etc.). Is this something you think would be possible?

[epwalsh]

The easiest way might be to just override the model implementation so that you can save the output from the text_field_embedders (which will contain the contextual hidden states from the transformer embedder): https://github.com/allenai/allennlp-models/blob/c57b931523503a31b360ab43a8e729586f0cb1bf/allennlp_models/coref/models/coref.py#L181