Error using conll_formatter in pipeline

[megamattc]

Hello,

I was wondering if the developer of the spacy_conll package or someone else in the community could explain where the error below comes from. It basically says that the conll_formatter cannot be run. The strange thing is is I have used spacy_conll for a long time now in my python scripts for spacy without problems until recently:

Traceback (most recent call last):
  File "/Users/matthewong/miniforge3/lib/python3.10/site-packages/spacy/language.py", line 1026, in __call__
    doc = proc(doc, **component_cfg.get(name, {}))  # type: ignore[call-arg]
  File "/Users/matthewong/miniforge3/lib/python3.10/site-packages/spacy_conll/formatter.py", line 115, in __call__
    self._set_span_conll(sent, sent_idx)
  File "/Users/matthewong/miniforge3/lib/python3.10/site-packages/spacy_conll/formatter.py", line 162, in _set_span_conll
    self._set_token_conll(token, token_idx)
  File "/Users/matthewong/miniforge3/lib/python3.10/site-packages/spacy_conll/formatter.py", line 190, in _set_token_conll
    token.pos_,
  File "spacy/tokens/token.pyx", line 869, in spacy.tokens.token.Token.pos_.__get__
KeyError: 3711981038

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/Users/matthewong/Documents/nlp_stuff/Akkadian/Akkadian-language-models/ak_tagger_parser_ud_lookup/packages/ak_AkkParser_Norm_1_2_5_9_15_anzu_barutu-0.0.0/prose_model_annotator_pipe.py", line 110, in <module>
    doc = nlp(line)
  File "/Users/matthewong/miniforge3/lib/python3.10/site-packages/spacy/language.py", line 1029, in __call__
    raise ValueError(Errors.E109.format(name=name)) from e
ValueError: [E109] Component 'conll_formatter' could not be run. Did you forget to call `initialize()`?

To explain my setup, I use

spacy 3.4.4
spacy-conll 3.2.0
python 3.10

The error occurs when I run the following script, the relevant parts of which are shown, with ERROR HERE marking the specific place the error occurs:

import spacy
import ak_AkkParser_Norm_1_2_5_9_15_anzu_barutu as akkModel
import pandas
import os, glob
import json
from spacy.language import Language

import spacy_conll
...


nlp = akkModel.load()

#Add conllu formatter
nlp.add_pipe("conll_formatter", last=True)


#Directories for input text files and where to store final conllu output

inputPath = './remainder'
outputPathIntermediate = './conllu_output_intermediate/'
outputPathFinal = './conllu_output_final/'

#Get all .txt files in input directory, apply spacy nlp model to get doc files, convert doc file to intermediate conllu format and store in directory given by outputPathIntermediate

for filename in glob.glob(os.path.join(inputPath, '*.txt')):
   with open(os.path.join(os.getcwd(), filename), 'r') as inputFile:
   
      ...

       #lines = inputFile.readlines() #Get lines from input .txt file. There should just be two lines, the header with saao info, then the entire Akkadian text as one line.

       line = inputFile.read()  # Process lines in a way that avoids turning blank lines, other anomalies into problems for Inception
       lines = line.split("\n")
       lines = [line for line in lines if (not line.isspace() and line != "")]

       for line in lines:
           ....

               print("Line is")
               print(line)
               line = line + " " #Important to avoid parsing errors later
               doc = nlp(line) <- ERROR HERE
               #print("doc._.conll_str")
               #print(doc._.conll_str)
               print(doc._.conll_str, file=outputFileIntermediate)

....

I can tell that the pipeline chokes on the first text line that is fed into it. I don't think I've changed anything in my config.cfg file for my model, and I can still train it on my conllu data as usual. Could there be some python packages interfering with things, as I may have installed some recently?

[BramVanroy]

This seems like an error that is unrelated to spacy_conll. As you can see in the error log:

  File "spacy/tokens/token.pyx", line 869, in spacy.tokens.token.Token.pos_.__get__
KeyError: 3711981038

Seems like there are some discrepancies in your model config or somewhere in your custom model.

[megamattc]

I see. So since there is a key error that occurs in spacy.tokens.token.Token.pos, might that mean that there is a token or symbol that is not recognized or processed properly in the text files fed to the model?

[BramVanroy]

I am not entirely sure what causes this. Does the same issue also arise if you do not use the CoNLL formatter (leave out the nlp.add_pipe line)?

[megamattc]

Good idea. I did that and other things. The pipeline with and without conll_formatter works in the python interpreter when I feed it a short string. Rather, It seems that the problem is with a number of .txt files within the current batch that I feed into the whole pipeline using my script to get morpho-syntactic annotations for. This is the first time this has happened to me over dozens of batches of similar text, and even within the current batch some files are processed fine. It seems something goes wrong for those files when I chop it up by newline.But the print out seems to show nothing weird. I'll have to get back to you when I find out what the problem is.

[megamattc]

In case you are curious, or know something I don't, I've copied two text files below, one which causes the problem and one which doesn't.

This one causes problems:

saao/saa09/P313634
x x+x it-tal-x x x x x x x x x x x lā talliki ana x x x x x x x x mā ūmē in ni u₂ kin x x x x x x x nakru akaššad ša Aššur-bani-apli x x x x x x mā šībi mātāti utaqqan x x x x x mā ina digilīya pānî x x x x x adanniš kakkabtu ša hurāṣi an-x x x x x paššūru 01 qâ kusāpi danni ina muhhi x x x x x sihhāru saʾʾu mat x x x x x x x x+x-ma-ni ša šaddūni x x x x x x x šapal kakkabtu ša hurāṣi x x x x x x x ina rēšēya x+x x x x x x x x x x x kakkibī x x x x x x x x x x x x x

This one doesn't cause problems:

saao/saa09/P336677
x x x x x x x x x x Aššur-ahu-iddina šar Mat-Aššur x x x kakkišāti ubâra ina pān šēpēka akarrar atta lā tapallah x x x x+x urkiūte akaššad x x x x x-ka tap-ta-an-x x x x x-ti ašakkan x x x x x+x-ka-bar ina x+x x x

I don't think I've had problems before with subscripts (u₂) or apostrophies (saʾʾu)...

[BramVanroy]

Unfortunately, I cannot help you with this problem as it does not seem specific to spacy-conll. Maybe the people at the spaCy discussions can help you.

[megamattc]

That's ok. Thanks for your help anyway!