ipdSummary: UnboundLocalError: local variable 'mods' referenced before assignment

[nick-youngblut]

I am running ipdSummary as such:

ipdSummary --identify m6A,m4C \
  --methylFraction    --numWorkers 8   --maxCoverage 100    \
  --log-file tests/ipdSummary/subreads/cov100/MT21.log \
  --gff tests/pdSummary/subreads/cov100/MT21.gff \
  --csv tests/ipdSummary/subreads/cov100/MT21.csv \
  --reference reference.fasta           
  MT21.bam 

and am getting the following uninformative error:

Traceback (most recent call last):
  File "/ebio/abt3_projects/software/bin/smrtlink/install/smrtlink-release_9.0.0.92188/bundles/smrttools/install/smrttools-release_9.0.0.92188/private/thirdparty/python3/python3_3.7.3/site-packages/kineticsTools/KineticWorker.py", line 141, in onChunk
    perSiteResults, (start, end))
  File "/ebio/abt3_projects/software/bin/smrtlink/install/smrtlink-release_9.0.0.92188/bundles/smrttools/install/smrttools-release_9.0.0.92188/private/thirdparty/python3/python3_3.7.3/site-packages/kineticsTools/KineticWorker.py", line 317, in _decodePositiveControl
    for (r, mod) in decoder.decode().items():
  File "/ebio/abt3_projects/software/bin/smrtlink/install/smrtlink-release_9.0.0.92188/bundles/smrttools/install/smrttools-release_9.0.0.92188/private/thirdparty/python3/python3_3.7.3/site-packages/kineticsTools/ModificationDecode.py", line 49, in decode
    return self.scoreMods(modCalls)
  File "/ebio/abt3_projects/software/bin/smrtlink/install/smrtlink-release_9.0.0.92188/bundles/smrttools/install/smrttools-release_9.0.0.92188/private/thirdparty/python3/python3_3.7.3/site-packages/kineticsTools/ModificationDecode.py", line 257, in scoreMods
    pos - self.post, pos + self.pre, modSeq)
  File "/ebio/abt3_projects/software/bin/smrtlink/install/smrtlink-release_9.0.0.92188/bundles/smrttools/install/smrttools-release_9.0.0.92188/private/thirdparty/python3/python3_3.7.3/site-packages/kineticsTools/MultiSiteCommon.py", line 150, in getContextMeans
    meanVector.append(self.gbmModel.getPredictions([ctx]))
  File "/ebio/abt3_projects/software/bin/smrtlink/install/smrtlink-release_9.0.0.92188/bundles/smrttools/install/smrttools-release_9.0.0.92188/private/thirdparty/python3/python3_3.7.3/site-packages/kineticsTools/ipdModel.py", line 273, in getPredictions
    modBits = baseToCode[s[i]]
KeyError: 67
> /ebio/abt3_projects/software/bin/smrtlink/install/smrtlink-release_9.0.0.92188/bundles/smrttools/install/smrttools-release_9.0.0.92188/private/thirdparty/python3/python3_3.7.3/site-packages/kineticsTools/ipdModel.py(273)getPredictions()
-> modBits = baseToCode[s[i]]
(Pdb)
Process KineticWorkerProcess-5:
Traceback (most recent call last):
  File "/ebio/abt3_projects/software/bin/smrtlink/install/smrtlink-release_9.0.0.92188/bundles/smrttools/install/smrttools-release_9.0.0.92188/private/thirdparty/python3/python3_3.7.3/lib/python3.7/multiprocessing/process.py", line 297, in _bootstrap
    self.run()
  File "/ebio/abt3_projects/software/bin/smrtlink/install/smrtlink-release_9.0.0.92188/bundles/smrttools/install/smrttools-release_9.0.0.92188/private/thirdparty/python3/python3_3.7.3/site-packages/kineticsTools/WorkerProcess.py", line 151, in run
    self._run()
  File "/ebio/abt3_projects/software/bin/smrtlink/install/smrtlink-release_9.0.0.92188/bundles/smrttools/install/smrttools-release_9.0.0.92188/private/thirdparty/python3/python3_3.7.3/site-packages/kineticsTools/WorkerProcess.py", line 128, in _run
    datum)  # pylint: disable=assignment-from-none
  File "/ebio/abt3_projects/software/bin/smrtlink/install/smrtlink-release_9.0.0.92188/bundles/smrttools/install/smrttools-release_9.0.0.92188/private/thirdparty/python3/python3_3.7.3/site-packages/kineticsTools/KineticWorker.py", line 155, in onChunk
    modDict = dict((x['tpl'], x) for x in mods if start <=
UnboundLocalError: local variable 'mods' referenced before assignment
Traceback (most recent call last):
  File "/ebio/abt3_projects/software/bin/smrtlink/install/smrtlink-release_9.0.0.92188/bundles/smrttools/install/smrttools-release_9.0.0.92188/private/thirdparty/python3/python3_3.7.3/site-packages/kineticsTools/KineticWorker.py", line 141, in onChunk
    perSiteResults, (start, end))
  File "/ebio/abt3_projects/software/bin/smrtlink/install/smrtlink-release_9.0.0.92188/bundles/smrttools/install/smrttools-release_9.0.0.92188/private/thirdparty/python3/python3_3.7.3/site-packages/kineticsTools/KineticWorker.py", line 339, in _decodePositiveControl
    for (r, mod) in decoder.decode().items():
  File "/ebio/abt3_projects/software/bin/smrtlink/install/smrtlink-release_9.0.0.92188/bundles/smrttools/install/smrttools-release_9.0.0.92188/private/thirdparty/python3/python3_3.7.3/site-packages/kineticsTools/ModificationDecode.py", line 49, in decode
    return self.scoreMods(modCalls)
  File "/ebio/abt3_projects/software/bin/smrtlink/install/smrtlink-release_9.0.0.92188/bundles/smrttools/install/smrttools-release_9.0.0.92188/private/thirdparty/python3/python3_3.7.3/site-packages/kineticsTools/ModificationDecode.py", line 257, in scoreMods
    pos - self.post, pos + self.pre, modSeq)
  File "/ebio/abt3_projects/software/bin/smrtlink/install/smrtlink-release_9.0.0.92188/bundles/smrttools/install/smrttools-release_9.0.0.92188/private/thirdparty/python3/python3_3.7.3/site-packages/kineticsTools/MultiSiteCommon.py", line 150, in getContextMeans
    meanVector.append(self.gbmModel.getPredictions([ctx]))
  File "/ebio/abt3_projects/software/bin/smrtlink/install/smrtlink-release_9.0.0.92188/bundles/smrttools/install/smrttools-release_9.0.0.92188/private/thirdparty/python3/python3_3.7.3/site-packages/kineticsTools/ipdModel.py", line 273, in getPredictions
    modBits = baseToCode[s[i]]
KeyError: 67
> /ebio/abt3_projects/software/bin/smrtlink/install/smrtlink-release_9.0.0.92188/bundles/smrttools/install/smrttools-release_9.0.0.92188/private/thirdparty/python3/python3_3.7.3/site-packages/kineticsTools/ipdModel.py(273)getPredictions()
-> modBits = baseToCode[s[i]]
(Pdb)
Process KineticWorkerProcess-2:
Traceback (most recent call last):
  File "/ebio/abt3_projects/software/bin/smrtlink/install/smrtlink-release_9.0.0.92188/bundles/smrttools/install/smrttools-release_9.0.0.92188/private/thirdparty/python3/python3_3.7.3/lib/python3.7/multiprocessing/process.py", line 297, in _bootstrap
    self.run()
  File "/ebio/abt3_projects/software/bin/smrtlink/install/smrtlink-release_9.0.0.92188/bundles/smrttools/install/smrttools-release_9.0.0.92188/private/thirdparty/python3/python3_3.7.3/site-packages/kineticsTools/WorkerProcess.py", line 151, in run
    self._run()
  File "/ebio/abt3_projects/software/bin/smrtlink/install/smrtlink-release_9.0.0.92188/bundles/smrttools/install/smrttools-release_9.0.0.92188/private/thirdparty/python3/python3_3.7.3/site-packages/kineticsTools/WorkerProcess.py", line 128, in _run
    datum)  # pylint: disable=assignment-from-none
  File "/ebio/abt3_projects/software/bin/smrtlink/install/smrtlink-release_9.0.0.92188/bundles/smrttools/install/smrttools-release_9.0.0.92188/private/thirdparty/python3/python3_3.7.3/site-packages/kineticsTools/KineticWorker.py", line 155, in onChunk
    modDict = dict((x['tpl'], x) for x in mods if start <=
UnboundLocalError: local variable 'mods' referenced before assignment
Traceback (most recent call last):
  File "/ebio/abt3_projects/software/bin/smrtlink/install/smrtlink-release_9.0.0.92188/bundles/smrttools/install/smrttools-release_9.0.0.92188/private/thirdparty/python3/python3_3.7.3/site-packages/kineticsTools/KineticWorker.py", line 141, in onChunk
    perSiteResults, (start, end))
  File "/ebio/abt3_projects/software/bin/smrtlink/install/smrtlink-release_9.0.0.92188/bundles/smrttools/install/smrttools-release_9.0.0.92188/private/thirdparty/python3/python3_3.7.3/site-packages/kineticsTools/KineticWorker.py", line 317, in _decodePositiveControl
    for (r, mod) in decoder.decode().items():
  File "/ebio/abt3_projects/software/bin/smrtlink/install/smrtlink-release_9.0.0.92188/bundles/smrttools/install/smrttools-release_9.0.0.92188/private/thirdparty/python3/python3_3.7.3/site-packages/kineticsTools/ModificationDecode.py", line 49, in decode
    return self.scoreMods(modCalls)
  File "/ebio/abt3_projects/software/bin/smrtlink/install/smrtlink-release_9.0.0.92188/bundles/smrttools/install/smrttools-release_9.0.0.92188/private/thirdparty/python3/python3_3.7.3/site-packages/kineticsTools/ModificationDecode.py", line 257, in scoreMods
    pos - self.post, pos + self.pre, modSeq)
  File "/ebio/abt3_projects/software/bin/smrtlink/install/smrtlink-release_9.0.0.92188/bundles/smrttools/install/smrttools-release_9.0.0.92188/private/thirdparty/python3/python3_3.7.3/site-packages/kineticsTools/MultiSiteCommon.py", line 150, in getContextMeans
    meanVector.append(self.gbmModel.getPredictions([ctx]))
  File "/ebio/abt3_projects/software/bin/smrtlink/install/smrtlink-release_9.0.0.92188/bundles/smrttools/install/smrttools-release_9.0.0.92188/private/thirdparty/python3/python3_3.7.3/site-packages/kineticsTools/ipdModel.py", line 273, in getPredictions
    modBits = baseToCode[s[i]]
KeyError: 65
Traceback (most recent call last):
  File "/ebio/abt3_projects/software/bin/smrtlink/install/smrtlink-release_9.0.0.92188/bundles/smrttools/install/smrttools-release_9.0.0.92188/private/thirdparty/python3/python3_3.7.3/site-packages/kineticsTools/KineticWorker.py", line 141, in onChunk
    perSiteResults, (start, end))
  File "/ebio/abt3_projects/software/bin/smrtlink/install/smrtlink-release_9.0.0.92188/bundles/smrttools/install/smrttools-release_9.0.0.92188/private/thirdparty/python3/python3_3.7.3/site-packages/kineticsTools/KineticWorker.py", line 317, in _decodePositiveControl
    for (r, mod) in decoder.decode().items():
  File "/ebio/abt3_projects/software/bin/smrtlink/install/smrtlink-release_9.0.0.92188/bundles/smrttools/install/smrttools-release_9.0.0.92188/private/thirdparty/python3/python3_3.7.3/site-packages/kineticsTools/ModificationDecode.py", line 49, in decode
    return self.scoreMods(modCalls)
  File "/ebio/abt3_projects/software/bin/smrtlink/install/smrtlink-release_9.0.0.92188/bundles/smrttools/install/smrttools-release_9.0.0.92188/private/thirdparty/python3/python3_3.7.3/site-packages/kineticsTools/ModificationDecode.py", line 257, in scoreMods
    pos - self.post, pos + self.pre, modSeq)
  File "/ebio/abt3_projects/software/bin/smrtlink/install/smrtlink-release_9.0.0.92188/bundles/smrttools/install/smrttools-release_9.0.0.92188/private/thirdparty/python3/python3_3.7.3/site-packages/kineticsTools/MultiSiteCommon.py", line 150, in getContextMeans
    meanVector.append(self.gbmModel.getPredictions([ctx]))
  File "/ebio/abt3_projects/software/bin/smrtlink/install/smrtlink-release_9.0.0.92188/bundles/smrttools/install/smrttools-release_9.0.0.92188/private/thirdparty/python3/python3_3.7.3/site-packages/kineticsTools/ipdModel.py", line 273, in getPredictions
    modBits = baseToCode[s[i]]
KeyError: 84
> /ebio/abt3_projects/software/bin/smrtlink/install/smrtlink-release_9.0.0.92188/bundles/smrttools/install/smrttools-release_9.0.0.92188/private/thirdparty/python3/python3_3.7.3/site-packages/kineticsTools/ipdModel.py(273)getPredictions()
-> modBits = baseToCode[s[i]]
(Pdb) > /ebio/abt3_projects/software/bin/smrtlink/install/smrtlink-release_9.0.0.92188/bundles/smrttools/install/smrttools-release_9.0.0.92188/private/thirdparty/python3/python3_3.7.3/site-packages/kineticsTools/ipdModel.py(273)getPredictions()
-> modBits = baseToCode[s[i]]
(Pdb)
Process KineticWorkerProcess-8:
Traceback (most recent call last):
  File "/ebio/abt3_projects/software/bin/smrtlink/install/smrtlink-release_9.0.0.92188/bundles/smrttools/install/smrttools-release_9.0.0.92188/private/thirdparty/python3/python3_3.7.3/lib/python3.7/multiprocessing/process.py", line 297, in _bootstrap
    self.run()
  File "/ebio/abt3_projects/software/bin/smrtlink/install/smrtlink-release_9.0.0.92188/bundles/smrttools/install/smrttools-release_9.0.0.92188/private/thirdparty/python3/python3_3.7.3/site-packages/kineticsTools/WorkerProcess.py", line 151, in run
    self._run()
  File "/ebio/abt3_projects/software/bin/smrtlink/install/smrtlink-release_9.0.0.92188/bundles/smrttools/install/smrttools-release_9.0.0.92188/private/thirdparty/python3/python3_3.7.3/site-packages/kineticsTools/WorkerProcess.py", line 128, in _run
    datum)  # pylint: disable=assignment-from-none
  File "/ebio/abt3_projects/software/bin/smrtlink/install/smrtlink-release_9.0.0.92188/bundles/smrttools/install/smrttools-release_9.0.0.92188/private/thirdparty/python3/python3_3.7.3/site-packages/kineticsTools/KineticWorker.py", line 155, in onChunk
    modDict = dict((x['tpl'], x) for x in mods if start <=
UnboundLocalError: local variable 'mods' referenced before assignment

Process KineticWorkerProcess-4:
Traceback (most recent call last):
  File "/ebio/abt3_projects/software/bin/smrtlink/install/smrtlink-release_9.0.0.92188/bundles/smrttools/install/smrttools-release_9.0.0.92188/private/thirdparty/python3/python3_3.7.3/lib/python3.7/multiprocessing/process.py", line 297, in _bootstrap
    self.run()
  File "/ebio/abt3_projects/software/bin/smrtlink/install/smrtlink-release_9.0.0.92188/bundles/smrttools/install/smrttools-release_9.0.0.92188/private/thirdparty/python3/python3_3.7.3/site-packages/kineticsTools/WorkerProcess.py", line 151, in run
    self._run()
  File "/ebio/abt3_projects/software/bin/smrtlink/install/smrtlink-release_9.0.0.92188/bundles/smrttools/install/smrttools-release_9.0.0.92188/private/thirdparty/python3/python3_3.7.3/site-packages/kineticsTools/WorkerProcess.py", line 128, in _run
    datum)  # pylint: disable=assignment-from-none
  File "/ebio/abt3_projects/software/bin/smrtlink/install/smrtlink-release_9.0.0.92188/bundles/smrttools/install/smrttools-release_9.0.0.92188/private/thirdparty/python3/python3_3.7.3/site-packages/kineticsTools/KineticWorker.py", line 155, in onChunk
    modDict = dict((x['tpl'], x) for x in mods if start <=
UnboundLocalError: local variable 'mods' referenced before assignment
Traceback (most recent call last):
  File "/ebio/abt3_projects/software/bin/smrtlink/install/smrtlink-release_9.0.0.92188/bundles/smrttools/install/smrttools-release_9.0.0.92188/private/thirdparty/python3/python3_3.7.3/site-packages/kineticsTools/KineticWorker.py", line 141, in onChunk
    perSiteResults, (start, end))
  File "/ebio/abt3_projects/software/bin/smrtlink/install/smrtlink-release_9.0.0.92188/bundles/smrttools/install/smrttools-release_9.0.0.92188/private/thirdparty/python3/python3_3.7.3/site-packages/kineticsTools/KineticWorker.py", line 339, in _decodePositiveControl
    for (r, mod) in decoder.decode().items():
  File "/ebio/abt3_projects/software/bin/smrtlink/install/smrtlink-release_9.0.0.92188/bundles/smrttools/install/smrttools-release_9.0.0.92188/private/thirdparty/python3/python3_3.7.3/site-packages/kineticsTools/ModificationDecode.py", line 49, in decode
    return self.scoreMods(modCalls)
  File "/ebio/abt3_projects/software/bin/smrtlink/install/smrtlink-release_9.0.0.92188/bundles/smrttools/install/smrttools-release_9.0.0.92188/private/thirdparty/python3/python3_3.7.3/site-packages/kineticsTools/ModificationDecode.py", line 257, in scoreMods
    pos - self.post, pos + self.pre, modSeq)
  File "/ebio/abt3_projects/software/bin/smrtlink/install/smrtlink-release_9.0.0.92188/bundles/smrttools/install/smrttools-release_9.0.0.92188/private/thirdparty/python3/python3_3.7.3/site-packages/kineticsTools/MultiSiteCommon.py", line 150, in getContextMeans
    meanVector.append(self.gbmModel.getPredictions([ctx]))
  File "/ebio/abt3_projects/software/bin/smrtlink/install/smrtlink-release_9.0.0.92188/bundles/smrttools/install/smrttools-release_9.0.0.92188/private/thirdparty/python3/python3_3.7.3/site-packages/kineticsTools/ipdModel.py", line 273, in getPredictions
    modBits = baseToCode[s[i]]
KeyError: 67
> /ebio/abt3_projects/software/bin/smrtlink/install/smrtlink-release_9.0.0.92188/bundles/smrttools/install/smrttools-release_9.0.0.92188/private/thirdparty/python3/python3_3.7.3/site-packages/kineticsTools/ipdModel.py(273)getPredictions()
-> modBits = baseToCode[s[i]]
(Pdb)
Process KineticWorkerProcess-7:
Traceback (most recent call last):
  File "/ebio/abt3_projects/software/bin/smrtlink/install/smrtlink-release_9.0.0.92188/bundles/smrttools/install/smrttools-release_9.0.0.92188/private/thirdparty/python3/python3_3.7.3/lib/python3.7/multiprocessing/process.py", line 297, in _bootstrap
    self.run()
  File "/ebio/abt3_projects/software/bin/smrtlink/install/smrtlink-release_9.0.0.92188/bundles/smrttools/install/smrttools-release_9.0.0.92188/private/thirdparty/python3/python3_3.7.3/site-packages/kineticsTools/WorkerProcess.py", line 151, in run
    self._run()
  File "/ebio/abt3_projects/software/bin/smrtlink/install/smrtlink-release_9.0.0.92188/bundles/smrttools/install/smrttools-release_9.0.0.92188/private/thirdparty/python3/python3_3.7.3/site-packages/kineticsTools/WorkerProcess.py", line 128, in _run
    datum)  # pylint: disable=assignment-from-none
  File "/ebio/abt3_projects/software/bin/smrtlink/install/smrtlink-release_9.0.0.92188/bundles/smrttools/install/smrttools-release_9.0.0.92188/private/thirdparty/python3/python3_3.7.3/site-packages/kineticsTools/KineticWorker.py", line 155, in onChunk
    modDict = dict((x['tpl'], x) for x in mods if start <=
UnboundLocalError: local variable 'mods' referenced before assignment
Child process exited with exitcode=1.  Aborting.

I've installed ipdSummary via:

wget https://downloads.pacbcloud.com/public/software/installers/smrtlink_9.0.0.92188.zip && unzip *.zip
 ./smrtlink_9.0.0.92188.run --rootdir  /path/to/install/location/ --install --smrttools-only

The amount of input sequence data is fairly low, which might be causing the error, but still, the error should then be more informative if that is the case.

[natechols]

Yeah, I suspect it's ending up with an uninitialized variable because of insufficient coverage, but still a bug in any case. I'll take a look.

[GDelevoye]

Quick suggestion here: The standard deviation can only be computed if coverage >= 3

[nick-youngblut]

@GDelevoye that doesn't explain why I get the error when using --maxCoverage 100 but not when using --maxCoverage 1

[natechols]

@nick-youngblut Looks like the --methylFraction option got broken in the process of the py3 porting. Sorry! I'll post a diff here once I have a fix.

[natechols]

@nick-youngblut Try this patch:

diff --git a/kineticsTools/MultiSiteCommon.py b/kineticsTools/MultiSiteCommon.py
index 96d06e8..aed1350 100644
--- a/kineticsTools/MultiSiteCommon.py
+++ b/kineticsTools/MultiSiteCommon.py
@@ -143,7 +143,7 @@ class MultiSiteCommon(object):
     def getContextMeans(self, start, end, sequence):
         meanVector = []
         for pos in range(start, end + 1):
-            ctx = sequence[(pos - self.pre):(pos + self.post + 1)].tostring()
+            ctx = sequence[(pos - self.pre):(pos + self.post + 1)].tostring().decode("ascii")
             if ctx in self.contextMeanTable:
                 meanVector.append(self.contextMeanTable[ctx])
             else:

That's in the file /ebio/abt3_projects/software/bin/smrtlink/install/smrtlink-release_9.0.0.92188/bundles/smrttools/install/smrttools-release_9.0.0.92188/private/thirdparty/python3/python3_3.7.3/site-packages/kineticsTools/MultiSiteCommon.py. (It fixes the crash using test inputs, but I haven't tried with real data yet.)