System requirements for training indian accent english over deepspeech pre trained model checkpoints

[JRMeyer]

>>> pra978
[March 28, 2018, 7:28am]

DeepSpeech does not perform great with Indian accent English. I am
trying to train it with more than 5000 audio files of Indian English
accent each a length of 5 sec.

What should be my system requirements? slash
Any idea how much time it will take?

[This is an archived TTS discussion thread from discourse.mozilla.org/t/system-requirements-for-training-indian-accent-english-over-deepspeech-pre-trained-model-checkpoints]

[JRMeyer]

>>> kdavis
[March 28, 2018, 10:08am]

I'm sure it will work with an Indian accent. It can work with various
accents and languages. So I doubt that an Indian accent will trip it up.
I think the issue is hyperparameters + data size + data quality.

Could you describe what you are doing in more detail?

[Archived Post]

[JRMeyer]

>>> pra978
[March 28, 2018, 11:02am]

Sorry. I think i was not clear in what i wanted to convey.

I tried DeepSpeech pre-trained model on some Indian accent audio files,
however it gave a WER of around slash ~35%.

To reduce this WER, i plan to train this model at checkpoints or with
frozen model approach with Indian accent audios ( slash ~ 10 hours).

My system specs are a 10 GB RAM MAC with 1.5 GB Intel Graphics card and
around 100 GB empty storage. slash
Will this be enough? Or i will have to use a different system with
better specs?

[Archived Post]

[JRMeyer]

>>> kdavis
[March 28, 2018, 5:45pm]

Oh, I see now.

I agree the pre-trained model is unlikely to work well on Indian
accented English and needs to be 'fine-tuned', as it looks like you are
about to do.

Unfortunately, Intel Graphics cards are not supported, a TensorFlow
limitation. So training will happen only on the CPU and likely take a
long time.

I'd suggest trying to get your hands on something like a 1080Ti or more
powerful if possible.

[Archived Post]

[JRMeyer]

>>> pra978
[April 3, 2018, 6:24am]

Yes, I am planning to fine-tune the model using the checkpoints.

I have around 88000 audio files ( slash ~5s in length each).

I am not able to arrange for a higher capability system for now. Is it
possible for me to fine-tune the pre-trained model using these files on
Google cloud/amazon AWS?

[Archived Post]

[JRMeyer]

>>> kdavis
[April 3, 2018, 6:48am]

Yes it should work on Google
cloud or Amazon AWS.

[Archived Post]

[JRMeyer]

>>> pra978
[April 10, 2018, 8:04pm]

, I am getting this
keyerror - 'ctcbeamsearchdecoderlm' while trying to restore the
checkpoints. Anyone has any idea about how to resolve this?

[Archived Post]

[JRMeyer]

>>> kdavis
[April 11, 2018, 7:04am]

Could you give the full text
of the error? It will help in debugging

[Archived Post]

[JRMeyer]

>>> pra978
[April 11, 2018, 8:06am]

i ran this for trying to import meta file from latest release:

saver = tf.train.import_meta_graph('model.ckpt-97999.meta')

and i got this error:

KeyError Traceback (most recent call last) slash
in () slash
slash - slash - slash -- slash > 1 saver = tf.train.import_meta_graph('model.ckpt-97999.meta')

/anaconda3/lib/python3.6/site-packages/tensorflow/python/training/saver.py
in import_meta_graph(meta_graph_or_file, clear_devices, import_scope,
slash DEEPSPEECH.cdx deepspeech.commands DEEPSPEECH.pages DEEPSPEECH.warc.gz discourse.mozilla.org html-to-markdown.sh shell-conver-html-to-split-posts.sh sorted-deepspeech-posts slash *kwargs) slash
1836 clear_devices=clear_devices, slash
1837 import_scope=import_scope, slash
1839 if meta_graph_def.HasField('saver_def'): slash
1840 return Saver(saver_def=meta_graph_def.saver_def, name=import_scope)

/anaconda3/lib/python3.6/site-packages/tensorflow/python/framework/meta_graph.py
in import_scoped_meta_graph(meta_graph_or_file, clear_devices, graph,
import_scope, input_map, unbound_inputs_col_name,
restore_collections_predicate) slash
658 importer.import_graph_def( slash
659 input_graph_def, name=(import_scope or ''), input_map=input_map, slash
661 slash
662 scope_to_prepend_to_names = '/'.join(

/anaconda3/lib/python3.6/site-packages/tensorflow/python/util/deprecation.py
in new_func( slash *args, slash DEEPSPEECH.cdx deepspeech.commands DEEPSPEECH.pages DEEPSPEECH.warc.gz discourse.mozilla.org html-to-markdown.sh shell-conver-html-to-split-posts.sh sorted-deepspeech-posts slash *kwargs) slash
314 'in a future version' if date is None else ('after %s' % date), slash
315 instructions) slash
317 return tf_decorator.make_decorator(func, new_func, 'deprecated', slash
318 slash _add_deprecated_arg_notice_to_docstring(

/anaconda3/lib/python3.6/site-packages/tensorflow/python/framework/importer.py
in import_graph_def(graph_def, input_map, return_elements, name,
op_dict, producer_op_list) slash
431 if producer_op_list is not None: slash
432 slash # TODO(skyewm): make a copy of graph_def so we're not mutating the
argument? slash
434 slash
435 graph = ops.get_default_graph()

/anaconda3/lib/python3.6/site-packages/tensorflow/python/framework/importer.py
in slash _RemoveDefaultAttrs(op_dict, producer_op_list, graph_def) slash
209 slash # Remove any default attr values that aren't in op_def. slash
210 if node.op in producer_op_dict: slash
212 producer_op_def = producer_op_dict slash [node.op slash ] slash
213 slash # We make a copy of node.attr to iterate through since we may
modify

KeyError: 'CTCBeamSearchDecoderWithLM'

[Archived Post]

[JRMeyer]

>>> pra978
[April 11, 2018, 10:56am]

I am trying to further train this model at checkpoints using Indian
accent datasets on a mac (macOS High Sierra 10.13.2). Will
'CTCBeamSearchDecoderWithLM' work on this?

[Archived Post]

[JRMeyer]

>>> kdavis
[April 12, 2018, 7:00am]

What command line arguments did you give DeepSpeech.py?

[Archived Post]

[JRMeyer]

>>> pra978
[April 13, 2018, 9:50am]

I didn't give any command line arguments to DeepSpeech.py.

I just ran this in jupyter Notebook :

saver = tf.train.import_meta_graph('model.ckpt-97999.meta')

and got this error:

KeyError: 'CTCBeamSearchDecoderWithLM'

[Archived Post]

[JRMeyer]

>>> kdavis
[April 13, 2018, 2:10pm]

Could you provide a link to the notebook?

[Archived Post]

[JRMeyer]

>>> reuben
[April 13, 2018, 3:08pm]

If you're writing your own training code you'll need to load the decoder
module as the training graph depends on it. See tf.load_op_library.

[Archived Post]

[JRMeyer]

>>> pra978
[April 16, 2018, 7:52am]

i just gave these 3 commands in the jupyter notebook. Nothing else.

import tensorflow as tf slash
sess=tf.Session() slash
saver = tf.train.import_meta_graph('model.ckpt-97999.meta')

and got this error:

KeyError: 'CTCBeamSearchDecoderWithLM'

[Archived Post]

[JRMeyer]

>>> pra978
[April 16, 2018, 8:17am]

I am not writing my own training code, i am just trying to load the
pre-trained model with the help of checkpoints provided in latest
release. I am doing this so that i can train some Indian accent audios
on this afterwards.

Do i still need to load the decoder module?

[Archived Post]

[JRMeyer]

>>> reuben
[April 21, 2018, 1:21pm]

> Do i still need to load the decoder module?

Yes.

[Archived Post]

[JRMeyer]

>>> reuben
[April 21, 2018, 1:23pm]



> I am not writing my own training code, i am just trying to load the
> pre-trained model with the help of checkpoints provided in latest
> release. I am doing this so that i can train some Indian accent audios
> on this afterwards.

In that case, I don't understand why you're trying to load the
checkpoint with your own code. Just use DeepSpeech.py and specify
--checkpoint_dir and it'll load the decoder module and the weights
appropriately.

[Archived Post]

[JRMeyer]

>>> pra978
[April 21, 2018, 1:38pm]

Thanks a lot. I will try that for sure. I was supposed to use it in
terminal and I was trying to do it in python which I see is possible but
kind of a longer process to follow.

Since this was not working out, I was trying to create my own model. I
got all the files necessary based upon a French model tutorial from one
of the discourse. When I am running the model, I am getting text error
which as I came to know happens due to wrong encoding of text files. I
will keep you updated on that. Thanks
![:blush:](

[Archived Post]

[JRMeyer]

>>> qtran
[August 9, 2018, 10:52pm]

, How are you going with
this? I'm trying to do the same with Australian accent.

[next page
→

[Archived Post]