GenerationModel

Status

This page is under development. Its objective is to show how a trigram model for generation ranking can be built. The first part is overlapping with DelphinTools and RedwoodsTop, and the second part describes a procedure made by ErikVelldal.

Preliminaries

The procedure requires three tools:

• The CMU Toolkit ([http://mi.eng.cam.ac.uk/~prc14/toolkit.html]),

• SMT QuickRun ([http://ufal.ms.mff.cuni.cz/~curin/SMT_QuickRun/]), and

• DelphinTools.

The installation of the CMU Toolkit may conflict with Logon, so you may need to comment out the Logon settings in the .bashrc temporarily file while the installation is going on.

Exporting trigrams from Logon

$ TSDBHOME=$LOGONROOT/lingo/lkb/src/tsdb/home
$ export PATH=$LOGONROOT/lingo/lkb/src/tsdb/home:$PATH

Creating the profiles

This step is needed if you do not have a profile. It shows how you can get a profile from the object sentences of a bilingual corpus file. (See also DelphinTools.) The first command below creates a new version of the bilingual corpus, where the object language comes first. The second command parses the sentences in the object language and stores the MRSs in 'bitxt.'

$ logon_do --bitext PATH/TO/BILINGUAL/CORPUS/FILE $TSDBHOME/bitxt
$ logon_do --count 2 --limit 5 --grammar jaen --task omrs $TSDBHOME/bitxt

Exporting triples from the profiles

This command extracts triples from the profile created above.

$ $TSDBHOME/export --binary  --condition "result-id=0" --format triples bitxt/omrs

Creating the generation model

This is ErikVelldals procedure for creating a generation model, and his comments are given below. (I (PetterHaugereid) have slightly modified the commands.)

Removing formatting

First we remove all formating inserted by the export code (to get only the tuples) and cat everything to a single file. Note that, in the pipe below, the script from the SMT_QuickRun package only inserts the "context cues" used by the CMU SLM toolkit, ie. the sentence boundaries <s> and </s>.

$ export PATH=~/logon/petterha/tools/SMT_QuickRun1.2/bin:$PATH
$ find $LOGONROOT/tmp/bitxt.omrs/ -name *.gz | xargs zcat | awk '!/(^[\;\{\}\[]|^[[:space:]]*$)/' | add_sent_marks.prl | gzip > /tmp/mrstuples.gz

Context cues

Produce a file holding the context cues, to be referenced by the CMU toolkit.

export PATH=~/logon/petterha/tools/CMU-Cam_Toolkit_v2/bin:$PATH
echo "<s>" > ccs; echo "</s>" >> ccs

Extract the vocabulary

zcat /tmp/mrstuples.gz | text2wfreq > mrs.wfreq; cat mrs.wfreq | wfreq2vocab -top 65535 > mrs.vocab

Train the model

zcat /tmp/mrstuples.gz | text2idngram -temp /tmp/ -n 3 -vocab mrs.vocab > mrs.idngram; idngram2lm -idngram mrs.idngram -n 3 -vocab mrs.vocab -binary mrs.binlm -calc_mem -context ccs -witten_bell