Release
===================

This package contains the code needed to replicate main results of
the EMNLP-13 paper *A Joint Learning Model of Word Segmentation, Lexical
Acquisition, and Phonetic Variability*, by Elsner, Goldwater,
Feldman and Wood. Please cite this paper when using the software.

An up-to-date version will be maintained at
`<https://bitbucket.org/melsner/beamseg>`_

Building
---------------

You'll need the Gnu Scientific Library and Boost. If you're not on a
64-bit machine, make directories *lib32* and *bin32*

Then type *make*

You'll get some warnings, but it should all build.

Running
---------------------

We've provided the transducer files used in this study.

For convenience, we also included a download of the modified-Brent
dataset files provided by Elsner et al at
`<http://www.ling.ohio-state.edu/~melsner/resources/acl12data.tgz>`_
and their readme file.

To run the model with EM::

  bin64/BeamSample --alpha 3000 --alpha 100 data/brentStest.surface --channel phone --read-channel data/initialChannel --write-channel output.channel --output output --em

This should replicate our main results in Table 1. Be aware that it will take
substantial time and memory. The beam sampler makes the program
*faster* but it doesn't make it *fast*.

To run the baseline, use *--channel none*. To run the oracle, use
*--read-channel data/oracleChannel*. To run unigrams, use::

  bin64/BeamSample --grams 1 --alpha 20 data/brentStest.surface --channel phone --read-channel data/oracleChannel --output output.unigrams

Notice that:

 * --grams sets the ngram size--- but gram sizes above 2 are not supported!
 * --alpha sets the alpha parameters of the Dirichlet processes; for bigrams, pass --alpha twice, with the first one being A0 and the second one A1
 * --channel selects the channel type
 * --read-channel selects a channel file (if needed)
 * --write-channel writes the learned channel
 * --em learns the channel
 * --lock-bounds does not sample word boundaries
 * --output selects the filename to write
 * --brent-reader reads data with one char per character, words delimited by spaces (as in the Goldwater distribution of Brent)
 * --help prints a list of command-line options (not all of them actually supported)

Scoring
---------------------

Your runs above should generate files *output.learned.surface* and
*output.learned.underlying*; the scorer tools take the stem *output*
and search for files with names in this format.

Run::

 python script/scoreSeg.py data/brentStest output

 1311 true words 1025 found mapped words 468 matched
 [not useful: unmapped tok] UP 47.78 UR 48.49 UF 48.13
 [maptok] MP 49.15 MR 49.88 MF 49.51
 [surf] SP 66.85 SR 67.84 SF 67.34
 [bds] BP 80.92 BR 82.63 BF 81.77
 [not useful: unmapped lex] LP 38.05 LR 29.75 LF 33.39
 [maplex] MLP 45.66 MLR 35.70 MLF 40.07

You can obtain the error analyses we use by running::

 python script/reportRealTokens.py data/brentStest output

Transducers
-------------------

Viewing transducers::

 bin64/PrintChannel data/oracleChannel

Making an oracle transducer::

 bin64/OracleTransducer --underlying data/brentStest.underlying data/brentStest.surface --channel-file data/initialChannel > oracleChannel

Making an initial transducer::

 bin64/InitialTransducer data/brentStest.surface > initialChannel

Test output
-------------------

We've given you our test output in the *test-runs* directory.

Warning
------------------

We didn't make much of an effort to remove all the developmental dead
ends, test options and so forth from this package. Apart from the
commands in this file, none of the other options for the various
scripts or programs are guaranteed to not crash, to do what they say
they do, or to do anything useful at all.

Code cleanups and more documentation will hopefully be forthcoming.
