#+TITLE: DAG transducer
#+DATE:
#+AUTHOR:
#+options: toc:nil

This project is the two-part system for our ACL2018 submission.

* Dataset
   There are three direcotry in directory ~data/export~. 
   - ~wsj00a~ is training data sample.
   - ~wsj20a~ is dev dataset sample
   - ~wsj21a~ is test dataset sample

   In directory ~output~, there are four files. They are the
   generation result of our complete experiment.
   - ~output/test-pred.txt~ is the final sentences
   - ~output/test-output.txt~ is the final lemma sequences
   - ~output/test-sentences.txt~ is the standard sentences
   - ~output/test-lemmas.txt~ is the standard lemma sequences


* Requirements
   - Python3 3.5 or newer
   - CMake 2.8 or newer
   - Modern C++ compiler that supports C++14
   - C++ Boost Library 1.58 or newer
   - Git
   - Torch (use for OpenNMT)

   Run following command in project root to install python packages
   #+begin_src sh
   pip install -r requirements.txt > /dev/null
   #+end_src


* Rule Extraction
   Run following command in project root:
   #+begin_src sh
   bash scripts/extract_rules
   #+end_src

   Or run following commands step by step:
   #+begin_src sh
   # Dump gzip data files to binary format
   python3 rule_extraction/reader.py
   # Generate intermediate graphs for train data
   python3 rule_extraction/data_stream.py
   # Extract induced-rules
   python3 rule_extraction/extract_compiled_rules.py
   # Generate extended-rules
   python3 rule_extraction/extend_rules.py
   #+end_src

   Serveral files will be generated in directory
   ~data~. ~data/rules.txt~ is the rules file for reading and
   ~data/compiled-rules.txt~ is the rules file for transducer.
   
   Every rule in ~rules.txt~ is like:
   #+begin_example
   {NP_VP:ARG1(a0, a1)} => _v_1 => {NP:ARG2(b0), NP:ARG1(c0)}
   a0 = c0
   a1 = "to" + L + b0
   #+end_example
   - Variables are directly put after state labels.
   - Direction of state is not displayed since we can induce it from
     statements.


* DAG transducer
   Run following command in project root to build and train DAG transducer:
   #+begin_src sh
   bash scripts/train_transducer
   #+end_src

   Or run following commands step by step:
   #+begin_src sh
   mkdir build && cd build
   cmake ../transducer -DCMAKE_BUILD_TYPE=Release && make
   cd ..
   # Train the main perception for selecting rules
   build/Release/transducer -c scripts/train.config -m data_driven
   # Train the perception for generator dynamic-rules
   build/Release/transducer -c scripts/train.config -m data_driven.generator
   # Transduce test dataset
   build/Release/transducer -c scripts/transduce.config -m data_driven
   #+end_src

   Three files will be generated in directory ~data~.
   - Two feature files: ~data/train.feat~ and ~data/dynamic.feat~
   - Transduction result for test dataset: ~data/test-output.txt~


* Morphology Generator
   First follw link [[http://torch.ch/docs/getting-started.html][Install Torch]] to install Torch. Make sure
   executable `th' is in =PATH=. Then install OpenNMT by running
   following commands in project root:
   #+begin_src sh
   luarocks install tds
   luarocks install bit32 # if using LuaJIT
   git clone https://github.com/OpenNMT/OpenNMT
   #+end_src

   Then run following command in project root:
   #+begin_src sh
   bash scripts/opennmt_run
   #+end_src

   Or run following commands step by step:
   #+begin_src sh
   # Generate standard sequence of lemmas for training
   python3 rule_extraction/process_sentence.py
   bash scripts/opennmt_prepare_data
   bash scripts/opennmt_train
   #+end_src

   The final model file will is ~data/final-model_epoch*.t7~. Run
   following command to generated final sentences:
   #+begin_src sh
   bash scripts/opennmt_infer data/final-model_epoch*.t7
   #+end_src

   - ~data/test-pred.txt~ is the output for test dataset
   - ~data/dev-pred.txt~ is the output for dev dataset
