szte.csd
Class Tokenizer

java.lang.Object
  extended by szte.csd.Tokenizer

public class Tokenizer
extends java.lang.Object

Tokenizer is an utility class for task-dependent tokenization and lemmatization.


Constructor Summary
Tokenizer()
           
Tokenizer(edu.stanford.nlp.process.DocumentPreprocessor preproc)
           
 
Method Summary
 void lemmatiseOn()
           
 void serialize()
           
 java.util.List<Sentence> tokenise(java.lang.String text)
           
 
Methods inherited from class java.lang.Object
equals, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait
 

Constructor Detail

Tokenizer

public Tokenizer()

Tokenizer

public Tokenizer(edu.stanford.nlp.process.DocumentPreprocessor preproc)
Method Detail

lemmatiseOn

public void lemmatiseOn()

tokenise

public java.util.List<Sentence> tokenise(java.lang.String text)

serialize

public void serialize()