Conventional n-best reranking techniques often suffer from the limited scope of the n-best list, which rules out many potentially good alternatives.
We instead propose forest reranking, a method that reranks a packed forest of exponentially many parses.
Since exact inference is intractable with non-local features, we present an approximate algorithm inspired by forest rescoring that makes discriminative training practical over the whole Tree-bank.
Our final result, an F-score of 91.7, outperforms both 50-best and 100-best reranking baselines, and is better than any previously reported systems trained on the Treebank.
1 Introduction
Discriminative reranking has become a popular technique for many NLP problems, in particular, parsing (Collins, 2000) and machine translation (Shen et al., 2005).
Typically, this method first generates a list of top-n candidates from a baseline system, and then reranks this n-best list with arbitrary features that are not computable or intractable to compute within the baseline system.
But despite its apparent success, there remains a major drawback: this method suffers from the limited scope of the n-best list, which rules out many potentially good alternatives.
For example 41% of the correct parses were not in the candidates of ~30-best parses in (Collins, 2000).
This situation becomes worse with longer sentences because the number of possible interpretations usually grows exponentially with the
* Part of this work was done while I was visiting Institute of Computing Technology, Beijing, and I thank Prof. Qun Liu and his lab for hosting me.
I am also grateful to Dan Gildea and Mark Johnson for inspirations, Eugene Charniak for help with his parser, and Wenbin Jiang for guidance on perceptron averaging.
This project was supported by NSF ITR EIA-0205456.
local non-local conventional reranking only at the root
DP-based discrim. parsing exact N/A this work: forest-reranking exact on-the-fly
Table 1: Comparison of various approaches for incorporating local and non-local features.
sentence length.
As a result, we often see very few variations among the n-best trees, for example, 50-best trees typically just represent a combination of5 to 6 binary ambiguities (since 25 < 50 < 26).
Alternatively, discriminative parsing is tractable with exact and efficient search based on dynamic programming (DP) if all features are restricted to be local, that is, only looking at a local window within the factored search space (Taskar et al., 2004; McDonald et al., 2005).
However, we miss the benefits of non-local features that are not representable here.
Ideally, we would wish to combine the merits of both approaches, where an efficient inference algorithm could integrate both local and non-local features.
Unfortunately, exact search is intractable (at least in theory) for features with unbounded scope.
So we propose forest reranking, a technique inspired by forest rescoring (Huang and Chiang, 2007) that approximately reranks the packed forest of exponentially many parses.
The key idea is to compute non-local features incrementally from bottom up, so that we can rerank the n-best subtrees at all internal nodes, instead of only at the root node as in conventional reranking (see Table 1).
This method can thus be viewed as a step towards the integration of discriminative reranking with traditional chart parsing.
Figure 1: A partial forest of the example sentence.
we achieved an F-score of 91.7, which is a 19% error reduction from the 1-best baseline, and outperforms both 50-best and 100-best reranking.
This result is also better than any previously reported systems trained on the Treebank.
2 Packed Forests as Hypergraphs
Informally, a packed parse forest, or forest in short, is a compact representation of all the derivations (i.e., parse trees) for a given sentence under a context-free grammar (Billot and Lang, 1989).
For example, consider the following sentence
where the numbers between words denote string positions.
Shown in Figure 1, this sentence has (at least) two derivations depending on the attachment of the prep. phrase PP^6 "with a mirror": it can either be attached to the verb "saw",
or be attached to "him", which will be further combined with the verb to form the same VP as above.
These two derivations can be represented as a single forest by sharing common sub-derivations.
Such a forest has a structure of a hypergraph (Klein and Manning, 2001; Huang and Chiang, 2005), where items like PP3;6 are called nodes, and deductive steps like (*) correspond to hyperedges.
head(e) G V is the consequent node in the deductive step, and tails(e) G V* is the list of antecedent nodes.
For example, the hyperedge for deduction (*) is notated:
We also denote IN (v) to be the set of incoming hyperedges of node v, which represent the different ways of deriving v. For example, in the forest in Figure 1, IN(VPi;6) is {ei,e2}, with e2 = ((VBDi,2, NP2,6), VPi,6).
We call |e| the arity of hyperedge e, which counts the number of tail nodes in e. The arity of a hypergraph is the maximum ar-ity over all hyperedges.
A CKY forest has an arity of 2, since the input grammar is required to be binary branching (cf. Chomsky Normal Form) to ensure cubic time parsing complexity.
However, in this work, we use forests from a Treebank parser (Char-niak, 2000) whose grammar is often flat in many productions.
For example, the arity of the forest in Figure 1 is 3.
Such a Treebank-style forest is easier to work with for reranking, since many features can be directly expressed in it.
There is also a distinguished root node TOP in each forest, denoting the goal item in parsing, which is simply So,i where S is the start symbol and l is the sentence length.
3 Forest Reranking
3.1 Generic Reranking with the Perceptron
We first establish a unified framework for parse reranking with both n-best lists and packed forests.
For a given sentence s, a generic reranker selects the best parse y among the set of candidates cand (s) according to some scoring function:
In n-best reranking, cand(s) is simply a set of n-best parses from the baseline parser, that is, cand(s) = {yi, y2,..., yn}.
Whereas in forest reranking, cand(s) is a forest implicitly representing the set of exponentially many parses.
As usual, we define the score of a parse y to be the dot product between a high dimensional feature representation and a weight vector w:
which is an instance of the WordEdges feature (see Figure 2(c) and Section 3.2 for details).
Using a machine learning algorithm, the weight vector w can be estimated from the training data where each sentence si is labelled with its correct ("gold-standard") parse y*.
As for the learner, Collins (2000) uses the boosting algorithm and Charniak and Johnson (2005) use the maximum entropy estimator.
In this work we use the averaged perceptron algorithm (Collins, 2002) since it is an online algorithm much simpler and orders ofmagni-tude faster than Boosting and MaxEnt methods.
Shown in Pseudocode 1, the perceptron algorithm makes several passes over the whole training data, and in each iteration, for each sentence Si, it tries to predict a best parse yi among the candidates cand(si) using the current weight setting.
Intuitively, we want the gold parse yi* to be picked, but in general it is not guaranteed to be within cand (si), because the grammar may fail to cover the gold parse, and because the gold parse may be pruned away due to the limited scope of cand(si).
So we define an oracle parse y+ to be the candidate that has the highest Parseval F-score with respect to the gold tree y *:1
where function F returns the F-score.
Now we train the reranker to pick the oracle parses as often as possible, and in case an error is made (line 6), perform an update on the weight vector (line 7), by adding the difference between two feature representations.
1If one uses the gold yl for oracle y+, the perceptron will continue to make updates towards something unreachable even when the decoder has picked the best possible candidate.
Pseudocode 1 Perceptron for Generic Reranking
oracle tree for si among cand(si)
In n-best reranking, since all parses are explicitly enumerated, it is trivial to compute the oracle tree.2 However, it remains widely open how to identify the forest oracle.
We will present a dynamic programming algorithm for this problem in Sec.
4.1.
We also use a refinement called "averaged parameters" where the final weight vector is the average of weight vectors after each sentence in each iteration over the training data.
This averaging effect has been shown to reduce overfitting and produce much more stable results (Collins, 2002).
3.2 Factorizing Local and Non-Local Features
A key difference between n-best and forest reranking is the handling of features.
In n-best reranking, all features are treated equivalently by the decoder, which simply computes the value of each one on each candidate parse.
However, for forest reranking, since the trees are not explicitly enumerated, many features can not be directly computed.
So we first classify features into local and non-local, which the decoder will process in very different fashions.
We define a feature / to be local if and only if it can be factored among the local productions in a tree, and non-local if otherwise.
For example, the Rule feature in Fig.
2(a) is local, while the Paren-tRule feature in Fig.
2(b) is non-local.
It is worth noting that some features which seem complicated at the first sight are indeed local.
For example, the WordEdges feature in Fig.
2(c), which classifies a node by its label, span length, and surrounding words, is still local since all these information are encoded either in the node itself or in the input sentence.
In contrast, it would become non-local if we replace the surrounding words by surrounding POS
2In case multiple candidates get the same highest F-score, we choose the parse with the highest log probability from the baseline parser to be the oracle parse (Collins, 2000).
VBD NP PP
Figure 2: Illustration of some example features.
Shaded nodes denote information included in the feature.
tags, which are generated dynamically.
More formally, we split the feature extractor f = (fi,..., fd) into f = (fL; fN) where fL and fv are the local and non-local features, respectively.
For the former, we extend their domains from parses to hy-peredges, where f (e) returns the value of a local feature f G fL on hyperedge e, and its value on a parsey factors across the hyperedges (local productions),
and we can pre-compute fL(e) for each e in a forest.
Non-local features, however, can not be pre-computed, but we still prefer to compute them as early as possible, which we call "on-the-fly" computation, so that our decoder can be sensitive to them at internal nodes.
For instance, the NGramTree feature in Fig.
2 (d) returns the minimum tree fragement spanning a bigram, in this case "saw" and "the", and should thus be computed at the smallest common ancestor of the two, which is the VP node in this example.
Similarly, the ParentRule feature in Fig.
2 (b) can be computed when the S subtree is formed.
In doing so, we essentially factor non-local features across subtrees, where for each subtree y' in a parse y, we define a unit feature f(y') to be the part of
Intuitively, we compute the unit non-local features at each subtree from bottom-up.
For example, for the binary-branching node Ai k in Fig.
3, the
Figure 3: Example of the unit NGramTree feature at node Ai k: ( A (B ... wj_1) (C ...
Wj)).
unit NGramTree instance is for the pair (wj_1, Wj) on the boundary between the two subtrees, whose smallest common ancestor is the current node.
Other unit NGramTree instances within this span have already been computed in the subtrees, except those for the boundary words of the whole node, Wi and Wk-1, which will be computed when this node is further combined with other nodes in the future.
3.3 Approximate Decoding via Cube Pruning
Before moving on to approximate decoding with non-local features, we first describe the algorithm for exact decoding when only local features are present, where many concepts and notations will be re-used later.
We will use D(v) to denote the top derivations of node v, where is its 1-best
derivation.
We also use the notation (e, j) to denote the derivation along hyperedge e, using the jith sub-derivation for tail ui, so (e, 1) is the best derivation along e. The exact decoding algorithm, shown in Pseudocode 2, is an instance of the bottom-up Viterbi algorithm, which traverses the hypergraph in a topological order, and at each node v, calculates its 1-best derivation using each incoming hyperedge e G IN(v).
The cost of e, c(e), is the score of its
Pseudocode 2 Exact Decoding with Local Features
Pseudocode 3 Cube Pruning for Non-local Features
19: j' — j + bl > bl is 1 only on the ith dim.
(pre-computed) local features w • fL(e).
This algorithm has a time complexity of O(E), and is almost identical to traditional chart parsing, except that the forest might be more than binary-branching.
For non-local features, we adapt cube pruning from forest rescoring (Chiang, 2007; Huang and Chiang, 2007), since the situation here is analogous to machine translation decoding with integrated language models: we can view the scores of unit nonlocal features as the language model cost, computed on-the-fly when combining sub-constituents.
Shown in Pseudocode 3, cube pruning works bottom-up on the forest, keeping a beam of at most k derivations at each node, and uses the k-best parsing Algorithm 2 of Huang and Chiang (2005) to speed up the computation.
When combining the sub-
derivations along a hyperedge e to form a new subtree y' = ( e, j) , we also compute its unit non-local feature values fN((e, j)) (line 25).
A priority queue (heap in Pseudocode 3) is used to hold the candidates for the next-best derivation, which is initialized to the set of best derivations along each hyper-edge (lines 7 to 9).
Then at each iteration, we pop the best derivation (lines 12), and push its successors back into the priority queue (line 14).
Analogous to the language model cost in forest rescoring, the unit feature cost here is a non-monotonic score in the dynamic programming backbone, and the derivations may thus be extracted out-of-order.
So a buffer buf is used to hold extracted derivations, which is sorted at the end (line 15) to form the list of top-k derivations D(v) of node v. The complexity of this algorithm is O(E + Vk log kN) (Huang and Chiang, 2005), where O(N) is the time for on-the-fly feature extraction for each subtree, which becomes the bottleneck in practice.
4 Supporting Forest Algorithms
Recall that the Parseval F-score is the harmonic mean of labelled precision P and labelled recall R:
where y and y* are the numbers of brackets in the test parse and gold parse, respectively, and |y n y*| is the number of matched brackets.
Since the harmonic mean is a non-linear combination, we can not optimize the F-scores on sub-forests independently with a greedy algorithm.
In other words, the optimal F-score tree in a forest is not guaranteed to be composed of two optimal F-score subtrees.
We instead propose a dynamic programming algorithm which optimizes the number of matched brackets for a given number of test brackets.
For example, our algorithm will ask questions like,
"when a test parse has 5 brackets, what is the maximum number of matched brackets? "
Pseudocode 4 Forest Oracle Algorithm
for all parses yv of node v with exactly t brackets:
When node v is combined with another node u along a hyperedge e = ((v, u), w), we need to combine the two oracle functions ora [v] and ora [u] by distributing the test brackets of w between v and u, and optimize the number of matched bracktes.
To do this we define a convolution operator <g> between two functions f and g:
For instance:
The oracle function for the head node w is then
where 1 is the indicator function, returning 1 if node w is found in the gold tree y*, in which case we increment the number of matched brackets.
We can also express Eq.
9 in a purely functional form
where f is a translation operator which shifts a function along the axes:
Above we discussed the case of one hyperedge.
If there is another hyperedge e' deriving node w, we also need to combine the resulting oracle functions from both hyperedges, for which we define a point-wise addition operator ©:
Shown in Pseudocode 4, we perform these computations in a bottom-up topological order, and finally at the root node TOP, we can compute the best global F-score by maximizing over different numbers of test brackets (line 7).
The oracle tree y+ can be recursively restored by keeping backpointers for each ora[v](t), which we omit in the pseudocode.
The time complexity of this algorithm for a sentence of l words is O(|E| • 12(a-1)) where a is the arity of the forest.
For a CKY forest, this amounts to O(13 • l2) = O(15), but for general forests like those in our experiments the complexities are much higher.
In practice it takes on average 0.05 seconds for forests pruned by p = 10 (see Section 4.2), but we can pre-compute and store the oracle for each forest before training starts.
Our forest pruning algorithm (Jonathan Graehl, p.c.) is very similar to the method based on marginal probability (Charniak and Johnson, 2005), except that ours prunes hyperedges as well as nodes.
Basically, we use an Inside-Outside algorithm to compute the Viterbi inside cost /?
(v) and the Viterbi outside cost a(v) for each node v, and then compute the merit a/(e) for each hyperedge:
Intuitively, this merit is the cost of the best derivation that traverses e, and the difference 5(e) = a//(e) — //(TOP) can be seen as the distance away from the globally best derivation.
We prune away all hyperedges that have 5(e) > p for a threshold p. Nodes with all incoming hyperedges pruned are also pruned.
The key difference from (Charniak and Johnson, 2005) is that in this algorithm, a node can "partially" survive the beam, with a subset of its hyperedges pruned.
In practice, this method prunes on average 15% more hyperedges than their method.
5 Experiments
We compare the performance of our forest reranker against n-best reranking on the Penn English Tree-bank (Marcus et al., 1993).
The baseline parser is the Charniak parser, which we modified to output a
Non-Local
instances
ParentRule
WordEdges
CoLenPar
HeadTree
Trigram0
NGramTree
HeadMod0
RightBranch
DistMod0
Table 2: Features used in this work.
Those with a ° are from (Collins, 2000), and others are from (Char-niak and Johnson, 2005), with simplifications.
packed forest for each sentence.3
We use the standard split of the Treebank: sections 02-21 as the training data (39832 sentences), section 22 as the development set (1700 sentences), and section 23 as the test set (2416 sentences).
Following (Charniak and Johnson, 2005), the training set is split into 20 folds, each containing about 1992 sentences, and is parsed by the Charniak parser with a model trained on sentences from the remaining 19 folds.
The development set and the test set are parsed with a model trained on all 39832 training sentences.
We implemented both n-best and forest reranking systems in Python and ran our experiments on a 64bit Dual-Core Intel Xeon with 3.0GHz CPUs.
Our feature set is summarized in Table 2, which closely follows Charniak and Johnson (2005), except that we excluded the non-local features Edges, NGram, and CoPar, and simplified Rule and NGramTree features, since they were too complicated to com-pute.4 We also added four unlexicalized local features from Collins (2000) to cope with data-sparsity.
3This is a relatively minor change to the Charniak parser, since it implements Algorithm 3 of Huang and Chiang (2005) for efficient enumeration of n-best parses, which requires storing the forest.
The modified parser and related scripts for handling forests (e.g. oracles) will be available on my homepage.
4In fact, our Rule and ParentRule features are two special cases of the original Rule feature in (Charniak and Johnson, 2005).
We also restricted NGramTree to be on bigrams only.
forest oracle
Figure 4: Forests (shown with various pruning thresholds) enjoy higher oracle scores and more compact sizes than n-best lists (on sec 23).
tures in the updated version.5 However, our initial experiments show that, even with this much simpler feature set, our 50-best reranker performed equally well as theirs (both with an F-score of 91.4, see Tables 3 and 4).
This result confirms that our feature set design is appropriate, and the averaged percep-tron learner is a reasonable candidate for reranking.
The forests dumped from the Charniak parser are huge in size, so we use the forest pruning algorithm in Section 4.2 to prune them down to a reasonable size.
In the following experiments we use a threshold of p = 10, which results in forests with an average number of 123.1 hyperedges per forest.
Then for each forest, we annotate its forest oracle, and on each hyperedge, pre-compute its local features.6 Shown in Figure 4, these forests have an forest oracle of 97.8, which is 1.1% higher than the 50-best oracle (96.7), and are 8 times smaller in size.
Table 3 compares the performance of forest rerank-ing against standard n-best reranking.
For both systems, we first use only the local features, and then all the features.
We use the development set to determine the optimal number of iterations for averaged perceptron, and report the F1 score on the test set.
With only local features, our forest reranker achieves an F-score of 91.25, and with the addition of non-
5http://www.cog.brown.edu/~mj/software.htm.
We follow this version as it corrects some bugs from their 2005 paper which leads to a 0.4% increase in performance (see Table 4).
6A subset oflocal features, e.g. WordEdges, is independent ofwhich hyperedge the node takes in a derivation, and can thus be annotated on nodes rather than hyperedges.
We call these features node-local, which also include part ofWord features.
n-best oracle
n-best reranking
forest reranking
features
pre-comp.
training
Table 3: Forestrerankingcomparedton-bestrerank-ing on sec.
23.
The pre-comp. column is for feature extraction, and training column shows the number of perceptron iterations that achieved best results on the dev set, and average time per iteration.
This improvement might look relatively small, but it is much harder to make a similar progress with n-best reranking.
For example, even if we double the size of the n-best list to 100, the performance
best oracle is only 0.5% higher than the 50-best one (see Fig.
In addition, the feature extraction step in 100-best reranking produces huge data files and takes 44 hours in total, though this part can be paral-lelized.8 On two CPUs, 100-best reranking takes 25 hours, while our forest-reranker can also finish in 26 hours, with a much smaller disk space.
Indeed, this demonstrates the severe redundancies as another disadvantage of n-best lists, where many subtrees are repeated across different parses, while the packed forest reduces space dramatically by sharing common sub-derivations (see Fig.
4).
To put our results in perspective, we also compare them with other best-performing systems in Table 4.
Our final result (91.7) is better than any previously reported system trained on the Treebank, although
7It is surprising that 50-best reranking with local features achieves an even higher F-score of 91.28, and we suspect this is due to the aggressive updates and instability of the perceptron, as we do observe the learning curves to be non-monotonic.
We leave the use ofmore stable learning algorithms to future work.
8The n-best feature extraction already uses relative counts (Johnson, 2006), which reduced file sizes by at least a factor 4.
this work
Table 4: Comparison of our final results with other best-performing systems on the whole Section 23.
Types D, G, and S denote discriminative, generative, and semi-supervised approaches, respectively.
McClosky et al. (2006) achieved an even higher ac-cuarcy (92.1) by leveraging on much larger unla-belled data.
Moreover, their technique is orthogonal to ours, and we suspect that replacing their n-best reranker by our forest reranker might get an even better performance.
Plus, except for n-best rerank-ing, most discriminative methods require repeated parsing of the training set, which is generally im-pratical (Petrov and Klein, 200S).
Therefore, previous work often resorts to extremely short sentences (< 15 words) or only looked at local features (Taskar et al., 2004; Henderson, 2004; Turian and Melamed, 200V).
In comparison, thanks to the efficient decoding, our work not only scaled to the whole Treebank, but also successfully incorporated non-local features, which showed an absolute im-provementof 0.44% over that of local features alone.
6 Conclusion
We have presented a framework for reranking on packed forests which compactly encodes many more candidates than n-best lists.
With efficient approximate decoding, perceptron training on the whole Treebank becomes practical, which can be done in about a day even with a Python implementation.
Our final result outperforms both 50-best and 100-best reranking baselines, and is better than any previously reported systems trained on the Treebank.
We also devised a dynamic programming algorithm for forest oracles, an interesting problem by itself.
We believe this general framework could also be applied to other problems involving forests or lattices, such as sequence labeling and machine translation.
