N-gram

This missing data means that the N-gram matrix of any corpus may have a large amount of zeros that should be actually fi lled with some non-zero probability value.

Recently Uploaded Slideshows tizyweb 2009

Smoothing techniques for N-gram models The term smoothing refer to such modi fi cations in the MLE estimates of N - gram probabilities that are addressed to move some probability mass from higher counts to zero-counts, making the overall distribution less jagged (Figure 2.5).

Recently Uploaded Slideshows tizyweb 2009

Finally, Good-Turing smoothing has been applied only if the correspond - ing N-gram counts were below a given threshold K.

Recently Uploaded Slideshows tizyweb 2009

One of these is backo ff N-gram modeling, and in particular the Katz backo ff algorithm [34].

Recently Uploaded Slideshows tizyweb 2009

Laplace smoothing The simplest way to do smoothing is to take the matrix of N-gram counts, before normalization, and add one to all the counts.

Recently Uploaded Slideshows tizyweb 2009

In practice, the Good-Turing estimate is not used by itself for N-gram smoothing, because it does not include the combination of higher-order models with lower-order models that is necessary for obtaining good per - formances.

Recently Uploaded Slideshows tizyweb 2009

An MLE estimate for the N-gram model's parameters can be thus obtained normalizing counts from a training corpus 1.

Recently Uploaded Slideshows tizyweb 2009

Statistical language modeling for text prediction 17 Once the model is de fi ned, we need a method to estimate these N-gram probabilities.

Recently Uploaded Slideshows tizyweb 2009

Moreover N-gram models behave in a completely di ff erent way if the words 'ordering changes.

Recently Uploaded Slideshows tizyweb 2009

The basic idea is again to choose the values of λj which maximize the likelihood of 3 An held-out corpus is an additional training corpus that is not used to set the N-gram counts, but reserved to estimate model parameters

Recently Uploaded Slideshows tizyweb 2009