[Review] Noise Contrastive Estimation: a new estimation principle for unnormalised statistical models

Michael Gutmann, Aapo Hyv¨arinen
Dept of Computer Science and HIIT, University of Helsinki


The idea is to perform nonlinear logistic regression to discriminate between the observed data and some artificially generated noise, using the model log-density function in the regression nonlinearity.
This leads to consistent estimator of the parameters.

Contrastive Divergence
Score Matching
1. Finding:
By minimising the expected squeared distance between the gradient of the log-density given by the model and the gradient of the observed data, we could estimate statistical models where the pdf is known only up to a multiplicative normalization constant.
2. Validation method
Demonstrating on Multivariate Gaussian and independent component analysis models, and by estimating an overcomplete filter set for natural image data.