[Review] Noise Contrastive Estimation: a new estimation principle for unnormalised statistical models

Authors
Michael Gutmann, Aapo Hyv¨arinen
Dept of Computer Science and HIIT, University of Helsinki

Abstract

The idea is to perform nonlinear logistic regression to discriminate between the observed data and some artificially generated noise, using the model log-density function in the regression nonlinearity.
This leads to consistent estimator of the parameters.

Contrastive Divergence
最適化方法の一つ、正規化定数がわからない確率分布のためのパラメーター推定方法。
Score Matching

1. Finding:
   By minimising the expected squeared distance between the gradient of the log-density given by the model and the gradient of the observed data, we could estimate statistical models where the pdf is known only up to a multiplicative normalization constant.
2. Validation method
   Demonstrating on Multivariate Gaussian and independent component analysis models, and by estimating an overcomplete filter set for natural image data.