Edited at

[Review] Foundation of Autoencoder

More than 1 year has passed since last update.


Autoencoders play a key role in deep neural network architectures for transfer learning and other tasks.

By analytically investigating the architecture of autoencoder, it leads us to certain general framework.

And in fact, learning the framework of autoencoder sheds the light on the understanding of deep architectures.

So in this article, I would like to review the research paper below.

Title: Autoencoders, Unsupervised Learning, and Deep Architectures

Author: Pierre Baldi

Publish Year: 2012

Link: http://proceedings.mlr.press/v27/baldi12a/baldi12a.pdf

My implementation



Initially autoencoder was invented by the collaborative work of Hinton and PDP group(Rumelheart) in 1986.

Recently it has got huge attention again because of the new discovery of the powerful architecture, which is a variational encoder.

The aim of this paper is to derive a better theoretical understanding of autoencoders and get a good insight of the nature of deep architectures.

A general Autoencoders Framework

Let us define the settings related with the framework in advance and then move on to the explanation of the architecture of it.


weight matrix: n->p

weight matrix: p->n

$X = ${$x_1, . . . , x_m$}

Target *optional
$Y = ${$y_1, . . . , y_m$}


Screen Shot 2018-05-23 at 16.16.12.png

So the main purpose of autoencoder is, again, to obtain a meaningful representation of the dataset, so we want a model to have a strong reproducibility.

Based on the settings above, I would like to build the mathematical equations below.

min \space E(A,B) = min_{A,B} \space \sum^m_{t=1} E(x_t) = min_{A,B} \space \sum^m_{t=1} \Delta(A o B(x_t), x_t)

Case: non auto-associative

min \space E(A,B) = min_{A,B} \space \sum^m_{t=1} E(x_t, y_t) = min_{A,B} \space \sum^m_{t=1} \Delta(A o B(x_t), y_t)\\

Simply saying...

The basic autoencoder is describe as below.

  • Input -> Hidden

    $y = f(W^Tx + b)$ where f is some activation function e.g. sigmoid/ReLU/Tanh...

  • Hidden -> Output

    $z = f(W'^Tz + b)$ where f is some activation function e.g. sigmoid/ReLU/Tanh...

  • Cost Function: Cross Entropy loss function

    $L(x,z) = H(B_x || B_y) = -\sum^d_{k=1} y_k \log z_k$
    And if we backpropagate the error from the output layer through the net, then it can learn how to reproduce the input.
    So the important part is not the generated output, but the weight matrix. Because while it learns, it squashes the features into weight matrix in numerical format.

More interesting way to understand the architecture is below's image.


image source: http://curiousily.com/data-science/2017/02/02/what-to-do-when-data-is-missing-part-2.html

Problem in Autoencoder

So as we have seen above, the mathematically this ability of representing the dataset is proved. But there remains the problem, which is that it tends to learn Identity Function.

Hence, it is not robust anymore. I have prepared the explanatory image of this phenomena below.

Screen Shot 2018-05-23 at 16.31.19.png

Source: https://stats.stackexchange.com/questions/130809/autoencoders-cant-learn-meaningful-features

keras implementation

Following this great post


Dataset: mnist images => Train:(60000, 784) Test:(10000, 784)




Layer (type) Output Shape Param #
input_1 (InputLayer) (None, 784) 0
encoding (Dense) (None, 32) 25120
decoding (Dense) (None, 784) 25872
Total params: 50,992
Trainable params: 50,992
Non-trainable params: 0


from keras.layers import Input, Dense

from keras.models import Model

# building a model
encoding_dim = 32

input_img = Input(shape=(784, ))
encoded = Dense(encoding_dim, activation='relu', name='encoding')(input_img)
decoded = Dense(784, activation='sigmoid', name='decoding')(encoded)
autoencoder = Model(input_img, decoded)
encoder = Model(input_img, encoded)
encoded_input = Input(shape=(encoding_dim, ))
decoded_layer = autoencoder.layers[-1]
decoder = Model(encoded_input, decoded_layer(encoded_input))
autoencoder.compile(optimizer='adam', loss='binary_crossentropy')

# summariseing the model architecture

# save the architecture in image
from keras.utils import plot_model
plot_model(autoencoder, to_file='model.png')

# get and prepare dataset
from keras.datasets import mnist
import numpy as np
(x_train, _), (x_test, _) = mnist.load_data()

x_train = x_train.astype('float32') / 255.
x_test = x_test.astype('float32') / 255.
x_train = x_train.reshape((len(x_train), np.prod(x_train.shape[1:])))
x_test = x_test.reshape((len(x_test), np.prod(x_test.shape[1:])))
print(x_train.shape, x_test.shape)

# fit the model to the dataset
autoencoder.fit(x_train, x_train, epochs=2, batch_size=256, shuffle=True, validation_data=(x_test, x_test))

# saving the model in a format of serialize weights to HDF5
print("Saved model to disk")

# presentation part: try reproducing the image to see how it worked
encoded_imgs = encoder.predict(x_test)
decoded_imgs = decoder.predict(encoded_imgs)

# showing the image
import matplotlib.pyplot as plt

n = 10
plt.figure(figsize=(20, 4))
for i in range(n):
ax = plt.subplot(2, n, i+1)
plt.imshow(x_test[i].reshape(28, 28))

# display reconstruction
ax = plt.subplot(2, n, i + 1 + n)
plt.imshow(decoded_imgs[i].reshape(28, 28))

Learning Result

Screen Shot 2018-05-24 at 12.44.12.png

I found that even in a few epochs, this model can achieve certain work.

Although this embeddings are really sparse matrix, the computational efficiency was retained on my machine.

Future Work

Since I have learned the pros and cons of autoencoders through the reviewing the paper, I would like to know how to solve the issue. In fact, there is already advanced research on this point and created the model for this as well.

So I will review that paper soon.