Detail of the paper

Title: Neural Fitted Q Iteration - First Experiences
with a Data Efficient Neural Reinforcement
Learning Method
Published Year: 2005
author: Martin Riedmiller
Link: http://ml.informatik.uni-freiburg.de/former/_media/publications/rieecml05.pdf

Introduction

This paper introduced the novel approach which combined the classical Q-learning and multi-layer perceptron. And this is one of the origin of the further development in the domain of Deep Reinforcement Learning.
Basically this is a function approximation using multi-layer perceptron to efficiently represent Q function.
And the author firstly clarified the advantages and disadvantages arising in this approach reported by other existing research conducted by [Tes92, Lin92, Rie00].

Advantages: Generalisation: since MLP can affect effectively globally, the model can represent the Q function efficiently compared to other local-oriented approach
Disadvantage: Danger of the divergence in Learning: to widely cover the observations to train the model globally, this is not be assured the convergence in learning.

Basic Concept

The basic idea underlying NFQ is the following: Instead of updating the neural
value function on-line (which leads to the problems described in the previous
section), the update is performed off-line considering an entire set of transition
experiences. Experiences are collected in triples of the form (s, a, s
) by interacting
with the (real or simulated) system1. Here, s is the original state, a is the
chosen action and s is the resulting state. The set of experiences is called the
sample set D.

Algorithm

Benchmarking

avoidance control task - keep the system somewhere within the ’valid’ region
of state space. Pole balancing is typically defined as such a problem, where
the task is to avoid that the pole crashes or the cart hits the boundary of
the track.
reaching a goal - the system has to reach a certain area in state space. As soon
as it gets there, the task is immediately finished. Mountaincar is typically
defined as getting the cart to a certain position up the hill.
regulator problem - the system has to reach a certain region in state space
and has to be actively kept there by the controller. This corresponds to the
problems typically tackled with methods of classical control theory.

Empirical Results

Pole Balancing Task

- The Mountain Car Benchmark

Conclusion

The author has concluded that utilising the experience reply, we can avoid the issues discussed above, and achieve the efficient training.

References

BM95] Boyan and Moore. Generalization in reinforcement learning: Safely approximating
the value function. In Advances in Neural Information Processing
Systems 7. Morgan Kaufmann, 1995.
[EPG05] D. Ernst and and L. Wehenkel P. Geurts. Tree-based batch mode reinforcement
learning. Journal of Machine Learning Research, 6:503–556, 2005.
[Gor95] G. J. Gordon. Stable function approximation in dynamic programming. In
A. Prieditis and S. Russell, editors, Proceedings of the ICML, San Francisco,
CA, 1995.
[Lin92] L.-J. Lin. Self-improving reactive agents based on reinforcement learning,
planning and teaching. Machine Learning, 8:293–321, 1992.
[LP03] M. Lagoudakis and R. Parr. Least-squares policy iteration. Journal of
Machine Learning Research, 4:1107–1149, 2003.
[RB93] M. Riedmiller and H. Braun. A direct adaptive method for faster backpropagation
learning: The RPROP algorithm. In H. Ruspini, editor, Proceedings
of the IEEE International Conference on Neural Networks (ICNN), pages
586 – 591, San Francisco, 1993.
[Rie00] M. Riedmiller. Concepts and facilities of a neural reinforcement learning control
architecture for technical process control. Journal of Neural Computing
and Application, 8:323–338, 2000.
[SB98] R. S. Sutton and A. G. Barto. Reinforcement Learning. MIT Press, Cambridge,
MA, 1998.
[Tes92] G. Tesauro. Practical issues in temporal difference learning. Machine Learning,
(8):257–277, 1992.