*This is an English translation of the original Japanese post by ChatGPT.
Overview
I would like to organize my thoughts on prediction and extrapolation by deep neural networks, or DNNs. This is not meant to be a mathematically rigorous discussion, but rather a summary of my intuitive understanding. If you notice any misunderstandings or points that should be corrected, I would appreciate it if you could let me know in the comments.
This article is intended for readers who have ever felt that the phrase “DNNs are bad at extrapolation” is slightly unsatisfying, and would like to clarify what it really means.
I can more or less understand what people are trying to say when they say that “DNNs are bad at extrapolation.” However, when I think about what is actually happening, I feel that it may be easier to understand this not as extrapolation in the classical sense, but rather as a question of OOD, or out-of-distribution, behavior.
In the following sections, I will summarize my thoughts on extrapolation and OOD. I hope this will be helpful to anyone who has felt something similar.
First, let’s consider a simple function fit
Let us first consider the fitting of a simple function. Suppose, for example, that we fit a function such as $$f(x)=a x^2 + b x + c$$ to a set of data. In this case, the form of the function is fixed from the beginning, so the only things we can adjust are the parameters $a$, $b$, and $c$. We tune these parameters so that the error with respect to the experimental data becomes small.
If the experimental data were obtained in the range $x_{min} < x < x_{max}$, then calculating values outside that range would be extrapolation. If the actual data distribution is close to $f(x)$, then the calculated values outside the range can also be trusted. On the other hand, if the actual distribution is far from $f(x)$, the values calculated outside the range of $x$ are constrained by the shape of $f(x)$, and therefore become less reliable the farther they are from the observed range.
In other words, the prediction performance depends on how close the assumed functional form is to the actual distribution.
What is different in DNNs?
With DNNs, the situation is a little different.
DNNs are also, in the end, performing function approximation. However, unlike a low-degree polynomial, they do not start with a simple, predefined functional form. They have a very large number of parameters and degrees of freedom, and as they learn from data, their shape changes in a direction that improves prediction performance.
In other words, unlike a simple function fit, a DNN acquires, as a result of training, a functional form that can capture the important features contained in the training data.
The actual data distribution is generally difficult to know. By learning from a large amount of data, a DNN can adjust the shape of a highly flexible function and achieve strong prediction performance. I think this is very useful for problems where it is difficult to define a simple functional form in advance.
In a simple function fit, prediction performance was affected by how close the functional form was to the actual distribution. In the case of a DNN, prediction performance depends on how well the functional form acquired through training matches the actual data distribution.
My discomfort with the phrase “DNNs are bad at extrapolation”
I understand that when people say DNNs are “bad at extrapolation,” they mean that performance tends to drop for data that deviates from the data distribution seen during training.
Indeed, when data collected under quite different conditions, or data that looks substantially different, is given as input, the prediction accuracy of a DNN tends to decrease. I think we can describe this as a drop in accuracy when data that is unfamiliar to the model, or substantially different from the data seen during training, is provided as input.
This is quite similar to predicting values outside the range of $x$ for a simple function, but I think we need to be careful about how we judge whether data is unfamiliar or far from what the model has learned.
This is the part where I felt some discomfort with the phrase “DNNs are bad at extrapolation.”
What seems unfamiliar to a DNN does not necessarily match human intuition
Data that lies away from the distribution seen during training is called OOD, or out-of-distribution, data.
What makes this a little complicated is that what seems unfamiliar to humans does not necessarily match what is unfamiliar to a DNN.
As humans, we tend to feel that something is “far from the training data” when it looks quite different, when the conditions under which it was obtained are different, or when the values of the input parameters are very different. Of course, that intuition may be correct in some cases. However, what a DNN ultimately uses is not the features as humans see them, but the feature representations it has acquired through training. If the input-output relationship is relatively simple, the features seen by the DNN may be close to human intuition. But depending on the situation, there may also be cases where they diverge significantly.
For this reason, even data that looks quite different to human eyes may be treated as similar in the internal representation of a DNN. Conversely, even data that looks almost identical to humans may be treated as quite different by a DNN.
Adversarial examples are, I think, an easy-to-understand example of this. Even when the added noise is so small that humans can hardly tell the difference, a DNN may classify the input as a different class. This also shows that what humans perceive as “similar” and what a DNN internally treats as “similar” are not necessarily the same.
It has been shown that DNNs trained on large amounts of data can achieve higher performance than conventional machine learning methods that rely on manually designed features. Even when a new kind of data appears, if the DNN can capture it based on features learned from existing data, it may still lead to a correct prediction.
The key point is that how unfamiliar a piece of data is to a DNN ultimately depends on what the DNN has learned from the data.
Being close in feature-vector space is not the same as being accurate
Of course, even if a feature vector is close to those of the trained data, that alone does not mean that the prediction accuracy will be high.
When the feature vector is close, it means that the input lies in a region that is close to what the DNN has already learned. However, that only means it is “familiar” from the DNN’s point of view. How well the model itself captures reality is a separate question. If biases or insufficiencies remain in the training process, then even for data that is close in feature space, the model may simply return predictions that smoothly extend those biases.
What closeness in feature space indicates is not prediction accuracy itself, but rather familiarity to the DNN, or a low degree of novelty.
Extrapolation and OOD
Based on the discussion so far, I think we can organize the idea as follows.
When people say that DNNs are “bad at extrapolation,” the real issue is not simply whether the input lies outside the input range in the classical sense, but rather how far it is from the training distribution from the DNN’s point of view. Moreover, this “distance” is not determined by human intuition, but depends on what kind of feature representation the DNN has learned internally.
I think we can say that it is not easy to judge, simply by looking at the input data as humans, whether the DNN will be able to predict it well, or whether the data is close to OOD for the DNN.
In that sense, I feel that the phrase “DNNs are bad at extrapolation” is easier to understand if we take it not as extrapolation in the classical sense, but as meaning that DNNs tend to lose performance for inputs that are OOD from the DNN’s point of view.
The meaning of incorporating prior knowledge into DNN training
In the simple function example, I mentioned that if the original data distribution is close to the assumed functional form $f(x)$, prediction performance outside the range of $x$ can improve. What kind of feature representation a model learns, and how it behaves in situations slightly outside the training distribution, should also depend on what we allow it to learn freely and what kinds of constraints we impose during training. If we have prior knowledge about the data distribution, we can expect that giving this knowledge to the learner in advance may help preserve prediction performance outside the range, or in other words, for OOD cases.
For example, when dealing with physical data, if we already know part of the underlying equations of motion or conservation laws, it seems very inefficient to make the model learn all of those relationships from data alone.
This leads to the idea of incorporating physical laws into training as constraints or prior knowledge, in order to improve the efficiency of data-driven learning and to give meaningful behavior under conditions that are slightly outside the training data. PINNs, or Physics-Informed Neural Networks, are one example of this way of thinking.
Closing thoughts
A DNN learns a distribution from the training data and makes predictions based on what it has learned. Whether a new input lies outside the training data distribution, that is, whether it is OOD, is ultimately determined by what the model has learned.
In cases where the input-output relationship is relatively clear, I think what a DNN learns may sometimes be close to human intuition. In such cases, it seems natural that prediction performance would drop for data that appears highly novel to human eyes.
On the other hand, for a DNN that has learned a complex input-output relationship that is difficult to model, it may be a little too rough to say, just because the conditions under which new data was obtained or the parameter values appear to differ from the training data, that “this is extrapolation, so the accuracy will be poor.”
What matters is how OOD that data is for the DNN — in other words, how novel it is when viewed from the learned representation space. However, a low degree of novelty only indicates that the data lies in a region familiar to the DNN; it does not guarantee accuracy itself.
I hope this helps anyone who has felt a similar sense of discomfort organize their thoughts.