Saturday, October 08, 2005

the problem of neural nets: the problem of people

The basic problem with artificial neural networks is very similar to the problem with people in the year 2005. A neural network is very sensitive to the order of training inputs it is presented. In fact, it is possible for a neural network to be presented training inputs in such an order that it forgets 'old input-output' pairs. This type of behavior is known as overfitting.

When did I start to overfit?

Sometime back in 10th grade of high school I was presented with a lot of input relating to my current scholastic endeavours. This overstimulation of the quantitative part of my brain has left me in a rather peculiar situation.

Why didn't anybody present me with a test set?

The problem of overstimulation of the quantitative part of the brain is that the distribution of analytic reasoning tasks is not representative of the distribution of tasks in the real-world. My schooling is analogous to the training of a neural network, where the purpose of the GPA is well aligned with the notion of a performance. However, as in the case of overfitting, the notion of a GPA fails to generalize to non-scholarly tasks and thus the performance of a pupil in a school systems falls short of predicting performance on common real-world tasks.

In the year 2005, many people overfit some aspect of life. I prefer to use the term 'overfit' as opposed to overspecialize because it draws upon the context of regression (fitting). When each person is treated independently of others, overfitting can only be seen as a bad thing. When does one person really want a neural network to overfit? However, in the context of society's machine, overfits are the ones who expand the horizon of modern life. It is as if overfitting was brought about by evolution. Evolution probably favored species who steadily produced a small, yet non-infinitesimal, percent of overfitters. Since people can be seen as the cream of the crop with respect to many evolutionary metrics, it is of no surprise that we are overfitting so well.

Overfitters unite!

I think people can learn about themselves by studying machine learning. In a good article titled "The Parallel Distributed Processing Approach to Semantic Cognition" by James L. McClelland and Timothy T. Rogers, degradation of semantic knowledge in humans (a condition known as sementic dementia) was compared to the behavior of a neural network. Traditionally scientists would study humans in an attempt to develop better computational techniques for tasks such as machine learning and machine vision, but it is important to study computational techniques because they can tell us something about ourselves.