Neural Networks have a long history in Computer Science.
While the concept was originally taken from nature, a lot
of work has since been performed with oversimplified models
and their mathematical treatment. The following is a short
overview of the main ideas in the field to motivate our approach.
Over the years, people have investigated various types of
neural networks and their features in such applications as
for example image and speech recognition. The models are in
competition with non-neural models such as support vector
machines or bayes networks.
All of these methods lack an integrated concept of time and
have shortcomings when we have a limited number of samples.
This makes it difficult to integrate or learn procedural
knowledge. In addition, they hardly explain effects that
make human learning so effective: Give a human five samples
and s/he will be able to identify this samples again afterwards.
Therefore besides the generalization of samples, the brain also
memorizes features of the samples themselves. This is in
contrast to the statistical or gradient descendent methods,
which might need several learning rounds or special tuning to
identify the learning samples correctly.
Several extensions of classical neural network types deal with
the integration of time. Examples are time delay neural networks
(TDNN) and recurrent neural networks. Besides these extensions of
classical network types, completely new and promising models were
invented and are being investigated. These new models are often
referenced as biologically inspired neural networks.
One of the most attractive ideas is the liquid state machines.
This is based on analog computing and introduces a concept of
time as an integral part of the model. The idea is to increase
the dimensionality of the feature space in time and combine the
features. When this is done correctly, the resulting space is
linearly separable, and an observer can be defined to classify
data, even when the data is varying with time.
Similarly to liquid state machines our model is also inspired by
physics. We use mechanical physics to model our system. This results
in a continuous rather than a discrete time model in our systems.
It also introduces a concept of time within the network that will
hopefully give us the same benefits as the time delays in liquid state
machines without some of the drawbacks that will be described further on.
When talking about biology inspired architectures we think not only
about a single neuron, but more about networks thereof. We take the
approach of modern physic that does not only investigate, but also
postulates some extensions to a model that can explain some of the effects.
On the network level we activate a whole subnet at the same time. A
very useful application for that is to activate different sets of
neurons during two learning samples. This solves the above problem
and we can identify feature sets that have been presented to the
system before. The same idea leads us to functions that allow the
deactivation of a whole subnet at a time. This is needed to focus
on certain features and prevents getting distracted by others.
Forgetting in other neural network models is implemented by just
adding more and more patterns until the ones that you want to forget
will be superseded by many others. Instead, we work with a decay function
that weakens the links over time when they are not regularly used.
To keep the networks stable we also use functions that limit activity
for sets of neurons. We allow pre-structuring of the networks. This is
done by defining the availability of connectivity between the nodes as
well as their capability to learn. This pre-structuring, for example,
would allow us to define networks of Quads such as mentioned in the Doerner
Neuron model or other architectures.
Besides these functional capabilities, our model seeks good scalability
and the possibility to implement nets with a very large number of neurons.
For that we try to limit the communication between areas by neglecting links
with little activity between components.
Our long-term goal is not only to investigate these neural network structures
separately but also to integrate them in the
PSI model. On our way to this goal
we investigate how our model can be tuned for learning time-dependent behavior.