Neural Network theory & the non-reducibility of brain operation to the single neuron

Research into potential systems of artificial intelligence now looks to the brain for models rather than looking to technology for ideas from which to model the brain. A number of scientists are looking at the development of artificial intelligence from the basis of a developing understanding of the architecture of the human brain. This work is now represented in two interlocking disciplines: Computational neurobiology: which involves understanding human/animal brains using computational models; and Neural Computing: or simulating and building a machine to emulate the real brain. The analysis is made on two levels: coarse grained, examining and elucidating networks of interacting subsystems which is largely a neurophysiological activity; and fine grained, building theories and models of actual artificial neural networks as subsystems.

By the 40's enough work had been done on describing the behaviour of the neuron for psychologists and mathematicians to make a serious attempt at a mathematical theory of the neuron, both natural and artificial.

The artificial neuron

The original neural network was based on work by Warren McCulloch and Walter Pitts published in 1943. They built up a logical calculus of sequences of nerve connections based on the point that a nerves' action potential only fires in an all-or-none manner if the treshold for that nerve has been exceeded.

They produced an artificial logical neuron network consisting of three kinds of neurons

1. Receptor, afferent or input neurons which receive the impulse to fire from a sensor.
2. Central or inner neurons which are synapsed onto from receptor and other neurons and synapse onto output and other neurons.
3. Effector neurons which receive impulses from both inner neurons and directly from receptors.

They described a set of rules for the operation of the neurons:

1. Propagation delay is assumed to be constant for all neurons,
2. Neurons fire at discrete moments, not continuously.
3. Each synapse output stage impinges onto only one synaptic input stage on a subsequent neuron.
4. Each neuron can have a number of input synaptic stages.
5. Synaptic input stages contribute to overcoming of a threshold below which the neuron will not fire.

An artificial neuron is set up to fire at any time t if and only if (e-i) exceeds h, where e is the number of excitatory synapses onto it at time t, i is the number of inhibitory ones and h is the firing threshold for that neuron.

Given a clearly defined set of input and output conditions it is possible to create an arbitrarily complex neural network from the three types of neurons, with appropriate thresholds at the various synapses of the network.

Compared with Biological Neurons

McCulloch and Pitts suggested that this network may as well describe the functioning of a human nervous system as much as it might describe an automaton. Nevertheless, the whole system is deterministic. The network is a scanning device which reads the input to output transform specification as if it were a dictionary, the 'meaning' of every possible input 'word' is determined by the dictionary of associated inputs and outputs in its repertoire.

"Given any finite dictionary of input stimuli and their associated meanings or output responses, we can...always make (on paper) a scanning device or neural network capable of consulting the dictionary and producing the listed meaning or response for each input 'word' denoting its associated stimulus." [Singh, 1965, p158]

The associations of input to output are altered by altering the pattern of interconnections between neurons of each layer.

This is really a look-up-table device using neurons to carry out logic hardware functions, all its input and output are predetermined, for each set of possible inputs and interconnections there is a fixed result. Obviously human intelligence is not so fixed, and there will always be shortfalls in any strictly defined neural system. Active human neural systems learn and adapt to the culture in which they grow, so the McCulloch and Pitts neuron is inadequate to describe what is really going on, but networks starting at this level can be set up to learn and adapt.

Jagjit Singh in his textbook on information theory speaks of the potential behaviour repertoire of natural neural systems as being impossible to reduce adequately to unambiguous description:

"Whether any existing mode of behaviour such as that of the natural automata like the living brains of animals can really be put 'completely and unambiguously' into words is altogether a different matter...Consider, for instance, one specific function of the brain among the millions it performs during the course of its life, the visual identification of analogous geometrical patterns. Any attempt at an 'unambiguous and complete' verbal description of the general concept of analogy, the basis of our visual faculty, will inevitably be too long to be of much use for even drawing (on paper) neuron networks having the wide diversity of visual responses the natural automata normally exhibit as a matter of course. No one in our present state of knowledge dare hazard a guess whether such an enterprise would require thousands or millions or any larger number of volumes. Faced with such a descriptive avalanche, one is tempted to substitute the deed for the description, treating the connection pattern of the visual brain itself as the simplest definition or 'description' of the visual analogy principle." [Singh, 1965, pp171-2]

These neural networks are essentially digital, computer-like models having profound differences from real neural systems. For example, in real neural systems the pulse trains carrying quantitative sensory information seem to be coded in pulse frequency modulation form, rather than digital representations of number; also the depth of connectionism seems to be much more efficient in our neural operations. That is, the number of layers of neurons: sensory input, processing and output (efferent) layers; is much less than appears necessary with artificial neural nets.

McCulloch and Pitts also spoke of neuron nets having circular interconnections in which "activity may be set up in a circuit and continue reverberating around it for an indefinite period of time, so that any realisable (result) may involve reference to past events of an indefinite degree of remoteness." [McCulloch & Pitts, 1943] thus producing a regenerative process which might be akin to learning and to memory.

In considering the differences between biological systems and automata von Neumann examined the problem of self-reproducing machines. He discerned that in systems below a certain level of complexity the product of those systems would always be less complex than the system itself, but with a sufficient degree of complexity the system can reproduce itself or even construct more complex entities.

"Since the physical basis of mindlike qualities resides in the patterns of organisation of biological materials occurring naturally in animals, there is no reason why similar qualities may not emerge (in the future) from patterns of organisation of similar or other materials specially rigged to exhibit those qualities." [Singh, 1965, p202].

One should note here that it is this statement about 'mindlike qualities residing in the physical' which it is the task for computational physiologists to prove in this work exploring neural nets.

As Charles Sherrington has remarked,

"It is a far cry from an electrical reaction in the brain to suddenly seeing the world around one with all its distances, its colours and chiaroscuro." [Singh, p203]

and in Penfield's work of direct electrical stimulation of the exposed cortex, the patient

"is aware that he moved his hand when the electrode is applied to the proper motor area, but he is never deluded into the belief that he willed the action." [Singh, p204]

That is, there will be action co-ordinating or integrating centres 'above' the direct control networks. The stimulated versus the willed movement are distinguished as having different antecedents. The complex systems of neural nets are organised hierarchically with layers of processing nets projecting to higher "integrating" layers and so on up to the cortical planning and control layers. Also many layers use descending projections to control what they are being fed in the way of information. This prevents swamping and allows attention and concentration on particular processes.