Multi-Layer Perceptron

Each year, one of our lecturers sets a coursework to write a C++ implementation of an MLP.  The students are only just learning C++ at this stage, and the MLP is new to them too.  It is strange though that in my year when we did this (the first year he set it) the marks were generally good, and yet subsequently they have been, well, not at all good (I don’t know about this year’s, but have heard rumours).

 

The requirement is to write the network as object oriented code.  Which should be your first choice anyway, in my opinion, despite the possible slight speed increases you could get for writing some mish-mashed array based implementation.  I was not, actually, all that happy with the one I wrote for that course work, so I thought I might revisit it sometime soon.

 

McCulloch-Pitts

Clever chaps, who came up with a biologically-inspired (not realistic) model of a neuron, which takes in input values, weighted by what I will refer to as synaptic weights (or just weights) and sums them.  It then calculates its output value by using a threshold function on the sum.  Oh, and it has a bias input of 1 (constant input) which is also weighted by the bias weight, and added in to the sum too.  If you think of the bias as having a negative weight, then it is the value which the inputs have to exceed to make the neuron fire.  This idea of firing is, of course, fine if you have a threshold function which is discontinuous, and actually produces a 0 output until a threshold is reached, and then produces (say) an output of 1.  This step-wise functionality is valid, but does not learn well, as the discontinuity causes us some problems.  So we generally use a function such as the logistic, or tanh, to give our thresholding, as they are both easily differentiable.

 

Networks

When you plug a few MCP style ‘neurons’ together, you get a multi-layer perceptron.  You can train it by showing it examples of inputs, letting it work out what it thinks the answer should be, and then calculating how much it was in error.  The errors are then fed back through the network, in a stage called back-propagation, where each ‘neuron’ passes a proportion of the error to the neurons which give it its inputs.  It then adjusts the weights, so that the largest contribution to the error is changed most.  A few iterations of this, and the network learns to give the right answers.  Too many iterations, and it learns the inputs too well, and is no longer able to generalise – but just the right amount of training leaves you with a network which is quite resistant to noise, can produce correct results from previously unseen data, and is typically even resilient in the face of having synaptic links, and even whole neurons, removed.

 

This sort of network is not the fastest on the block, and nor does it provide the optimal separating hyper-plane, but it is quite flexible and easy to implement.  It can be used for pattern recognition and classification problems, and also for extrapolating results from data.

 

I’ll post on this in more detail later, as this is just to whet your appetite.  I will probably also give some examples of MLPs in use too…