Multilayer perceptron

General

A multilayer perceptron is a feedforward artificial neural network. This means the signal inside the neural network flows from input layer passing hidden layers to output layer. While training the error correction of neural weights are done in the opposite direction. This is done by the backpropagation algorithm.

Activation

At first a cumulative input is calculated by the following equation:

$s=\sum _{k=1}^{n}i_{k}\cdot w_{k}$

Sigmoid activation function

$o={\rm {{sig}(s)={\frac {1}{1+{\rm {e^{-s}}}}}}}$

Tangens hyperbolicus activation function

$o=tanh(s)$

using output range between -1 and 1, or

$o={\frac {tanh(s)+1}{2}}$

using output range between 0 and 1.

$s$ cumulative input

$w$ weight of input

$i$ value of input

$n$ number of inputs

$k$ number of neuron

Error of neural network

If the neural network is initialized by random weights it has of course not the expected output. Therefore training is necessary. While supervised training known inputs and their corresponded output values are presented to the network. So it is possible to compare the real output with the desired output. The error is described as the following algorithm:

$E={1 \over 2}\sum _{i=1}^{n}(t_{i}-o_{i})^{2}$

$E$ network error

$n$ count of input patterns

$t_{i}$ desired output

$o_{i}$ calculated output

Backpropagation

The learning algorithm of a single layer perceptron is easy compared to a multilayer perceptron. The reason is that just the output layer is directly connected to the output, but not the hidden layers. Therefore the calculation of the right weights of the hidden layers is difficult mathematically. To get the right delta value for changing the weights of hidden neuron is described in the following equation:

$\Delta w_{ij}=-\alpha \cdot {\partial E \over \partial w_{ij}}=\alpha \cdot \delta _{j}\cdot x_{i}$

$E$ network error

$\Delta w_{ij}$ delta value $w_{ij}$ of neuron connection $i$ to $j$

$\alpha$ learning rate

$\delta _{j}$ the error of neuron $j$

$x_{i}$ input of neuron $i$

$t_{j}$ desired output of output neuron $j$

$o_{j}$ real output of output neuron $j$.

Programming solution of backpropagation

In this PHP implementation of multilayer perceptron the following algorithm is used for weight changes in hidden layers:

$s_{kl}=\sum _{l=1}^{n}w_{k}\cdot \Delta w_{l}\cdot \beta$

$\Delta w_{k}=o_{k}\cdot (1-o_{k})\cdot s_{kl}$

$w_{mk}=w_{mk}+\alpha \cdot i_{m}\cdot \Delta w_{k}$

$\alpha$ learning rate

$\beta$ momentum

$k$ neuron k

$l$ neuron l

$m$ weight m

$i$ input

$o$ output

$n$ count of neurons

To avoid overfitting of neural network the training procedure is finished if real output value has a fault tolerance of 1 per cent of desired output value.

Choosing learning rate and momentum

The proper choosing of learning rate ($\alpha$) and momentum ($\beta$) is done by experience. Both values have a range between 0 and 1. This PHP implementation uses a default value of 0.5 for $\alpha$ and 0.95 for $\beta$. Theses factors can be change by runtime.