Multilayer perceptron: Difference between revisions

From Artificial Neural Network for PHP
 
(17 intermediate revisions by the same user not shown)
Line 7: Line 7:
At first a cumulative input is calculated by the following equation:
At first a cumulative input is calculated by the following equation:


<math>s = \sum^{n}_{k=1} i_{k} \cdot w_{k}</math>
:<math>s = \sum^{n}_{k=1} i_{k} \cdot w_{k}</math>

Considering the ''BIAS'' value the equation is:

:<math>s = (\sum^{n}_{k=1} i_{k} \cdot w_{k}) + BIAS \cdot w_{k}</math>

:<math>BIAS = 1</math>


=== Sigmoid activation function ===
=== Sigmoid activation function ===


<math>o = \rm{sig}(s) = \frac{1}{1 + \rm e^{-s}}</math>
:<math>o = \rm{sig}(s) = \frac{1}{1 + \rm e^{-s}}</math>


=== Hyperbolic tangent activation function ===
=== Hyperbolic tangent activation function ===


<math>o = tanh(s)</math>
:<math>o = tanh(s)</math>


using output range between -1 and 1, or
using output range between -1 and 1, or


<math>o = \frac{tanh(s) + 1}{2}</math>
:<math>o = \frac{tanh(s) + 1}{2}</math>


using output range between 0 and 1.
using output range between 0 and 1.
Line 33: Line 39:
If the neural network is initialized by random weights it has of course not the expected output. Therefore training is necessary. While supervised training known inputs and their corresponded output values are presented to the network. So it is possible to compare the real output with the desired output. The error is described as the following algorithm:
If the neural network is initialized by random weights it has of course not the expected output. Therefore training is necessary. While supervised training known inputs and their corresponded output values are presented to the network. So it is possible to compare the real output with the desired output. The error is described as the following algorithm:


<math>E={1\over2} \sum^{n}_{i=1} (t_{i}-o_{i})^{2}</math>
:<math>E={1\over2} \sum^{n}_{i=1} (t_{i}-o_{i})^{2}</math>


:<math>E</math> network error
:<math>E</math> network error
Line 44: Line 50:
The learning algorithm of a single layer perceptron is easy compared to a multilayer perceptron. The reason is that just the output layer is directly connected to the output, but not the hidden layers. Therefore the calculation of the right weights of the hidden layers is difficult mathematically. To get the right delta value for changing the weights of hidden neuron is described in the following equation:
The learning algorithm of a single layer perceptron is easy compared to a multilayer perceptron. The reason is that just the output layer is directly connected to the output, but not the hidden layers. Therefore the calculation of the right weights of the hidden layers is difficult mathematically. To get the right delta value for changing the weights of hidden neuron is described in the following equation:


<math>\Delta w_{ij}= -\alpha \cdot {\partial E \over \partial w_{ij}} = \alpha \cdot \delta_{j} \cdot x_{i}</math>
:<math>\Delta w_{ij}= -\alpha \cdot {\partial E \over \partial w_{ij}} = \alpha \cdot \delta_{j} \cdot x_{i}</math>


:<math>E</math> network error
:<math>E</math> network error
Line 56: Line 62:
== Programming solution of backpropagation ==
== Programming solution of backpropagation ==


In this PHP implementation of multilayer perceptron the following algorithm is used for weight changes in hidden layers:
In this PHP implementation of multilayer perceptron the following algorithm is used for weight changes in hidden layers and output layer.


=== Weight change of output layer ===
<math>s_{kl} = \sum^{n}_{l=1} w_{k} \cdot \Delta w_{l} \cdot \beta</math>


<math>\Delta w_{k} = o_{k} \cdot (1 - o_{k}) \cdot s_{kl}</math>
:<math>\Delta w_{k} = o_{k} \cdot (a_{k} - o_{k}) \cdot (1 - o_{k})</math>


<math>w_{mk} = w_{mk} + \alpha \cdot i_{m} \cdot \Delta w_{k}</math>
:<math>w_{mk} = w_{mk} + \alpha \cdot i_{m} \cdot \Delta w_{k}</math>

:<math>k</math> neuron k
:<math>o</math> output
:<math>i</math> input
:<math>a</math> desired output
:<math>w</math> weight
:<math>m</math> weight m

=== Weight change of hidden layers ===

:<math>s_{kl} = \sum^{n}_{l=1} w_{k} \cdot \Delta w_{l} \cdot \beta</math>

:<math>\Delta w_{k} = o_{k} \cdot (1 - o_{k}) \cdot s_{kl}</math>

:<math>w_{mk} = w_{mk} + \alpha \cdot i_{m} \cdot \Delta w_{k}</math>


:<math>\alpha</math> learning rate
:<math>\alpha</math> learning rate
Line 68: Line 89:
:<math>k</math> neuron k
:<math>k</math> neuron k
:<math>l</math> neuron l
:<math>l</math> neuron l
:<math>w</math> weight
:<math>m</math> weight m
:<math>m</math> weight m
:<math>i</math> input
:<math>i</math> input
Line 73: Line 95:
:<math>n</math> count of neurons
:<math>n</math> count of neurons


=== Momentum ===
To avoid overfitting of neural network the training procedure is finished if real output value has a fault tolerance of 1 per cent of desired output value.

To avoid oscillating weight changes the momentum factor <math>\beta</math> is defined. Therefore the calculated weight change would not be the same always.

=== Overfitting ===

To avoid overfitting of neural networks in this PHP implementation the training procedure is finished if real output value has a fault tolerance of 1 per cent of desired output value.

=== Choosing learning rate and momentum ===

The proper choosing of learning rate (<math>\alpha</math>) and momentum (<math>\beta</math>) is done by experience. Both values have a range between 0 and 1. This PHP implementation uses a default value of 0.5 for <math>\alpha</math> and 0.95 for <math>\beta</math>. <math>\alpha</math> and <math>\beta</math> cannot be zero. Otherwise no weight change will be happen and the network would never reach an errorless level. Theses factors can be changed by runtime.

=== Dynamic learning rate ===

To convergent the network faster to its lowest error, use of dynamic learning rate may be a good way.

:<math>w_{mk} = w_{mk} + \alpha \cdot \gamma \cdot i_{m} \cdot \Delta w_{k}</math>

:<math>\alpha \cdot \gamma = [0.5 .. 0.9]</math>

:<math>\alpha</math> learning rate
:<math>\beta</math> momentum
:<math>\gamma</math> dynamic learning rate factor
:<math>k</math> neuron k
:<math>w</math> weight
:<math>m</math> weight m
:<math>i</math> input

=== Weight decay ===

Normally weights grow up to large numbers. But in fact this is not necessary. The weight decay algorithm tries to avoid large weights. Through large weights maybe the network convergence takes too long.

The weight change algorithm without weight decay is the following:

:<math>\Delta w_{i}(t) = \alpha \cdot \frac{\part E(t)}{\part w_{i}(t)}</math>

By subtracting a value the weight change will be reduce in relation to the last weight.

:<math>\Delta w_{i}(t) = \alpha \cdot \frac{\part E(t)}{\part w_{i}(t)} - \lambda \cdot w_{i}(t-1)</math>

:<math>\lambda = [0.03 .. 0.05]</math>

:<math>w</math> weight
:<math>i</math> neuron
:<math>E</math> error function
:<math>t</math> time (training step)
:<math>\alpha</math> learning rate
:<math>\lambda</math> weight decay factor

=== Quick propagation algorithm ===

The Quickprop algorithm calculates the weight change by using the quadratic function <math>f(x) = x^2</math>. Two different error values of two different weights are the two points of a secant. Relating this secant to a quadratic function it is possible to calculate its minimum <math>f'(x) = 0</math>. The x-coordinate of the minimum point is the new weight value.

:<math>S(t) = \frac{\part E}{\part w_{i}(t)}</math>

:<math>\Delta w_{i}(t) = \alpha \cdot \frac{\part E}{\part w_{i}(t)}</math> (normal backpropagation)

:<math>\frac{\Delta w_{i}(t)}{\alpha} = \frac{\part E}{\part w_{i}(t)}</math>

:<math>S(t) = \frac{\part E}{\part w_{i}(t)} = \frac{\Delta w_{i}(t)}{\alpha}</math>

:<math>\Delta w_{i}(t) = \frac{S(t)}{S(t-1) - S(t)} \cdot \Delta w_{i}(t-1)</math> (quick propagation)

:<math>w</math> weight
:<math>i</math> neuron
:<math>E</math> error function
:<math>t</math> time (training step)
:<math>\alpha</math> learning rate

To avoid too big changes the maximum weight change is limited by the following equation:

:<math>\Delta w_{i}(t) \leq \mu \cdot \Delta w_{i}(t-1)</math>

:<math>\mu = [1.75 .. 2.25]</math>

:<math>w</math> weight
:<math>i</math> neuron
:<math>t</math> time (training step)
:<math>\mu</math> maximal weight change factor

=== RProp (Resilient Propagation) ===

The RProp algorithm just refers to the direction of the gradient.

<math>\Delta w_{ij}(t) = \begin{cases}

-\Delta p_{ij}, & \text{if } \frac{\part E}{\part w_{ij}} > 0 \\
+\Delta p_{ij}, & \text{if } \frac{\part E}{\part w_{ij}} < 0 \\
0, & \text{if } \frac{\part E}{\part w_{ij}} = 0

\end{cases}</math>

<math>\Delta p_{ij}(t) = \begin{cases}

\alpha^+ \cdot \Delta w_{ij}(t-1), & \text{if } \frac{\part E}{\part w_{ij}}(t-1) \cdot \frac{\part E}{\part w_{ij}}(t) > 0 \\
\alpha^- \cdot \Delta w_{ij}(t-1), & \text{if } \frac{\part E}{\part w_{ij}}(t-1) \cdot \frac{\part E}{\part w_{ij}}(t) < 0 \\
\Delta w_{ij}(t-1), & \text{if } \frac{\part E}{\part w_{ij}}(t-1) \cdot \frac{\part E}{\part w_{ij}}(t) = 0

\end{cases}</math>

:<math>\alpha</math> learning rate

:<math>w</math> weight

:<math>p</math> weight change

:<math>\alpha^+ = 1.2</math>

:<math>\alpha^- = 0.5</math>

:<math>\Delta w(0) = 0.5</math>

:<math>\Delta w(t)_{max} = 50</math>

:<math>\Delta w(t)_{min} = 0</math>

=== RProp+ ===

The RProp+ algorithm reduce the previous weight change from the last weight change if the mathematical sign of the gradient changes.

<math>\Delta w_{ij}(t) = \begin{cases}

\alpha^+ \cdot \Delta w_{ij}(t-1), & \text{if } \frac{\part E}{\part w_{ij}}(t-1) \cdot \frac{\part E}{\part w_{ij}}(t) > 0 \\
\Delta w_{ij}(t-1) - \Delta w_{ij}(t-2), & \text{if } \frac{\part E}{\part w_{ij}}(t-1) \cdot \frac{\part E}{\part w_{ij}}(t) < 0 \\
\Delta w_{ij}(t-1), & \text{if } \frac{\part E}{\part w_{ij}}(t-1) \cdot \frac{\part E}{\part w_{ij}}(t) = 0

\end{cases}</math>

=== iRProp+ ===

The iRProp+ is a improve RProp+ algorithm with a little change. Before reducing the previous weight change from the last weight change, the network error will be calculated and compared. If the network error increases from <math>E(t-2)</math> to <math>E(t-1)</math>, then the procedure of RProp+ will be done. Otherwise no change will be done, because if <math>E(t-1)</math> has a lower value than <math>E(t-2)</math> the weight change seems to be correct to convergent the neural network.

<math>\Delta w_{ij}(t) = \begin{cases}

\alpha^+ \cdot \Delta w_{ij}(t-1), & \text{if } \frac{\part E}{\part w_{ij}}(t-1) \cdot \frac{\part E}{\part w_{ij}}(t) > 0 \\
\Delta w_{ij}(t-1) - \Delta w_{ij}(t-2), & \text{if } \frac{\part E}{\part w_{ij}}(t-1) \cdot \frac{\part E}{\part w_{ij}}(t) < 0 \text{ and if } E(t) > E(t-1) \\
\Delta w_{ij}(t-1), & \text{if } \frac{\part E}{\part w_{ij}}(t-1) \cdot \frac{\part E}{\part w_{ij}}(t) = 0

\end{cases}</math>

== Binary and linear input ==

If binary input is used easily the input value is 0 for ''false'' and 1 for ''true''.

:<math>0 : False</math>

:<math>1 : True</math>

Using linear input values normalization is needed:

:<math>i = \frac{f - f_{min}}{f_{max} - f_{min}}</math>

:<math>i</math> input value for neural network
:<math>f</math> real world value

This PHP implementation is supporting input normalization.

== Binary and linear output ==

The interpretation of output values just makes sense for the output layer. The interpretation is depending on the use of the neural network. If the network is used for classification, so binary output is used. Binary has two states: True or false. The network will produce always linear output values. Therefore these values has to be converted to binary values:

:<math>o < 0.5 : False</math>

:<math>o >= 0.5 : True</math>

:<math>o</math> output value

If using linear output the output values have to be normalized to a real value the network is trained for:

:<math>f = o \cdot (f_{max} - f_{min}) + f_{min}</math>

:<math>f</math> real world value
:<math>o</math> real output value of neural network

The same normalization equation for input values is used for output values while training the network.

:<math>o = \frac{f - f_{min}}{f_{max} - f_{min}}</math>


:<math>o</math> desired output value for neural network
== Choosing learning rate and momentum ==
:<math>f</math> real world value


This PHP implementation is supporting output normalization.
The proper choosing of learning rate (<math>\alpha</math>) and momentum (<math>\beta</math>) is done by experience. Both values have a range between 0 and 1. This PHP implementation uses a default value of 0.5 for <math>\alpha</math> and 0.95 for <math>\beta</math>. Theses factors can be change by runtime.

Latest revision as of 11:25, 22 December 2009

General

A multilayer perceptron is a feedforward artificial neural network. This means the signal inside the neural network flows from input layer passing hidden layers to output layer. While training the error correction of neural weights are done in the opposite direction. This is done by the backpropagation algorithm.

Activation

At first a cumulative input is calculated by the following equation:

Failed to parse (SVG (MathML can be enabled via browser plugin): Invalid response ("Math extension cannot connect to Restbase.") from server "https://wikimedia.org/api/rest_v1/":): {\displaystyle s = \sum^{n}_{k=1} i_{k} \cdot w_{k}}

Considering the BIAS value the equation is:

Failed to parse (SVG (MathML can be enabled via browser plugin): Invalid response ("Math extension cannot connect to Restbase.") from server "https://wikimedia.org/api/rest_v1/":): {\displaystyle s = (\sum^{n}_{k=1} i_{k} \cdot w_{k}) + BIAS \cdot w_{k}}
Failed to parse (SVG (MathML can be enabled via browser plugin): Invalid response ("Math extension cannot connect to Restbase.") from server "https://wikimedia.org/api/rest_v1/":): {\displaystyle BIAS = 1}

Sigmoid activation function

Failed to parse (SVG (MathML can be enabled via browser plugin): Invalid response ("Math extension cannot connect to Restbase.") from server "https://wikimedia.org/api/rest_v1/":): {\displaystyle o = \rm{sig}(s) = \frac{1}{1 + \rm e^{-s}}}

Hyperbolic tangent activation function

Failed to parse (SVG (MathML can be enabled via browser plugin): Invalid response ("Math extension cannot connect to Restbase.") from server "https://wikimedia.org/api/rest_v1/":): {\displaystyle o = tanh(s)}

using output range between -1 and 1, or

Failed to parse (SVG (MathML can be enabled via browser plugin): Invalid response ("Math extension cannot connect to Restbase.") from server "https://wikimedia.org/api/rest_v1/":): {\displaystyle o = \frac{tanh(s) + 1}{2}}

using output range between 0 and 1.

Failed to parse (SVG (MathML can be enabled via browser plugin): Invalid response ("Math extension cannot connect to Restbase.") from server "https://wikimedia.org/api/rest_v1/":): {\displaystyle s} cumulative input
Failed to parse (SVG (MathML can be enabled via browser plugin): Invalid response ("Math extension cannot connect to Restbase.") from server "https://wikimedia.org/api/rest_v1/":): {\displaystyle w} weight of input
Failed to parse (SVG (MathML can be enabled via browser plugin): Invalid response ("Math extension cannot connect to Restbase.") from server "https://wikimedia.org/api/rest_v1/":): {\displaystyle i} value of input
Failed to parse (SVG (MathML can be enabled via browser plugin): Invalid response ("Math extension cannot connect to Restbase.") from server "https://wikimedia.org/api/rest_v1/":): {\displaystyle n} number of inputs
Failed to parse (SVG (MathML can be enabled via browser plugin): Invalid response ("Math extension cannot connect to Restbase.") from server "https://wikimedia.org/api/rest_v1/":): {\displaystyle k} number of neuron

Error of neural network

If the neural network is initialized by random weights it has of course not the expected output. Therefore training is necessary. While supervised training known inputs and their corresponded output values are presented to the network. So it is possible to compare the real output with the desired output. The error is described as the following algorithm:

Failed to parse (SVG (MathML can be enabled via browser plugin): Invalid response ("Math extension cannot connect to Restbase.") from server "https://wikimedia.org/api/rest_v1/":): {\displaystyle E={1\over2} \sum^{n}_{i=1} (t_{i}-o_{i})^{2}}
Failed to parse (SVG (MathML can be enabled via browser plugin): Invalid response ("Math extension cannot connect to Restbase.") from server "https://wikimedia.org/api/rest_v1/":): {\displaystyle E} network error
Failed to parse (SVG (MathML can be enabled via browser plugin): Invalid response ("Math extension cannot connect to Restbase.") from server "https://wikimedia.org/api/rest_v1/":): {\displaystyle n} count of input patterns
Failed to parse (SVG (MathML can be enabled via browser plugin): Invalid response ("Math extension cannot connect to Restbase.") from server "https://wikimedia.org/api/rest_v1/":): {\displaystyle t_{i}} desired output
Failed to parse (SVG (MathML can be enabled via browser plugin): Invalid response ("Math extension cannot connect to Restbase.") from server "https://wikimedia.org/api/rest_v1/":): {\displaystyle o_{i}} calculated output

Backpropagation

The learning algorithm of a single layer perceptron is easy compared to a multilayer perceptron. The reason is that just the output layer is directly connected to the output, but not the hidden layers. Therefore the calculation of the right weights of the hidden layers is difficult mathematically. To get the right delta value for changing the weights of hidden neuron is described in the following equation:

Failed to parse (SVG (MathML can be enabled via browser plugin): Invalid response ("Math extension cannot connect to Restbase.") from server "https://wikimedia.org/api/rest_v1/":): {\displaystyle \Delta w_{ij}= -\alpha \cdot {\partial E \over \partial w_{ij}} = \alpha \cdot \delta_{j} \cdot x_{i}}
Failed to parse (SVG (MathML can be enabled via browser plugin): Invalid response ("Math extension cannot connect to Restbase.") from server "https://wikimedia.org/api/rest_v1/":): {\displaystyle E} network error
Failed to parse (SVG (MathML can be enabled via browser plugin): Invalid response ("Math extension cannot connect to Restbase.") from server "https://wikimedia.org/api/rest_v1/":): {\displaystyle \Delta w_{ij}} delta value Failed to parse (SVG (MathML can be enabled via browser plugin): Invalid response ("Math extension cannot connect to Restbase.") from server "https://wikimedia.org/api/rest_v1/":): {\displaystyle w_{ij}} of neuron connection Failed to parse (SVG (MathML can be enabled via browser plugin): Invalid response ("Math extension cannot connect to Restbase.") from server "https://wikimedia.org/api/rest_v1/":): {\displaystyle i} to Failed to parse (SVG (MathML can be enabled via browser plugin): Invalid response ("Math extension cannot connect to Restbase.") from server "https://wikimedia.org/api/rest_v1/":): {\displaystyle j}
Failed to parse (SVG (MathML can be enabled via browser plugin): Invalid response ("Math extension cannot connect to Restbase.") from server "https://wikimedia.org/api/rest_v1/":): {\displaystyle \alpha} learning rate
Failed to parse (SVG (MathML can be enabled via browser plugin): Invalid response ("Math extension cannot connect to Restbase.") from server "https://wikimedia.org/api/rest_v1/":): {\displaystyle \delta_{j}} the error of neuron Failed to parse (SVG (MathML can be enabled via browser plugin): Invalid response ("Math extension cannot connect to Restbase.") from server "https://wikimedia.org/api/rest_v1/":): {\displaystyle j}
Failed to parse (SVG (MathML can be enabled via browser plugin): Invalid response ("Math extension cannot connect to Restbase.") from server "https://wikimedia.org/api/rest_v1/":): {\displaystyle x_{i}} input of neuron Failed to parse (SVG (MathML can be enabled via browser plugin): Invalid response ("Math extension cannot connect to Restbase.") from server "https://wikimedia.org/api/rest_v1/":): {\displaystyle i}
Failed to parse (SVG (MathML can be enabled via browser plugin): Invalid response ("Math extension cannot connect to Restbase.") from server "https://wikimedia.org/api/rest_v1/":): {\displaystyle t_{j}} desired output of output neuron Failed to parse (SVG (MathML can be enabled via browser plugin): Invalid response ("Math extension cannot connect to Restbase.") from server "https://wikimedia.org/api/rest_v1/":): {\displaystyle j}
Failed to parse (SVG (MathML can be enabled via browser plugin): Invalid response ("Math extension cannot connect to Restbase.") from server "https://wikimedia.org/api/rest_v1/":): {\displaystyle o_{j}} real output of output neuron Failed to parse (SVG (MathML can be enabled via browser plugin): Invalid response ("Math extension cannot connect to Restbase.") from server "https://wikimedia.org/api/rest_v1/":): {\displaystyle j} .

Programming solution of backpropagation

In this PHP implementation of multilayer perceptron the following algorithm is used for weight changes in hidden layers and output layer.

Weight change of output layer

Failed to parse (SVG (MathML can be enabled via browser plugin): Invalid response ("Math extension cannot connect to Restbase.") from server "https://wikimedia.org/api/rest_v1/":): {\displaystyle \Delta w_{k} = o_{k} \cdot (a_{k} - o_{k}) \cdot (1 - o_{k})}
Failed to parse (SVG (MathML can be enabled via browser plugin): Invalid response ("Math extension cannot connect to Restbase.") from server "https://wikimedia.org/api/rest_v1/":): {\displaystyle w_{mk} = w_{mk} + \alpha \cdot i_{m} \cdot \Delta w_{k}}
Failed to parse (SVG (MathML can be enabled via browser plugin): Invalid response ("Math extension cannot connect to Restbase.") from server "https://wikimedia.org/api/rest_v1/":): {\displaystyle k} neuron k
Failed to parse (SVG (MathML can be enabled via browser plugin): Invalid response ("Math extension cannot connect to Restbase.") from server "https://wikimedia.org/api/rest_v1/":): {\displaystyle o} output
Failed to parse (SVG (MathML can be enabled via browser plugin): Invalid response ("Math extension cannot connect to Restbase.") from server "https://wikimedia.org/api/rest_v1/":): {\displaystyle i} input
Failed to parse (SVG (MathML can be enabled via browser plugin): Invalid response ("Math extension cannot connect to Restbase.") from server "https://wikimedia.org/api/rest_v1/":): {\displaystyle a} desired output
Failed to parse (SVG (MathML can be enabled via browser plugin): Invalid response ("Math extension cannot connect to Restbase.") from server "https://wikimedia.org/api/rest_v1/":): {\displaystyle w} weight
Failed to parse (SVG (MathML can be enabled via browser plugin): Invalid response ("Math extension cannot connect to Restbase.") from server "https://wikimedia.org/api/rest_v1/":): {\displaystyle m} weight m

Weight change of hidden layers

Failed to parse (SVG (MathML can be enabled via browser plugin): Invalid response ("Math extension cannot connect to Restbase.") from server "https://wikimedia.org/api/rest_v1/":): {\displaystyle s_{kl} = \sum^{n}_{l=1} w_{k} \cdot \Delta w_{l} \cdot \beta}
Failed to parse (SVG (MathML can be enabled via browser plugin): Invalid response ("Math extension cannot connect to Restbase.") from server "https://wikimedia.org/api/rest_v1/":): {\displaystyle \Delta w_{k} = o_{k} \cdot (1 - o_{k}) \cdot s_{kl}}
Failed to parse (SVG (MathML can be enabled via browser plugin): Invalid response ("Math extension cannot connect to Restbase.") from server "https://wikimedia.org/api/rest_v1/":): {\displaystyle w_{mk} = w_{mk} + \alpha \cdot i_{m} \cdot \Delta w_{k}}
Failed to parse (SVG (MathML can be enabled via browser plugin): Invalid response ("Math extension cannot connect to Restbase.") from server "https://wikimedia.org/api/rest_v1/":): {\displaystyle \alpha} learning rate
Failed to parse (SVG (MathML can be enabled via browser plugin): Invalid response ("Math extension cannot connect to Restbase.") from server "https://wikimedia.org/api/rest_v1/":): {\displaystyle \beta} momentum
Failed to parse (SVG (MathML can be enabled via browser plugin): Invalid response ("Math extension cannot connect to Restbase.") from server "https://wikimedia.org/api/rest_v1/":): {\displaystyle k} neuron k
Failed to parse (SVG (MathML can be enabled via browser plugin): Invalid response ("Math extension cannot connect to Restbase.") from server "https://wikimedia.org/api/rest_v1/":): {\displaystyle l} neuron l
Failed to parse (SVG (MathML can be enabled via browser plugin): Invalid response ("Math extension cannot connect to Restbase.") from server "https://wikimedia.org/api/rest_v1/":): {\displaystyle w} weight
Failed to parse (SVG (MathML can be enabled via browser plugin): Invalid response ("Math extension cannot connect to Restbase.") from server "https://wikimedia.org/api/rest_v1/":): {\displaystyle m} weight m
Failed to parse (SVG (MathML can be enabled via browser plugin): Invalid response ("Math extension cannot connect to Restbase.") from server "https://wikimedia.org/api/rest_v1/":): {\displaystyle i} input
Failed to parse (SVG (MathML can be enabled via browser plugin): Invalid response ("Math extension cannot connect to Restbase.") from server "https://wikimedia.org/api/rest_v1/":): {\displaystyle o} output
Failed to parse (SVG (MathML can be enabled via browser plugin): Invalid response ("Math extension cannot connect to Restbase.") from server "https://wikimedia.org/api/rest_v1/":): {\displaystyle n} count of neurons

Momentum

To avoid oscillating weight changes the momentum factor Failed to parse (SVG (MathML can be enabled via browser plugin): Invalid response ("Math extension cannot connect to Restbase.") from server "https://wikimedia.org/api/rest_v1/":): {\displaystyle \beta} is defined. Therefore the calculated weight change would not be the same always.

Overfitting

To avoid overfitting of neural networks in this PHP implementation the training procedure is finished if real output value has a fault tolerance of 1 per cent of desired output value.

Choosing learning rate and momentum

The proper choosing of learning rate (Failed to parse (SVG (MathML can be enabled via browser plugin): Invalid response ("Math extension cannot connect to Restbase.") from server "https://wikimedia.org/api/rest_v1/":): {\displaystyle \alpha} ) and momentum (Failed to parse (SVG (MathML can be enabled via browser plugin): Invalid response ("Math extension cannot connect to Restbase.") from server "https://wikimedia.org/api/rest_v1/":): {\displaystyle \beta} ) is done by experience. Both values have a range between 0 and 1. This PHP implementation uses a default value of 0.5 for Failed to parse (SVG (MathML can be enabled via browser plugin): Invalid response ("Math extension cannot connect to Restbase.") from server "https://wikimedia.org/api/rest_v1/":): {\displaystyle \alpha} and 0.95 for Failed to parse (SVG (MathML can be enabled via browser plugin): Invalid response ("Math extension cannot connect to Restbase.") from server "https://wikimedia.org/api/rest_v1/":): {\displaystyle \beta} . Failed to parse (SVG (MathML can be enabled via browser plugin): Invalid response ("Math extension cannot connect to Restbase.") from server "https://wikimedia.org/api/rest_v1/":): {\displaystyle \alpha} and Failed to parse (SVG (MathML can be enabled via browser plugin): Invalid response ("Math extension cannot connect to Restbase.") from server "https://wikimedia.org/api/rest_v1/":): {\displaystyle \beta} cannot be zero. Otherwise no weight change will be happen and the network would never reach an errorless level. Theses factors can be changed by runtime.

Dynamic learning rate

To convergent the network faster to its lowest error, use of dynamic learning rate may be a good way.

Failed to parse (SVG (MathML can be enabled via browser plugin): Invalid response ("Math extension cannot connect to Restbase.") from server "https://wikimedia.org/api/rest_v1/":): {\displaystyle w_{mk} = w_{mk} + \alpha \cdot \gamma \cdot i_{m} \cdot \Delta w_{k}}
Failed to parse (SVG (MathML can be enabled via browser plugin): Invalid response ("Math extension cannot connect to Restbase.") from server "https://wikimedia.org/api/rest_v1/":): {\displaystyle \alpha \cdot \gamma = [0.5 .. 0.9]}
Failed to parse (SVG (MathML can be enabled via browser plugin): Invalid response ("Math extension cannot connect to Restbase.") from server "https://wikimedia.org/api/rest_v1/":): {\displaystyle \alpha} learning rate
Failed to parse (SVG (MathML can be enabled via browser plugin): Invalid response ("Math extension cannot connect to Restbase.") from server "https://wikimedia.org/api/rest_v1/":): {\displaystyle \beta} momentum
Failed to parse (SVG (MathML can be enabled via browser plugin): Invalid response ("Math extension cannot connect to Restbase.") from server "https://wikimedia.org/api/rest_v1/":): {\displaystyle \gamma} dynamic learning rate factor
Failed to parse (SVG (MathML can be enabled via browser plugin): Invalid response ("Math extension cannot connect to Restbase.") from server "https://wikimedia.org/api/rest_v1/":): {\displaystyle k} neuron k
Failed to parse (SVG (MathML can be enabled via browser plugin): Invalid response ("Math extension cannot connect to Restbase.") from server "https://wikimedia.org/api/rest_v1/":): {\displaystyle w} weight
Failed to parse (SVG (MathML can be enabled via browser plugin): Invalid response ("Math extension cannot connect to Restbase.") from server "https://wikimedia.org/api/rest_v1/":): {\displaystyle m} weight m
Failed to parse (SVG (MathML can be enabled via browser plugin): Invalid response ("Math extension cannot connect to Restbase.") from server "https://wikimedia.org/api/rest_v1/":): {\displaystyle i} input

Weight decay

Normally weights grow up to large numbers. But in fact this is not necessary. The weight decay algorithm tries to avoid large weights. Through large weights maybe the network convergence takes too long.

The weight change algorithm without weight decay is the following:

Failed to parse (SVG (MathML can be enabled via browser plugin): Invalid response ("Math extension cannot connect to Restbase.") from server "https://wikimedia.org/api/rest_v1/":): {\displaystyle \Delta w_{i}(t) = \alpha \cdot \frac{\part E(t)}{\part w_{i}(t)}}

By subtracting a value the weight change will be reduce in relation to the last weight.

Failed to parse (SVG (MathML can be enabled via browser plugin): Invalid response ("Math extension cannot connect to Restbase.") from server "https://wikimedia.org/api/rest_v1/":): {\displaystyle \Delta w_{i}(t) = \alpha \cdot \frac{\part E(t)}{\part w_{i}(t)} - \lambda \cdot w_{i}(t-1)}
Failed to parse (SVG (MathML can be enabled via browser plugin): Invalid response ("Math extension cannot connect to Restbase.") from server "https://wikimedia.org/api/rest_v1/":): {\displaystyle \lambda = [0.03 .. 0.05]}
Failed to parse (SVG (MathML can be enabled via browser plugin): Invalid response ("Math extension cannot connect to Restbase.") from server "https://wikimedia.org/api/rest_v1/":): {\displaystyle w} weight
Failed to parse (SVG (MathML can be enabled via browser plugin): Invalid response ("Math extension cannot connect to Restbase.") from server "https://wikimedia.org/api/rest_v1/":): {\displaystyle i} neuron
Failed to parse (SVG (MathML can be enabled via browser plugin): Invalid response ("Math extension cannot connect to Restbase.") from server "https://wikimedia.org/api/rest_v1/":): {\displaystyle E} error function
Failed to parse (SVG (MathML can be enabled via browser plugin): Invalid response ("Math extension cannot connect to Restbase.") from server "https://wikimedia.org/api/rest_v1/":): {\displaystyle t} time (training step)
Failed to parse (SVG (MathML can be enabled via browser plugin): Invalid response ("Math extension cannot connect to Restbase.") from server "https://wikimedia.org/api/rest_v1/":): {\displaystyle \alpha} learning rate
Failed to parse (SVG (MathML can be enabled via browser plugin): Invalid response ("Math extension cannot connect to Restbase.") from server "https://wikimedia.org/api/rest_v1/":): {\displaystyle \lambda} weight decay factor

Quick propagation algorithm

The Quickprop algorithm calculates the weight change by using the quadratic function Failed to parse (SVG (MathML can be enabled via browser plugin): Invalid response ("Math extension cannot connect to Restbase.") from server "https://wikimedia.org/api/rest_v1/":): {\displaystyle f(x) = x^2} . Two different error values of two different weights are the two points of a secant. Relating this secant to a quadratic function it is possible to calculate its minimum Failed to parse (SVG (MathML can be enabled via browser plugin): Invalid response ("Math extension cannot connect to Restbase.") from server "https://wikimedia.org/api/rest_v1/":): {\displaystyle f'(x) = 0} . The x-coordinate of the minimum point is the new weight value.

Failed to parse (SVG (MathML can be enabled via browser plugin): Invalid response ("Math extension cannot connect to Restbase.") from server "https://wikimedia.org/api/rest_v1/":): {\displaystyle S(t) = \frac{\part E}{\part w_{i}(t)}}
Failed to parse (SVG (MathML can be enabled via browser plugin): Invalid response ("Math extension cannot connect to Restbase.") from server "https://wikimedia.org/api/rest_v1/":): {\displaystyle \Delta w_{i}(t) = \alpha \cdot \frac{\part E}{\part w_{i}(t)}} (normal backpropagation)
Failed to parse (SVG (MathML can be enabled via browser plugin): Invalid response ("Math extension cannot connect to Restbase.") from server "https://wikimedia.org/api/rest_v1/":): {\displaystyle \frac{\Delta w_{i}(t)}{\alpha} = \frac{\part E}{\part w_{i}(t)}}
Failed to parse (SVG (MathML can be enabled via browser plugin): Invalid response ("Math extension cannot connect to Restbase.") from server "https://wikimedia.org/api/rest_v1/":): {\displaystyle S(t) = \frac{\part E}{\part w_{i}(t)} = \frac{\Delta w_{i}(t)}{\alpha}}
Failed to parse (SVG (MathML can be enabled via browser plugin): Invalid response ("Math extension cannot connect to Restbase.") from server "https://wikimedia.org/api/rest_v1/":): {\displaystyle \Delta w_{i}(t) = \frac{S(t)}{S(t-1) - S(t)} \cdot \Delta w_{i}(t-1)} (quick propagation)
Failed to parse (SVG (MathML can be enabled via browser plugin): Invalid response ("Math extension cannot connect to Restbase.") from server "https://wikimedia.org/api/rest_v1/":): {\displaystyle w} weight
Failed to parse (SVG (MathML can be enabled via browser plugin): Invalid response ("Math extension cannot connect to Restbase.") from server "https://wikimedia.org/api/rest_v1/":): {\displaystyle i} neuron
Failed to parse (SVG (MathML can be enabled via browser plugin): Invalid response ("Math extension cannot connect to Restbase.") from server "https://wikimedia.org/api/rest_v1/":): {\displaystyle E} error function
Failed to parse (SVG (MathML can be enabled via browser plugin): Invalid response ("Math extension cannot connect to Restbase.") from server "https://wikimedia.org/api/rest_v1/":): {\displaystyle t} time (training step)
Failed to parse (SVG (MathML can be enabled via browser plugin): Invalid response ("Math extension cannot connect to Restbase.") from server "https://wikimedia.org/api/rest_v1/":): {\displaystyle \alpha} learning rate

To avoid too big changes the maximum weight change is limited by the following equation:

Failed to parse (SVG (MathML can be enabled via browser plugin): Invalid response ("Math extension cannot connect to Restbase.") from server "https://wikimedia.org/api/rest_v1/":): {\displaystyle \Delta w_{i}(t) \leq \mu \cdot \Delta w_{i}(t-1)}
Failed to parse (SVG (MathML can be enabled via browser plugin): Invalid response ("Math extension cannot connect to Restbase.") from server "https://wikimedia.org/api/rest_v1/":): {\displaystyle \mu = [1.75 .. 2.25]}
Failed to parse (SVG (MathML can be enabled via browser plugin): Invalid response ("Math extension cannot connect to Restbase.") from server "https://wikimedia.org/api/rest_v1/":): {\displaystyle w} weight
Failed to parse (SVG (MathML can be enabled via browser plugin): Invalid response ("Math extension cannot connect to Restbase.") from server "https://wikimedia.org/api/rest_v1/":): {\displaystyle i} neuron
Failed to parse (SVG (MathML can be enabled via browser plugin): Invalid response ("Math extension cannot connect to Restbase.") from server "https://wikimedia.org/api/rest_v1/":): {\displaystyle t} time (training step)
Failed to parse (SVG (MathML can be enabled via browser plugin): Invalid response ("Math extension cannot connect to Restbase.") from server "https://wikimedia.org/api/rest_v1/":): {\displaystyle \mu} maximal weight change factor

RProp (Resilient Propagation)

The RProp algorithm just refers to the direction of the gradient.

Failed to parse (SVG (MathML can be enabled via browser plugin): Invalid response ("Math extension cannot connect to Restbase.") from server "https://wikimedia.org/api/rest_v1/":): {\displaystyle \Delta w_{ij}(t) = \begin{cases} -\Delta p_{ij}, & \text{if } \frac{\part E}{\part w_{ij}} > 0 \\ +\Delta p_{ij}, & \text{if } \frac{\part E}{\part w_{ij}} < 0 \\ 0, & \text{if } \frac{\part E}{\part w_{ij}} = 0 \end{cases}}

Failed to parse (SVG (MathML can be enabled via browser plugin): Invalid response ("Math extension cannot connect to Restbase.") from server "https://wikimedia.org/api/rest_v1/":): {\displaystyle \Delta p_{ij}(t) = \begin{cases} \alpha^+ \cdot \Delta w_{ij}(t-1), & \text{if } \frac{\part E}{\part w_{ij}}(t-1) \cdot \frac{\part E}{\part w_{ij}}(t) > 0 \\ \alpha^- \cdot \Delta w_{ij}(t-1), & \text{if } \frac{\part E}{\part w_{ij}}(t-1) \cdot \frac{\part E}{\part w_{ij}}(t) < 0 \\ \Delta w_{ij}(t-1), & \text{if } \frac{\part E}{\part w_{ij}}(t-1) \cdot \frac{\part E}{\part w_{ij}}(t) = 0 \end{cases}}

Failed to parse (SVG (MathML can be enabled via browser plugin): Invalid response ("Math extension cannot connect to Restbase.") from server "https://wikimedia.org/api/rest_v1/":): {\displaystyle \alpha} learning rate
Failed to parse (SVG (MathML can be enabled via browser plugin): Invalid response ("Math extension cannot connect to Restbase.") from server "https://wikimedia.org/api/rest_v1/":): {\displaystyle w} weight
Failed to parse (SVG (MathML can be enabled via browser plugin): Invalid response ("Math extension cannot connect to Restbase.") from server "https://wikimedia.org/api/rest_v1/":): {\displaystyle p} weight change
Failed to parse (SVG (MathML can be enabled via browser plugin): Invalid response ("Math extension cannot connect to Restbase.") from server "https://wikimedia.org/api/rest_v1/":): {\displaystyle \alpha^+ = 1.2}
Failed to parse (SVG (MathML can be enabled via browser plugin): Invalid response ("Math extension cannot connect to Restbase.") from server "https://wikimedia.org/api/rest_v1/":): {\displaystyle \alpha^- = 0.5}
Failed to parse (SVG (MathML can be enabled via browser plugin): Invalid response ("Math extension cannot connect to Restbase.") from server "https://wikimedia.org/api/rest_v1/":): {\displaystyle \Delta w(0) = 0.5}
Failed to parse (SVG (MathML can be enabled via browser plugin): Invalid response ("Math extension cannot connect to Restbase.") from server "https://wikimedia.org/api/rest_v1/":): {\displaystyle \Delta w(t)_{max} = 50}
Failed to parse (SVG (MathML can be enabled via browser plugin): Invalid response ("Math extension cannot connect to Restbase.") from server "https://wikimedia.org/api/rest_v1/":): {\displaystyle \Delta w(t)_{min} = 0}

RProp+

The RProp+ algorithm reduce the previous weight change from the last weight change if the mathematical sign of the gradient changes.

Failed to parse (SVG (MathML can be enabled via browser plugin): Invalid response ("Math extension cannot connect to Restbase.") from server "https://wikimedia.org/api/rest_v1/":): {\displaystyle \Delta w_{ij}(t) = \begin{cases} \alpha^+ \cdot \Delta w_{ij}(t-1), & \text{if } \frac{\part E}{\part w_{ij}}(t-1) \cdot \frac{\part E}{\part w_{ij}}(t) > 0 \\ \Delta w_{ij}(t-1) - \Delta w_{ij}(t-2), & \text{if } \frac{\part E}{\part w_{ij}}(t-1) \cdot \frac{\part E}{\part w_{ij}}(t) < 0 \\ \Delta w_{ij}(t-1), & \text{if } \frac{\part E}{\part w_{ij}}(t-1) \cdot \frac{\part E}{\part w_{ij}}(t) = 0 \end{cases}}

iRProp+

The iRProp+ is a improve RProp+ algorithm with a little change. Before reducing the previous weight change from the last weight change, the network error will be calculated and compared. If the network error increases from Failed to parse (SVG (MathML can be enabled via browser plugin): Invalid response ("Math extension cannot connect to Restbase.") from server "https://wikimedia.org/api/rest_v1/":): {\displaystyle E(t-2)} to Failed to parse (SVG (MathML can be enabled via browser plugin): Invalid response ("Math extension cannot connect to Restbase.") from server "https://wikimedia.org/api/rest_v1/":): {\displaystyle E(t-1)} , then the procedure of RProp+ will be done. Otherwise no change will be done, because if Failed to parse (SVG (MathML can be enabled via browser plugin): Invalid response ("Math extension cannot connect to Restbase.") from server "https://wikimedia.org/api/rest_v1/":): {\displaystyle E(t-1)} has a lower value than Failed to parse (SVG (MathML can be enabled via browser plugin): Invalid response ("Math extension cannot connect to Restbase.") from server "https://wikimedia.org/api/rest_v1/":): {\displaystyle E(t-2)} the weight change seems to be correct to convergent the neural network.

Failed to parse (SVG (MathML can be enabled via browser plugin): Invalid response ("Math extension cannot connect to Restbase.") from server "https://wikimedia.org/api/rest_v1/":): {\displaystyle \Delta w_{ij}(t) = \begin{cases} \alpha^+ \cdot \Delta w_{ij}(t-1), & \text{if } \frac{\part E}{\part w_{ij}}(t-1) \cdot \frac{\part E}{\part w_{ij}}(t) > 0 \\ \Delta w_{ij}(t-1) - \Delta w_{ij}(t-2), & \text{if } \frac{\part E}{\part w_{ij}}(t-1) \cdot \frac{\part E}{\part w_{ij}}(t) < 0 \text{ and if } E(t) > E(t-1) \\ \Delta w_{ij}(t-1), & \text{if } \frac{\part E}{\part w_{ij}}(t-1) \cdot \frac{\part E}{\part w_{ij}}(t) = 0 \end{cases}}

Binary and linear input

If binary input is used easily the input value is 0 for false and 1 for true.

Failed to parse (SVG (MathML can be enabled via browser plugin): Invalid response ("Math extension cannot connect to Restbase.") from server "https://wikimedia.org/api/rest_v1/":): {\displaystyle 0 : False}
Failed to parse (SVG (MathML can be enabled via browser plugin): Invalid response ("Math extension cannot connect to Restbase.") from server "https://wikimedia.org/api/rest_v1/":): {\displaystyle 1 : True}

Using linear input values normalization is needed:

Failed to parse (SVG (MathML can be enabled via browser plugin): Invalid response ("Math extension cannot connect to Restbase.") from server "https://wikimedia.org/api/rest_v1/":): {\displaystyle i = \frac{f - f_{min}}{f_{max} - f_{min}}}
Failed to parse (SVG (MathML can be enabled via browser plugin): Invalid response ("Math extension cannot connect to Restbase.") from server "https://wikimedia.org/api/rest_v1/":): {\displaystyle i} input value for neural network
Failed to parse (SVG (MathML can be enabled via browser plugin): Invalid response ("Math extension cannot connect to Restbase.") from server "https://wikimedia.org/api/rest_v1/":): {\displaystyle f} real world value

This PHP implementation is supporting input normalization.

Binary and linear output

The interpretation of output values just makes sense for the output layer. The interpretation is depending on the use of the neural network. If the network is used for classification, so binary output is used. Binary has two states: True or false. The network will produce always linear output values. Therefore these values has to be converted to binary values:

Failed to parse (SVG (MathML can be enabled via browser plugin): Invalid response ("Math extension cannot connect to Restbase.") from server "https://wikimedia.org/api/rest_v1/":): {\displaystyle o < 0.5 : False}
Failed to parse (SVG (MathML can be enabled via browser plugin): Invalid response ("Math extension cannot connect to Restbase.") from server "https://wikimedia.org/api/rest_v1/":): {\displaystyle o >= 0.5 : True}
Failed to parse (SVG (MathML can be enabled via browser plugin): Invalid response ("Math extension cannot connect to Restbase.") from server "https://wikimedia.org/api/rest_v1/":): {\displaystyle o} output value

If using linear output the output values have to be normalized to a real value the network is trained for:

Failed to parse (SVG (MathML can be enabled via browser plugin): Invalid response ("Math extension cannot connect to Restbase.") from server "https://wikimedia.org/api/rest_v1/":): {\displaystyle f = o \cdot (f_{max} - f_{min}) + f_{min}}
Failed to parse (SVG (MathML can be enabled via browser plugin): Invalid response ("Math extension cannot connect to Restbase.") from server "https://wikimedia.org/api/rest_v1/":): {\displaystyle f} real world value
Failed to parse (SVG (MathML can be enabled via browser plugin): Invalid response ("Math extension cannot connect to Restbase.") from server "https://wikimedia.org/api/rest_v1/":): {\displaystyle o} real output value of neural network

The same normalization equation for input values is used for output values while training the network.

Failed to parse (SVG (MathML can be enabled via browser plugin): Invalid response ("Math extension cannot connect to Restbase.") from server "https://wikimedia.org/api/rest_v1/":): {\displaystyle o = \frac{f - f_{min}}{f_{max} - f_{min}}}
Failed to parse (SVG (MathML can be enabled via browser plugin): Invalid response ("Math extension cannot connect to Restbase.") from server "https://wikimedia.org/api/rest_v1/":): {\displaystyle o} desired output value for neural network
Failed to parse (SVG (MathML can be enabled via browser plugin): Invalid response ("Math extension cannot connect to Restbase.") from server "https://wikimedia.org/api/rest_v1/":): {\displaystyle f} real world value

This PHP implementation is supporting output normalization.

Retrieved from "https://ann.thwien.de/wiki/index.php?title=Multilayer_perceptron&oldid=261"