Multilayer perceptron - Revision history

Thwien: /* Dynamic learning rate */

2009-12-22T11:25:07Z

Dynamic learning rate

← Older revision		Revision as of 11:25, 22 December 2009
Line 122:		Line 122:
	:<math>m</math> weight m		:<math>m</math> weight m
	:<math>i</math> input		:<math>i</math> input

	This PHP implementation supports dynamic learning rate by default.

	=== Weight decay ===		=== Weight decay ===

2008-01-19T18:31:36Z

iRProp+

← Older revision		Revision as of 18:31, 19 January 2008
Line 232:		Line 232:

	\alpha^+ \cdot \Delta w_{ij}(t-1), & \text{if } \frac{\part E}{\part w_{ij}}(t-1) \cdot \frac{\part E}{\part w_{ij}}(t) > 0 \\		\alpha^+ \cdot \Delta w_{ij}(t-1), & \text{if } \frac{\part E}{\part w_{ij}}(t-1) \cdot \frac{\part E}{\part w_{ij}}(t) > 0 \\
	\Delta w_{ij}(t-1) - \Delta w_{ij}(t-2), & \text{if } \frac{\part E}{\part w_{ij}}(t-1) \cdot \frac{\part E}{\part w_{ij}}(t) < 0 \\		\Delta w_{ij}(t-1) - \Delta w_{ij}(t-2), & \text{if } \frac{\part E}{\part w_{ij}}(t-1) \cdot \frac{\part E}{\part w_{ij}}(t) < 0 \text{ and if } E(t) > E(t-1) \\
	\Delta w_{ij}(t-1), & \text{if } \frac{\part E}{\part w_{ij}}(t-1) \cdot \frac{\part E}{\part w_{ij}}(t) = 0 ~~\text{ and if } E(t) > E(t-1)~~		\Delta w_{ij}(t-1), & \text{if } \frac{\part E}{\part w_{ij}}(t-1) \cdot \frac{\part E}{\part w_{ij}}(t) = 0

	\end{cases}</math>		\end{cases}</math>

2008-01-19T15:14:12Z

RProp (Resilient Propagation)

← Older revision		Revision as of 15:14, 19 January 2008
Line 183:		Line 183:
	<math>\Delta w_{ij}(t) = \begin{cases}		<math>\Delta w_{ij}(t) = \begin{cases}

	-\Delta w_{ij}, & \text{if } \frac{\part E}{\part w_{ij}} > 0 \\		-\Delta p_{ij}, & \text{if } \frac{\part E}{\part w_{ij}} > 0 \\
	+\Delta w_{ij}, & \text{if } \frac{\part E}{\part w_{ij}} < 0 \\		+\Delta p_{ij}, & \text{if } \frac{\part E}{\part w_{ij}} < 0 \\
	0, & \text{if } \frac{\part E}{\part w_{ij}} = 0		0, & \text{if } \frac{\part E}{\part w_{ij}} = 0

	\end{cases}</math>		\end{cases}</math>

	<math>\Delta w_{ij}(t) = \begin{cases}		<math>\Delta p_{ij}(t) = \begin{cases}

	\alpha^+ \cdot \Delta w_{ij}(t-1), & \text{if } \frac{\part E}{\part w_{ij}}(t-1) \cdot \frac{\part E}{\part w_{ij}}(t) > 0 \\		\alpha^+ \cdot \Delta w_{ij}(t-1), & \text{if } \frac{\part E}{\part w_{ij}}(t-1) \cdot \frac{\part E}{\part w_{ij}}(t) > 0 \\
Line 198:		Line 198:

	:<math>\alpha</math> learning rate		:<math>\alpha</math> learning rate

			:<math>w</math> weight

			:<math>p</math> weight change

	:<math>\alpha^+ = 1.2</math>		:<math>\alpha^+ = 1.2</math>

2008-01-18T19:32:54Z

Programming solution of backpropagation

@@ Line 176: / Line 176: @@
 :<math>t</math> time (training step)
 :<math>\mu</math> maximal weight change factor
 == Binary and linear input ==

2008-01-18T14:33:09Z

Quick propagation algorithm

@@ Line 165: / Line 165: @@
 :<math>t</math> time (training step)
 :<math>\alpha</math> learning rate
 == Binary and linear input ==

2008-01-18T14:15:34Z

Programming solution of backpropagation

@@ Line 145: / Line 145: @@
 :<math>\alpha</math> learning rate
 :<math>\lambda</math> weight decay factor
 == Binary and linear input ==

2008-01-17T18:55:12Z

Weight decay

← Older revision		Revision as of 18:55, 17 January 2008
Line 131:		Line 131:
	The weight change algorithm without weight decay is the following:		The weight change algorithm without weight decay is the following:

	:<math>\~~delta~~ w_{i}(t) = \alpha \cdot \frac{\part E(t)}{\part w_{i}(t)}</math>		:<math>\Delta w_{i}(t) = \alpha \cdot \frac{\part E(t)}{\part w_{i}(t)}</math>

	By subtracting a value the weight change will be reduce in relation to the last weight.		By subtracting a value the weight change will be reduce in relation to the last weight.

	:<math>\~~delta~~ w_{i}(t) = \alpha \cdot \frac{\part E(t)}{\part w_{i}(t)} - \lambda \cdot w_{i}(t-1)</math>		:<math>\Delta w_{i}(t) = \alpha \cdot \frac{\part E(t)}{\part w_{i}(t)} - \lambda \cdot w_{i}(t-1)</math>

	:<math>\lambda = [0.03 .. 0.05]</math>		:<math>\lambda = [0.03 .. 0.05]</math>

2008-01-17T18:37:21Z

Weight decay

← Older revision		Revision as of 18:37, 17 January 2008
Line 131:		Line 131:
	The weight change algorithm without weight decay is the following:		The weight change algorithm without weight decay is the following:

	:<math>\delta w_{i}(t) = \frac{\part F(t)}{\part w_{i}(t)}</math>		:<math>\delta w_{i}(t) = \alpha \cdot \frac{\part E(t)}{\part w_{i}(t)}</math>

	By subtracting a value the weight change will be reduce in relation to the last weight.		By subtracting a value the weight change will be reduce in relation to the last weight.

	:<math>\delta w_{i}(t) = \frac{\part F(t)}{\part w_{i}(t)} - \lambda \cdot w_{i}(t-1)</math>		:<math>\delta w_{i}(t) = \alpha \cdot \frac{\part E(t)}{\part w_{i}(t)} - \lambda \cdot w_{i}(t-1)</math>

	:<math>\lambda = [0.03 .. 0.05]</math>		:<math>\lambda = [0.03 .. 0.05]</math>
Line 141:		Line 141:
	:<math>w</math> weight		:<math>w</math> weight
	:<math>i</math> neuron		:<math>i</math> neuron
	:<math>F</math> error function		:<math>E</math> error function
	:<math>t</math> time (training step)		:<math>t</math> time (training step)
			:<math>\alpha</math> learning rate
	:<math>\lambda</math> weight decay factor		:<math>\lambda</math> weight decay factor

2008-01-17T18:34:24Z

Programming solution of backpropagation

← Older revision		Revision as of 18:34, 17 January 2008
Line 124:		Line 124:

	This PHP implementation supports dynamic learning rate by default.		This PHP implementation supports dynamic learning rate by default.

			=== Weight decay ===

			Normally weights grow up to large numbers. But in fact this is not necessary. The weight decay algorithm tries to avoid large weights. Through large weights maybe the network convergence takes too long.

			The weight change algorithm without weight decay is the following:

			:<math>\delta w_{i}(t) = \frac{\part F(t)}{\part w_{i}(t)}</math>

			By subtracting a value the weight change will be reduce in relation to the last weight.

			:<math>\delta w_{i}(t) = \frac{\part F(t)}{\part w_{i}(t)} - \lambda \cdot w_{i}(t-1)</math>

			:<math>\lambda = [0.03 .. 0.05]</math>

			:<math>w</math> weight
			:<math>i</math> neuron
			:<math>F</math> error function
			:<math>t</math> time (training step)
			:<math>\lambda</math> weight decay factor

	== Binary and linear input ==		== Binary and linear input ==

2008-01-17T12:47:56Z

Programming solution of backpropagation

@@ Line 74: / Line 74: @@
 :<math>i</math> input
 :<math>a</math> desired output
 :<math>m</math> weight m
@@ Line 88: / Line 89: @@
 :<math>k</math> neuron k
 :<math>l</math> neuron l
 :<math>m</math> weight m
 :<math>i</math> input
@@ Line 104: / Line 106: @@
 The proper choosing of learning rate (<math>\alpha</math>) and momentum (<math>\beta</math>) is done by experience. Both values have a range between 0 and 1. This PHP implementation uses a default value of 0.5 for <math>\alpha</math> and 0.95 for <math>\beta</math>. <math>\alpha</math> and <math>\beta</math> cannot be zero. Otherwise no weight change will be happen and the network would never reach an errorless level. Theses factors can be changed by runtime.
 == Binary and linear input ==