> ')&7 "bjbjUU .:7|7|l,.,X 2fx
x
x
x
x
x
x
$! #x
x
x
x
x
x
x
x
bx
x
x
V@x
ZFS( 0X _x>$>$Neural Networks
The McCulloch and Pitts (MCP) neural computing unit
This is a simple model neuron with a number of inputs and one output. Both the inputs and outputs are binary.
Each input has a weight factor. If the weighted sum of the inputs (which is called the net input and given the symbol h) exceeds the threshold (symbol (), then the output is 1. Otherwise, the output is zero.
Perceptrons
A perceptron is a simple neural network: it consists of layers of perceptron units combined in a feed-forward manner. Connections are only made between adjacent layers.
Perceptron units are similar to MCP units, but may have binary or continuous inputs and outputs.
A perceptron with only one layer of units is called a simple perceptron.
Equation for a simple perceptron:
For a simple perceptron where each unit has N inputs, the output of unit i is given by:
EMBED Equation.3
Oi = output from unit i
g() = activation function
wij = weight of input j to unit i
xj = value of input j
(i = threshold of unit i
hi = net input to unit i (weighted sum of all N inputs)
The activation function, g is the function that relates the net input to the output. For a binary perceptron, this may be a Heaviside function (i.e. step from 0 to 1 at threshold) or a sgn function (step from 1 to 1 at threshold). Other activation functions are also possible.
An alternative notation to describe the threshold is to treat it as a special weight vector. This is given the index j = 0 and x0 is defined as 1. The threshold is then given the notation wi0 and the equation becomes:
EMBED Equation.3
Dot Product Representation
The behaviour of perceptrons can be described using vector notation. In this case, we define a vector of inputs x and a vector of weights w
The net input, h can then be described as the scalar product (dot product) of the inputs and weights:
EMBED Equation.3
The value of h then gives a measure of the similarity of the two vectors. So if the weight vector is used to represent some stored pattern, the value of h gives a measure of similarity between the current input and the stored pattern.
Binary Perceptron units
Taking the case where the perceptron units have binary outputs, we can construct a dividing plane between the output states as a function of the input states. For a two-input unit, this is a line which indicates the transition between the two output states.
The dividing line occurs when h = 0, and it is always at right-angles to the weight vector.
For a problem to be solved by a simple perceptron, it must be possible to draw a dividing line. This is called the condition of linear seperability.
XOR is an example of a function which is not linearly seperable.
Simple perceptrons can be designed using analytical methods: from a truth table, one can construct a dividing line and determine values for weights and threshold that satisfy it.
Learning Rules
For complex problems, the networks cannot be designed analytically. Neural networks can learn by interaction with the environment. Learning uses an iterative process to make incremental adjustments to the weights, using some performance metric.
Supervised learning is based on comparisons between actual outputs and desired output, and requires a teacher of some sort.
Unsupervised learning is when the network creates its own categories based on common features in the input data.
Notation:
The symbol P is used to represent a desired output. Pi( represents the desired response of unit i to an input pattern (.
Supervised Learning
Error-correction rule:
input patterns are applied one at a time
weights are adjusted if actual output differs from desired output
incorrect weights are adjusted by a term proportional to (P O)
weight wij are adjusted by adding a factor (wij
EMBED Equation.3
r is called the learning rate and controls the speed of the learning process. (wij
is zero if P = O
Gradient Descent learning rule
Define a cost function and use a weight adjustment proportional to the derivative of the cost function.
Widrow-Hoff delta rule (specific gradient descent rule)
Select input pattern (
Calculate net input hi and output Oi(
If Pi(=Oi( go back to start
Calculate error: EMBED Equation.3
Adjust weights according to EMBED Equation.3 where EMBED Equation.3
Repeat for next input pattern
Do not adjust weights when x is zero
The equations assume the threshold is represented as the weight of input zero.
me is the margin of error
Associative Networks
fully-connected
symmetrical weights
no connection from unit to itself
inputs and outputs are binary
The input pattern is imposed on the units, and then each unit is updated in turn until the network stabilises on a particular pattern.
Setting the weights for a given pattern:
Stability criterion: net input must have same sign as desired output.
EMBED Equation.3
If 50% of the inputs have the same sign as the pattern bits, then the net input will have the correct sign, and the network will converge to the pattern as a stable attractor.
Storing multiple patterns
If we wish to store K patterns in the network, we use a linear combination of terms, one term for each pattern:
EMBED Equation.3
xi represents patterns that we wish to store within a networks memory.
Hopfields energy function
This describes the network as it is updated. Changes in the state of the network reduce the energy, and attractor states correspond to local minima in the function.
EMBED Equation.3
Unsupervised Hebbian learning
Ojas rule:
EMBED Equation.3
Competitive Learning
Only one output within the network is activated for each network pattern. The unit which is activated is known as the winning unit, and is denoted as i*. The winning unit is the one with the biggest net input: EMBED Equation.3
This technique is used for categorising data, as similar inputs should fire the same output.
The learning rule for these networks is to only update the weights for the winning unit:
EMBED Equation.3
Kohonens algorithm and feature mapping
Competitive learning gives rise to a topographic mapping of inputs to outputs: nearby outputs are activated by nearby inputs. This can give rise to a self-organising feature mapping.
The algorithm involves updating all the weights in the network, but an extra term, called the neighbourhood function makes bigger adjustments in units surrounding the winning unit:
EMBED Equation.3
EMBED Equation.3 is the neighbourhood function, and is equal to 1 for the winning unit.
Shown graphically, the Kohonen network spreads itself like elastic over the feature space, providing a high density of units in areas where there are a high density of input patterns.
Multilayer Binary Perceptron Networks
These include hidden layers, i.e. units whose outputs are not directly accessible. The problem of linear seperability does not apply in these networks.
The first layer categorises the data according to dividing lines. The second layer then combines these to form convex feature spaces. Convex means that the space does not contain holes or indentations. A third layer can then combine several convex spaces together to describe an arbitrary feature space.
Continuous multilayer perceptrons
Instead of having binary outputs, use a sigmoid (0/1) or tanh(-1/+1) activation function to give a continuous output. These functions have a parameter called ( which controls the slope of the transition around the zero point.
Back propagation
This is the technique used to perform gradient descent learning in a multilayer perceptron network. Basically, gradient descent learning is performed first on the connections between the output layer and the hidden layer, and then between the hidden layer and the inputs. You need to perform partial derivatives of the cost function with respect to the two sets of connections.
-.<FNO78TUdexyz{|}~-/QR
+
,
?
@
A
B
jEHUjf@
CJUVaJmH sH jq6] 6H*]
jEHUjf@
CJUVaJmH sH jU jq6]GEF=cd|78N O *
+
C
"C
D
_
RjUVnp
aVWf[HIS"d
&F
RSfghiwx
L_[n^_;<=?KLOP'(= jm
jEHUjf@
CJUVaJmH sH jD jm6] jm6H*] 6H*]
jEHUjqf@
CJUVaJmH sH jU6] 6>*]Bd@QRq)Ok(w
&F
&F
&F=>?@KLMNORSTUWXYj|}wxyV
jEHUjZf@
CJUVaJmH sH
jEHUjf@
CJUVaJmH sH
jsEHUjff@
CJUVaJmH sH
j?
EHUj3f@
CJUVaJmH sH jU jm6H*6H*6] jm6H*] 6H*]6]2Vnwx01VWjklmnopqxy_ka78Kߧy
jEHUjYg@
CJUVaJmH sH
jEHUj>g@
CJUVaJmH sH 6H*]
j&EHUjf@
CJUVaJmH sH
jjEHUjf@
CJUVaJmH sH 6H*]6]
jEHUjBf@
CJUVaJmH sH jU-7Ofg%U V x Z![!l!""""KLMNOPcdef!!" jb6]
j"EHUjGg@
UVmH sH jU
jY EHUj,g@
CJUVaJmH sH ,1h. A!"#$%Dd
J
CA?"2>
+"hHކdD8`!
+"hHކdJ``\xڥ+EA^J!HP
Ys=QY(YVv66V/XxD3\.t3}73gD!PA{/{ͤR]z|\
P
>!!=lv21ZũQw.3^6qQ*7lH]+ChP$98Skdn~?"|O"hо˯s|fk T_s=>ԍ(!*"qEIzx48(#1:ʌ(MLN9Aʍ4֬\* 9>/C>Ł8v~0;V+~L7#T{CŘ-W:{)}f:QfI>bɯ|sI~%)ycF%?_[VF_Zb9P[ogMdsBd9]s-hir-|-wvQ#!
YETyHoӫ*vM'%/ې7<>#V?
+f~؎ݯ?̒k&LSCkV!ahMI$DdTJ
CA?"2b͑]̔*b8`!Zb͑]̔* XJ(xcdd``fd2
ĜL0H
bd?(ㆫaR`:*d3H1)fYˀ0nĒʂT/&`b]F"L'2X@V53L0LL
'] ZZ|3@aox$SQ Od|+?>PP@penR~CP@+v${O0Tlm%4!
1b;LLJ% 4E+n&`]
ML|DdJ
CA?"2N@^
?8`!N@^
?ʢhxcdd`` @c112BYL%bpu_µ`ܞY@*R*dnHfnj_jBP~nb.wCe4
J+ 0<3H?T㇂W竀m(ǰD07qRaǊw₦.pJ[N{LLJ% HX,{Dd J
CA?"2(q6}{8 8`!q6}{8@Cxڕ.P9SL;cf,DDZ`MH1udGc kBs
!"#$%I(+,-/.0124356798:;<>=?@ABCEDFGHTUJKLMNOPQRSVXYZ[\]^_`abcdefghiRoot EntryO F*Data
%WordDocumentN.:ObjectPoolQ0y_1080470790F0y0yOle
CompObjfObjInfo
!$'*-./258;<?BCDGJKLORUXY\_`dfghijkmnopr
FMicrosoft Equation 3.0DS EquationEquation.39q>X,
Oi
=ghi
()=gwij
xj
"ij=1N
"
()Equation Native _1080471327F Ole
CompObj
f
FMicrosoft Equation 3.0DS EquationEquation.39q>,
Oi
=ghi
()=gwij
xjj=0N
"
()ObjInfo
Equation Native _1080471665F..Ole
CompObjfObjInfoEquation Native G_1080473729 'F@;@;
FMicrosoft Equation 3.0DS EquationEquation.39q>+X,
h=w" x
FMicrosoft Equation 3.0DS EquationEquation.39qOle
CompObjfObjInfoEquation Native >L>
wij
=rPi
"Oi
()xj
FMicrosoft Equation 3.0DS EquationEquation.39q_1080475955FOle
CompObjfObjInfoEquation Native V_1080476006"FHHOle
"CompObj #f>:X,
e=me
+h
FMicrosoft Equation 3.0DS EquationEquation.39q>XH|
wij
=ri
xjObjInfo!%Equation Native &t_1080476066$F ϖ ϖOle
(
FMicrosoft Equation 3.0DS EquationEquation.39q>X
i
=Pi
"Oi
()me
+hi
()CompObj#%)fObjInfo&+Equation Native ,_10804767621)FUUOle
0CompObj(*1fObjInfo+3Equation Native 4s
FMicrosoft Equation 3.0DS EquationEquation.39q>WL>
wij
=1Nxi
xj
FMicrosoft Equation 3.0DS Eq_1080483138.F`ܙ`ܙOle
6CompObj-/7fObjInfo09uationEquation.39q>+$
wij
=1Nxi
xj=1K
"
FMicrosoft Equation 3.0DS EquationEquation.39qEquation Native :_1080483719,;3FccOle
=CompObj24>fObjInfo5@Equation Native A_10804840208FOle
E>L>
H="12wij
si
sj
+i
sii
"j
"i
"
i`"j
FMicrosoft Equation 3.0DS EquationEquation.39qCompObj79FfObjInfo:HEquation Native I_10805463666E=F >Ȑ>
wj
=rVxj
"Vwj
()V=wj
xjj
"
FMicrosoft Equation 3.0DS EquationEquation.39qOle
MCompObj<>NfObjInfo?PEquation Native Qc6G-$
hi
=wi
" x
FMicrosoft Equation 3.0DS EquationEquation.39q6{L>
wi*_1080546649BF``Ole
SCompObjACTfObjInfoDVEquation Native W_1080547372@JGFOle
ZCompObjFH[fj
=r(xj
"wi*j
)
FMicrosoft Equation 3.0DS EquationEquation.39q6,
wij
=r(i*
,i)(xjObjInfoI]Equation Native ^_1080547399LFOle
a
"wij
)6/x>w
(i*
,i)Oh+'0 ,
HT
`lt|ObjInfoKMbEquation Native cK1TableW>$SummaryInformation(Peodwzr^ q@G&!*Î Q%&,tŐB,+SAMfWJKb?Ai8IIEno;-U:ys@U7JHuG ~2${0P_\1,I;# O3樝\2a.uX0ueo/=sF}>}ezxzy>Y_F-Of7?+ػ9-,l;+[_n
lG<}~nr57]~Jެ^gZ#~SQN~:#m;{g0>8`!|)>m;{g0>Jxcdd``g2
ĜL0##0KQ*
Wy A?dmv@=P5<%!@5@_L ĺEʙA,a .ȝCTNA $37X/\!(?71k f3ˈ;H&ƿ
g efAkG}L`&O6,ߔż;0` w0AQx
l!6s \PhbD@*l1N
]=Ĥ\Y\qAck\~`nDd|J
CA?"2՞KC%(5v=g8`!՞KC%(5v=gN`
`PxڥS;LPi%?1Nt'!q2FLI i"Hj7F7u#nl.&.ΰZߏ
~BsyK@
c?BBTD}m% x1i9cEkYX!pEl(Nwnt ʲ]+ňA"R-I>{xknxNt
n%jѻ}uklK(M"mOOzG(CmI[$RnS8g5-:VL[yXrg7]~Fwid8̧cS)ΣMwe!M:;T?F7~~:GrfDbCu!SJ/q frr82T78I{" Dd|J
CA?"2YHZ8`!YHZN`h{
xڥ;KAggDsAD-QhDca%-b0g" L~[AB@:j#66>r+G{fg3c^0|lvbBBEqQ6 fbqYlZlq6(d7۫n1P$r)"NbLhaVR:_ڔB`\ZU傳GO6RgJ>m$p+^8 A~\y^J7(v
znϣAkxl}
G#ycy{\qT21C.@ *qa<^)tĩ&ygx$T/.u8H+@QI9@\ .plܶ+\[kF&&\S G10#3X<_fDdXJ
CA?"2Y٫5l\
>478`!Y٫5l\
>4J 7jxcdd``^ @c112BYL%bpu
X.pd4+aV_b~ggdۣU 1A{0=P{dUG9AL(g6TL|&ze
W9@*,vG/U!3
W&0;0s$.|7TYV K%4qS&T[=Ĥ\Y\te[0sDd$ J
CA?"2'OdY 8`!'OdYjhwxڥKPS""XQA`X!`Cb-Y'U$߽$Hxyy}w.!AY')Fb#)i`9Z)]FKZLM,Cn8*l}.a=8iy='s]W4q~et+]}
;Ro]@F뺏ԼXKYsZ
WG9:bҷsq14Wtua).QOy|\˳µb2e_;gnhg07ΐxwmpBIs<2ryK&07,=a6Ғ]GҤ1 <6Ddh0
#A2Jr%y;Ÿl8#8`!Jr%y;Ÿl8ٸ@|TxڝR=KA}3{~șJ"$,IchtD80!]8R0?"?B+snuaͽ7N`'q[8My9*:Gx«[X6paY<0%.pZ]x:{ދ6! &`^-{LO9ĵ7L?e_RW(/L?ݴ Ebz%0U6BƁ.Ino@)oePy_65eqۼH;AOJSnF'Tn]Neural NetworkseurMichael Prior-JonesichichNormal.dotoMichael Prior-Jones21hMicrosoft Word 9.0@ț@YE@x|՜.+,0hp|
l6
K
Neural NDocumentSummaryInformation8l(CompObjqjetworksTitle
FMicrosoft Word Document
MSWordDocWord.Document.89q
i8@8NormalCJ_HaJmH sH tH <A@<Default Paragraph Font
:EF=cd|78NO*+CD_RjUVnp a
VWf[H
I
S
"d@QRq)Ok(wVnwx017Ofg%UVxZ[l0000000000000000000000000000000000000000000000000000000 0 0 0 0 00000000 0 0 0 0 0 000000 0 0 0 00000000000000000000000000000000000000000000000000
=VK"C
d""dxz+?ARfh|Vjlx7KMOce:::::::::::::::=G "TU|~p{]g S
_
<?KMVXnpgqyaklwMichael Prior-JonesvC:\Documents and Settings\Michael Prior-Jones\Application Data\Microsoft\Word\AutoRecovery save of Neural Networks.asdMichael Prior-JonesvC:\Documents and Settings\Michael Prior-Jones\Application Data\Microsoft\Word\AutoRecovery save of Neural Networks.asdMichael Prior-JonesED:\My Documents\Electronics\Notes\year 3 revision\Neural Networks.docMichael Prior-JonesED:\My Documents\Electronics\Notes\year 3 revision\Neural Networks.docMichael Prior-JonesED:\My Documents\Electronics\Notes\year 3 revision\Neural Networks.docMichael Prior-JonesED:\My Documents\Electronics\Notes\year 3 revision\Neural Networks.docMichael Prior-JonesED:\My Documents\Electronics\Notes\year 3 revision\Neural Networks.docMichael Prior-JonesED:\My Documents\Electronics\Notes\year 3 revision\Neural Networks.docMichael Prior-JonesED:\My Documents\Electronics\Notes\year 3 revision\Neural Networks.docMichael Prior-JonesED:\My Documents\Electronics\Notes\year 3 revision\Neural Networks.docw2$ (ޚb36&h^`OJQJo(h^`OJQJo(ohpp^p`OJQJo(h@@^@`OJQJo(h^`OJQJo(oh^`OJQJo(h^`OJQJo(h^`OJQJo(ohPP^P`OJQJo(h^`OJQJo(h^`OJQJo(ohpp^p`OJQJo(h@@^@`OJQJo(h^`OJQJo(oh^`OJQJo(h^`OJQJo(h^`OJQJo(ohPP^P`OJQJo(h^`OJQJo(h^`OJQJo(ohpp^p`OJQJo(h@@^@`OJQJo(h^`OJQJo(oh^`OJQJo(h^`OJQJo(h^`OJQJo(ohPP^P`OJQJo( (wb3 @P@UnknownGz Times New Roman5Symbol3&z Arial?5 z Courier New;Wingdings"qhqdFdfx|
6!$0K2QNeural NetworksMichael Prior-JonesMichael Prior-Jones