Overview
Neural Network

Custom Control 1.1

Overview

This custom control implements a feed forward multilayer neural network device with training by error backpropagation.

It allows you to define,train and use neural network devices in your programs easily. It also supports save and load networks to disk for later training or use.

Properties        Methods        Events       Error Codes    New in version 1.1

Comments:

Input and output values are float numbers ranging from 0.0 to 1.0 for inputs and 0.1 to 0.9 for outputs, see Outputs() property for details.

The control uses a 'standard' Action property to control the net.

Net Use:

The usual process in implementing neural nets has 3 phases, Design, Train & Use.

Design phase.

The Design phase is simple. Set the topology of net (properties I, O, and optionally H0 and H1) and set Action property to 1 (Create). This creates and initializes the network in a random state in memory.

Train phase.

In the Train phase, you must define a training set of examples , set the Rate, Momentum and MaxError properties, and repeatedly set the Action property to 6 (Train), till the Error property is small enough for your needs.

To support the training set, the control uses a ListBox-like mechanism. It has  List(), ListCount, and ListIndex properties, and the AddItem, RemoveItem and Clear methods, like a normal listbox. (Clear  allowed only in VB 3.0 compliance). List values are string values specifiying the input plus the output values, separated by a space, tab or linefeed. The decimal point, must be expressed with the '.' (point) character. For instance, in a network with 3 inputs and 2 outputs, a valid string might be '.125 .9 .1 .55 .9'. Alternatively, you can set the Inputs() and Outputs() properties and set the action to 2 (AddItem). This adds the current input and output values at the bottom of the List() property.

Once created or trained, the network configuration can be saved on disk for later recall. The file name is specified in the Filename property, and setting action property to 4 (Save) or 5 (SaveTrain) causes the memory image to be written on disk. Note that a network saved with the Save action cannot learn further, but consumes much less memory than a network saved with SaveTrain action. Remember that no save or load of List() values is done, the process to store the training set is completely left to the user.

Use phase.

The Use phase implies getting into the memory a trained network. This can be done in two ways: a) you already have a network created and trained (via action Create and Train); b) You set the Filename property and use the Load action =3.

Once in memory, all you have to do is set input values with the Inputs() table property, and set the action to 7 (Work), or alternatively use the Refresh method. The result is in the Outputs() table property.

Training considerations:

Training a neural network requires a massive floating point use. The objective is to minimize the error of an unknown function defined by input-output couples in the training set, starting at a random network state.

Several cases can occur depending on training set values, initial random state, network topology, and rate and momentum values. They can be grouped in two major cases.

1 - The net can reach a global minimun error.

In this case, network topology is good enough to satisfy the training needs, although the training phase can be slow, and the network can require a very large number of training cycles.

2 - The net cannot reach the global minimun error.

In this case, network topology can be good enough, but initial starting point and rate and momentum values lead to a local minimun, and the network gets trapped in it. Or worse, the net topology is too simple and cannot learn the training set, no matter what number of training cycles used.

As a general rule of thumb, two behaviors can be detected. The net error does not change, no matter what training cycles we use. Or the net error oscillates 'ad aeternum' between two or more local minimuns.

 Avoiding local minima.

To try to dodge the local minima problem, remember that if a deterministic function is defined by the training set, the net can learn it, but if we are trying to learn a non-deterministic function, the error precision becomes decisive. For example, if the training set consist of two examples, one, where input vector A leads to output vector B, AND two, where the same input vector A leads to output vector C, the error cannot be smaller than a difference function between vectors B and C.

Use small rate values. This usually leads to a large number of training cycles, but reduces the probability of getting stuck in an oscilative behavior.

Choosing a network topology.

Start trying a three-layer net with few (ex. 2) hidden neurons. If the net cannot attain the global minimun, increase the number of hidden neurons.

If global minimun cannot be reached, perhaps because you are trying to optimize a function with discontinuities, try with a four-layer network, starting with a few hidden neurons in both layers, and increase the number of hidden neurons as needed. Usually the hidden layer 0 (H0 property), should be bigger than hidden layer 1(H1 property).


SetNNVector Function

This function allows you to write an entire layer of neurons in a single call.

Declare:

Declare Function SetNNVector  Lib "NN.VBX" ( Hctl as Control, Layer as Integer, VBVector() as Double ) as Integer

Parameters:

Hctl:	Must be a created NN control.

Layer:	This Integer indicates which layer we want to write

Legal values:

0 = Inputs()

1 = Hiddens0()

2 = Hiddens1()

3 = Outputs()

Vector:	A one-dimensional array of doubles, big enough to hold the desired values.

Return value:

Four possible values:

0 = Function was succesful

20003 = Network not created

20013 = Bad Layer or Bad Array

20014 = Bad Hctl

Description:

This function checks parameters, and if they are all right, copies the VB array of double values, starting at the lowest bound, to the neurons in the referenced layer.

Note that no checking of valid values is made, so it is possible to write 'illegal' values..!

Note:

The host operating system must conform to at least VB 2.0 compliance.


Functions

In addition to standard properties and methods , version 1.1, adds two declared functions to speed operation when reading or writing layers of neurons.

GetNNVector()

SetNNVector()

These functions have been coded, to speed network interface, because after training, the programs tend to spend most of its time setting or getting input and/or output values.

Due to the limitations of VBX's interface, and VB memory allocation strategy, it is not possible to specify a direct buffer where one can set or read values, so these functions uses VB arrays of doubles to pass information.

Note:

Using VB arrays forces the host operating system to conform to at least VB 2.0 compliance.


GetNNVector Function

This function allows you to read an entire layer of neurons in a single call.

Declare:

Declare Function GetNNVector  Lib "NN.VBX" ( Hctl as Control, Layer as Integer, VBVector() as Double ) as Integer

Parameters:

Hctl:	Must be a created NN control.

Layer:	This Integer indicates which layer we want to read

Legal values:

0 = Inputs()

1 = Hiddens0()

2 = Hiddens1()

3 = Outputs()

Vector:	A one-dimensional array of doubles, big enough to hold the desired values.

Return value:

Four possible values:

0 = Function was succesful

20003 = Network not created

20013 = Bad layer or Bad Array

20014 = Bad Hctl

Description:

This function checks parameters, and if they are all right, copies the neuron values of the referenced layer , to the VB array of doubles, starting at the lowest bound.

Note:

The host operating system must conform to at least VB 2.0 compliance.


New in version 1.1

Four new properties are available in version 1.1

Ins

Outs

Hiddens0

Hiddens1

Ins and Outs properties duplicate 1.0 version  I and O properties. They are provided only for compatibility with Delphi environment .

Hiddens0 and Hiddens1 are provided to access the hidden layers' values.

Two new functions have been declared to speed up reading ort writting layer values.

GetNNVector()

SetNNVector()


Properties

−	I

−	H0

−	H1

−	O

−	Inputs

−	Outputs

−	Action

−	Filename

−	List

−	ListCount

−	Listindex

−	Error

−	Rate

−	Momentum

−	MaxError

−	Ins

−	Outs

−	Hiddens0

−	Hiddens1


I Property

Type int

This property indicates the number of input neurons of the net.

It can only be set before activating the Create action, or after the Destroy action has been used.

Properties


H0 Property

Type int

This property indicates the number of neurons in the first hidden layer.

It can only be set before activating the Create action, or after the Destroy action has been used.

Properties


H1 Property

Type int

This property indicates the number of neurons in the second hidden layer.

It can only be set before activating the Create action, or after the Destroy action has been used.

Properties


O Property

Type int

This property indicates the number of output neurons of the net.

It can only be set before activating the Create action, or after the Destroy action has been used.

Properties


Hiddens1 Property
Hiddens1() Property

Type float

This is an array property to access the hidden layer1 of the net. Legal values are from 0. to 1. . Legal index values are from 0 to H1 property value -1.

Properties


Hiddens0 Property
Hiddens0() Property

Type float

This is an array property to access the hidden layer0 of the net. Legal values are from 0. to 1. . Legal index values are from 0 to H0 property value -1.

Properties


Inputs Property
Inputs() Property

Type float

This is an array property to access the input neurons of the net. Legal values are from 0. to 1. . Legal index values are from 0 to I property value -1.

Properties


Outputs Property
Outputs() Property

Type float

This is an array property to access the output neurons of the net.

Legal index values are from 0 to O property value -1.

Unlike the other layers, Inputs() , Hiddens0() , Hiddens1() , legal values when writing are from 0.1 to 0.9, , but 0.0 to 1.0 when reading.

There is a practical reason for this different range:

Output values are written only when we want to specify the target values for an input-output couple in the training set .

In the training process, output values have to reflect the expected target values, and neurons use a sigmoid function that matches a positive infinite value to 1. , and a negative infinite value to 0..

If a target value of 1. or 0. is specified, an infinite high or low value has to be applied to the neuron, this leads to use very strong conexion values, and numerical stability suffers in  floating point computation systems!

But remenber that it is possible to read a value higher than 0.9 or below 0.1, because the neuron fires from 0.0 to 1.0 !

Properties


Action Property

Type int   (write only)

By setting this property, the user controls the net behavior.

Available actions:

−	0 - None

−	1 - Create

−	2 - AddItem

−	3 - Load

−	4 - Save

−	5 - SaveTrain

−	6 - Train

−	7 - Work

−	8 - Destroy

Properties


Filename Property

Type stringz

This property holds the filename to be used in the Load , Save, and SaveTrain actions. No checking of valid filename is done. If the user sets an invalid filename, an error will be triggered in the Load , Save  or SaveTrain action process.

Properties


List Property
List() Property

Tipe string

This is a property array to hold the training sets. Although it is a string property, at runtime, strings are parsed and float values are stored, or alternatively, stored float values are writen to return a string value. Due to the conversion process, it may happen that you will read a string value different than the written one.

Legal values are a string with input float values plus output float values. For Example, in a net with 3 inputs and 3 output, a valid string could be "0. 1. 0. .1 .5 .9".

Note that values must conform with the legal ranges for inputs() and outputs(), otherwise, an error is triggered.

Note also that the '.' (point) character is used to indicate the decimal point, and space, tab or linefeed characters are used as token separators.

Properties


ListCount Property

Type int

This property holds the count of training sets present in the net. Read only.

Properties


ListIndex Property

Type int

This property is similar to the standard ListIndex property of ListBoxes, but in this control, setting this property to a legal value (below ListCount property value), causes the net to present the List() property value referenced by ListIndex, in the Inputs() and Outputs() properties.

Properties


Error Property

Type float

This property holds the error value found in a Train action. This value is computed as a standard mean square error between output vector values and target vector values.

Properties


Rate Property

Type float

This property holds the rate of gradient correction used to modify connexion values in the training cycle. It is usually set between 0. and 1.

Even if this isn't the standard method, you can change its value between training cycles.

Properties


Momentum Property

Type float

This property holds the momentum (inertia) constant used to correct the network connexions. It is usually set between 0. and 1.

Even if it is not the standard method, you can change its value between training cycles.

Properties


Outs Property

Type int

This property is included in version 1.1 to access the 0 property in Delphi environments.

See new in version 1.1.

Properties


Ins Property

Type int

This property is included in version 1.1 to access the I property in Delphi environments.

See new in version 1.1.

Properties


MaxError Property

Type float

This property holds the maximun error allowed in a train action. If after train is done the Error property is bigger than this value, the network corrects its weights. If not, training is ignored, and a Trained event is fired, so it is a waste of time to keep using the Train action if Error is lower than this value.

Properties


Network Topology

The neural network custom control implements a multilayer design. Each layer is fully connected to the next, and an (invisible) bias neuron is added at each layer.

To keep the complexity at the lowest level possible, this version does not support other interconnecting possibilities.

You can choose among between a 2, 3 or 4 layer design by setting the  I , O , H0 , and H1  properties.

To select a 2 layer design, leave the H0 and H1 properties at zero.

To select a 3 layer design, leave only the H0 or H1 properties at zero.

To select a 4 layer design, set the I, O, H0, and H1 properties to a non-zero value.

Memory considerations & Limitations.

This simple scheme does not imply a limitation of network capabilities. It is assumed on paper that any degree of complexity can be achieved with a 4-layer net.

You can set any value in the I, O, H0 and H1 properties, but  the number of connexions between two adjacent layers must be less than 8000. According to the connexion scheme explained before, this number is computed as:

   (neurons_in_previous_layer + 1) * neurons_in_next_layer

This limitation arises from the fact that any connexion is an 8 byte value in work mode, and three 8-byte value in train mode. To avoid the use of huge pointers, and speed up operation in this version, any layer vector must fit into a segment area, so, that leads to the 8000 limit.


Trainnig
Training

The net uses a 'standard' gradient/momentum method to implement training.

It consists of two phases:

Phase One.

Traing set (List() property values) is presented to the net and the output values are compared with expected target values. With this info, gradient vectors and Error property are computed, and if Error is bigger than MaxError property value, the net activates the phase two. Otherwise, the control fires a Trained Event to indicate that no correction will be applied to connexions.

Phase Two.

In this phase, the net uses the gradient vectors computed in phase one, the previous momentum vectors, and the Rate and Momentum property values, to correct the connexions in order to approach the solution and save the actual momentum vector for the next training cycle.


Trainnig Sets
Training Sets

In work time, a neural network device is capable of reproducing in response to its actual inputs values the output values 'learned' in the train phase . To support the training set, or collection of input-output couples, this control implements a ListBox-like mechanism.

Any couple of inputs-outputs values is stored as an entry in the List() property. To support the insert , remove or access of items (examples), three properties ( List(), ListCount, ListIndex),  three methods ( AddItem , RemoveItem, Clear ), and an Action value (AddItem (2)) are implemented.

In any Train action, the entire set of List values is considered, but you can change the List contents between Train actions.

No load or save of List() values is supported. The user must implement his own system to store the training set.


Methods

•	AddItem

•	RemoveItem

•	Clear

•	Refresh


AddItem method

This method is similar to the standard AddItem method of ListBoxes, except for the format of the string value. See List() property for details.

Methods


RemoveItem method

This method is similar to the RemoveItem method of ListBoxes.

Methods


Clear method

This method is similar to the standard Clear method of ListBoxes. It deallocates the current  training set. and set the Listcount property to 0, and ListIndex property to -1.

Avalilable only in VB 3.0 compliance systems. VC++, Delphi environments supports only VB 1.0 compliance !

Methods


Refresh method

This method is used to activate the network. It is equivalent to setting the Action property to 7 (Work action).

Methods


Create action

Value 1

This action checks the I , O, H0 and H1 propeties, and if legal, allocates the net, initiates the connexions at a random state, and zeroes the momentum vectors. Once allocated, the net topology cannot be changed. You must use the Destroy action to free resources and return the net to a not created state.

Actions


AddItem action

Value 2

This action adds the current input and output values ( Inputs() and Outputs() properties) to the bottom of List() values. It is implemented to facilitate the training set definition.

Actions


Load action

Value 3

This action uses the Filename property to load a network previously saved with the Save or SaveTrain action.

Actions


Save action

Value 4

This action uses the Filename property to store in disk the current net configuration, excluding training vectors. This results in a small image, but  note that networks saved with this option cannot learn further.

This action is recommended when the net has been completely trained and no further learning is required.

Actions


SaveTrain action

Value 5

This action uses the Filename property to store actual network configuration in disk, including training vectors. Nets saved with this action can be loaded later, and can continue the learning process.

Note that though the training vectors are saved, the training sets (List() property values) aren't saved. The user must implement his own processing to save and load the training sets. This is done to make the control usage more flexible, allowing the user to integrate the control with external data.

Actions


Train action

Value 6

This action starts a training cycle. Once finished, the Error property indicates the Error amount found, and if this Error value is less than the MaxError property, a Trained Event will be fired.

Actions


Work action

Value 7

This action activates the network. Actual Inputs() values are presented, and  Hiddens0() , Hiddens1() and Output() values are computed.

Actions


Destroy action

Value 8

This action deallocates the net resources, frees List() values, and returns the net to an initial not-created state. Only in this state can the network topology be changed.

Actions


Events

The control implements a unique event in response to the Train action.

Trained Event

If the Error property falls below the MaxError property in response to the Train action, the control fires the Trained Event to indicate that no further correction will be applied to connexions. This event has no parameters.


Error Codes

The following set of error codes are supported.

20000 Network already created, Use Destroy Action.

20001 Invalid Value.

20002 No Available Memory.

20003 Network not created, Use Create Action.

20004 Reserved.

20005 Too many connexions. Value not allowed.

20006 Cannot open file.

20007 Cannot write file.

20008 Cannot read file.

20009 Incompatible Neural Network version.

20010 Input or Output len < 1.

20011 Listcount<1. Nothing to learn.

20012 Network cannot learn.

Function error codes:

20013 Bad Layer or Array.

20014 Bad Hctl.