MATLAB NEURAL NETWORK TOOLBOX REVIEW

Review By Lou Mendelsohn

THE MATH WORKS INC.
Cochituate Place
24 Prime Park Way
Natick, MA 01760
Phone: 508 653-1415
Fax:508 653-2997
Product: Software designed to work within the MATLAB environment to develop neural networks for industry, education and research.
Equipment requirements: MATLAB technical computing environment, 80386 or higher processor, 387/487 math coprocessor, Windows 3.1 and Windows-supported mouse. 8 MB of free space on hard drive. 4 MB of extended memory, (8 MB or more is required to use MATLAB 4.0 3-D color graphics and image processing capabilities.)
Recommended equipment: At least 8 MB of RAM, 8-bit graphics adapter and display (to achieve 256 simultaneous colors), Microsoft Windows-supported graphics accelerator card and printer.
Other platforms: Apple Macintosh, Sun SPARC, DEC RISC, Dec ALPHA, SGI, HP9000 Series 300/400/700, IBM Rs/6000 and VAX/VMS, CONVEX and CRAY.
Other operating systems:Apple Macintosh, UNIX and VMS.
Price: $895

The Neural Network Toolbox is one of 15 application-specific software toolboxes designed to run in the MATLAB technical computing environment. (See the product review for MATLAB for Windows 4.0.) Designed as a tool for industry, education and research, it provides the user with a means of developing various neural networks and graphically analyzing both their training process and results.

When installed, the functions that the Toolbox provides are seamlessly integrated into MATLAB. The Neural Network Toolbox is composed of a number of M-files, which are the standard script and function files for MATLAB, which in turn are written in ASCII. This means that the user can directly examine the inner workings of all the toolbox functions and examples. Using these files as guides, the user can experiment with the Toolbox to become familiar with the product.

The MATLAB Neural Network Toolbox has not attempted to provide a definitive set of neural network functions. Instead, due to the rapidly developing nature of the field, the goal has been to provide the user with all the necessary elements for network development and exploration, so that he or she can find what works and what doesn’t for a specific application. The Neural Network Toolbox comes with an excellent user’s guide filled with a wide variety of examples. Each example is also included in the Toolbox itself so it can be run in MATLAB, while the user follows along in the guide. The manner in which these examples are presented, coupled with the fact that the user can examine the code that executes them, allows the user to become proficient with the Toolbox’s capabilities.

THE HARD FACTS
The M-files that make up the Neural Network Toolbox fall into three primary categories: transfer function specifications for network simulation, learning rules for updating network parameters and training functions for network training on data.

The transfer function for a given neuron provides the means by which the inputs of that neuron are converted to outputs with desired characteristics. Available transfer functions in the Toolbox include competitive layer, hard limit, symmetric hard limit, log-sigmoid, linear, saturated linear and tan-sigmoid functions. One of the most commonly used transfer functions is the log-sigmoid function, which is used to scale the input of a neuron from the range of plus or minus infinity to the range of zero to one. In addition, because it is differentiable, it is commonly used in back-propagation networks, one of the most widely used training paradigms. In addition, custom-tailored transfer functions can be created by the user.

The learning rules provided in the Toolbox are designed to alter the weights and biases of the network neurons. These rules include perception, Widrow-Hoff, back-propagation, associative learning, competitive learning, feature maps and Hopfield. The user is also able to create an entirely new learning rule or one that is a combination of the learning rules provided. The Neural Network Toolbox also provides a number of enhancements to the back-propagation paradigm, which include the addition of a momentum term, a method for improving initial weight conditions and an adaptive learning rate. Such features have been shown to increase training speeds on some problems. In addition, The Math Works is also working on the next version of the Toolbox, with new features to include a significant speed improvement in training back propagation networks plus the addition of two new paradigms, radial basis functions and learning vector quantization. At present, the new release is stated for January 1994.

The training functions, of course, vary from one type of learning to the next, but all require similar information to perform the training task — typically, an initial weight matrix, the initial bias vector, a matrix of input vectors, target vectors, the number of epochs between sampling of the error for display, the maximum number of epochs to train, the error goal of the network and the learning rate. The training function then returns new weights and biases once the maximum number of epochs has been reached or the network error has dropped below a preset error goal.

TRAINING A NETWORK
A network does through three phases during the training process in MATLAB: the presentation phase, the check phase and the learning phase. The presentation phase presents the matrix of input vectors and then calculates the matrix of output vectors. During the check phase, the output vectors are compared to the target vectors. If all input vectors generate the correct output vectors, the training stops. If the input vectors do not generate the correct output vectors, the training process continues with the learning phase, which will adjust weights and biases with the specified learning rule and then begin the presentation phase again. For many applications it may be desirable to apply multiple input vectors simultaneously and retrieve the network response for each of the vectors. For example, batching input vectors in a back propagation network will give the true gradient of the network. In addition, batching operations are often more efficient. Factors such as biases and weighted inputs are automatically accommodated by indicating the use of a batch process to the transfer function.

GRAPHING
One of the most convenient features found when working with neural networks in MATLAB is the extensive graphing capabilities. MATLAB contains both elementary and specialized plotting functions for two-dimensional graphs and a variety of functions to display three-dimensional data. Some of the three-dimensional functions plot lines, while others draw surfaces and wire frames. These functions can be used to display the error surface of a network, the learning rate and function approximations. Figure 1 shows the error surface of a back-propagation network. The path (from upper left to lower right) shows the change in weight bias and sum-squared error over the course of training.

Since MATLAB’s graphs can change dynamically, it is possible to view network information graphically during the training process. For instance, one of the examples included with the Toolbox performs function optimization using back-propagation. Figure 2 depicts the network output for this example before training has begun. Figure 3 shows the network output at epoch 2,000. The network output after 6,000 epochs can be seen in Figure 4. With MATLAB’S graphic capabilities, the user can observe network performance like this as it occurs.

In addition to MATLAB’s display and plotting capabilities, the Neural Network Toolbox includes specialized graphing functions. The ploterr function plots the sum-squared error of a network against the training epochs, while the plotlr function plots the adaptive learning rate of a network against the training epochs. A feature map diagram for a layer of neurons can be plotted using the plotmap function, and the hintonw function plots a Hinton diagram, which graphically displays the elements of a weight matrix. Figure 5 shows the Hinton diagram for the first layer of a four-input, three-hidden neuron network. The area of each square is proportional to the magnitude of the weight connection between the input and hidden neurons associated with that square. Positive weights are distinguished by a plus through the square. This type of diagram is designed to help make the relative significance of elements in a large weight matrix more apparent, thus making it easier to see which weights have the greatest effect on a neuron’s output. These methods of monitoring the training of a network and displaying the end results graphically give the user an opportunity to visualize information that may not be obvious when presented in a numerical format.

SUMMARY
The flexibility and seamless integration of the Neural Network Toolbox into the MATLAB environment allows the user to explore a broad range of network paradigms and architectures while also providing a variety of methods for quantitatively and graphically monitoring network training and analyzing results. This product will provide those relatively new to neural networks with all the functionality and examples they need to experiment with this technology. The veteran neural network researcher should be able to utilize the mathematical, statistical and data visualization capabilities built into MATLAB to analyze input data, network performance and results in a variety of ways.

Lou Mendelsohn, 813 973-0496, fax 813 973-2700, is president of Market Technologies , Wesley Chapel, FL., a research, software development and consulting firm involved in the application of artificial intelligence to financial market analysis.

Reprinted from Technical Analysis of
Stocks & Commodities magazine. (C) 1994 Technical Analysis, Inc.,
4757 California Avenue S.W., Seattle, WA 98116-4499, (800) 832-4642.