Deep Learning is a branch of Machine Learning that is gaining more and more popularity with time. In this post, you will read about some important Deep Learning algorithms.
The multilayer perceptron (MLP) is a type of formal neural network, organized in several layers in which information flows only from the input layer to the output layer; it is therefore, a feedforward network. Each layer consists of a variable number of neurons, the neurons of the last layer (called “output”) being the outputs of the global system.
Multilayer Perceptron with Backpropagation
In the multilayer perceptron with backpropagation, the neurons of a layer are connected to all the neurons of the adjacent layers. These links are subjected to a coefficient, altering the effect of information on the destination neuron. Thus, the weight of each of these links is the key element for the operation of the network: the implementation of a Multilayer Perceptron to solve a problem therefore requires the determination of the best weights applicable to each of the inter-neuronal connections. Here, this determination is carried out through a backpropagation algorithm.
Convolutional Neural Networks
In machine learning, a convolutional neural network or neural network convolution (CNN or ConvNet) is a type of artificial neural network acyclic ( feed-forward ), wherein the connection pattern between the neurons is inspired by the visual cortex of the animals. The neurons in this region of the brain are arranged so that they correspond to the regions that overlap during tiling of the visual field. Their operation is inspired by biological processes and they consist of a multilayer stack of perceptrons, the purpose of which is to pretreat 3 small amounts of information. Convolutional neural networks have wide applications in image and video recognition, recommendation systems and natural language processing.
Consider the analysis of a monochrome image (in 2 dimensions – width and height) or in color (in 3 dimensions, considering the RGB image with 3 depth units, the third of which corresponds to the stacking of 3 images according to each color; red, green and blue).
A convolutional neural network consists of two types of artificial neurons, arranged in “layers” and successively processing the information:
- The processing neurons, which process a limited portion of the image (called the “receptive field”) through a convolution function.
- The pooling neurons or so-called pooling exits (total or partial).
Nonlinear and point correction can be applied between each layer to improve the relevance of the result.
The set of outputs of a treatment layer makes it possible to reconstitute an intermediate image, which will serve as a basis for the next layer.
Features and Benefits
A major advantage of convolutional networks is the use of a unique weight associated with the signals entering all the neurons of the same convolution core. This method reduces the memory footprint, improves performance 3 and also allows translation processing invariance. This is the main advantage of the convolutional neuron network with respect to the multilayer perceptron, considering each independent neuron and thereby, assigning a different weight to each incoming signal.
When the input volume varies in time (video or sound), it becomes interesting to add a parameter along the time scale in the setting of neurons. In this case, we will speak of a neural network with time delay( TDNN ).
Compared to other image classification algorithms, convolutional neural networks use relatively little pre-processing. This means that the network is responsible for changing its own filters by itself (learning without supervision), which is not the case for other more traditional algorithms. The absence of initial parameterization and human intervention is a major asset of the CNN.
Recurrent Neural Networks
A network of recurrent neurons is understood as a network of artificial neurons with recurrent connections. A network of recurrent neurons consists of interconnected units ( neurons ) interacting non-linearly and for which, there is at least one cycle in the structure. The units are connected by arches (synapses) that have a weight. The output of a neuron is a nonlinear combination of its inputs.
Recurrent neural networks are suitable for input data of variable size. They are particularly suitable for time series analysis. They are used in automatic speech recognition or handwriting recognition – more generally in pattern recognition or in machine translation. “Unfolded”, they are comparable to conventional neural networks with equality constraints between the network weights. Training techniques of this network are the same as for conventional networks ( backpropagation of the gradient), however recurrent neural networks face the problem of disappearance of the gradient to learn to memorize past events. Particular architectures respond to this latter problem, in particular Long Short-term memory networks. The behaviors of recurrent neural networks can be studied with the theory of bifurcations, but the complexity of this study increases very rapidly with the number of neurons.
Long short-term memory networks
A long short-term memory (LSTM) network is the most frequently used recurrent neural network architecture.
The idea associated with LSTM is that each computational unit is not only tied to a hidden state h but also, to a state c of the cell that plays the role of memory. The passage of ct-1 at ct is done by the transfer to constant gain and is equal to . In this way, errors propagate to the previous steps (up to 1000 steps in the past) without the disappearance of gradient. The state of the cell can be modified through a gate that allows or blocks the update ( input gate ). Similarly, a gate checks if the cell state is communicated at the output of the LSTM unit ( output gate ). The most widespread version of LSTM also uses a gate for allowing the resetting of the cell state ( forget gate ).