Chernodub A. Training the dynamic neural networks for long-term predictions

The thesis is dedicated to the problems of training dynamic neural networks for forecasts of time series and control of non-linear dynamic plants. Analysis of current dynamic neural network architectures, methods of their training and methods of neurocontrol were performed. As a result, novel methods were proposed for learning long-term dependencies in the training data for feedforward neural networks with tapped delay lines and in recurrent neural networks. For feedforward networks, a method called "Forecasted Propagation Through Time" was developed for increasing the accuracy of multi-step-ahead predictions. For recurrent networks, a pseudoregularization method was developed. It controls the norm of backpropagated gradients and prevents the vanishing gradients effect that decreases long-term memory inside the recurrent networks. This method of gradient pseudoregularization was also adopted to the neurocontrol problem. Controlling the norm of backpropagated gradients inside neuroemulators increases the quality of training neurocontrollers by removing the vanishing gradients effect in the deep "neuroemulator + neurocontroller" neural network bundle.