Dudar V. Training algorithms and invariance to geometric transformations of neural networks

Thesis is devoted to the development and theoretical foundation of new algorithms for training and achieving invariance to geometric transformations of neural networks. A two-stage subspace trust region approach is developed for deep neural network training. Proposed method uses second order information (second derivatives of the error fucntion) for making steps in the weight space. Experiments show that proposed method works faster than exsiting methods of second order and methods of first order in case network is deep enough. We developed new regularization method called Column Drop. It can be interpreted in terms of training set augmentation and representing convolutional neural network as an ensemble of smaller models acting on different subimages of the input. This method imposes some limitations on neural architecture. It case these conditions are satisfied, it shows better resuts than dropout and data augmentation on test set.