Vdovychenko R. Sparse Distributed Representation of Structured Data in Neural Networks

The dissertation is focused on studying phenomenological models of human and animal memory by presenting data with a specific structure (hierarchical, semantic, etc.) in neural networks. The main task of the dissertation is constructing and analysing a hybrid semantic store that would be able to store complete data (for example, structures of interconnected and consecutive key-value pairs) in a neural network. Memory designs to solve this problem were proposed in the 1990s but are not practical due to insufficient scalability and low storage density. The proposed CS-SDM model fills the existing gap between two phenomenological approaches to biological memory modelling by using the third theory - Compressive Sensing. All the statements above determine the relevance of the dissertation work. The scientific novelty of the work consists of developing and researching the characteristics of a new hybrid model of sparse-distributed memory CS-SDM. For the first time, this model combined two directions of phenomenological modelling of memory, providing conditions for the effective use of sparse-distributed memory of the SDM type. Also, applying the theory of Compressive Sensing (CS) was proposed for the first time to model natural memory. The effectiveness of CS-SDM has been proven both formally and experimentally. CS-SDM is the first artificial neural network that entirely and practically suitable capacity allows you to store structured data, which is suitable for preserving semantics. The CS-SDM model has numerous applications. CS-SDM, with a practically usable capacity, allows you to store structured data, which opens prospects for its use in various tasks of artificial intelligence and as a component of neural network models in machine learning. Also, CS-SDM can be used in those fields of human activity where artificial intelligence is used: robotics, semantic search, content generation in social networks, medical diagnostics, etc. In the course of the research, an open-source software library was developed that implements CS-SDM on graphics processors (on the NVIDIA CUDA platform) and also contains implementations of the Kanerva and Jackel SDM designs adapted to the preservation of sparse vectors. The library is implemented as part of the essential software of the SCIT supercomputer complex in the Center for collective use of the equipment of the SCIT supercomputer complex (CCKO SKK "SCIT") at the V.M. Glushkov Institute of Cybernetics of the National Academy of Sciences of Ukraine. The library code is open and available to other researchers on the GitHub platform.