Nikitin V. Methods of increasing the efficiency of data consistency in information systems

The dissertation work is devoted to the development of methods for distributed document-oriented databases that allow speeding up data reconciliation and improving collision resistance in the process of searching for inconsistent data in various network information systems, such as IoT, heterogeneous multicomputer systems, analytical systems of administrative management, financial systems, research on environmental safety and nature management, and others. Also, special software for conducting experiments was implemented. The following results were obtained as a result of the dissertation work. A method of ensuring data consistency in distributed NoSQL document-oriented databases using a transactional clock has been developed. It receives transactions from client applications and stores them in appropriate queues. Queues are processed depending on the priority of the transaction. With the highest priority, the queue is processed earlier, compared to those with a lower priority. This allows you to figure out critical data (for example, transmitting financial transactions) that should be processed first. When the queue is processed, the transaction clock merges the transactions into a resulting transaction. To do this, it uses the creation time of the transaction, which allows them to be globally ordered and merged in the order in which they were created, rather than received by the transactional clock. After receiving the resulting transaction, it is transmitted to other replicas on which writing takes place. It should be noted that when using a transactional clock, read operations take place directly from replicas, which allows to reduce the load on the host machine. The active anti-entropy method has been improved using a Bloom spectral filter and a hashing algorithm instead of a Merkle tree. Its creation is due to the fact that the classical mechanism of active anti-entropy uses many expensive hashing operations. Also, when hashing a large amount of data, the probability of collisions increases, which can affect the timely identification of inconsistencies. If this happens, the system may be in an inconsistent state because the start of the reconciliation procedure will occur later. Therefore, for distributed NoSQL document-oriented databases, it was decided to use the gossip protocol, which consists in a decentralized method of node interaction. This provides reliability, compared to a centralized approach, because the failure of one node will not affect the availability of the system. The problem with using a centralized approach is that failing a master requires some latency due to the consensus protocol used to elect a new master from existing replicas. A decentralized approach allows the information system to be available for record operations, although this makes it difficult to maintain consistency. To search for inconsistencies, a certain snapshot is used, which consists of a spectral Bloom filter and a hash value. The algorithm to produce the Bloom spectral filter was modified specifically for the developed method of active anti-entropy, which makes it possible to speed up the identification of data inconsistencies in the method of active anti-entropy. The speed of forming the spectral Bloom filter has been increased by using an algorithm based on prime numbers instead of using hash functions. The results of the experiments show that the developed algorithm has higher collision resistance comparing to single hash function and higher speed comparing to usage of a several hashing functions. Also, a collision-resistant hashing method was developed specifically for the developed active anti-entropy method. Its purpose is to reduce the number of collisions when hashing data that differs in size. This is very important for active antientropy, as it allows early mismatch detection. A special transaction clock service with its own application programming interface has been implemented for the possibility of using the developed method of ensuring consistency using a transaction clock with a distributed MongoDB database in information systems. A special service of active antientropy with its own application programming interface has been implemented for the use in information systems of the developed method of active antientropy with the MongoDB distributed database. A prototype of the financial information system was implemented, in which the distributed database consists of eight nodes. With its use, the results of the study of the developed methods of ensuring data consistency were obtained. The application programming interface was implemented using the Python3 programming language. Docker and docker-compose were used to create test environment and orchestrate the necessary components.