Bondarenko K. Mathematical models and computational methods for detecting anomalies in security systems

Bondarenko K.O. Mathematical models and computational methods for detecting anomalies in security systems - Qualification scientific work on the rights of manuscript. Dissertation for the degree of Doctor of Philosophy in speciality 125 Cybersecurity and Information Protection, field of knowledge 12 - Information Technology, National Technical University "Kharkiv Polytechnic Institute", Ministry of Education and Science of Ukraine, Kharkiv, 2024. The dissertation is devoted to solving the problem of ensuring an adequate level of security of protected objects by developing and implementing mathematical models and computational methods for detecting anomalies in security systems. By using the developed models and methods of data mining and neural networks for anomaly detection, it becomes possible to detect and prevent attacks unknown to the security system, which is a prerequisite for improving the cybersecurity of any system. The object of research is the processes of detecting anomalies in information security systems. The subject of the research is mathematical models and computational methods for detecting anomalies in security systems based on neural network and data mining (classification trees) methods. The purpose of the dissertation is to develop mathematical models and computational methods for detecting anomalies in security systems that ensure an increase in the level of security of information protection systems. The system itself should be easy to use and configure, as well as easily transferable between different software systems. The introduction substantiates the relevance of the topic of the dissertation research, formulates the purpose of the research and the scientific and applied tasks necessary to achieve it, shows the connection of the research with scientific programmes and topics, presents the scientific novelty of the results obtained, their practical value and the personal contribution of the applicant. Information on the testing of the results of the work, the personal contribution of the applicant and his publications is provided. The first chapter analyses the current state of anomaly detection in security systems, discusses network anomalies, their origin and taxonomy. The sources of anomalies in security systems are identified. A comparison of anomalies with cyber attacks on computer systems and networks is provided, and the cause-and-effect relationship between intruder attacks, network anomalies and their consequences for the security of an organisation's network is presented. A reflection of the impact of network service anomalies on security and quality of service goals is built. The second section analyses the existing theoretical models of anomaly detection: operational model, mean and standard deviation model, multivariate model, Markov process model, time series model. An intrusion detection algorithm is proposed. The attributes of measures and methods of anomaly detection are analysed, which allowed to determine the appropriate methods of anomaly detection. The analysis of anomaly metrics based on proximity measures allowed to justify the choice of the Mahalonobis proximity measure as the basis of anomaly metrics. Section 3 analyses various methods of anomaly detection based on machine learning. The correspondence between the used machine learning methods for artificial neural networks and cybersecurity tasks is formulated. A mathematical model of anomaly and intrusion detection based on genetic algorithms is developed. In the fourth section, an approach is proposed that consistently classifies known attack traffic into different types of attacks and simultaneously separates anomalies from normal traffic. The application of the abuse detection model to the KDD CUP 99 dataset is demonstrated. It is proposed to use a genetic algorithm to select appropriate parameter values, optimise the RF classifier and improve the accuracy of classification of normal and anomalous network traffic and its implementation using the built artificial neural network of a multi-level perceptron and methods of building classification trees in the Statistica package. The conclusions of the dissertation outline the main results arising from the research, present and characterise the performance indicators when using the proposed solutions. The following scientific results were obtained as a result of the study: 1. For the first time, the choice of the Mahalanobis metric as a basis for determining anomalies is substantiated. 2. The system of cause-and-effect relationships between intruder attacks, network anomalies and their consequences for the security of an organisation's network has been improved 3. The mathematical model for detecting anomalies and intrusions based on genetic algorithms has been improved 4. An improved approach to the sequential classification of known attack traffic into different types of attacks.