Sulema Y. Methods, Models, and Means of Researched Objects Digital Twins Multimodal Data Processing

The dissertation is devoted to solving a topical scientific and technical problem of the efficiency increasing of temporal multimodal data processing in computer systems. In the work, a theoretical basis for representing the temporal multimodal data of a digital twin to describe a researched object is developed. An apparatus of the algebraic system of aggregates is developed as a tool for such a description. A distinguishing feature of the proposed algebraic system is a consideration of the elements order when performing operations as well as the introduction of ordering operations and special relations (frequency, interval) between the elements. The operations and relations between the elements of the proposed algebraic system, including fuzzy elements, are defined. The concept of a multi-image is proposed for the formal specification of digital twin models and the description of the researched object. A method for creating a multi-image of a researched object is developed; it consists in the sequential implementation of procedures for the data structure forming, the multi-image specification decomposition, temporal multimodal data preparing, data fusion, sorting, and singling. A method for multi-images synchronization is proposed; it consists in the implementation of procedures for synchronization template forming, multi-images ordering, and pairwise synchronization of multi-images. A method for digital twin graphical data adaptive compression is developed; it is based on the application of morphological operations and defining regions of interest. A method of confidential storage of digital twin data is formulated; a distinguishing feature of this method is the application of the data protection procedure depending on the data category. The application of these methods enables developing the complete set of software for digital twins technology support. Three digital twin models are developed: the muxel model, the temporal linked model, and the multilevel ontological model. These models allow to represent the object at different levels of abstraction. The muxel model is designed to provide detailed information about the researched object. The temporal linked model is used to represent the object in the dynamics of changes in its state. In a multilevel ontological model, the researched object is represented as a composition of its components. The combining of these models enables the representation of the researched object as detailed as possible. The paradigm of multi-images programming for digital twins temporal multimodal data processing is proposed; its distinguishing feature is the use of a new entity, which is a multi-image of the researched object, as the main component of the computational process that allows to simplify software development for digital twins temporal multimodal data processing systems. Features of this paradigm are the use of the apparatus of the algebraic system of aggregates for processing temporal multimodal data; the use of temporal multimodal data synchronization and aggregation procedures as a main component of the computational model; the focus on using heterogenous data sources. To implement the multi-image programming paradigm, a basic computational model, and a computational model for digital twins technology are proposed. The ASAMPL programming language for processing temporal multimodal data is developed; its syntax and semantics are presented, and the application of the ASAMPL programming language for solving applied problems is demonstrated. An architecture of the software system for digital twins temporal multimodal data processing is proposed and the options of its application for software development for health care and distance education are shown and discussed. A feature of the developed architecture is the use of a two-level procedure for synchronization and aggregation of temporal multimodal data. The proposed approach provides the efficient processing of temporal multimodal data as well as it enables reducing programming man-hours due to simplifying the process of applied software development for digital twins temporal multimodal data processing systems.