Nikolaiev S. Information technology for human heart rate variability remote non-contact determination from video stream

The dissertation is devoted to the creation of the modern video processing information technology that uses widely distributed web- and other video cameras for contactless remote measurement of heart rate variability that might provide an adequate level of heart health without changing person’s lifestyle and schedule. This task is extremely important for personalized, prognostic and preventive medicine paradigm, which is a global trend in the 21st century that stands for constant human heart monitoring before any illness has been detected. In the dissertation during the process of research and IT creation the following tasks have been solved: finding stable faces’ coordinates on the frames, taken from the webcam video stream; developing probabilistic skin detectors that with the help of RBF neural networks are capable of separating facial skin pixel mask and the background of the frames; obtaining the signal of skin tint change in a form of time series that capture slight fluctuations of oxygenated blood spectral characteristics during each heartbeat cycle; applying amplitude differential correction for removing cameras frame-to-frame adaptation ripples and blind source separation methods to find a projected component of skin tint signal to luminance independent plane that corresponds to rPPG signal; resampling of unevenly sampled raw rPPG signal and its further frequency filtration with a finite impulse response filter for noise reduction; annotating R-peaks that correspond to the moments in time of heart beats on the filtered rPPG signal and validating them for obtaining RR intervals. A newly proposed similarity criterion Q is used for estimating the time series of heart contractions moments (R-peaks) annotation accuracy compared to the reference series obtained from Holter monitor device in clinical conditions. During the experimental phase people had been sitting in front of cameras with put on Holter monitoring device and as the result simultaneously two datasets were recorded: R-peaks from developed IT with corresponding high-quality video and R-peaks from Holter monitor ECG. People of different age groups and genders participated in the experiment. Experiments were repeated for calm conditions and immediately after 5 minutes of physical exercises. This setup allows to evaluate the quality and adjust the parameters of the proposed IT. The results of the final finetuned developed system show high accuracy of individual heart contractions recognition in time and thus the high quality of obtained HRV. Developed IT can be used, for example, in insurance medicine providing SaaS solution to enable early prediction and prevention of health issues and therefore greatly reduce insurance cases and losses. The essence of the service is to detect as soon as possible the (pre)illness conditions of insurance company clients via remote constant rPPG monitoring. The web-based service with thick client using the proposed information technology was designed for these insurance companies. The thick client in the developed IT is used to process video stream from widely distributed webcams on clients’ devices and to perform rPPG analysis, enabling remote contactless continuous geographically distributed HRV monitoring service. The general architecture and the implementation details, including database structure, classes diagrams, communication protocols of the service for both commercial and non-commercial use are presented in the thesis. This approach allows to get additional source of information and better reevaluate risks. The experimental evaluations of the proposed system were conducted on car drivers during intercity rides where strong correlations of stress index and tiredness levels were found depending on route length. Experiments were conducted on farmers during tractor driving in the field that has also shown correlations with farmers’ internal states and as the result it can be stated that in this case more specialized cameras are needed to cope with noise from tractor vibrations. Another set of experiments was conducted on students before and during exams for stress testing. It was shown that behind poker faces of students – some of them experience extraordinarily high stress while others are not stressed at all. Questionnaire of these students has proved that measurements of the system are in accordance with what students were experiencing during the experiment. It was clearly shown that an experienced professor who was conducting the exam could not distinguish internal state of some students with mere external visual observation. Also, one set of experiments was related to extracting rPPG signal from recorded videos. It was successfully shown during analysis of D. Trump and H. Clinton debates video recordings that it is possible to retrieve rPPG signals having high-quality video source.