Kondratiuk S. Ukrainian dactyl alphabet gesture modeling and recognition using crossplatform technologies

For this purpose, for the first time, a technology developed using cross-platform tools for modeling gestures of the Ukrainian dactyl alphabet, animation of transitions between states of gesture units and combination of gestures (words) was proposed. The developed technology reproduces a sequence of gestures using a virtual spatial model of the hand and performs dactyl recognition from the input stream of the camera using a convoluted neural network trained on the assembled image set, based on MobileNetv2 architecture, and selected optimal layer configuration and parameters. Accuracy of more than 97% was achieved on the collected test data set. As part of the proposed technology for the preparation of a gesture recognition model, a data set of the Ukrainian dactyl alphabet was collected for the first time, consisting of more than 50,000 images collected from 50 people in different environmental conditions, lighting conditions and supplemented by data augmentation methods. The chosen architecture of the MobileNetv2 convolutional neural network was improved by three-dimensional convolutions, which allowed to teach the recognition model with 0.97 f1-score quality on the test data set of the Ukrainian dactyl alphabet, which is on a par with modern developments in this language in other languages. The results of simulation and experimental research, obtained in the work, can be used to develop advanced human-computer interfaces.