ASTRA: A MULTIMODAL HAND-GESTURE AND VOICE-COMMAND SYSTEM FOR HANDS-FREE COMPUTER CONTROL WITH AFFORDABLE HARDWARE

Authors

  • Abdurashidova Kamola Turg’unbayevna Author
  • urmamatov Samandar Author

Keywords:

human-computer interaction; multimodal fusion; gesture recognition; voice control; MediaPipe; 3D-CNN; LSTM; Transformer ASR; Dynamic Bayesian Network; adaptive pinch; exponential smoothing; privacyfirst design; accessibility.

Abstract

Touchless human-computer interaction (HCI) is increasingly demanded in sterile environments, accessibility scenarios for users with motor impairments, and immersive workflows. Existing solutions either rely on costly proprietary depth sensors (e.g., Leap Motion) or are tied to cloud services that introduce latency and privacy concerns. This paper presents ASTRA, a multimodal HCI system that combines hand-gesture and voice control on commodity hardware (a Logitech 720p webcam and a USB microphone).

References

1. Tran D., Bourdev L., Fergus R., et al. Learning spatiotemporal features with 3D

convolutional networks // Proceedings of the IEEE International Conference

on Computer Vision (ICCV). — 2015. — P. 4489–4497.

2. Hochreiter S., Schmidhuber J. Long short-term memory // Neural Computation.

— 1997. — Vol. 9, № 8. — P. 1735–1780.

3. Lugaresi C., Tang J., Nash H., et al. MediaPipe: a framework for building

perception pipelines // arXiv preprint arXiv:1906.08172. — 2019.

4. Weichert F., Bachmann D., Rudak B., Fisseler D. Analysis of the accuracy and

robustness of the Leap Motion controller // Sensors. — 2013. — Vol. 13, № 5.

— P. 6380–6393.

5. Vaswani A., Shazeer N., Parmar N., et al. Attention is all you need // Advances

in Neural Information Processing Systems (NeurIPS). — 2017. — P. 5998–

6008.

6. Devlin J., Chang M.-W., Lee K., Toutanova K. BERT: pre-training of deep

bidirectional transformers for language understanding // arXiv preprint

arXiv:1810.04805. — 2018.

7. Liu Y., Ott M., Goyal N., et al. RoBERTa: a robustly optimized BERT

pretraining approach // arXiv preprint arXiv:1907.11692. — 2019.

8. Bradski G. The OpenCV library // Dr. Dobb's Journal of Software Tools. —

2000. — Vol. 25. — P. 120–125.

9. Rabiner L. R. A tutorial on hidden Markov models and selected applications in

speech recognition // Proceedings of the IEEE. — 1989. — Vol. 77, № 2. —

P. 257–286.

10. Bolt R. A. “Put-That-There”: voice and gesture at the graphics interface //

Proceedings of the 7th Annual Conference on Computer Graphics and

Interactive Techniques (SIGGRAPH). — 1980. — P. 262–27011. Oviatt S. L. Multimodal interfaces // The Human-Computer Interaction

Handbook: Fundamentals, Evolving Technologies and Emerging Applications.

2nd ed. — Boca Raton: CRC Press, 2007. — P. 413–432.

12. Murphy K. P. Dynamic Bayesian Networks: Representation, Inference and

Learning. PhD Thesis. — University of California, Berkeley, 2002. — 281 p.

13. Blei D. M., Kucukelbir A., McAuliffe J. D. Variational inference: a review for

statisticians // Journal of the American Statistical Association. — 2017. —

Vol. 112, № 518. — P. 859–877.

14. Kingma D. P., Welling M. Auto-encoding variational Bayes // arXiv preprint

arXiv:1312.6114. — 2013.

15. Goodfellow I., Bengio Y., Courville A. Deep Learning. — Cambridge, MA:

MIT Press, 2016. — 800 p.

16. Chollet F. Deep Learning with Python. 2nd ed. — Shelter Island, NY:

Manning, 2021. — 504 p.

17. Howard A. G., Zhu M., Chen B., et al. MobileNets: efficient convolutional

neural networks for mobile vision applications // arXiv preprint

arXiv:1704.04861. — 2017.

18. Jacob B., Kligys S., Chen B., et al. Quantization and training of neural

networks for efficient integer-arithmetic-only inference // Proceedings of the

IEEE Conference on Computer Vision and Pattern Recognition (CVPR). —

2018. — P. 2704–2713.

19. Hinz S., Hintermayer T. ZeroMQ: messaging for many applications //

Software: Practice and Experience. — 2013. — Vol. 43, № 7. — P. 845–860..

Published

2026-05-08