ASTRA: A MULTIMODAL HAND-GESTURE AND VOICE-COMMAND SYSTEM FOR HANDS-FREE COMPUTER CONTROL WITH AFFORDABLE HARDWARE
Keywords:
human-computer interaction; multimodal fusion; gesture recognition; voice control; MediaPipe; 3D-CNN; LSTM; Transformer ASR; Dynamic Bayesian Network; adaptive pinch; exponential smoothing; privacyfirst design; accessibility.Abstract
Touchless human-computer interaction (HCI) is increasingly demanded in sterile environments, accessibility scenarios for users with motor impairments, and immersive workflows. Existing solutions either rely on costly proprietary depth sensors (e.g., Leap Motion) or are tied to cloud services that introduce latency and privacy concerns. This paper presents ASTRA, a multimodal HCI system that combines hand-gesture and voice control on commodity hardware (a Logitech 720p webcam and a USB microphone).
References
1. Tran D., Bourdev L., Fergus R., et al. Learning spatiotemporal features with 3D
convolutional networks // Proceedings of the IEEE International Conference
on Computer Vision (ICCV). — 2015. — P. 4489–4497.
2. Hochreiter S., Schmidhuber J. Long short-term memory // Neural Computation.
— 1997. — Vol. 9, № 8. — P. 1735–1780.
3. Lugaresi C., Tang J., Nash H., et al. MediaPipe: a framework for building
perception pipelines // arXiv preprint arXiv:1906.08172. — 2019.
4. Weichert F., Bachmann D., Rudak B., Fisseler D. Analysis of the accuracy and
robustness of the Leap Motion controller // Sensors. — 2013. — Vol. 13, № 5.
— P. 6380–6393.
5. Vaswani A., Shazeer N., Parmar N., et al. Attention is all you need // Advances
in Neural Information Processing Systems (NeurIPS). — 2017. — P. 5998–
6008.
6. Devlin J., Chang M.-W., Lee K., Toutanova K. BERT: pre-training of deep
bidirectional transformers for language understanding // arXiv preprint
arXiv:1810.04805. — 2018.
7. Liu Y., Ott M., Goyal N., et al. RoBERTa: a robustly optimized BERT
pretraining approach // arXiv preprint arXiv:1907.11692. — 2019.
8. Bradski G. The OpenCV library // Dr. Dobb's Journal of Software Tools. —
2000. — Vol. 25. — P. 120–125.
9. Rabiner L. R. A tutorial on hidden Markov models and selected applications in
speech recognition // Proceedings of the IEEE. — 1989. — Vol. 77, № 2. —
P. 257–286.
10. Bolt R. A. “Put-That-There”: voice and gesture at the graphics interface //
Proceedings of the 7th Annual Conference on Computer Graphics and
Interactive Techniques (SIGGRAPH). — 1980. — P. 262–27011. Oviatt S. L. Multimodal interfaces // The Human-Computer Interaction
Handbook: Fundamentals, Evolving Technologies and Emerging Applications.
2nd ed. — Boca Raton: CRC Press, 2007. — P. 413–432.
12. Murphy K. P. Dynamic Bayesian Networks: Representation, Inference and
Learning. PhD Thesis. — University of California, Berkeley, 2002. — 281 p.
13. Blei D. M., Kucukelbir A., McAuliffe J. D. Variational inference: a review for
statisticians // Journal of the American Statistical Association. — 2017. —
Vol. 112, № 518. — P. 859–877.
14. Kingma D. P., Welling M. Auto-encoding variational Bayes // arXiv preprint
arXiv:1312.6114. — 2013.
15. Goodfellow I., Bengio Y., Courville A. Deep Learning. — Cambridge, MA:
MIT Press, 2016. — 800 p.
16. Chollet F. Deep Learning with Python. 2nd ed. — Shelter Island, NY:
Manning, 2021. — 504 p.
17. Howard A. G., Zhu M., Chen B., et al. MobileNets: efficient convolutional
neural networks for mobile vision applications // arXiv preprint
arXiv:1704.04861. — 2017.
18. Jacob B., Kligys S., Chen B., et al. Quantization and training of neural
networks for efficient integer-arithmetic-only inference // Proceedings of the
IEEE Conference on Computer Vision and Pattern Recognition (CVPR). —
2018. — P. 2704–2713.
19. Hinz S., Hintermayer T. ZeroMQ: messaging for many applications //
Software: Practice and Experience. — 2013. — Vol. 43, № 7. — P. 845–860..