Multimodal Sign Language Recognition Using Convolutional Neural Networks

The primary objective of the proposed sign language recognition system is to automatically interpret hand gestures into meaningful linguistic representations by analyzing their shape, spatial orientation, motion trajectory, and relative position. The work focuses on the development of a real-time, vision-based interpreter capable of recognizing and classifying alphabet gestures from American Sign Language (ASL). The system is designed as a practical, deployable assistive solution suitable for real-world human–computer interaction scenarios. The implementation employs MediaPipe Hands, a high-precision hand-tracking framework, to detect and extract 21 three-dimensional hand landmarks from live video streams captured through a standard webcam. These landmark coordinates serve as discriminative spatial features representing finger articulation and palm geometry. Instead of directly processing raw image frames, the extracted landmark vectors are normalized and structured into feature arrays to reduce computational complexity and improve invariance to scale, rotation, and background variations. For gesture classification, a Convolutional Neural Network (CNN)-based architecture is utilized to learn hierarchical feature representations from the processed landmark data. The dataset was constructed by recording multiple gesture samples under diverse environmental conditions, including varying illumination levels and backgrounds, to enhance generalization capability. The collected data underwent preprocessing steps such as noise filtering, coordinate normalization, temporal segmentation, data augmentation, and labeling to ensure robustness and improved training efficiency. The model was trained using supervised learning techniques with categorical cross-entropy loss and optimized through adaptive gradient-based optimization algorithms. To enhance usability, a text-to-speech (TTS) module is integrated into the system pipeline. Upon successful classification of a gesture, the predicted alphabet character is immediately converted into synthesized speech output, enabling seamless verbal communication. This multimodal feedback mechanism significantly improves accessibility for individuals with hearing or speech impairments.