Fusion of Visual and Motion Modalities for Sign Language Recognition Using Deep Learning

Sign language recognition (SLR) plays a crucial role in bridging the communication gap between the deaf and hearing communities. However, accurate interpretation of sign gestures remains challenging due to variations in hand shape, movement dynamics, occlusion, lighting conditions, and signer-specific differences. This research proposes a deep learning–based framework that integrates the fusion of visual and motion modalities to enhance recognition performance. The visual modality captures spatial features such as hand configuration and facial expressions using convolutional neural networks (CNNs), while the motion modality extracts temporal dynamics through optical flow representations and sequential modeling techniques such as Long Short-Term Memory (LSTM) networks. A multimodal fusion strategy is implemented at the feature level to combine spatial and temporal information effectively, enabling robust gesture classification. The proposed system is evaluated on benchmark sign language datasets, demonstrating improved accuracy, robustness, and generalization compared to unimodal approaches. Experimental results indicate that multimodal fusion significantly enhances recognition performance, particularly for dynamic and continuous gestures. The framework offers a scalable and real-time solution suitable for assistive communication systems, human–computer interaction, and inclusive educational technologies. This study highlights the effectiveness of deep multimodal integration in advancing intelligent sign language interpretation systems.