Cloud-based services have revolutionized data storage and processing tasks. However, these services raise security concerns as service providers may misuse the user’s stored data. Privacy loss is particularly problematic for hearing and speech impaired individuals that may need to use cloud infrastructure for sign language recognition (SLR). Addressing these challenges, this paper presents a privacy-preserving sign language recognition (P2SLR) as a cloud-service that operates over cloud infrastructure without revealing the individual’s visual information to the cloud service provider (CSP). The proposed P2SLR system is realized through two innovations: (a) block-based probabilistic image encryption scheme that combines fractional-order chaotic system (FOCS) and singular value decomposition (SVD) to obfuscate the visual information in video frames, and (b) a cloud-residing deep convolutional neural network (DCNN) based recognition architecture with a modified classifier to recognize gestures from encrypted video. The proposed scheme is validated for American, Argentinian, and German sign languages. The proposed scheme achieved recognition accuracy in the range 90:76 - 98:09%, comparable to existing state-of-the-art SLR techniques in the plain domain (PD). The proposed image encryption scheme is secure under standard cryptographic image attacks, protecting the individual’s identity. P2SLR is the first move towards developing a secure SLR system over the cloud.