IoT-driven smart assistive communication system for the hearing impaired with hybrid deep learning models for sign language recognition (2025)

This study presents a SACHI-SLRHDL methodology in IoT. The SACHI-SLRHDL technique aims to develop an effective SLR technique that assists people with hearing impairments by creating an intelligent solution. It comprises four distinct processes: image pre-processing, improved MobileNetV3 for feature extractor, hybrid DL classification process, and AROA-based parameter tuning. Figure1 depicts the entire flow of the SACHI-SLRHDL methodology.

Image pre-processing: BF model

Initially, the SACHI-SLRHDL approach utilizes BF for image pre-processing to improve the excellence of the captured images by decreasing noise while preserving edges²⁹. This model was chosen for image pre-processing due to its superior capability to preserve edges while reducing noise, which is significant for maintaining the integrity of SL gestures. Unlike conventional smoothing techniques, BF effectively smooths out noise without blurring the crucial details, ensuring that key features of the SL images remain intact. Furthermore, BF works well in scenarios with varying lighting conditions and complex backgrounds, which is common in real-world applications. This makes it ideal for pre-processing SL images that may suffer from such challenges. Moreover, the BF model is computationally efficient, allowing it to be implemented in real-time systems, which is significant for SLR tasks. By improving image quality without compromising key spatial details, BF assists in enhancing the overall performance of subsequent DL methods in SLR. Figure2 specifies the BF architecture.

BF is an innovative image pre-processing model that enhances the excellence of images using SLR techniques. It aids in reducing noise while defending edges, which is vital for precisely seizing the hand gestures in SL. In the structure of IoT-based SLR, BF certifies that the seized images from IoT devices, like sensors or cameras, are clear and free from falsification. This pre-processing stage considerably progresses the accuracy of feature extraction by upholding significant spatial details in the imageries. By eliminating unrelated noise, BF permits the detection method to concentrate on profound gestures, safeguarding superior performance in real SLT. Therefore, it contributes to the efficacy of IoT-enabled models in helping persons with hearing loss.

Feature extraction: improved MobileNetV3

Next, the improved MobileNetV3 model extracts relevant features from input images³⁰. This technique was chosen due to its capability to balance high performance with low computational cost, making it appropriate for real-time applications like SLR. Unlike larger networks that require extensive computational resources, MobileNetV3 presents effectual processing without sacrificing accuracy, which is significant for deployment on resource-constrained devices such as IoT systems. Its optimized architecture uses depthwise separable convolutions, which mitigate the number of parameters and computational complexity, making it faster and more efficient. Furthermore, MobileNetV3 performs exceptionally well in extracting discriminative features from images, which is crucial for accurately recognizing SL gestures. By implementing the improved MobileNetV3, the model attains high recognition accuracy while maintaining efficiency, even under varying conditions. This makes it a robust choice compared to conventional, heavier CNN architectures. Figure3 illustrates the MobileNetV3 model.

This work designated MobileNetV3 from the MobileNet series. The MobileNetV3 method keeps its lightweight features, whereas enduring uses the depth-wise separable convolutional and reversed residual module from the MobileNetV2 method. It improves the bottleneck architecture by combining the Squeeze-and‐Excitation (SE) units, reinforcing the importance of significant characteristics and overcoming unimportant ones. Furthermore, the novel hard‐swish activation function has been accepted to enhance the system architecture. The MobileNetV3 approach is accessible in larger and smaller versions according to the availability of resources, and this work utilizes the MobileNetV3‐larger approach as a base.

However, combining the SE modules into the bottleneck architecture of MobileNetV3-Large has developed the model’s performance; the SE modules choose information amongst channels to define the significance of all channels. Nevertheless, it manages the important positioning information within the visual fields. Therefore, this method can only capture local feature information, resulting in problems like scattered fields of interest and narrow performances. To deal with these restrictions, the ECA unit increases the SE modules by preventing the reduction of dimensions and taking cross‐channel interaction information more effectively.

Despite developing the ECA unit through the SE modules, it still selects the information amongst channels. During the paper, the SE module is substituted in the MobileNetV3 architecture using the CA module to enhance MobileNetV3. The complete framework of the enhanced MobileNetV3-CA method is presented. The module of CA can concentrate the attention models on the field of interest over the efficient position in the pixel-coordinated method, thus gaining information that considers either position or channel, decreasing the attention to interference data and enhancing the feature appearance capability of the technique. The fundamental architecture of the CA module is presented\(\:.\) For a specified feature graph \(\:X\), the width is \(\:W\), the channel counts are \(\:C,\) and the height is \(\:H\). The module of CA initially pools the input \(\:X\) in dual spatial control, like width and height, to get feature mapping in either direction. Then, it connects the feature mapping from these dual ways in spatial sizes and then variations the sizes to the unique \(\:Clr\) utilizing the \(\:1\)x\(\:1\) convolutional transformation. Then, it uses Swish activation and batch normalization processes to get the middle feature mapping comprising information from either direction, as the equation below exposes.

Here, \(\:f\) denotes intermediate feature mapping gained by encoder spatial information in dual ways, \(\:\delta\:\) represents the Swish activation function, and \(\:{F}_{1}\) refers to the function of convolution transformation of \(\:1\)x\(\:1\). Now, \(\:{x}_{c}\) denotes feature data of the particular location of the feature graph in channel \(\:c,\)\(\:h\) denotes the specific height of feature mapping, and \(\:j\) symbolizes feature mapping width by the value range between \(\:[0,\:W].\) Also, \(\:w\) represents the particular width of the feature mapping, and \(\:i\) mean feature mapping height through the value ranges from \(\:[0,\:H].\)\(\:F\) can be separated into dual tensors, \(\:{h}^{f}\) and \(\:{w}^{f}\), along with the spatial dimensions in dual ways. Over dual \(\:1\)x\(\:1\) convolution transformation functions, \(\:{h}^{f}\) and \(\:{w}^{f}\) are transformed into tensors by the equivalent channel counts as the input \(\:X\). Lastly, it multiplies the lengthy attention weight with \(\:X\) to obtain the CA module output, as the equation below shows.

While \(\:{y}_{c}\) denotes the output of the \(\:c‐th\) channel, \(\:0\) means the activation function of the sigmoid, and \(\:{F}_{h}\) and \(\:{F}_{w}\) represent convolution transformation functions in width and height.

Classification process: hybrid DL models

For the SLR process, the hybrid of the CNN-BiGRU-A classifier is employed³¹. This hybrid model was chosen for its ability to handle spatial and temporal information effectively, which is significant for accurate SLR. CNN outperforms in extracting spatial features from images, while BiGRU captures sequential dependencies, making it ideal for comprehending the temporal aspect of sign gestures. Adding AMs allows the model to concentrate on the most crucial features in a sequence, improving recognition accuracy by mitigating noise and irrelevant data. This integration enables the model to process dynamic, real-world SL data more effectually than conventional methods that may only focus on one aspect (spatial or temporal) at a time. Furthermore, this hybrid methodology ensures that the model can handle the complexity and variability of SL gestures, giving superior performance to simpler architectures. The integration of these techniques presents a robust solution to the challenges in SLR, particularly for continuous and dynamic gestures. Figure4 portrays the structure of the CNN-BiGRU-A model.

The CNN-BiGRU‐A method comprises 3 core elements. Initially, CNN is applied to remove local temporal features from the time sequences subsiding information, assisting the process in recognizing short-term forms within the data through various monitoring scores. Bi-GRU handles longer‐range dependency in the time sequences, permitting the method to consider previous or upcoming subsiding tendencies, which develops complete prediction precision. Finally, the AM concentrates on the most significant time intervals, allocating high weight to important moments of change and improving the performance of the models by ordering primary data. This mixture allows the method to capture composite subsiding patterns successfully and makes precise predictions. For instance, in a mining region using composite subsiding behaviour, CNN identifies fast, localized variations at different observing points. BiGRU then trajectories longer‐range tendencies by combining historical or present data, assisting the model in identifying gradually growing subsiding patterns. The AM emphasizes moments of abrupt change, leading the model’s concentration to crucial changes, like abrupt growths in subsiding rate. Mutually, these modules guarantee timely and precise predictions, making the method helpful in dynamical mining atmospheres.

The CNN module contains various layers, which work together to remove essential patterns from the input data. This convolution layer recognizes particular attributes within the data by calculating weighting amounts, whereas pooling and activation layers present nonlinearity and decrease data sizes, effectively allowing the system to identify composite subsiding patterns. Normalization and fully connected (FC) layers enhance the last predictions by normalization, enhancing training speed or model strength. The major equations are as shown:

During Eq.(3), \(\:(I*K{)}_{ij}\) signifies the output feature mapping value at location \(\:(i,\:j)\) after the convolutional process. \(\:I\:\)denotes input data, \(\:K\) represents the convolutional kernel, and \(\:i,j\), and \(\:m,\)\(\:n\) represent position indices over output feature mapping and the convolutional kernel. In Eq.(4), \(\:{P}_{\text{i}\text{j}}\) characterizes the output feature mapping value at location \(\:(i,\:j)\) after the pooling process. Also\(\:,\:m\:and\:n\) represent position indices of the pooling window. During Eq.(5), \(\:O\) refers to output, \(\:I\) means input features, \(\:W\) stands for weighted matrix, \(\:b\) signifies biased matrix, and \(\:\sigma\:\) denotes activation function. Equation(6) characterizes the normalization layer task, while \(\:x\) signifies input data, \(\:\widehat{x}\) symbolizes the standardized input data, \(\:{\sigma\:}_{B}^{2}\)and\(\:{\mu\:}_{B}\) means the variance and mean of the present minibatch data, correspondingly. \(\:\epsilon\:\) denotes constant for numerical accuracy, whereas \(\:\gamma\:\) and \(\:\beta\:\) are learnable parameters.

The GRU module enhances prediction precision by controlling the flow of information over its update and reset gates, which define the related data to keep or discard at all steps. This mechanism permits the method to effectively take important patterns in subsiding information regarding either recent inputs or previous observations. For instance, in predicting subsiding tendencies, the GRU utilizes previous and present data scores to recognize a steady pattern that improves the model’s capability for predicting upcoming subsiding precisely. Applying the hidden layer (HL) \(\:{h}_{t-1}\) from the preceding time step and the present input \(\:{x}_{t}\), the GRU approach can be signified as shown:

Here, \(\:{W}_{r},{\:W}_{r}\) characterizes the weighted matrices, \(\:{b}_{z},{b}_{r}\) symbolize the biased vectors, and \(\:\sigma\:\) means the sigmoid activation function. The reset gate \(\:{r}_{t}\) defines which data from the preceding HL \(\:{h}_{t-1}\) must be discarded; however, the update gate \(\:{z}_{t}\) selects the mixing ratio of the newer and older memories.

Then, the last output is gained by computing the candidate HL \(\:{\stackrel{\sim}{h}}_{t}\) and the HL \(\:{h}_{t}\). The HL is then distributed to another layer or applied as the previous output.

Here \(\:W\) denotes the weighted matrix, \(\:{\stackrel{\sim}{h}}_{t}\) can be approximated utilizing \(\:{x}_{t}\) and \(\:{r}_{t}\) to gain the promising HL. Lastly, \(\:{\stackrel{\sim}{h}}_{t}\) and \(\:{h}_{t}\) are weighted to obtain the last condition fusion degree, \(\:\odot\:\) recognizes the Hadamard functions.

This bidirectional model allows the method to recognize patterns within the data more efficiently, which is helpful in tracking modifications that subside over time.

The AM enhances this ability by specially targeting significant portions of the data. It allocates greater weight to crucial data at every step, selecting main characteristics that might specify essential variations. For instance, in a real-time situation, Bi-GRU’s main characteristics are tendencies in previous or upcoming contexts, whereas the AM highlights unexpected moves or crucial points within the information, like regions where rates quickly rise. This integration permits the method to better adjust to useful requirements in mining regions using wide-ranging behaviours. These particular equations are as shown in (11)-(13) :

Now, \(\:{\alpha\:}_{ij}\) characterizes the attention score computed amongst the output of the encoder at the \(\:j\:th\) time step and the decoding layer at \(\:i\:th\) time steps. \(\:h\) signifies the HL at all-time steps, \(\:W\) symbolizes the weighted matrix related to the input or HL, and \(\:{\alpha\:}_{i}\) signifies the final attention weighting gained over the AM.

This prediction process includes three significant stages: Initially, data can be pre-processed in the CNN layer over pooling and convolution to make feature-rich data. Next, these vectors are given to a layer of BiGRU that takes either short‐ or long‐term designs within the data and avoids gradient problems. Lastly, the AM allocates weight to the main features, decreasing unrelated data and enhancing model efficacy. This allows the method to concentrate on essential patterns in subsiding data, leading to precise predictions.

Parameter optimizing process: AROA

Finally, the AROA optimally adjusts the hyperparameter values of the CNN-BiGRU-A approach, resulting in more excellent classification performance³². This methos was chosen for parameter optimization due to its ability to balance exploration and exploitation during the optimization process effectively. Unlike conventional optimization methods that may get stuck in local minima, AROA uses attraction and repulsion mechanisms to explore the solution space more thoroughly and avert suboptimal solutions. This is beneficial for DL models with complex parameter spaces. Furthermore, AROA’s simplicity and efficiency make it a robust choice for optimizing resource-intensive models like DL networks without needing extensive computational power. The algorithm’s flexibility in fine-tuning hyperparameters enhances model convergence and accuracy, improving overall performance. The ARO model’s adaptability and capability to optimize parameters such as learning rates, batch sizes, and network architecture make it superior to more conventional optimization techniques like grid or random search. AROA is a practical and effectual solution for improving the model’s performance in SLR tasks. Figure5 demonstrates the structure of the AROA model.

This method naturally imitates the phenomenon of attraction-repulsion. The initial phase in AROA is to make the first value of \(\:n\) individuals\(\:X\).

During Eq.(14), \(\:{X}_{i}\) refers to the value of the \(\:{i}^{th}\) individual, \(\:{X}_{low}\) and\(\:\:{X}_{up}\:\)mean lower and upper limits of the searching space, individually. \(\:rand\) denotes the randomly generated vector.

Then, each fitness value \(\:{X}_{i}\) is calculated and the best is defined based on the testing problem. The following stage in AROA utilizes the theory of attraction and repulsion, which relies on the distance between individuals \(\:X\). Hence, the value of \(\:X\) can be upgraded by computing the fitness levels of neighbouring individuals. The distance between \(\:{i}^{th}\) and \(\:{j}^{th}\) has been calculated as shown:

Meanwhile, \(\:{X}_{i}\) and \(\:{X}_{j}\) correspondingly provide the values of \(\:{i}^{th}\) and \(\:{j}^{th}\) individuals, and \(\:dim\) denotes \(\:{X}_{i}\)’s dimension counts\(\:.\)

The following operation is to update the attraction-repulsion operator \(\:\left({n}_{i}\right)\) according to the distance from the \(\:{i}^{th}\) individual to the furthermost member of \(\:X\)\(\:\left({d}_{i,\:\text{m}\text{a}\text{x}}\right)\) and \(\:{d}_{i,j}\in\:D\). This can be described as shown:

Here, \(\:c\) stands for the step size, and \(\:s\) signifies the function, which controls the direction of the change based on fitness value, and the \(\:s\) value is upgraded as:

Additionally, the \(\:k\) value, in Eq.(17), denotes the neighbour number, which reduces with excess the iterations and it can be upgraded as:

Now, \(\:t\) means present iteration and \(\:{t}_{\:\text{m}\text{a}\text{x}\:}\)stands for maximal iteration counts.

The next step is to utilize attraction to determine the optimal solution. This process characterizes the exploration stage as equivalent to other MH models determining the possible area. The Attraction operator \(\:\left(\right)\) can be described as:

Whereas \(\:{X}_{best}\) signifies optimal solution, \(\:{a}_{1}\) designates randomly generated vectors. The parameter \(\:{r}_{1}\in\:\left[\text{0,1}\right]\) means a randomly generated number, and \(\:{p}_{1}\) indicates probability thresholds. The parameter \(\:m\) has been utilized to mimic the impact of the best solution, and it is necessary for controlling the balance between exploitation and exploration; it is outlined as follows:

Consequently, the exploration phase of AROA has been employed to improve the probability of defining the optimal solution. This process can be described as shown:

While \(\:{r}_{B}\) represents the operator that signifies the Brownian motion by upgrading the standard deviation based on the searching area limits, and it can be described as:

Whereas \(\:{u}_{1}\) denotes a binary vector. \(\:N\) signifies the randomly generated vector value after a normal distribution, and \(\:f{r}_{1}\) symbolizes the contact value.

Besides, \(\:{r}_{tri}\) denotes the second operator, which relies on trigonometric functions and the individual that can be chosen using the roulette wheel selection. This can be outlined as shown:

Here, \(\:f{r}_{2}\) denotes the multiplier, \(\:{u}_{2}\) refers to binary vectors, \(\:{and\:r}_{4}\) and \(\:{r}_{5}\) are randomly generated numbers amongst\(\:\:(0\), 1). \(\:{a}_{2}\) relates to a randomly generated vector comprising values ranging between \(\:(0\)-1). \(\:{X}_{w}\) denotes a randomly chosen solution from \(\:X.\)

During Eq.(23), \(\:{r}_{R}\) refers to the third operator applied to improve the value of \(\:{X}_{i}\), and it can be well-defined as:

\(\:{u}_{3}\) denotes the binary vector gained using the threshold \(\:t{r}_{3}\) used for every solution. \(\:{a}_{3}\) indicates randomly selected vector values, and zero stands for matrix unit.

Additionally, the eddy formation theory can be used to improve the solution, and this can be expressed as:

Here, \(\:{r}_{7}\) signifies randomly formed integers variant from \(\:(0\)-1), and \(\:{e}_{f}\) signifies probability cutoffs. \(\:{u}_{4}\) indicates a binary vector gained by the threshold of \(\:1-{e}_{f}\), and \(\:{a}_{4}\) represents a vector containing arbitrary numbers. However, \(\:{r}_{8}\) and \(\:{r}_{9}\) are agent indexes randomly selected from \(\:X\), and \(\:{c}_{f}\) means parameter upgraded as shown:

After, the memory is measured as the subsequent impact applied to upgrade the solutions. This can be directed by comparing the novel value of the solution using its old value and preservative the best of them as expressed in the subsequent Eq.(29).

The AROA originates a fitness function (FF) to improve classifier performance. It defines an optimistic number to characterize the higher efficiency of the candidate solution. Here, the decrease in classification rate of error is deliberated as FF. Its formulation is mathematically expressed in Eq.(30).