Document Type : Research Paper
Authors
1 PhD Candidate, Department of Computer Engineering, Iran University of Science and Technology, Tehran, Iran
2 Professor, Department of Computer Engineering, Iran University of Science and Technology, Tehran, Iran
Abstract
Keywords
Note: Figures, tables, formulas, and diagrams are not displayed in the HTML version of this article. For the complete and properly formatted content, please download and refer to the PDF version.
Natural Language Processing (NLP) is a major branch of artificial intelligence that focuses on enabling computers to understand, process, and generate human language. Given that language is central to human communication, the creation of models able to use natural language has been a long-standing goal in computing (Kamath et al. 2019). Nowadays, NLP approaches are extensively used in various fields such as information retrieval, text categorization, machine translation, and intelligent decision support systems. One of the most basic techniques in NLP is stemming, which is used to find the stem of a word. Stemming reduces the size of an index, makes matching more efficient, and improves the performance of downstream NLP tasks by minimizing morphological variation. Hence, stemming is important in applications like information retrieval, text categorization, question answering, and text mining (Alshalabi et al. 2022).
Arabic is also a challenging language for NLP due to its complex morphology, rich and highly productive morphological system with large numbers of word forms, complex orthography, and a high degree of speech ambiguity. Arabic words are usually inflected into a stem derived from one of ten well-established triconsonantal or (rarely) quadrilateral roots, to which various affixes and, rarely, circumfixes may be added, encoding additional syntactic information, but not necessarily meaning. Also, Arabic orthography permits the existence of several readings for the same surface form and correspondingly gives rise to homographs and context-sensitive meanings. These features make stem extraction much more difficult and reduce the efficiency of standard stemming approaches.
Several Arabic stemming methods have been introduced in the literature, including rule-based heavy stemmers and light statistical ones (Galal et al. 2019; Kanan et al. 2022; Hamza et al. 2021). Though these methods have performed with different degrees of success, most of them are based on manually created linguistic rules and pre-defined lists of affixes that limit their adaptability and robustness in dealing with irregular cases and ambiguous word formation. This does not, however, apply to processing classical Arabic text, which exhibits more complex word morphology and orthographic variation than Modern Standard Arabic.
Recent developments in deep learning, especially in sequence modeling architectures, have achieved impressive results across many NLP tasks. Character-level seq2seq models, in particular, demonstrate a strong capability in learning fine-grained morphological regularities without being explicitly fed with linguistic rules. These advances motivate the current research, which investigates deep learning for Arabic stemming and considers a data-driven character-level approach.
To formulate the research hypothesis for this paper, we pose the following question: Is it feasible to design a model that accurately extracts the stem of an Arabic word, and what method yields the best results? To address this research question, we hypothesize that deep learning can enhance the accuracy of word stemming. Specifically, we propose using a sequence-to-sequence (seq2seq) model. Seq2seq models, commonly used in tasks like machine translation, handle inputs and outputs of varying and unaligned lengths. The standard approach involves an encoder-decoder architecture with two main components: an encoder, comprising several recurrent neural networks (RNNs), to encode the input word sequence of characters, and a decoder, also comprising several RNNs, to generate the corresponding output sequence.
The contributions of this work can be summarized as follows:
We organize Related Works into two sections to cover research and ideas related to our paper.
Abuein et al. (2024) proposed a new deep learning (DL)-based Arabic sarcasm detection framework for Twitter, called ArSa-Tweets, comprising five different deep learning models. A new robust preprocessing phase was introduced to improve the performance of the proposed sarcasm detection model. Further, this work presented a new golden corpus called ArSa-data to support sarcasm analysis of Arabic Twitter, which was considered a publicly available sarcastic corpus for researchers in this domain. Baseline experiments were conducted by deploying five NLP models: LSTM, Multi-headed CNN-LSTM-GRU, BERT, AraBert-V01, and AraBert-V02. To validate the proposed Arabic sarcasm detection system, several preprocessing steps were conducted in the proposed ArSa-Tweets system to enhance performance.
A recent study introduced a lemmatization algorithm based on Recurrent Neural Network (RNN) models specifically designed for the Urdu language. The model was tested on two datasets: the Universal Dependencies Corpus of Urdu (UDU) and the Urdu Monolingual Corpus (UMC). Lemmatization was performed using RNN models, with semantic and syntactic embeddings generated through the Word2Vec model and edit trees. Various models were tested under specific hyper parameters, including Bidirectional Long Short-Term Memory (Bi-LSTM), Bidirectional Gated Recurrent Units (Bi-GRU), Attention-Free Encoder-Decoder (AFED), and Bidirectional Gated Recurrent Neural Network (Bi-GRNN). The experimental results demonstrated that the Attention-Free Encoder-Decoder model achieved accuracy, recall, precision, and F-score values of 0.96, 0.95, 0.95, and 0.95, respectively, surpassing the performance of existing models (Hafeez et al. 2023). Das et al. (2023) investigated the effectiveness of various machine learning methods, including deep learning and hybrid models, for text categorization in Bangla and English. The study focused on sentiment analysis of comments from DARAZ, a well-known Bengali e-commerce platform containing reviews in both Bangla and English. Seven machine learning and deep learning models, including Long Short-Term Memory (LSTM), Bidirectional LSTM (Bi-LSTM), Convolutional 1D (Conv1D), and a hybrid Conv1D-LSTM model, were implemented. To improve model accuracy, preprocessing techniques were applied to a modified text corpus.
Shahade et al. (2023) proposed a deep learning approach for multilingual opinion mining, utilizing a hybrid fine-tuned Smith algorithm combined with the Adam optimizer (HFS-AO). This approach supported three languages—English, Marathi, and Hindi—which were selected through a web scraping algorithm. The collected data were annotated using the Zero-shot instance-weighting technique and underwent comprehensive preprocessing, including noise removal, data cleaning and reduction, transformation, stop-word removal, tokenization, stemming, lemmatization, and Part-of-Speech (POS) tagging. Al-Khatib et al. (2023) presented an Arabic light stemming algorithm, called Tashaphyne 0.4. This algorithm extracted the most roots and stems from words in Arabic text. Thus, the algorithm acted as a stemmer, root extractor, and segmentation tool simultaneously. The model consisted of three phases: Preparation, Root Extraction, and Stem Extraction. Kanan et al. (2022) introduced an improved P-stemmer by integrating it with various classifiers. Due to a shortage of Arabic datasets, the authors created an Arabic dataset from diverse online news sources to evaluate performance.
Almanaseer et al. (2021) applied a Deep Belief Network (DBN) for Arabic text diacritical restoration. DBN, a deep learning algorithm consisting of multiple Restricted Boltzmann Machines (RBMs), demonstrated effective data modeling and feature extraction. The approach utilized greedy layer-wise training followed by fine-tuning to achieve competitive classification performance. Ezhilarasi and Maheswari (2021) designed a lemmatizer for analyzing 11th-century paleographic stone inscriptions. Their approach utilized a Bi-LSTM Sequential Model for POS Tag Classification, effectively categorizing and predicting tags for new script words through tag assignment and prediction. Nuţu (2021) proposed a deep learning sequence-to-sequence method to advance automatic Romanian lemmatization. This study compared 24 systems using various combinations of recurrent, attention, and convolutional layers, with input consisting of word-lemma pairs at word and trigram levels. Two scenarios were explored for trigrams: predicting the lemma for each word in the sequence or only for the middle word. The lemmatizers were evaluated on two Romanian datasets: the belletristic subset of the CoRoLa corpus and the Romanian Explicative Dictionary (DEX).
Bounhas et al. (2020) focused on creating a morpho-semantic knowledge graph from Arabic vocalized corpora. The authors employed a tool suite to analyze and disambiguate classical Arabic texts, incorporating short diacritics to reduce ambiguities. The morphological analysis combined MADAMIRA and the Ghwanmeh stemmer to extract multiple lexicons. Alnaied et al. (2020) introduced the Arabic Morphology Information Retrieval (AMIR) system for extracting Arabic stems. AMIR applied a set of rules related to Arabic letter relationships to identify stems for text indexing in Arabic retrieval systems. The system was evaluated using the EveTAR (2016) dataset, covering various Arabic event detection types. Galal et al. (2019) introduced a model called GStem to gather similar words that share the same root based on the Arabic extra letters as a preprocessing layer for the CNN. To evaluate the model, the authors collected an Arabic news dataset and used it in the experiments. They tested the WE from scratch using the collected training samples, and did not rely on any pre-trained data for word2vec or for GStem. The experiments showed that using GStem as a preprocessing step increased the accuracy of the CNN model because the number of distinct words was reduced.
Zalmout and Habash (2017) employed Bidirectional LSTM (Bi-LSTM) models for morphological tagging and language modeling, utilizing the outcomes of these models to rank the analyses of a morphological analyzer. The study incorporated various sub-word and morphological features at different linguistic levels in the tagger, including both word-based and character-based embeddings, and compared this system to a state-of-the-art benchmark. The dataset used for this study was the Penn Arabic Treebank (PATB parts 1, 2, and 3). Abdelali et al. (2016) introduced a new stemmer called Fast and Accurate Arabic Word Segmentation (FARASA). FARASA employs Support Vector Machine (SVM)-based ranking to determine the optimal segmentation of words into prefixes, stems, and suffixes. Hammo (2007) introduced a morphological analysis of Qur’anic texts, showing the Arabic morphology for some surahs from the Qur’an. This study also focused on comparing and contrasting English and Arabic morphology to determine the points where they differ by conducting morphological analysis for both languages.
Noor Dataset comprises over 260,000 unique words sourced from a variety of materials, including the Holy Qur’an and classical Arabic texts.
In the domain of computational linguistics, particularly within the context of Arabic natural language processing, the development of robust and efficient algorithms for text normalization and stemming is crucial. To address these challenges, the proposed method introduces a novel approach implemented as a Python module using PyTorch, a popular deep learning library. The method leverages neural network architectures, specifically Long Short-Term Memory (LSTM) networks, to achieve high accuracy in stemming Arabic texts. The proposed model aims to improve preprocessing quality for downstream natural language processing tasks such as machine translation, sentiment analysis, and text classification.
The dataset utilized in this study consists of Arabic word–stem pairs. It is derived from the Noor dataset, which includes classical Arabic texts and Qur’anic vocabulary, totaling approximately 260,000 words. This dataset provides a rich and diverse representation of Arabic linguistic patterns, particularly those found in classical Arabic, making it suitable for training and evaluating stemming models. Each instance in the dataset is represented as a pair of words stored in a text file, where each line contains a tab-separated pair: the original (unstemmed) word and its corresponding stem. This structure allows the model to learn a direct mapping between surface word forms and their normalized stem representations in a supervised learning setting.
Before proceeding, it is important to clarify the meaning of the term "stem" as used in this paper. In Arabic NLP, there is often confusion between root extraction and stemming. Root extractors aim to identify the root (typically consisting of three letters), for example, K-T-B from maktabah. Our task is different. We follow the light stemming approach, where the stem is a surface form that can stand as a word after removing affixes. For example, the stem of al-maktabāt is maktabah, not K-T-B.
For the experiments conducted in this work, a subset of the dataset was used, specifically the file QuranWords2.txt, which contains word–stem pairs extracted from Qur’anic texts. This subset was intentionally selected due to its morphological richness and complexity, providing a challenging benchmark for evaluating the effectiveness of the proposed model. The dataset was divided into 90% for training and 10% for testing. This split ensures that the model is evaluated on unseen data to assess its generalization capability. The dataset includes a wide range of morphological variations, including prefixes, suffixes, and inflectional patterns commonly found in classical Arabic.
It is important to note that the dataset focuses on classical Arabic; therefore, the model’s performance may be domain-dependent, and further evaluation on other Arabic text domains is required to assess its generalizability.
Before feeding the data into the model, it undergoes normalization. The Normalizer Builder class from the Piraye library is employed to perform several normalization tasks:
These normalization steps help reduce the variability in the dataset, which could otherwise lead to an unnecessarily large model size and overfitting due to the high dimensionality of the input data. The dataset was divided into two main sets: 90% for training and 10% for testing, ensuring a reliable evaluation of the model’s performance.
After normalization and tokenization, each word is split into individual characters, and an end-of-sequence (EOS) token is appended. The pad sequence function takes a sequence of characters, a maximum sequence length, and a padding character.
The architecture of the model plays a critical role in determining its effectiveness and efficiency in handling the challenges of Arabic text stemming. The proposed model, named Char Stemmer, utilizes a neural network architecture that leverages the strengths of Long Short-Term Memory (LSTM) networks and embedding layers.
The first component of the Char Stemmer model is the embedding layer. This layer transforms sparse representations of input characters (character indices) into dense, continuous vector representations with an embedding dimension of 512 in this implementation.
The output from the encoder is fed into an LSTM decoder, which aims to generate the stemmed sequence of characters. Unlike the encoder, the decoder is not bidirectional but uses the concatenated hidden states from the encoder to initialize its internal state. At each time step, the decoder predicts the next character in the stemmed sequence based on its current state and the encoder's output.
The final component of the Char Stemmer model is the output layer. This layer maps the high-dimensional output of the decoder to the size of the vocabulary, accurately predicting the probability distribution over all possible characters at each position in the output sequence. The model then applies a SoftMax function to this output to predict the most probable next character in the stemmed word. For training the model, a Cross-Entropy loss function is employed. This loss function is appropriate for classification tasks such as character prediction, where it measures the difference between the predicted probability distribution and the ground-truth distribution.
The Char Stemmer model applies complex rules that assume the script and morphology of Arabic. Through the combination of embedding layers with bi-directional and uni-directional LSTMs, the model effectively captures both local character interactions and longer-range dependencies essential for accurate stemming. The selection of the loss function is also consistent with the goal of the model of effectively learning to predict sequences of characters. These architectural components make the proposed approach capable of addressing Arabic text stemming challenges and provide a strong framework for different NLP tasks related to Arabic texts.
The character-level processing begins with a dense embedding layer that transforms sparse character representations into 512-dimensional continuous vector spaces. This embedding mechanism captures semantic relationships between Arabic characters, providing a rich foundation for morphological analysis. Figure 1 displays the proposed model for extracting the stems.
The model uses the Adam optimizer (torch.optim.Adam), a popular choice for training deep neural networks due to its adaptive learning rate capabilities. This optimizer helps the model converge faster and more effectively than standard gradient descent methods. The learning rate and other parameters of Adam are set to their default values, which provide a good starting point.
The loss function utilized is the Cross-Entropy Loss, which is suitable for classification tasks such as predicting the next character in the stemming sequence. During training, the loss is calculated between the model’s predictions and the ground-truth labels for each batch. This loss value indicates how well the model is performing; a lower loss implies better model predictions.
The prepared and preprocessed dataset is loaded using the Arabic Stemmer Dataset class. The dataset is split into training and test subsets, with 90% of the data used for training and the remaining 10% reserved for evaluation. For efficient training, the data is batched using PyTorch's DataLoader. A batch size of 32 is used.
Before training begins, the Char Stemmer model is instantiated with predefined hyperparameters, including the input size (the size of the character vocabulary), the hidden size (1024 for the LSTM layers), and the output size (equal to the input size, representing the character predictions).
The model was trained for 20 epochs, which provided a balance between convergence and computational cost. An embedding dimension of 512 and a hidden layer size of 1024 were used to capture complex morphological patterns. A batch size of 32 was selected to ensure stable training. The dataset was split into 90% for training and 10% for testing to evaluate generalization performance. A detailed summary of all hyperparameter settings is provided in Table 1.
Table 1. Hyperparameter settings of the proposed model
The model was trained on an NVIDIA GPU for approximately 12 hours. For early stopping, a patience of 5 epochs was used based on the validation loss. Dropout of 0.3 was applied after each LSTM layer to prevent overfitting. Figure 1 and Figure 2 illustrate the proposed model for Arabic stemming.
Figure 1. The proposed model (Char Stemmer)
The training procedure for the Char Stemmer model is designed to ensure robust learning without overfitting. By carefully managing data loading, model configuration, optimization, and iterative learning through epochs, the procedure lays a strong foundation for the model to learn the complex mappings required for effective Arabic text stemming. This structured approach not only optimizes the training phase but also provides a foundation for comprehensive evaluation and deployment in practical NLP applications.
Figure 2. The architecture of the Char Stemmer model
Assessing the performance of the proposed model is critical to understanding its effectiveness and identifying areas for improvement. This section discusses the metrics used to assess the model's performance and provides an analysis of the results obtained after training on a dataset composed of classical Arabic texts.
The performance of the proposed model is evaluated using several key metrics:
The final training epoch yielded a training loss of 0.2748 and an accuracy of 93.88%. The model was evaluated on unseen data, resulting in a test loss of 0.2889 and an accuracy of 93.88%. These results indicate that the model effectively learned the stemming task with a high degree of accuracy and minimal overfitting. Table 2 summarizes these results.
Table 2. The result of char stemmer model
However, the precision (82.01%), recall (70.46%), and F1-score (75.28%) on the test data suggest some challenges in the model’s ability to generalize across all aspects of the data. While the precision is relatively high, indicating that the model’s predictions are usually correct when it predicts a character, the lower recall and F1-score point to difficulties in consistently identifying all correct characters, especially in more ambiguous or rare stemming scenarios found in classical texts.
In our experiments, the model was trained for 20 epochs, which provided a balance between training time and convergence without overfitting. We used an embedding dimension of 512, allowing the model to capture rich character-level representations while keeping the computational cost manageable. The hidden layer size was set to 1024 units to enhance the model’s capacity for learning complex morphological patterns in Arabic words. The batch size was set to 32, offering a compromise between stable gradient updates and training speed. The dataset was divided into 90% for training and 10% for testing, ensuring that the model was evaluated on unseen data to assess its generalization performance. Figure 3 illustrates the confusion matrix for the Char Stemmer model.
Figure 3. Confusion matrix for the Char Stemmer model
The performance evaluation of the Char Stemmer model demonstrates its effectiveness in processing and stemming classical Arabic texts with high accuracy. However, the detailed metrics also highlight areas for potential improvement, particularly in enhancing recall without sacrificing precision. Future work could explore more sophisticated model architectures, richer contextual embeddings, or advanced training techniques such as transfer learning from classical to modern texts to further refine the model’s performance.
The Char Stemmer model, as detailed in the previous sections, presents a novel approach to stemming Arabic text, particularly focusing on the complexities of classical Arabic morphology. This discussion explores the reasons behind the model's effectiveness, its limitations, and the theoretical implications of the chosen methodology.
The Char Stemmer model attained an overall accuracy of 93.88%, with precision and recall of 82.01% and 70.46%, respectively. Despite its strong performance, errors primarily stem from the morphological richness of Qur’anic Arabic. Complex affixation patterns and irregular inflections challenge the model’s ability to identify correct stems. Ambiguities arise when words share surface forms but differ in meaning or grammatical category, such as the word malik, which can be either a noun (king) or a verb (he has). Rare or infrequent words present additional difficulties due to their limited representation in the training data. Moreover, the absence of diacritics hampers the disambiguation of words with identical orthography but distinct pronunciations. Finally, the character-level approach limits contextual understanding, which is vital for accurate lemmatization in ambiguous cases.
While the Char Stemmer achieves high accuracy, its recall and F1-score indicate room for improvement, especially in handling rare and ambiguous cases inherent in classical texts. These limitations suggest a few potential areas for enhancement:
We selected a set of well-known stemmers for comparison. We used these stemmers because they achieved high accuracy in previous studies. The stemmers used in the comparison are as follows:
Following the evaluation of the proposed model, we compared it with five Arabic stemmers using the same Noor dataset. Table 3 presents a comprehensive comparison across multiple evaluation metrics.
Table 3. Comparative performance of Arabic stemmers
Figure 4. Comparison of the evaluated stemmers
We also conducted a brief ablation study to evaluate the contribution of each component of the proposed model. We trained two simpler versions: one with a regular LSTM encoder instead of a Bi-LSTM (without backward context), and another without dropout. The results are shown in Table 4.
The Bi-LSTM version achieved 93.88% accuracy. When the encoder was replaced with a regular LSTM, accuracy dropped to approximately 91.2%. This result suggests that capturing contextual information from both prefix and suffix directions improves stemming performance. Without dropout, the model still performed well on the training data but showed signs of overfitting, as reflected by a larger gap between training and test losses. Therefore, the final model incorporates dropout.
Table 4. Results of the ablation study
To further evaluate the performance of the proposed Char Stemmer model, a qualitative analysis was conducted by examining sample outputs and comparing them with selected baseline stemmers. Table 5 presents several examples of Arabic words along with their corresponding ground-truth stems, the outputs of the proposed model, and the outputs of representative baseline stemmers.
Table 5. Qualitative comparison of stemming results
In conclusion, the proposed model represents a significant step forward in the computational processing of Arabic texts, combining advanced machine learning techniques with a deep understanding of linguistic principles. While there are areas for improvement, the current implementation provides a strong foundation for further exploration and development in Arabic NLP, particularly for classical and morphologically complex texts.
The Char Stemmer model introduced in this paper represents a significant advancement in the field of Arabic natural language processing, specifically addressing the challenges associated with stemming classical Arabic texts. The model leverages a sophisticated deep learning architecture, encompassing robust preprocessing methods, character-level embedding, bidirectional LSTM networks, and a sequential decoding process. These components collectively contribute to the model’s ability to accurately and effectively predict the stems of Arabic words, demonstrating high overall accuracy in both training and testing phases.
This paper has successfully demonstrated that deep learning techniques, particularly those based on LSTM networks, are well suited for handling the complexities of Arabic morphology and script variations. The bidirectional approach of the LSTM encoder captures contextual nuances from both directions of the input sequences, enhancing the model's sensitivity to the intricate patterns that define Arabic stemming. Furthermore, the embedding layer plays a crucial role by transforming discrete character representations into rich, continuous vector spaces, enabling the model to learn subtle linguistic distinctions.
)