Effective Class-Imbalance Learning Based on SMOTE and Convolutional Neural Networks

Abstract

Imbalanced Data (ID) is a problem that deters Machine Learning (ML) models from achieving satisfactory results. ID is the occurrence of a situation where the quantity of the samples belonging to one class outnumbers that of the other by a wide margin, making such models’ learning process biased towards the majority class. In recent years, to address this issue, several solutions have been put forward, which opt for either synthetically generating new data for the minority class or reducing the number of majority classes to balance the data. Hence, in this paper, we investigate the effectiveness of methods based on Deep Neural Networks (DNNs) and Convolutional Neural Networks (CNNs) mixed with a variety of well-known imbalanced data solutions meaning oversampling and undersampling. Then, we propose a CNN-based model in combination with SMOTE to effectively handle imbalanced data. To evaluate our methods, we have used KEEL, breast cancer, and Z-Alizadeh Sani datasets. In order to achieve reliable results, we conducted our experiments 100 times with randomly shuffled data distributions. The classification results demonstrate that the mixed Synthetic Minority Oversampling Technique (SMOTE)-Normalization-CNN outperforms different methodologies achieving 99.08% accuracy on the 24 imbalanced datasets. Therefore, the proposed mixed model can be applied to imbalanced binary classification problems on other real datasets.

1. Introduction

Learning a classifier from an imbalanced dataset is an important topic and still a complicated problem in supervised learning algorithms. In other words, a class imbalance is a customary long-standing challenge in classification problems [1,2,3,4,5], which deals with a dataset that contains an asymmetrically larger number of samples of the majority class. The imbalanced datasets appear in vast real-world research, such as life sciences [6], facial age approximation [7], anomaly detection [8], determining counterfeit credit card transactions [9], medical imaging [10], DNA sequence identification [11], and so forth. For an imbalanced binary classification problem, samples are typically characterized by two classes, namely majority and minority.

1.1. Context of the Study

In general terms, the minority class often illustrates samples of higher importance and interest rather than the majority class. Nevertheless, compared to the minority class, the majority class usually has a more significant number of samples in a meaningful way, and sometimes, the situation may be extremely serious.

Different situations can occur in confronting the imbalanced datasets, and four common cases are depicted in Figure 1, where the blue-filled circles represent the samples of the majority class; in contrast, the red circles denote the minority class [1]. It has been shown that the type of data complexity is the principal determining factor of classification performance reduction [2].

Most of the classical classification methods, such as decision trees [2,3,4], KNN [5,6], and Repeated Incremental Pruning to Produce Error Reduction (RIPPER) [7,8], usually train models that maximize the accuracy of proposed algorithms, sometimes ignoring the minority class [9,10,11]. Hence, several techniques have been designed and implemented to handle the imbalanced binary classification problems. Among these techniques, oversampling and undersampling are well-known [12,13,14,15]. Yet, the common undersampling and oversampling algorithms modify the initial class distribution of the dataset by excluding the majority class samples or expanding the minority class samples.

Cost-sensitive learning algorithms were among the solutions for the above-mentioned issues of imbalanced data [16,17,18]. Such algorithms designate misclassification cost errors for multiple classes, mainly lower costs for the samples of the majority class and higher for the minority class. In addition, Bagging [19] and Boosting [20] methods, which are based on ensemble learning algorithms, are among the other commonly used methods to handle imbalanced class problems [7,8,21,22].

In this paper, we use several undersampling and oversampling methods in the process of implementing our methodology, which is briefly introduced in the sequel:

• RUS: Among undersampling methods, random undersampling (RUS) is the simplest one, in which the samples of the majority class are randomly removed until suitable balanced data are obtained [23].

• Tomek Links: Some of the undersampling techniques focus on overlap elimination. For example, the Tomek Links [24] method, which is a modification of the Condensed Nearest Neighbor rule, is one of these methods.

• One-Sided Selection: As a development of the Tomek Links algorithm, one can refer to the One-Sided Selection or briefly OSS method [25] that merges Tomek Links and the Condensed Nearest Neighbor algorithms.

• Near Miss is another popular undersampling method that randomly removes the majority of class samples. When two samples classified in different classes are very close to each other, it removes the sample belonging to the larger class [5].

• ROS: Among the oversampling algorithms, random oversampling (ROS) is the simplest one that merely selects and copies the samples from the minority class randomly, leading to more balanced data [26].

• SMOTE: The best-known oversampling method is the Synthetic Minority Oversampling Technique (SMOTE) [14,27,28] which leverages the kNN algorithm to identify the neighbors of minority class samples and generates the new sample by selecting the kth neighbor randomly [29].

1.2. Research Gap

It is worth noting that the methods mentioned above may cause some unexpected issues. For example, undersampling techniques may ignore some valuable data, which could be vital for training a classifier. In contrast, oversampling algorithms may cause overfitting. Furthermore, for cost-sensitive learning techniques, it is not straightforward to determine the exact misclassification cost, and different misclassification costs might result in different induced outcomes. Moreover, Bagging and Boosting algorithms may exclude some valuable data while they propose sampling methods in every single iteration, and they may face an overfitting problem. Consequently, the classification results obtained by these methods are not stable.

1.3. Motivation and Contribution

In order to address these problems, this paper proposes two DL-based methods mixed with different resampling methods for better tackling the issue of an imbalanced dataset. The existing DL-based methods, especially CNN architectures, have been employed in a wide variety of challenges, and they have proven to be extremely powerful in terms of learning balanced datasets. Their efficacy has not been satisfactorily investigated when tackling imbalanced datasets [30].

CNNs are types of architectures that contain convolutional blocks and can provide an end-to-end classification algorithm. These blocks are a stack of different layers, namely convolutional layers, pooling layers, and activation functions. The most significant attributes of such models are their learning capacity with fewer parameters and translational invariance concerning the input data. In CNNs, the input data are fed to multiple convolutional blocks, which are named mainly backbone as a whole, and then followed by a sequence of fully connected layers to be classified.

The training procedure is performed using Focal Loss (FL), which optimizes the abstraction learned by the models to handle complex samples better.

In particular, the main contributions of this paper are threefold. First, 24 popular imbalanced datasets from KEEL Dataset Repository, breast cancer dataset from KDD Cup, and Z-Alizadeh Sani are chosen. The proposed pipeline is trained and validated 100 times to achieve more reliable results. Second, this paper is the first research work that extensively investigates the most efficient mix of Deep Neural Networks (DNN) with the famous resampling method, SMOTE, for imbalanced data. Lastly, the mixed SMOTE-Normalization-CNN methodology has proved to produce superior results in comparison with related research works in terms of accuracy, precision, recall, Geometric mean (G-Mean), specificity, Area Under the Receiver Operating Characteristic Curve (AUC-ROC), and Kappa.

The rest of the paper is outlined as follows: Section 2 provides a brief review of the existing works on imbalanced datasets; Section 3 presents the details of our proposed methodology; Section 4 includes the implementation setup and the evaluation process and then focuses on the results obtained by the proposed methods; Section 5 presents a discussion on comparing our method to the others, and the last section draws a reasoned conclusion and future work.

2. Related Work

To better handle imbalanced data, Li et al. [31] proposed a developed AdaBoost algorithm called Adaboost-A, which is based on the AUC evaluation metric. In fact, by considering the impact of misclassification probabilities and AUC, the algorithm mentioned above betters the computational performance of the Adaboost algorithm. Furthermore, this study proposed an ensemble learning algorithm named PSOPD-AdaBoost-A, by which the multipliers of Adaboost weak classifiers were optimized.

In [32], the authors provide a detailed exploratory comparison between the problem of handling class overlapping and class imbalances using a full range of class overlaps along with a large scale of class imbalance degrees. The rest of this study contains a thorough review of the current methods and solutions for handling the imbalanced data classification problem, characterized by two categories: distribution-based and class overlap-based algorithms.

The literature [33] proposed an undersampling technique that mainly uses the well-known Naïve Bayes classifier. Based on a random primary selection, this classifier is leveraged to select the most informational samples among the existing training dataset. At first, the model is trained on a small training set, and after that, by an iterative teaching method over the current samples, the base model is taught. The practical outcomes showed that the proposed undersampling technique is comparable to other resampling techniques.

In [34], Dablain et al. proposed a novel deep oversampling method called DeepSMOTE, which contains three principle parts and mainly uses the properties of the effective SMOTE method. Despite being simple, the method is efficient and powerful. An encoder/decoder structure, being SMOTE-based and including an improved loss function, form the three parts of this method. The results of this study show the advantages of the proposed method, especially in GAN-based oversampling cases.

The research study [35] investigates the effects of resampling on the performances of multi-class Artificial Neural Networks. Different resampling techniques (both undersampling and oversampling) were examined on several cybersecurity datasets. Furthermore, to determine the results of the proposed methods, various evaluation metrics were used. Finally, four observed patterns were reported that compare the impacts of resampling on the evaluation metrics and model training duration.

In another study [36], the authors dealt with the issue of the probable bias and tendency of a learning classifier toward the majority of samples for an imbalanced dataset. This work implemented an innovative three-dimensional framework that includes a discriminator, a generator, and a classifier, together with decision boundary regularization. The remarkable aspect of the proposed method is training a generator in association with a classifier. The reporting results show better performance of the technique than the existing methods.

To improve the efficiency and functioning of undersampling methods for imbalanced data, Xie et al. [37] proposed a new undersampling technique that leverages consecutive density peaks to gradually take out samples from the majority class of imbalanced data. In order to determine the importance of the samples of the majority class, two factors were considered, which generate a sequence of samples for learning classifiers. The study compared the implemented algorithm to six well-known undersampling methods over 40 public benchmarks, and the results verify the outperformance of the proposed technique.

In order to design learning classifiers that provide stable performances on imbalanced datasets, the study [38] proposed three different methods. These methods are mainly based on genetic algorithms which automatically specify the ratios of samples for oversampling, undersampling, and hybrid sampling techniques. The implemented algorithms were examined on 14 imbalanced datasets, and the results show that they achieved the best AUC compared to random sampling methods.

To decrease the domination of the majority class samples, the study [39] implemented a novel hybrid method called CDSMOTE, which uses class decomposition and oversampling on the minority class samples. Contrary to general undersampling algorithms, this proposed method keeps the majority class samples, leading to more balanced data. The algorithm was examined on 60 imbalanced public datasets, and the results show comparable performance compared to the existing algorithms.

By introducing a new algorithm called SMOTE-LOF, Maulidevi and Surendro [40] attempted to refine SMOTE. This method distinguishes the noise that arises when dealing with imbalanced datasets by adding the Local Outlier Factor (LOF). By examining the proposed algorithm over different imbalanced datasets, the results were compared to SMOTE. Unlike small data samples, for a large-scale dataset with a small imbalance ratio, SMOTE-LOF outperforms SMOTE.

In order to annihilate the overlap between the majority class and the minority class in an imbalanced dataset and obtain a balanced and normalized class distribution, the study [29] implemented two innovative density-based methods. These methods were density-based undersampling (DB_US) and density-based hybrid sampling (DB_HS). The first method applies merely an undersampling algorithm, while the second implements both undersampling and oversampling approaches. In addition, the balanced datasets were modeled employing Random Forest (RF) and Support Vector Machine (SVM) classifiers. As a result, the two proposed methods eliminated high-density samples from the majority class and omitted the noises of both classes. The performance of these methods was examined on 16 imbalanced datasets.

In the literature [41], the authors proposed a novel classification method called the Bagging Supervised Autoencoder Classifier (BSAC) to model credit scoring problems. This algorithm essentially leverages the superlative implementation of a supervised autoencoder based on the axioms of multi-task learning. Furthermore, BSAC tackles the issue of imbalanced datasets by engaging a variation of the Bagging procedure based on undersampling techniques. The examinations of benchmark and real-world credit scoring datasets show the robustness and efficiency of BSAC.

To improve the performance of the basic antlion optimization (ALO), in [42], a novel modified antlion optimization method (MALO) was introduced. This algorithm adds an extra variable that depends on the step size of the ants as revising the antlion position. Furthermore, MALO is modified to the issues of sample reduction to achieve better performance due to various metrics. MALO was examined on several benchmarks and balanced and imbalanced datasets. The results show the outperformance of MALO against the primary ALO method and some other comparable algorithms.

In [43], Yang et al., implemented a sampling level technique called the gravitational balanced multiple kernel learning (GBMKL) algorithms, which merges the gravity approach to produce the gravitation-balanced midpoint samples (GBMS) placed on the classification boundary. Moreover, to better the generalization efficiency, the classification boundary was modified according to the nearest neighbors of the boundary (NNB) samples. Finally, two regularization terms that correspond to GBMS and NNB were formulated to prevent overfitting. The resulting method was examined on 54 artificial and real-life imbalanced datasets, and the outcomes show the dominance of the implemented method.

Tanimoto et al. in [44] studied the near-miss positive samples in the class of imbalanced datasets. They showed that if the true positive samples are severely limited, the accuracy of the proposed model could be increased by obtaining modified label-like side information positivity to identify near-miss samples from true negatives. Furthermore, the proposed method follows learning using privileged information that leverages side information for training the desired model devoid of predicting the side information itself. The results of the experiments show the outperformance of the method in contrast to the existing algorithms.

The research study [45] proposed new development of SMOTE by merging it with the Kalman filter. After applying SMOTE to the given dataset, the implemented algorithm, called Kalman-SMOTE (KSMOTE), excludes the noisy samples in the resultant dataset that simultaneously contains the initial data and the synthetically added samples. The method was examined on a broad range of datasets, and the results show that the implemented algorithm outperforms the existing methods.

Since oversampling techniques cannot usually achieve high performance in the presence of noise, the study [46] implemented an innovative oversampling algorithm called IR-SMOTE that handles this issue. By sorting the majority class samples and the k-means clustering algorithm, the noise in minority class clusters is eliminated. After that, using the kernel density estimation method, the amount of synthetic samples is compatibly designated to each cluster. Finally, regarding random-SMOTE, the desired algorithm was improved to add new samples with ensured diversity.

The literature [30] studied the performance of convolutional neural networks (CNNs) in the presence of imbalanced data for classification problems. In order to explore this probable impact, the research used MNIST, CIFAR-10, and ImageNet as benchmarks, alongside undersampling, oversampling, two-phase training, and thresholding. The results show that imbalanced data have a detrimental effect on the performance of the proposed method. Furthermore, one should implement oversampling to the level that removes the imbalance, while the extent of the imbalance determines the ideal undersampling ratio. In addition, oversampling does not lead to the overfitting of CNNs.

Fault diagnosis of complex equipment, which plays an important role in industries, is a crucial technology, and CNN is a general tool for this purpose. In this case, faults are not common, which leads to imbalanced data, and therefore, one cannot propose CNN methods directly. In order to address this problem, a hierarchical training CNN is implemented in [47]. At first, the method uses a number-resampling technique to balance data. Then, a magnet-loss pretraining algorithm is provided to handle the overlap between diverse faults. The proposed method was examined on the public dataset CWRU with an accuracy of 94.28%.

3. Methodology

In this paper, we have used our methods applied to various datasets collected from benchmark repositories such as the KEEL (https://sci2s.ugr.es/keel/imbalanced.php, accessed on 17 March 2023), breast cancer (https://www.kdd.org/kdd-cup, accessed on 17 March 2023), and Z-Alizadeh Sani (https://archive.ics.uci.edu/ml/datasets/Z-Alizadeh+Sani, accessed on 17 March 2023) datasets in order to address the class imbalance problem. Figure 2 demonstrates an overview of our proposed methodology, whose details are included in this section.

Based on Figure 2, the main steps in our methodology include preprocessing, classification, and analysis of models.

3.1. Dataset Preprocessing

As stated before, the most acute problem in classifying imbalanced data is that classifiers become biased toward the majority class. There are several methods to overcome this issue which are generally called resampling techniques. By adding minority class samples or removing samples from the majority class, resampling turns the data into a more balanced one. In this regard, there are two principal methods: oversampling and undersampling.

Oversampling algorithms generate new samples, duplicated or synthetic, that belong to the minority class. In contrast, the undersampling techniques delete samples that belong to the majority class to afford balance to the dataset [23].

As a preprocessing step in our methodology, we have utilized various well-known oversampling and undersampling techniques for balancing the dataset. Normalization and split datasets are the next steps in data preprocessing. These are elaborated on in the following.

3.1.1. Oversampling Techniques

Random Over-Sampling (ROS)

The first and simplest method in this field is random oversampling (ROS), which aims to help the distribution of datasets by increasing the number of samples in the majority class until the class distributions tend to balance. This approach is non-heuristic, meaning that it does not boast any intelligent decision boundaries. Random oversampling is usually applied to the level that excludes the imbalance. By merely regenerating samples from the minority class, ROS tackles a balance in the training model. However, duplicating similar samples may lead to the problem of overfitting, particularly for the samples belonging to the minority classes [26,48]. Figure 3 shows an illustration of the oversampling technique.

Synthetic Minority Oversampling Technique

Synthetic Minority Oversampling Technique (SMOTE) [14,49,50,51] is another resampling technique that aims to increase the amount of minority class samples by creating synthetic samples in the minority class and is applied for balancing datasets with a highly unbalanced ratio. In order to avoid the issue of overfitting, the synthetic generation of new samples differed from the multiplication algorithm.

The main idea behind SMOTE is to generate new samples of data in the minority class by interpolation between samples of this class that are in close vicinity of each other [4,52]. Thus, SMOTE increases the number of minority class examples within an imbalanced dataset and consequently enables the classifier to achieve better generalizability. The formal procedure for SMOTE can be explained as follows: Firstly, N, which is the desired amount of oversampling, should be set to an integer number. This number can be opted for in that the dataset becomes balanced with a ratio of 1:1 within the different classes. Then, three main steps should be taken iteratively. These steps are 1: Randomly selecting a sample that belongs to the minority class, 2: The K (default 5) nearest neighbors of this sample should be selected, 3: N of these K neighbors are selected randomly for interpolation and generating new samples [53]. An intuition of how SMOTE works is shown in Figure 4.

3.1.2. Undersampling Techniques

Random Under-Sampling

The simplest technique among under-sampling methods is Random Under-Sampling (RUS) which is a data-level approach. Here, the algorithm tries to reduce the number of the majority class samples to balance data. In RUS, we randomly select samples within the majority class and delete them, which makes the distribution of a class-imbalanced dataset with a highly unbalanced ratio more balanced. RUS is a non-heuristic approach that does not behave as smart as some other algorithms. Its main drawback is the high probability of losing valuable information within a dataset [4]. More precisely, the principal issue in proposing this method is that there is no control over what information about the majority class is being thrown away. As a result, the samples that contain information and details about the decision boundary may be removed, and that valuable information is lost [23]. An overview of RUS is shown in Figure 5.

Tomek Links

Tomek Links (TL) [24] is another effective undersampling technique used for balancing the data. TLs are pairs of samples that are very close two each other, but they belong to different classes. These samples are contiguous to the borderline between classes. In mathematical language, given a pair of samples $\left(S_i.S_j\right)$ from the dataset, we say that there is a TL between the two samples if at least one of the two following inequalities is satisfied:

(1)$\mathsf{\delta }\left({\mathrm{S}}_{\mathrm{i}}.{\mathrm{S}}_{\mathrm{l}}\right) < \mathsf{\delta }\left({\mathrm{S}}_{\mathrm{i}}.{\mathrm{S}}_{\mathrm{j}}\right), \mathsf{\delta }\left({\mathrm{S}}_{\mathrm{j}}.{\mathrm{S}}_{\mathrm{l}}\right) < \mathsf{\delta }\left({\mathrm{S}}_{\mathrm{i}}.{\mathrm{S}}_{\mathrm{j}}\right)$

where $\delta \left(x.y\right)$ is the distance between $x$ and $y$ [54].

Generally, one of the two samples that form a TL is considered a noisy sample, or the two samples together are considered borderline [4]. In this case, by eliminating the samples of the majority class that belong to the pairs forming TLs, the distance between the two classes increases, and the dataset becomes more balanced [23]. See Figure 6, which shows how TLs can be used to reduce the number of samples in the majority class.

One-Sided Selection (OSS)

One-Sided Selection (OSS) [25] is proposed as an undersampling technique whose main idea is to combine TL and Condensed Nearest Neighbor Rule. In order to address the issue of imbalanced datasets, this approach leaves the minority class samples completely intact. It filters out the redundant samples in the majority class through a modification of the condensed nearest-neighbor rule [55].

In OSS, $\delta \left(x.y\right)$ is supposed to be a distance value that meets the requirements for being a TL, where $x$ is chosen from the majority class, and $y$ is selected from the minority. This way, two scenarios can happen: (1) a TL is found to be on the class boundary when both $x$ and $y$ exist in the right class regions (2) a TL is found to be inside one of the class regions when either $x$ or $y$ lies in the wrong region. OSS was introduced to decrease the number of majority class samples by omitting the data points which are borderline or noisy [56]. Figure 7 illustrates a diagram of the OSS technique.

NearMiss

The last undersampling technique proposed in this study and introduced here is NearMiss [5]. This method is based on the K-nearest neighbors algorithm and categorized as NearMiss-1, NearMiss-2, and NearMiss-3. The main idea behind NearMiss is to consider the mean distances of samples from the majority class to the samples from the minority class.

Contrary to randomly removing samples from the majority class, these methods eliminate these samples intelligibly. NearMiss-1 removes the majority class samples whose mean distances to the three nearest samples of the minority class are minimal. On the other hand, NearMiss-2 deletes the samples from the majority class with minimal average distances to the three farthest minority samples. Finally, NearMiss-3 selects a certain number of the closest samples of the majority class regarding every minority class sample [57].

As claimed in [5], the results of experiments showed that NearMiss-2 has a better performance than NearMiss-1 and NearMiss-3. Furthermore, it outperforms the RUS technique [23].

It is worth noting that NearMiss can be fine-tuned in two aspects: The variant that can be chosen from 1, 2, and 3. In addition, the number of neighbors to consider for calculating the mean distances is three as the default. An outline of the NearMiss algorithms is shown in Figure 8.

3.1.3. Normalization

Normalization is one of the most crucial preprocessing steps for any challenge in machine learning. It can be performed by scaling or transforming the original data to balance the contributions of different features in data samples. In this study, we have normalized the input data to make a distribution between zero and one.

3.1.4. Split Dataset

Further, due to the low number of samples in datasets which makes the classification result extremely unstable, we have trained and evaluated our models for 100 runs. In each run, first, we randomly shuffle and split the data into training and testing sets and train them according to the model for 2000 epochs and then evaluate it.

3.2. Models

In this section, we introduce our two proposed Artificial Neural Network (ANN)-based models, including Deep Neural Network and Convolutional Neural Network.

3.2.1. Proposed Deep Neural Network

Deep Neural Networks (DNNs) have recently become among the favorite approaches in various fields in the domain of Artificial Intelligence [58]. These networks, which are famously called models, are characterized by several layers that contain a huge number of computational units. These units, which are interconnected, meaning that the output of one unit is the input of the other, are conceived as the imitation of the physiological brain’s structure. In mathematical terms, they are a set of parametrized linear and non-linear transformations capable of being adjusted in order to output abstractions of the input data [59]. This capability comes from the amalgamation of multiple layers full of perceptrons. Although a single perceptron cannot handle data that are not linearly separable, they are the basis of Multi-Layer Perceptrons (MLPs), whose ability to transform highly non-linear data makes them a powerful and efficient tool in machine learning [60].

Furthermore, the first proposed method in this paper is a DNN-based model. The architecture of this model is demonstrated in Figure 9.

As is observed in Figure 9, our DNN model comprises different layers, including a fully connected one followed by an activation function and a batch normalization layer. Then, a fully connected layer, an activation function, and a batch normalization followed by a dropout and a single neuron fully connected layer come after.

3.2.2. Proposed Convolutional Neural Network

Convolutional operations are the main components in Convolutional Neural Network (CNN)-based models. These operations enable CNNs to extract and learn the salient features existent in the input data [61]. CNN comprises different layers that output feature maps, resulting in sliding different kernels on the input and applying activation functions [62]. Compared with DNNs, the major advantage of CNNs over DNNs is their capability to reduce the computational cost in each layer. The convoluted features extracted by these models are compact representations of the input data, which can be further used in downstream tasks such as classification [63].

In this paper, the second method proposed for the binary classification of the input data is a CNN-based model. The architecture of this model is demonstrated in Figure 10.

As is seen in Figure 10, our proposed model consists of 4 layers (two 1-dimensional convolutional layers and two fully connected layers). After each hidden layer, a non-linear activation function (ReLU) is applied to the output. To make our training process more efficient, we have experimented with several loss functions, among which Focal Loss (FL) [64] claimed better supervision of the network. In fact, it was invented to address the issue of class imbalance. FL belongs to the cost-sensitive methods which were originally introduced in the case of object detection, where the imbalance between background and salient object is often frequent. FL is a modification to the cross-entropy loss in that during the training procedure; the neural network receives more cost for wrongly predicting complex training samples.

More precisely, the cross-entropy loss function is among the most common loss functions in deep learning that originates from information theory. It is seemingly identical to the negative log-likelihood loss function, and for the binary classification problems, the binary cross-entropy loss function, denoted by $l_{BCE}$ is as follows:

(2)$l_{BCE}(y,\stackrel{⌢}{y})=-(y\log (\stackrel{⌢}{y})+(1-y)\log (1-\stackrel{⌢}{y}))$

Here $y,\stackrel{⌢}{y}\in {\left\{0,1\right\}}^N$, where N is the number of samples, $\stackrel{⌢}{y}$ is the predicted value, and $y$ denotes the ground truth table.

The problem with the cross-entropy loss function is that in the case of imbalance classification, the larger class overwhelms the loss by dominating the gradient [65]. Hence, to obtain the Focal loss function, one can simplify and rewrite Equation (2) in the following way:

(3)$\mathrm{CE}\left(\mathrm{p},\mathrm{y}\right)=\left\{\begin{array}{cc}\hfill -\log \left(\mathrm{p}\right), & \mathrm{if} \mathrm{y}=1\hfill \\ \hfill -\log \left(1-\mathrm{p}\right), & \mathrm{if} \mathrm{y}=0 \hfill \end{array}\right\}$

Name the probability of predicting the ground truth class $p_t$ and define $p_t$ as:

(4)${\mathrm{CEp}}_{\mathrm{t}}=\left\{\begin{array}{cc}\hfill \mathrm{p}, & \mathrm{if} \mathrm{y}=1\hfill \\ \hfill 1-\mathrm{p}, & \mathrm{if} \mathrm{y}=0 \hfill \end{array}\right\}$

Therefore, $l_{BCE}$ can be rewritten and simplified as:

(5)$l_{BCE}\left(p,y\right)=CE\left(p_t\right)=-\log \left(p_t\right)$

Finally, FL augments a modulating factor $\alpha {\left(1-p_t\right)}^{\gamma }$ to the binary class entropy loss function, where γ > 0 is a tunable focusing parameter which yields the following equation:

(6)$FL\left(p_t\right)=-{\alpha }_t{\left(1-p_t\right)}^{\gamma }\log \left(p_t\right)$

4. Experimental Results

This section comprised simulation setup, dataset description, split dataset, evaluation metrics, and classification results.

4.1. Simulation Setup

This section includes the implementation details of our proposed methods. The tools used in this paper are listed in Table 1.

Moreover, in our implementation, we used the Adam algorithm to optimize the models’ parameters with a learning rate of 0.001. For the loss function, FL is used with the alpha parameter set to 0.25 and gamma parameters set to 2. Further, the list of hyperparameters of the DNN and CNN models and the description of the parameters for oversampling techniques is described in detail in Table 2, Table 3 and Table 4.

4.2. Dataset Description

In order to examine our proposed methods, we used the KEEL [66] dataset repository, breast cancer, and Z-Alizadeh Sani datasets. As is depicted in Table 5, the datasets comprise different imbalanced datasets for classification tasks.

Based on Table 5, the first column indicates the number of attributes of each dataset. The second, the sum of positive and negative samples, is calculated as all samples. Furthermore, the imbalance ratio between minority or positive and majority or negative classes is assigned in the third column. Meanwhile, the imbalance ratio is achieved by dividing negative samples into positive samples. As described in Section 3.1.3 (split dataset), the dataset was randomly shuffled and split into training and testing sets which the dataset was trained according to the model for 2000 epochs. The generated models were trained and evaluated for 100 runs.

4.3. Evaluation Metrics

This section includes the elaboration of the metrics which is used to evaluate the performance of our proposed models. A fundamental classification metric tool is the Confusion Matrix. This tool is a way of demonstrating the number of correctly and incorrectly predicted samples by a classifier. It is usually a table that contains the actual and predicted state of samples compared to each other. Figure 11 depicts such a matrix for a binary classifier. This matrix includes four items, namely True Positive (TP), True Negative (TN), False Positive (FP), and False Negative (FN). The formal description of these four items is as follows:

• True Positive (TP): The number of samples that belongs to the positive class and are correctly predicted as positive by the classifier

• True Negative (TN): The number of samples that belongs to the negative class and are correctly predicted as negative by the classifier

• False Positive (FP): Number of samples that belongs to the negative class, even though they are predicted as positive by the classifier

• False Negative (FN): Number of samples that belongs to the positive class, even though they are predicted as negative by the classifier

Figure 11

Confusion matrix.

[Figure omitted. See PDF]

Based on Figure 11, the classes of the minority and majority are remarked as positive and negative classes, respectively. Therefore, a confusion matrix is used to obtain performance metrics for the models on the imbalanced datasets. We utilized eight metrics, such as accuracy, precision, recall, F1-score, G-Mean, specificity, AUC-ROC, and Kappa, for evaluating the DNN and CNN models [29,42,67,68,69,70,71].

4.3.1. Accuracy

Accuracy is the ratio of the number of samples that are predicted correctly to the total of the input samples, as formulated in (7).

(7)$\mathrm{Accuracy}=\frac{\mathrm{TP}+\mathrm{TN}}{\mathrm{FP}+\mathrm{FN}+\mathrm{TP}+\mathrm{TN}}$

4.3.2. Specificity

The specificity is the proportion of true-negative samples to the overall number of true-negative and false-positive samples. The specificity or True Negative Rate (TNR) of a classifier is calculated using Equation (8).

(8)$\mathrm{Specificity}=\frac{\mathrm{TN}}{\mathrm{TN}+\mathrm{FP}}$

4.3.3. Recall

A recall is another measurement that shows the ratio of predicted positive samples to all the relevant samples, meaning the samples which have been actually positive. The recall is a significant metric for imbalanced datasets, demonstrating the learning accuracy of the positive class. It is calculated by Equation (9).

(9)$\mathrm{sensitivity}=\frac{\mathrm{TP}}{\mathrm{TP}+\mathrm{FN}}$

4.3.4. G-Mean

The G-mean is exploited as an accuracy metric as it can gauge the accuracy rates of majority and minority classes. It is achieved by Equation (10).

(10)$\mathrm{G}-\mathrm{Mean}=\sqrt{\mathrm{Sensitivity}\times \mathrm{Specificity}}$

4.3.5. Precision

Precision shows how well a classifier’s performance is in terms of predicting positive samples. As Equation (11) shows, it is easily calculated by dividing the number of true positives by the total number of predicted samples as positive.

(11)$\mathrm{Precision}=\frac{\mathrm{TP}}{\mathrm{TP}+\mathrm{FP}}$

4.3.6. F1-Score

F1-Score, which is also called F-score or F-measure, indicates the balance which exists between recall and precision for a classifier. The closer it is to one, the more balance between precision and recall exists. F1-Score can be obtained by Equation (12).

(12)$\mathrm{F}1-\mathrm{Score}=\frac{2\mathrm{TP}}{2\mathrm{TP}+\mathrm{FP}+\mathrm{FN}}$

4.3.7. Kappa

The kappa metric considers the random classification model accuracy to evaluate the obtained classification accuracy. It is an important metric that indicates whether the accuracy of the classifier is at the level of reliability. The values of the Kappa are between −1 to 1. On the other hand, three reliability levels of Kappa have been exploited to assess the accuracy as follows:

• Kappa $\ge 75$: Robust consistency, high reliable accuracy.

• $0.4 \le$ Kappa < 0.75: the accuracy’s reliance level is generally.

• Kappa < 0.4: Accuracy is unreliable.

The kappa formula has been specified in (13).

(13)$\mathrm{Kappa}=\frac{\mathrm{Accuracy}-\mathrm{random}}{1-\mathrm{random}}$

4.3.8. AUC-ROC

The AUC-ROC is a crucial measurement to evaluate the performance of generated classification models. A ROC plot represents the trade-off between true positives and false positives, which actually indicates the correlation between specificity and recall. Furthermore, AUC specifies the amount of separability power of the classifier. The AUC range is from 0 to 1. Therefore, the higher the AUC means the model has better performance at recognizing the minority and majority classes.

4.4. Classification Results

In this section, we demonstrate our experimental results based on the evaluation metrics such as accuracy, precision, recall, F1-score, G-Mean, specificity, AUC, and Kappa. The results have been elaborated by obtaining the average for each metric on three imbalanced datasets, including the KEEL repository, breast cancer, and Z-Alizadeh Sani for classification tasks.

The results are given in Table 6, Table 7, Table 8, Table 9, Table 10 and Table 11 for six models, such as SMOTE + NORM. + CNN/DNN, TL + Normalization (NORM.) + CNN/DNN, OSS + NORM. + CNN/DNN, NearMiss + NORM. + CNN/DNN, ROS + NORM. + CNN/DNN, and RUS + NORM. + CNN/DNN, respectively. We marked the best results in boldface.

According to the results obtained, the proposed SMOTE + NORM. + CNN model outperforms other models in terms of eight metrics on the datasets. As a result, the mixed SMOTE-NORM.-CNN model demonstrates the impact of using SMOTE in our CNN model so that the overall performance has been enhanced.

Moreover, in our experiment, the ROC plots based on the best AUC scores gained through the models are shown in Figure 12a–z on the datasets. Due to Table 6, Table 7, Table 8, Table 9, Table 10 and Table 11 and ROC plots, the mixed SMOTE-NORM.-CNN model has the best AUC value.

Table 6

SMOTE + NORM. + CNN/DNN.

Dataset	Acc (CNN/DNN)		Pre (CNN/DNN)		Rec (CNN/DNN)		F1 (CNN/DNN)		G-Mean (CNN/DNN)		Spe (CNN/DNN)		AUC (CNN/DNN)		Kap (CNN/DNN)
ecoli1	99.11	99.38	99.13	99.40	99.11	99.38	99.11	99.38	99.11	99.38	98.83	99.17	99.00	99.25	98.21	98.77
ecoli2	99.60	99.76	99.60	99.77	99.60	99.76	99.60	99.76	99.60	99.76	99.44	99.54	99.03	99.43	99.19	99.53
ecoli3	99.53	99.64	99.54	99.65	99.53	99.64	99.53	99.64	99.53	99.64	99.21	99.39	99.20	99.31	99.06	99.29
ecoli-0_vs_1	99.91	99.83	99.92	99.84	99.91	99.83	99.91	99.83	99.91	99.83	99.93	99.79	99.27	99.10	99.83	99.66
glass0	99.26	99.19	99.27	99.26	99.26	99.19	99.26	99.19	99.26	99.19	99.27	99.00	99.31	99.53	99.27	98.38
glass1	99.31	99.23	99.32	99.25	99.31	99.23	99.31	99.23	99.31	99.23	99.32	99.64	99.25	99.34	99.30	98.46
glass6	100.00	100.00	100.00	100.00	100.00	100.00	100.00	100.00	100.00	100.00	100.00	100.00	100.00	100.00	100.00	100.00
Haberman	95.60	95.44	95.61	96.30	95.60	95.77	95.60	96.03	95.60	96.10	95.61	96.45	97.03	98.05	95.00	96.01
iris0	100.00	100.00	100.00	100.00	100.00	100.00	100.00	100.00	100.00	100.00	100.00	100.00	99.99	99.99	100.00	100.00
new-thyroid1	99.95	99.96	99.96	99.96	99.95	99.96	99.60	99.96	99.96	99.96	99.95	99.92	99.90	99.95	99.96	99.92
new-thyroid2	99.95	99.97	99.96	99.97	99.95	99.97	99.95	99.97	99.95	99.97	99.96	99.94	99.38	99.61	99.94	99.94
page-blocks0	99.42	99.39	99.42	99.39	99.42	99.39	99.42	99.39	99.42	99.39	99.38	99.25	99.27	99.28	99.23	98.97
pima	99.35	99.15	99.36	99.15	99.35	99.14	99.35	99.15	99.35	99.21	99.30	99.37	99.81	99.25	99.46	99.03
segment0	99.99	99.99	99.99	99.99	99.99	99.99	99.99	99.99	99.99	99.99	99.99	99.99	99.67	99.31	99.99	99.99
vehicle0	99.95	99.97	99.95	99.97	99.95	99.97	99.95	99.97	99.95	99.97	99.89	99.95	99.83	99.60	99.88	99.95
vehicle1	99.81	99.76	99.81	99.76	99.81	99.76	99.81	99.76	99.81	99.76	99.87	99.67	99.90	99.15	99.90	99.52
vehicle2	99.99	99.99	99.99	99.99	99.99	99.99	99.99	99.99	99.99	99.99	99.99	99.99	99.84	99.57	99.97	99.98
vehicle3	99.85	99.74	99.86	99.75	99.85	99.74	99.85	99.74	99.85	99.74	99.82	99.62	99.80	99.20	99.91	99.48
wisconsin	99.80	99.84	99.81	99.85	99.80	99.84	99.80	99.84	99.80	99.84	99.80	99.75	99.46	99.51	99.64	99.60
yeast1	98.98	98.71	98.98	98.72	98.98	98.71	98.98	98.71	98.98	98.71	98.98	99.81	98.80	98.62	98.67	98.35
yeast3	100.00	100.00	100.00	100.00	100.00	100.00	100.00	100.00	100.00	100.00	100.00	100.00	100.00	100.00	99.90	99.94
yeast-2_vs_4	99.95	99.91	99.96	99.92	99.95	99.91	99.95	99.91	99.95	99.91	99.94	99.92	99.80	99.45	99.73	99.67
penbased	99.99	99.99	99.99	99.99	99.99	99.99	99.99	99.99	99.99	99.99	99.88	99.81	99.98	99.96	99.99	99.99
nursery	88.71	87.42	88.72	87.43	88.71	87.42	88.71	87.42	88.71	87.42	88.62	87.30	90.00	89.92	88.40	87.30
breast cancer	99.67	98.84	99.68	98.85	99.67	98.84	99.67	98.84	99.67	98.84	99.46	98.67	99.42	99.14	99.48	98.62
Z-Alizadeh Sani	98.57	97.91	98.58	97.92	98.57	97.91	98.57	97.91	98.57	97.85	98.42	97.84	99.14	99.02	98.21	98.04
Average	99.08	98.96	99.09	99.00	99.08	98.97	99.09	98.98	99.08	98.98	99.03	98.99	99.08	99.02	98.92	98.78

NORM.: Normalization, Accuracy: Acc, Precision: Pre, Recall: Rec F1-Score: F1, Specificity: Spe, Kappa: Kap, The best results for each metric are shown in the tables. Bold font indicates the best result obtained for each metric of the models.

Table 7

TL + NORM. + CNN/DNN.

Dataset	Acc (CNN/DNN)		Pre (CNN/DNN)		Rec (CNN/DNN)		F1 (CNN/DNN)		G-Mean (CNN/DNN)		Spe (CNN/DNN)		AUC (CNN/DNN)		Kap (CNN/DNN)
ecoli1	99.05	98.98	99.06	98.99	99.05	98.98	99.05	98.98	99.05	98.99	99.00	98.98	99.02	98.80	99.10	98.96
ecoli2	99.51	99.76	99.52	99.77	99.51	99.76	99.51	99.76	99.51	99.76	99.45	99.54	99.03	99.70	99.19	99.53
ecoli3	99.23	99.74	99.24	99.75	99.23	99.74	99.23	99.74	99.23	99.74	99.20	99.41	99.05	99.52	99.00	99.30
ecoli-0_vs_1	99.81	99.75	99.82	99.76	99.81	99.75	99.81	99.75	99.81	99.75	99.82	99.63	99.60	99.17	99.80	99.61
glass0	99.05	98.98	99.06	98.99	99.05	98.98	99.05	98.98	99.05	98.99	99.02	99.00	99.00	98.90	99.02	98.38
glass1	98.83	98.32	98.84	98.33	98.83	98.32	98.83	98.32	98.83	98.32	98.83	98.42	99.15	99.32	98.80	98.32
glass6	100.00	100.00	100.00	100.00	100.00	100.00	100.00	100.00	100.00	100.00	100.00	100.00	100.00	100.00	100.00	100.00
haberman	94.20	95.24	94.21	95.25	94.20	95.25	94.20	95.25	94.20	95.24	94.25	95.25	97.23	97.23	95.00	96.35
iris0	100.00	100.00	100.00	100.00	100.00	100.00	100.00	100.00	100.00	100.00	100.00	100.00	100.00	100.00	100.00	100.00
new-thyroid1	99.75	99.78	99.76	99.79	99.75	99.78	99.75	99.78	99.75	99.78	99.76	99.74	99.80	99.60	99.54	99.61
new-thyroid2	99.89	99.85	99.90	99.86	99.89	99.85	99.89	99.85	99.89	99.85	99.88	99.82	99.51	99.74	99.80	99.74
page-blocks0	98.62	99.39	98.63	99.39	98.62	99.39	98.62	99.39	98.62	99.39	98.60	99.25	99.37	99.28	99.21	98.97
pima	99.27	99.10	99.28	99.11	99.27	99.10	99.27	99.10	99.27	99.10	99.24	99.10	99.12	99.10	99.05	99.04
segment0	99.99	99.99	99.99	99.99	99.99	99.99	99.99	99.99	99.99	99.99	99.99	99.99	99.67	99.31	99.99	99.99
vehicle0	99.90	99.97	99.91	99.97	99.90	99.97	99.90	99.97	99.90	99.97	99.90	99.95	99.85	99.60	99.87	99.95
vehicle1	99.70	99.75	99.71	99.76	99.70	99.75	99.70	99.75	99.70	99.75	99.77	99.65	99.78	99.66	99.68	99.50
vehicle2	99.99	99.99	99.99	99.99	99.99	99.99	99.99	99.99	99.99	99.99	99.99	99.99	99.83	99.63	99.97	99.98
vehicle2	99.80	99.76	99.81	99.75	99.80	99.74	99.81	99.74	99.80	99.74	99.80	99.62	99.81	99.20	99.91	99.56
wisconsin	99.50	99.84	99.51	99.85	99.50	99.84	99.50	99.84	99.50	99.84	99.50	99.75	99.51	99.51	99.40	99.60
yeast1	98.98	98.71	98.98	98.72	98.98	98.71	98.98	98.71	98.98	98.71	98.98	99.81	98.80	98.62	98.67	98.35
yeast3	100.00	100.00	100.00	100.00	100.00	100.00	100.00	100.00	100.00	100.00	100.00	100.00	100.00	100.00	99.92	99.93
yeast-2_vs_4	99.85	99.89	99.86	99.90	99.85	99.89	99.85	99.89	99.85	99.89	99.84	99.88	99.70	99.62	99.64	99.60
penbased	99.99	99.99	99.99	99.99	99.99	99.99	99.99	99.99	99.99	99.99	99.81	99.77	99.90	99.85	99.99	99.99
Nursery	89.84	88.21	89.85	88.22	89.84	88.21	89.84	88.21	89.84	88.21	89.70	88.11	91.05	90.95	89.20	88.00
breast cancer	99.05	98.70	99.06	98.71	99.05	98.70	99.05	98.70	99.05	98.70	98.90	98.55	99.26	99.07	98.89	98.57
Z-Alizadeh Sani	98.34	97.79	98.35	97.80	98.34	97.79	98.34	97.79	98.34	97.79	98.29	97.62	98.80	98.34	98.11	97.88
Average	98.92	98.95	98.93	98.90	98.92	98.90	98.92	98.90	98.92	98.90	98.90	98.87	99.07	98.98	98.87	98.79

NORM.: Normalization, Accuracy: Acc, Precision: Pre, Recall: Rec F1-Score: F1, Specificity: Spe, Kappa: Kap, The best results for each metric are shown in the tables. Bold font indicates the best result obtained for each metric of the models.

Table 8

OSS + NORM. + CNN/DNN.

Dataset	Acc (CNN/DNN)		Pre (CNN/DNN)		Rec (CNN/DNN)		F1 (CNN/DNN)		G-Mean (CNN/DNN)		Spe (CNN/DNN)		AUC (CNN/DNN)		Kap (CNN/DNN)
ecoli1	98.70	98.55	98.71	98.56	98.70	98.55	98.70	98.55	98.70	98.55	98.65	98.50	99.12	99.24	98.75	98.45
ecoli2	98.41	98.62	98.42	98.63	98.41	98.62	98.41	98.62	98.41	98.62	98.36	98.60	99.13	99.50	98.25	98.54
ecoli3	99.20	99.63	99.21	99.64	99.20	99.63	99.20	99.63	99.20	99.63	99.15	99.52	99.26	99.60	99.03	99.31
ecoli-0_vs_1	99.59	99.80	99.60	99.81	99.59	99.80	99.59	99.80	99.59	99.80	99.48	99.74	99.40	99.41	99.58	99.21
glass0	99.15	99.10	99.16	99.11	99.15	99.10	99.15	99.10	99.15	99.10	99.10	99.10	99.01	99.18	98.99	99.00
glass1	98.78	98.40	98.79	98.41	98.78	98.40	98.78	98.40	98.78	98.40	98.62	98.38	99.05	98.99	98.98	98.84
glass6	100.00	100.00	100.00	100.00	100.00	100.00	100.00	100.00	100.00	100.00	100.00	100.00	100.00	100.00	100.00	100.00
haberman	94.50	95.14	94.51	95.15	94.50	95.15	94.50	95.15	94.50	95.14	94.46	95.30	98.00	97.20	94.89	96.30
iris0	100.00	100.00	100.00	100.00	100.00	100.00	100.00	100.00	100.00	100.00	100.00	100.00	100.00	100.00	100.00	100.00
new-thyroid1	99.70	99.63	99.71	99.64	99.70	99.63	99.70	99.63	99.70	99.63	99.62	99.60	99.77	99.05	99.80	99.61
new-thyroid2	99.80	99.80	99.81	99.81	99.80	99.80	99.80	99.80	99.80	99.80	99.74	99.74	99.20	99.55	99.22	99.74
page-blocks0	98.43	99.21	98.44	99.22	98.43	99.21	98.43	99.21	98.43	99.21	98.21	99.15	98.50	99.40	99.20	98.84
pima	99.50	99.11	99.51	99.12	99.50	99.11	99.50	99.11	99.50	99.11	99.48	99.09	99.32	99.05	99.14	99.09
segment0	99.99	99.99	99.99	99.99	99.99	99.99	99.99	99.99	99.99	99.99	99.99	99.99	99.38	99.20	99.99	99.99
vehicle0	99.70	99.80	99.71	99.81	99.70	99.80	99.70	99.80	99.70	99.80	99.70	99.74	99.63	99.62	99.87	99.70
vehicle1	99.30	99.48	99.31	99.49	99.30	99.48	99.30	99.48	99.30	99.48	99.26	99.40	99.12	99.20	99.40	99.20
vehicle2	99.99	99.99	99.99	99.99	99.99	99.99	99.99	99.99	99.99	99.99	99.99	99.99	99.60	99.40	99.95	99.95
vehicle2	99.60	99.70	99.61	99.71	99.60	99.70	99.60	99.70	99.60	99.70	99.58	99.56	99.74	99.22	99.54	99.47
wisconsin	99.50	99.75	99.51	99.76	99.50	99.75	99.50	99.75	99.50	99.75	99.50	99.70	99.51	99.60	99.40	99.51
yeast1	98.85	98.70	98.86	98.71	98.85	98.71	98.85	98.71	98.85	98.71	98.81	99.69	98.40	98.31	98.52	98.22
yeast3	100.00	100.00	100.00	100.00	100.00	100.00	100.00	100.00	100.00	100.00	100.00	100.00	100.00	100.00	99.81	99.70
yeast-2_vs_4	99.85	99.89	99.86	99.90	99.85	99.89	99.85	99.89	99.85	99.89	99.84	99.88	99.70	99.62	99.64	99.60
penbased	99.99	99.99	99.99	99.99	99.99	99.99	99.99	99.99	99.99	99.99	99.76	99.69	99.84	99.73	99.99	99.99
nursery	89.27	88.88	89.28	88.89	89.27	88.88	89.27	88.88	89.27	88.88	89.15	88.65	90.31	91.23	89.06	88.70
breast cancer	99.15	98.81	99.16	98.82	99.15	98.81	99.15	98.81	99.15	98.81	99.01	98.69	99.19	98.99	99.03	98.73
Z-Alizadeh Sani	97.56	97.43	97.57	97.44	97.56	97.43	97.56	97.43	97.56	97.43	97.27	97.21	98.24	98.20	97.20	97.10
Average	98.78	98.82	98.79	98.83	98.78	98.82	98.78	98.82	98.78	98.82	98.72	98.80	98.93	98.94	98.73	98.72

NORM.: Normalization, Accuracy: Acc, Precision: Pre, Recall: Rec F1-Score: F1, Specificity: Spe, Kappa: Kap, The best results for each metric are shown in the tables. Bold font indicates the best result obtained for each metric of the models.

Table 9

NearMiss + NORM. + CNN/DNN.

Dataset	Acc (CNN/DNN)		Pre (CNN/DNN)		Rec (CNN/DNN)		F1 (CNN/DNN)		G-Mean (CNN/DNN)		Spe (CNN/DNN)		AUC (CNN/DNN)		Kap (CNN/DNN)
ecoli1	99.16	99.20	99.17	99.21	99.16	99.20	99.16	99.20	99.16	99.20	99.10	99.15	99.41	99.31	99.11	99.11
ecoli2	99.60	99.88	99.61	99.89	99.60	99.88	99.60	99.88	99.60	99.88	99.52	99.80	99.23	99.55	99.33	99.88
ecoli3	99.33	99.74	99.34	99.75	99.33	99.74	99.33	99.74	99.33	99.74	99.29	99.41	99.15	99.52	99.02	99.30
ecoli-0_vs_1	99.60	99.78	99.61	99.79	99.60	99.78	99.60	99.78	99.60	99.78	99.54	99.74	99.35	99.37	99.57	99.54
glass0	99.20	99.25	99.21	99.26	99.20	99.25	99.20	99.25	99.20	99.25	99.13	99.18	99.11	99.18	99.18	99.06
glass1	99.15	99.20	99.16	99.21	99.15	99.20	99.15	99.20	99.15	99.20	99.10	99.17	99.18	99.25	99.06	99.14
glass6	100.00	100.00	100.00	100.00	100.00	100.00	100.00	100.00	100.00	100.00	100.00	100.00	100.00	100.00	100.00	100.00
haberman	94.80	95.36	94.81	95.37	94.80	95.37	94.80	95.37	94.80	95.37	94.62	95.32	98.00	97.25	95.00	95.35
iris0	100.00	100.00	100.00	100.00	100.00	100.00	100.00	100.00	100.00	100.00	100.00	100.00	100.00	100.00	100.00	100.00
new-thyroid1	99.63	99.65	99.64	99.66	99.63	99.65	99.63	99.65	99.63	99.64	99.59	99.61	99.74	99.80	99.50	99.55
new-thyroid2	99.90	99.87	99.91	99.88	99.90	99.87	99.90	99.87	99.90	99.87	99.84	99.87	99.50	99.70	99.87	99.78
page-blocks0	99.20	99.43	99.21	99.44	99.20	99.43	99.20	99.43	99.20	99.43	99.15	99.39	99.35	99.24	99.11	99.24
pima	99.30	99.15	99.31	99.16	99.30	99.15	99.30	99.15	99.30	99.15	99.25	99.11	99.25	99.11	99.15	99.08
segment0	99.99	99.99	99.99	99.99	99.99	99.99	99.99	99.99	99.99	99.99	99.99	99.99	99.70	99.51	99.99	99.99
vehicle0	99.99	99.99	99.99	99.99	99.99	99.99	99.99	99.99	99.99	99.99	99.99	99.99	99.78	99.62	99.99	99.99
vehicle1	99.99	99.99	99.99	99.99	99.99	99.99	99.99	99.99	99.99	99.99	99.99	99.99	99.84	99.80	99.99	99.99
vehicle2	99.99	99.99	99.99	99.99	99.99	99.99	99.99	99.99	99.99	99.99	99.99	99.99	99.77	99.64	99.99	99.99
vehicle3	99.99	99.99	99.99	99.99	99.99	99.99	99.99	99.99	99.99	99.99	99.99	99.99	99.40	99.50	99.99	99.99
wisconsin	99.42	99.50	99.43	99.51	99.42	99.50	99.42	99.50	99.42	99.50	99.35	99.42	99.20	99.55	99.32	99.41
yeast1	99.30	99.25	99.31	99.26	99.30	99.25	99.30	99.25	99.30	99.25	99.24	99.15	99.58	98.37	99.24	98.35
yeast3	100.00	100.00	100.00	100.00	100.00	100.00	100.00	100.00	100.00	100.00	100.00	100.00	100.00	100.00	99.81	99.70
yeast-2_vs_4	99.90	99.91	99.91	99.92	99.90	99.91	99.90	99.91	99.90	99.91	99.87	99.92	99.74	99.85	99.62	99.67
penbased	99.99	99.99	99.99	99.99	99.99	99.99	99.99	99.99	99.99	99.99	99.81	99.76	99.92	99.88	99.99	99.99
nursery	88.37	88.18	88.38	88.19	88.37	88.18	88.37	88.18	88.37	88.18	88.21	88.02	90.37	89.86	88.02	88.00
breast cancer	99.23	98.92	99.24	98.93	99.23	98.92	99.23	98.92	99.23	98.92	99.11	98.75	99.05	99.02	99.09	99.02
Z-Alizadeh Sani	98.50	98.24	98.51	98.25	98.50	98.24	98.50	98.24	98.50	98.10	98.39	98.05	98.27	98.10	98.21	98.13
Average	98.98	99.01	98.98	99.02	98.98	99.01	98.98	99.01	98.98	99.01	98.92	98.95	99.07	98.99	98.89	98.89

NORM.: Normalization, Accuracy: Acc, Precision: Pre, Recall: Rec F1-Score: F1, Specificity: Spe, Kappa: Kap, The best results for each metric are shown in the tables. Bold font indicates the best result obtained for each metric of the models.

Table 10

ROS + NORM. + CNN/DNN.

Dataset	Acc (CNN/DNN)		Pre (CNN/DNN)		Rec (CNN/DNN)		F1 (CNN/DNN)		G-Mean (CNN/DNN)		Spe (CNN/DNN)		AUC (CNN/DNN)		Kap (CNN/DNN)
ecoli1	98.52	98.49	98.53	98.50	98.52	98.49	98.52	98.49	98.52	98.49	98.34	98.28	99.08	98.57	98.87	98.64
ecoli2	98.81	98.70	98.82	98.71	98.81	98.70	98.81	98.70	98.81	98.70	98.72	98.61	99.38	99.33	98.76	98.50
ecoli3	98.90	98.64	98.91	98.65	98.90	98.64	98.90	98.64	98.90	98.64	98.78	98.56	99.16	99.22	99.12	99.21
ecoli-0_vs_1	98.81	98.64	98.82	98.65	98.81	98.64	98.81	98.64	98.81	98.64	98.70	98.53	99.06	99.27	98.54	98.40
glass0	98.30	98.17	98.31	98.18	98.30	98.17	98.30	98.17	98.30	98.17	98.19	98.08	99.25	99.15	98.11	99.16
glass1	98.27	98.16	98.28	98.17	98.27	98.16	98.27	98.16	98.27	98.16	98.20	98.10	99.08	99.19	98.15	98.02
glass6	100.00	100.00	100.00	100.00	100.00	100.00	100.00	100.00	100.00	100.00	100.00	100.00	100.00	100.00	100.00	100.00
Haberman	95.45	95.50	95.46	95.51	95.45	95.50	95.45	95.50	95.45	95.50	95.30	95.40	98.00	98.26	95.24	95.40
iris0	100.00	100.00	100.00	100.00	100.00	100.00	100.00	100.00	100.00	100.00	100.00	100.00	100.00	100.00	100.00	100.00
new-thyroid1	98.59	98.63	98.60	98.64	98.59	98.63	98.59	98.63	98.59	98.63	98.46	98.60	99.01	99.14	98.68	99.85
new-thyroid2	98.87	98.79	98.88	98.80	98.87	98.79	98.87	98.79	98.87	98.79	98.54	98.50	99.08	99.01	98.80	98.72
page-blocks0	98.40	98.16	98.41	98.17	98.40	98.16	98.40	98.16	98.40	98.16	98.26	98.09	99.12	99.06	98.60	98.47
pima	98.32	99.02	98.33	99.03	98.32	99.02	98.32	99.02	98.32	99.02	98.24	98.95	99.20	99.03	99.23	99.01
segment0	99.99	99.99	99.99	99.99	99.99	99.99	99.99	99.99	99.99	99.99	99.99	99.99	99.35	99.29	99.99	99.99
vehicle0	98.98	98.90	98.99	98.91	98.98	98.90	98.98	98.90	98.98	98.90	98.86	98.80	98.60	98.57	98.86	98.80
vehicle1	98.58	98.44	98.59	98.45	98.58	98.44	98.58	98.44	98.58	98.44	98.50	98.38	99.16	99.10	98.42	98.37
vehicle2	99.99	99.99	99.99	99.99	99.99	99.99	99.99	99.99	99.99	99.99	99.99	99.99	99.55	99.34	99.95	99.95
vehicle2	99.31	99.15	99.32	99.16	99.31	99.15	99.31	99.15	99.31	99.15	99.23	99.10	99.21	99.11	99.20	99.05
wisconsin	99.72	99.70	99.73	99.71	99.72	99.70	99.72	99.70	99.72	99.70	99.60	99.55	99.23	99.19	99.50	99.45
yeast1	98.94	98.60	98.95	98.61	98.94	98.60	98.94	98.60	98.94	98.60	98.80	98.73	99.03	99.00	98.90	98.72
yeast3	100.00	100.00	100.00	100.00	100.00	100.00	100.00	100.00	100.00	100.00	100.00	100.00	100.00	100.00	99.91	99.96
yeast-2_vs_4	98.93	99.00	98.94	99.00	98.93	99.00	98.93	99.00	98.93	99.00	98.81	98.90	98.86	99.10	98.99	98.92
penbased	99.99	99.99	99.99	99.99	99.99	99.99	99.99	99.99	99.99	99.99	99.73	99.60	99.89	99.87	99.99	99.99
nursery	88.19	88.03	88.20	88.04	88.19	88.03	88.19	88.03	88.19	88.03	88.02	87.93	90.19	89.95	87.94	87.83
breast cancer	99.00	98.72	99.01	98.73	99.00	98.72	99.00	98.72	99.00	98.72	98.81	98.61	98.73	98.52	98.67	98.43
Z-Alizadeh Sani	96.87	96.60	96.88	96.61	96.87	96.60	96.87	96.60	96.87	96.60	96.72	96.50	97.19	97.00	96.72	96.49
Average	98.45	98.38	98.45	98.39	98.45	98.38	98.45	98.38	98.45	98.38	98.33	98.29	98.78	98.74	98.42	98.43

NORM.: Normalization, Accuracy: Acc, Precision: Pre, Recall: Rec F1-Score: F1, Specificity: Spe, Kappa: Kap, The best results for each metric are shown in the tables. Bold font indicates the best result obtained for each metric of the models.

Table 11

RUS + NORM. + CNN/DNN.

Dataset	Acc (CNN/DNN)		Pre (CNN/DNN)		Rec (CNN/DNN)		F1 (CNN/DNN)		G-Mean (CNN/DNN)		Spe (CNN/DNN)		AUC (CNN/DNN)		Kap (CNN/DNN)
ecoli1	98.26	98.17	98.27	98.18	98.26	98.17	98.26	98.17	98.26	98.17	98.19	98.07	99.02	99.00	98.64	98.60
ecoli2	98.51	98.43	98.52	98.44	98.51	98.43	98.51	98.43	98.51	98.43	98.39	98.37	99.13	99.06	98.71	98.60
ecoli3	98.84	98.70	98.85	98.71	98.84	98.70	98.84	98.70	98.84	98.70	98.76	98.63	99.19	99.16	98.80	98.62
ecoli-0_vs_1	98.77	98.60	98.78	98.61	98.77	98.60	98.77	98.60	98.77	98.60	98.70	98.51	99.03	99.01	98.65	98.57
glass0	98.36	98.13	98.37	98.14	98.36	98.13	98.36	98.13	98.36	98.13	98.24	98.03	99.28	99.06	98.17	98.11
glass1	98.49	98.53	98.50	98.54	98.49	98.53	98.49	98.53	98.49	98.53	98.33	98.44	99.16	98.39	98.27	98.45
glass6	100.00	100.00	100.00	100.00	100.00	100.00	100.00	100.00	100.00	100.00	100.00	100.00	100.00	100.00	100.00	100.00
haberman	94.73	94.61	94.74	94.62	94.73	94.61	94.73	94.61	94.73	94.61	94.65	94.52	99.01	98.89	94.29	94.20
iris0	99.99	99.99	99.99	99.99	99.99	99.99	99.99	99.99	99.99	99.99	99.99	99.99	99.84	99.70	99.96	99.95
new-thyroid1	98.29	98.43	98.30	98.44	98.30	98.43	98.30	98.43	98.30	98.43	98.05	98.27	99.32	99.43	99.27	98.43
new-thyroid2	98.80	98.30	98.81	98.31	98.80	98.30	98.80	98.30	98.80	98.30	98.69	98.15	98.99	98.52	99.01	98.78
page-blocks0	98.31	98.14	98.32	98.15	98.31	98.14	98.31	98.14	98.31	98.14	98.15	98.04	98.87	98.45	98.32	98.14
pima	98.19	98.05	98.20	98.06	98.19	98.05	98.19	98.05	98.19	98.05	98.06	97.94	98.85	98.56	98.00	97.94
segment0	99.99	99.99	99.99	99.99	99.99	99.99	99.99	99.99	99.99	99.99	99.99	99.99	99.29	99.21	99.99	99.99
vehicle0	98.84	98.73	98.85	98.74	98.84	98.73	98.84	98.73	98.84	98.73	98.63	98.73	99.15	98.99	98.70	98.52
vehicle1	98.46	98.32	98.47	98.33	98.46	98.32	98.46	98.32	98.46	98.32	98.32	98.24	98.21	98.10	98.24	98.10
vehicle2	99.99	99.99	99.99	99.99	99.99	99.99	99.99	99.99	99.99	99.99	99.99	99.99	99.33	99.05	99.82	99.70
vehicle3	99.28	99.13	99.29	99.14	99.28	99.13	99.28	99.13	99.28	99.13	99.18	99.05	99.08	98.99	99.11	98.94
wisconsin	99.64	99.48	99.65	99.49	99.64	99.48	99.64	99.48	99.64	99.48	99.61	99.34	99.64	99.27	99.29	99.16
yeast1	98.99	98.70	99.00	98.71	98.99	98.70	98.99	98.70	98.99	98.70	98.79	98.61	99.02	98.89	98.75	98.52
yeast3	100.00	100.00	100.00	100.00	100.00	100.00	100.00	100.00	100.00	100.00	100.00	100.00	100.00	100.00	99.84	99.62
yeast-2_vs_4	98.82	98.70	98.83	98.71	98.82	98.70	98.82	98.70	98.82	98.62	98.70	98.70	98.82	98.50	98.70	98.63
penbased	99.99	99.99	99.99	99.99	99.99	99.99	99.99	99.99	99.99	99.99	99.92	99.87	99.90	99.87	99.99	99.99
nursery	88.06	88.00	88.07	88.01	88.06	88.00	88.06	88.00	88.06	88.00	87.96	87.80	89.96	89.85	87.84	87.95
breast cancer	98.91	98.70	98.92	98.71	98.91	98.70	98.91	98.70	98.91	98.70	98.70	98.60	98.50	98.43	98.36	98.27
Z-Alizadeh Sani	96.30	96.15	96.31	96.16	96.30	96.15	96.30	96.15	96.30	96.15	96.16	96.02	96.98	97.00	96.12	96.01
Average	98.33	98.22	98.34	98.23	98.33	98.22	98.33	98.22	98.33	98.22	98.23	98.15	98.75	98.59	98.26	98.14

NORM.: Normalization, Accuracy: Acc, Precision: Pre, Recall: Rec F1-Score: F1, Specificity: Spe, Kappa: Kap, The best results for each metric are shown in the tables. Bold font indicates the best result obtained for each metric of the models.

Figure 12

ROC plots for the models from (a–z).

[Figure omitted. See PDF]

[Figure omitted. See PDF]

[Figure omitted. See PDF]

[Figure omitted. See PDF]

[Figure omitted. See PDF]

5. Discussion

The classification performance usually drops and faces different failures in the presence of imbalanced datasets. However, imbalanced datasets exist in a broad range of real-life research. Hence, for imbalanced binary classification problems, samples are usually categorized into two classes, majority and minority. Generally, the minority class often illustrates the more significant and crucial samples and interests rather than the majority class samples. Nevertheless, compared to the minority class, the majority class has a larger amount of samples, and in some cases, the situation may be exceedingly serious. Therefore, handling these problems efficiently has become a crucial and significant topic in machine and deep learning methods.

To overcome these challenges, we proposed two methods that are based on DNN and CNN algorithms. At first, several classical and well-known undersampling and oversampling methods, such as RUS, Tomek Links, OSS, Near Miss, ROS, and SMOTE, were used in the data preprocessing procedure. Furthermore, to achieve better performance, we normalized current datasets. Then, we considered the focal loss function in the process of training the desired models, which are widely implemented in neural network frameworks for class imbalance problems. Moreover, due to the limited amount of datasets samples which causes unstable classification results, we have trained and evaluated our models for 100 runs and 2000 epochs. In the end, we analyzed our proposed models concerning the accuracy, precision, recall, F1-score, G-Mean, specificity, AUC, and Kappa as evaluation metrics. Based on 24 imbalanced datasets, the average performance score of the evaluation metrics of the executed models is indicated in Table 12.

According to Table 12, it can be founded that the mixed SMOTE + NORM + CNN model has the best performance with 99.08% accuracy, 99.09% precision, 99.08% sensitivity, 99.09% F1-score, 99.08% G-Mean, 99.03% specificity, 99.08% AUC, and 98.92% kappa. Furthermore, the comparison of performance metrics between our study and study [29] on the same imbalanced dataset is demonstrated in Table 13.

According to Table 13, the results show the efficiency of our proposed model performance on 16 imbalances that 99.00% recall, 99.00% G-Mean, and 98.98% F1-score attained by the SMOTE + NORM + CNN model. In addition, the results of the proposed SMOTE + NORM. + CNN model are compared with the related works on the Z-Alizadeh Sani dataset, as represented in Table 14.

Based on Table 14, the outcomes show the dominance of the Proposed SMOTE + NORM. + CNN model compared with other studies. The hybrid model of SMOTE + NORM. + CNN verifies the best performance with 98.57% accuracy, 98.58% recall, 98.57% F1-score, 98.58% precision, 98.42% specificity, and 99.14% AUC. In addition to what I have just explained, I can add that the F1-score metric is reputable with the imbalanced Z-Alizadeh Sani dataset because this metric shows the balance between recall and precision for classifiers.

6. Conclusions and Future Work

An unbalanced dataset of the majority and minority classes is a challenging issue when samples belonging to one or more classes are not evenly distributed. Especially, imbalanced datasets reasons deep learning-based models to obtain biased results for binary classification. To address this issue, we presented oversampling and undersampling techniques such as SMOTE, TL, OSS, NearMiss, ROS, and RUS. Among these techniques, SMOTE is the most common robust that targets the growth of the amount of minority class samples by generating synthetic samples, which is employed for balancing datasets with an extremely unbalanced ratio. In this study, six deep learning-based models were used to classify the majority and minority classes. We investigated SMOTE + NORM. + CNN/DNN, TL + NORM. + CNN/DNN, OSS + NORM. + CNN/DNN, NearMiss + NORM. + CNN/DNN, ROS + NORM. + CNN/DNN, and RUS + NORM. + CNN/DNN. To evaluate these models, we utilized KEEL, breast cancer, and Z-Alizadeh Sani datasets. The results show that the mixed SMOTE-NORM-CNN model significantly outperforms other models achieving 99.08% accuracy, 99.09% precision, 99.08% sensitivity, 99.09% F1-score, 99.08% G-Mean, 99.03% specificity, 99.08% AUC, and 98.92% kappa on 24 imbalanced datasets. Furthermore, the proposed model has been compared to the study [29], and the mixed model is suitable for the same dataset. Furthermore, we investigated the related methodologies on the Z-Alizadeh Sani dataset. The results indicate that our proposed methodology is more robust. In the future, we are planning to extend our technique to deal with multi-class problems. Moreover, we shall try to exploit different versions of SMOTE (e.g., SMOTE-Cov) with generative DL models, including Variational Autoencoder (VAE) and a Generative Adversarial Network (GAN), to explore their effectiveness in comparison to our model on other real datasets. Our model’s major trade-off/limitation is that it has a longer training time and higher computational cost than traditional ML techniques.

Author Contributions

J.H.J. designed the study. J.H.J., A.M. and M.A.N. performed conceptualization and methodology. The software and analysis of data have been performed by A.M. Furthermore, J.H.J., A.M., M.A.N., S.S.O. and S.H. wrote the original draft and visualized the figures. The final manuscript has been reviewed and edited by S.H. and S.S.O. All authors have read and agreed to the published version of the manuscript.

Institutional Review Board Statement

Not applicable.

Informed Consent Statement

Not applicable.

Data Availability Statement

These links for datasets: https://sci2s.ugr.es/keel/imbalanced.php (accessed on 17 March 2023), https://www.kdd.org/kdd-cup (accessed on 17 March 2023), https://archive.ics.uci.edu/ml/datasets/Z-Alizadeh+Sani (accessed on 17 March 2023).

Conflicts of Interest

The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Footnotes

Disclaimer/Publisher’s Note: The statements, opinions and data contained in all publications are solely those of the individual author(s) and contributor(s) and not of MDPI and/or the editor(s). MDPI and/or the editor(s) disclaim responsibility for any injury to people or property resulting from any ideas, methods, instructions or products referred to in the content.

Figures and Tables

Figure 1. The schematic diagram of several data distributions in various scenarios with two-dimensional binary-class-imbalanced data. (a) Imbalanced Ratio, (b) Class Surrunding, (c) Class Overlapping, (d) Small Disjucts.

Figure 2. The framework overview of our proposed methodology in this study.

Figure 3. Illustration of Random Over-Sampling technique. This algorithm randomly selects instances from the minority class, with replacement, and adds them to the training dataset.

Figure 4. Illustration of SMOTE Oversampling for Imbalanced Classification. By interpolating between samples of the minority class that is close to one another, SMOTE generates new samples from the minority class.

Figure 5. Illustration of Random Under-Sampling technique. This method selects examples from the majority class at random and removes them from the training dataset.

Figure 6. Illustration of Tomek Links. The distance between the two classes widens as the samples of the majority class of the pairs forming TLs are eliminated, resulting in a more evenly distributed dataset. (a) Original Distribution of Data, (b) Providing the Tomek Links, (c) Removing the links.

Figure 7. Illustration of One-Sided Selection. Through a modification of the condensed closest neighbor rule, this method preserves the entirety of the minority class samples while eliminating the redundant majority class samples.

Figure 8. Illustration of NearMiss. This method refers to a collection of undersampling algorithms that select instances based on the distance of majority-class samples to minority-class ones.

Figure 9. The architecture of the proposed DNN-based classifier.

Figure 10. The architecture of the proposed CNN-based classifier.

Table 1

The implementation details.

Programming Language	Python 3.9
Deep Learning Library	PyTorch 1.9
CPU	Intel® Core™ i7-10700 CPU @ 2.90 GHz × 16
GPU	NVIDIA Corporation GP104 [GeForce GTX 1070]
RAM	64 GB

Table 2

The list of hyperparameters of the DNN model.

Layer	Layer Type	Input Features	Out Features
1	Linear (Dense)	Input shape	64
2	Relu	N/A	N/A
3	batch normalization	64	-
4	Linear (Dense)	64	64
5	Relu	N/A	N/A
6	batch normalization	64	-
7	dropout	rate: 0.3	-
8	Linear (Dense)	64	1

N/A: Not applicable.

Table 3

The list of hyperparameters of the CNN-based model.

Layer	Layer Type	Input Channels	Output Channels	Kernel Size	Stride
1	Convolution	1	16	3	1
2	Relu	N/A	N/A	N/A	N/A
3	Convolution	16	4	2	1
4	Relu	N/A	N/A	N/A	N/A
5	Linear (Dense)	16	50	N/A	N/A
6	Relu	N/A	N/A	N/A	N/A
7	Linear (Dense)	16	1	N/A	N/A

N/A: Not applicable.

Table 4

The description of the parameters for oversampling techniques.

Name of Parameter	Description	Choice
Sampling strategy	Resample which one of the classes	All
Random state	Set a fixed state to reproduce the same distribution of the data	None
K Neighbors	Corresponding to the number of neighbors to use for synthesizing new data	10
Number of Jobs	Corresponding to the number of cores of Central Processing Unit (CPU) for use	None
Number of features	Corresponding to the number of input features	Based on the number of features existent in each dataset

Table 5

Datasets description in detail.

No.	Name	Attribute	All Samples	Imbalanced Ratio
1	Wisconsin	9	683	1.86
2	Pima	8	768	1.87
3	iris0	4	150	2.00
4	glass0	9	214	2.06
5	glass1	9	214	1.82
6	glass6	9	214	6.38
7	yeast1	8	1484	2.46
8	Haberman	3	306	2.78
9	vehicle1	18	846	2.90
10	vehicle2	18	846	2.88
11	vehicle3	18	846	2.99
12	ecoli1	7	336	3.36
13	ecoli2	7	336	5.46
14	ecoli3	7	336	8.6
15	new-thyroid1	5	215	5.14
16	new-thyroid2	5	215	5.14
17	segment0	19	2308	6.02
18	yeast3	8	1484	8.10
19	page-blocks0	10	5472	8.79
20	yeast-2_vs_4	8	514	9.08
21	penbased	10	10,992	9.41
22	Nursery	5	12,690	2.2
23	breast cancer	117	102,294	163.2
24	Z-Alizadeh Sani	55	303	2.48

Table 12

The average performance score of the evaluation metrics of the executed models.

Models	Acc	Pre	Sen	F1	G-Mean	Spe	AUC	Kap
TL + NORM. + CNN	98.92	98.93	98.92	98.92	98.92	98.90	99.07	98.87
TL + NORM. + DNN	98.95	98.90	98.90	98.90	98.90	98.87	98.98	98.79
OSS + NORM. + CNN	98.78	98.79	98.78	98.78	98.78	98.72	98.93	98.73
OSS + NORM. + DNN	98.82	98.83	98.82	98.82	98.82	98.80	98.94	98.72
NearMiss + NORM. + CNN	98.98	98.98	98.98	98.98	98.98	98.92	99.07	98.89
NearMiss + NORM. + DNN	99.01	99.02	99.01	99.01	99.01	98.95	98.99	98.89
RUS + NORM. + CNN	98.33	98.34	98.33	98.33	98.33	98.23	98.75	98.26
RUS + NORM. + DNN	98.22	98.23	98.22	98.22	98.22	98.15	98.59	98.14
SMOTE + NORM. + CNN	99.08	99.09	99.08	99.09	99.08	99.03	99.08	98.92
SMOTE + NORM. + DNN	98.96	99.00	98.97	98.98	98.98	98.99	99.02	98.78

Boldface specifies that SMOTE + NORM. + CNN is the most robust model.

Table 13

The comparison of performance metrics on the same dataset.

Study	Method	Rec (%)	G-Nean (%)	F1 (%)
Mayabadi and Saadatfar, [29]	DB_HS + SVM	95.80	88.30	81.40
	DB_HS + RF	98.10	92.00	83.80
	DB_US + SVM	92.70	88.50	81.50
	DB_US + RF	95.60	93.80	87.90
Current Study	SMOTE + NORM + CNN	99.00	99.00	98.98

Table 14

The comparison of the metrics results between the proposed study and related studies on the Z-Alizadeh Sani dataset.

Study	Method	Acc (%)	Rec (%)	F1 (%)	Pre (%)	Spe (%)	AUC (%)
Alizadehsani et al. 2012 [72]	Sequential Minimal Optimization	92.09	97.22	NC	NC	79.31	NC
Alizadehsani et al. 2012 [73]	Ensemble of Naïve Bayes and Sequential Minimal Optimization	88.52	91.12	NC	NC	82.05	NC
Alizadehsani et al. 2013 [74]	Information gain + Sequential Minimal Optimization	94.08	96.30	NC	NC	88.51	NC
Babič et al., 2017 [75]	Suppoort vector machine	86.67	NC	NC	NC	NC	NC
Arabasadi et al., 2017 [76]	Neural network + Genetic algorithm	93.85	97.00	NC	NC	92.00	NC
Li et al., 2018 [77]	Naïve Bayes + Genetic algorithm	88.16	88.00	NC	NC	87.78	NC
Abdar et al., 2019 [78]	nested ensemble nu-Support Vector Classification + genetic search algorithm + multi-step data balancing	94.66	94.70	94.70	94.70	NC	96.60
Abdar et al., 2019 [79]	N2Genetic optimizer-nuSupport Vector Machine	93.08	NC	91.51	NC	NC	NC
Khan et al., 2019 [80]	Neural network + Gini Index for feature selection + Backward Weight Optimization	88.49	NC	NC	NC	NC	NC
Kolukısa et al., 2019 [81]	Ensemble Classifier with Fisher Linear Discriminant Analysis	92.07	94.00	94.40	NC	87.40	95.30
Nasarian et al., 2020 [82]	Heterogeneous hybrid feature selection algorithm + SMOTE + Extreme gradient boosting	92.58	92.99	90.62	92.59	NC	NC
Shahid and Singh, 2020 [83]	Hybrid Particle Swarm Optimization-emotional neural networks coupled with feature selection	88.34	91.85	92.12	92.37	78.98	NC
Ghiasi et al., 2020 [84]	Classification and Regression tree	92.41	98.61	NC	NC	77.01	NC
Joloudari et al., 2020 [85]	Random trees	91.47	NC	NC	NC	NC	96.70
Zomorodi-moghadam et al., 2021 [86]	Hybrid Particle Swarm Optimization	84.25	NC	NC	NC	NC	NC
Ashish et al., 2021 [87]	Support Vector Machine -Extreme gradient boosting + Random forest	93.86	NC	91.86	93.86	NC	NC
Zhang et al., 2022 [71]	Extreme gradient boosting + Feature construction + SMOTE	94.70	96.10	94.60	93.40	93.20	98.00
Gupta et al., 2022 [88]	Fixed analysis of mixed data + Binary Bat Algorithm + Ensemble of Random Forest and Extra Trees	97.37	98.15	98.15	NC	95.45	96.80
The proposed study	Mixed SMOTE-NORM.-CNN	98.57	98.58	98.57	98.58	98.42	99.14

NC: Not considered.

References

1. Li, Y.; Zhang, J.; Zhang, S.; Xiao, W.; Zhang, Z. Multi-objective optimization-based adaptive class-specific cost extreme learning machine for imbalanced classification. Neurocomputing; 2022; 496, pp. 107-120. [DOI: https://dx.doi.org/10.1016/j.neucom.2022.05.008]

2. He, H.; Garcia, E.A. Learning from imbalanced data. IEEE Trans. Knowl. Data Eng.; 2009; 21, pp. 1263-1284.

3. Weiss, G.M. Mining with rarity: A unifying framework. ACM Sigkdd Explor. Newsl.; 2004; 6, pp. 7-19. [DOI: https://dx.doi.org/10.1145/1007730.1007734]

4. Batista, G.E.A.P.A.; Prati, R.C.; Monard, M.C. A study of the behavior of several methods for balancing machine learning training data. ACM SIGKDD Explor. Newsl.; 2004; 6, pp. 20-29. [DOI: https://dx.doi.org/10.1145/1007730.1007735]

5. Mani, I.; Zhang, I. kNN approach to unbalanced data distributions: A case study involving information extraction. Proceedings of the Workshop on Learning from Imbalanced Datasets (ICML 2003); Washington, DC, USA, 21 August 2003; pp. 1-7.

6. Liu, W.; Chawla, S. Class confidence weighted knn algorithms for imbalanced data sets. Proceedings of the Pacific-Asia Conference on Knowledge Discovery and Data Mining; Shenzhen, China, 24–27 May 2011; Springer: Berlin/Heidelberg, Germany, 2011; pp. 345-356.

7. Chawla, N.V.; Lazarevic, A.; Hall, L.O.; Bowyer, K.W. SMOTEBoost: Improving prediction of the minority class in boosting. Proceedings of the European Conference on Principles of Data Mining and Knowledge Discovery; Cavtat-Dubrovnik, Croatia, 22–26 September 2003; Springer: Berlin/Heidelberg, Germany, 2003; pp. 107-119.

8. Seiffert, C.; Khoshgoftaar, T.M.; Van Hulse, J.; Napolitano, A. RUSBoost: A Hybrid Approach to Alleviating Class Imbalance. IEEE Trans. Syst. Man Cybern. Part A Syst. Hum.; 2009; 40, pp. 185-197. [DOI: https://dx.doi.org/10.1109/TSMCA.2009.2029559]

9. Provost, F. Machine learning from imbalanced data sets 101. Proceedings of the AAAI’2000 Workshop on Imbalanced Data Sets; Austin, TX, USA, 31 July 2000; AAAI Press: Palo Alto, CA, USA, 2000; pp. 1-3.

10. Sun, Y.; Kamel, M.S.; Wong, A.K.; Wang, Y. Cost-sensitive boosting for classification of imbalanced data. Pattern Recognit.; 2007; 40, pp. 3358-3378. [DOI: https://dx.doi.org/10.1016/j.patcog.2007.04.009]

11. Liu, X.-Y.; Wu, J.; Zhou, Z.-H. Exploratory Undersampling for Class-Imbalance Learning. IEEE Trans. Syst. Man Cybern. Part B; 2008; 39, pp. 539-550. [DOI: https://dx.doi.org/10.1109/tsmcb.2008.2007853]

12. Barandela, R.; Sánchez, J.S.; Garcıa, V. Rangel, Strategies for learning in class imbalance problems. Pattern Recognit.; 2003; 36, pp. 849-851. [DOI: https://dx.doi.org/10.1016/S0031-3203(02)00257-1]

13. Tahir, M.A.; Kittler, J.; Yan, F. Inverse random under sampling for class imbalance problem and its application to multi-label classification. Pattern Recognit.; 2012; 45, pp. 3738-3750. [DOI: https://dx.doi.org/10.1016/j.patcog.2012.03.014]

14. Chawla, N.V.; Bowyer, K.W.; Hall, L.O.; Kegelmeyer, W.P. SMOTE: Synthetic Minority Over-sampling Technique. J. Artif. Intell. Res.; 2002; 16, pp. 321-357. [DOI: https://dx.doi.org/10.1613/jair.953]

15. García, S.; Herrera, F. Evolutionary Undersampling for Classification with Imbalanced Datasets: Proposals and Taxonomy. Evol. Comput.; 2009; 17, pp. 275-306. [DOI: https://dx.doi.org/10.1162/evco.2009.17.3.275] [PubMed: https://www.ncbi.nlm.nih.gov/pubmed/19708770]

16. Weiss, G.M.; McCarthy, K.; Zabar, B. Cost-sensitive learning vs. sampling: Which is best for handling unbalanced classes with unequal error costs?. Dmin; 2007; 7, 24.

17. Zhou, Z.-H.; Liu, X.-Y. Training cost-sensitive neural networks with methods addressing the class imbalance problem. IEEE Trans. Knowl. Data Eng.; 2005; 18, pp. 63-77. [DOI: https://dx.doi.org/10.1109/TKDE.2006.17]

18. Seiffert, C.; Khoshgoftaar, T.M.; Van Hulse, J.; Napolitano, A. A Comparative Study of Data Sampling and Cost Sensitive Learning. Proceedings of the IEEE International Conference on Data Mining Workshops; Pisa, Italy, 5–19 December 2008; pp. 46-52. [DOI: https://dx.doi.org/10.1109/icdmw.2008.119]

19. Breiman, L. Bagging predictors. Mach. Learn.; 1996; 24, pp. 123-140. [DOI: https://dx.doi.org/10.1007/BF00058655]

20. Freund, Y.; Schapire, R.E. Experiments with a new boosting algorithm. Icml; 1996; 96, pp. 148-156.

21. Guo, H.; Viktor, H.L. Learning from imbalanced data sets with boosting and data generation: The databoost-im approach. ACM Sigkdd Explor. Newsl.; 2004; 6, pp. 30-39. [DOI: https://dx.doi.org/10.1145/1007730.1007736]

22. Hido, S.; Kashima, H.; Takahashi, Y. Roughly balanced bagging for imbalanced data. Stat. Anal. Data Min. ASA Data Sci. J.; 2009; 2, pp. 412-426. [DOI: https://dx.doi.org/10.1002/sam.10061]

23. Durahim, A.O. Comparison of sampling techniques for imbalanced learning. Yönet. Bilişim Sist. Derg.; 2016; 2, pp. 181-191.

24. Tomek, I. Two Modifications of CNN. IEEE Trans. Syst. Man Cybern.; 1976; SMC-6, pp. 769-772. [DOI: https://dx.doi.org/10.1109/tsmc.1976.4309452]

25. Kubat, M.; Matwin, S. Addressing the curse of imbalanced training sets: One-sided selection. Icml; 1997; 97, 179.

26. Azadbakht, M.; Fraser, C.S.; Khoshelham, K. Synergy of sampling techniques and ensemble classifiers for classification of urban environments using full-waveform LiDAR data. Int. J. Appl. Earth Obs. Geoinf.; 2018; 73, pp. 277-291. [DOI: https://dx.doi.org/10.1016/j.jag.2018.06.009]

27. Czarnowski, I. Weighted Ensemble with one-class Classification and Over-sampling and Instance selection (WECOI): An approach for learning from imbalanced data streams. J. Comput. Sci.; 2022; 61, 101614. [DOI: https://dx.doi.org/10.1016/j.jocs.2022.101614]

28. Chen, Q.; Zhang, Z.-L.; Huang, W.-P.; Wu, J.; Luo, X.-G. PF-SMOTE: A novel parameter-free SMOTE for imbalanced datasets. Neurocomputing; 2022; 498, pp. 75-88. [DOI: https://dx.doi.org/10.1016/j.neucom.2022.05.017]

29. Mayabadi, S.; Saadatfar, H. Two density-based sampling approaches for imbalanced and overlapping data. Knowl.-Based Syst.; 2022; 241, 108217. [DOI: https://dx.doi.org/10.1016/j.knosys.2022.108217]

30. Buda, M.; Maki, A.; Mazurowski, M.A. A systematic study of the class imbalance problem in convolutional neural networks. Neural Netw.; 2018; 106, pp. 249-259. [DOI: https://dx.doi.org/10.1016/j.neunet.2018.07.011]

31. Li, K.; Zhou, G.; Zhai, J.; Li, F.; Shao, M. Improved PSO_AdaBoost Ensemble Algorithm for Imbalanced Data. Sensors; 2019; 19, 1476. [DOI: https://dx.doi.org/10.3390/s19061476]

32. Vuttipittayamongkol, P.; Elyan, E.; Petrovski, A. On the class overlap problem in imbalanced data classification. Knowl.-Based Syst.; 2020; 212, 106631. [DOI: https://dx.doi.org/10.1016/j.knosys.2020.106631]

33. Aridas, C.K.; Karlos, S.; Kanas, V.G.; Fazakis, N.; Kotsiantis, S.B. Uncertainty Based Under-Sampling for Learning Naive Bayes Classifiers Under Imbalanced Data Sets. IEEE Access; 2019; 8, pp. 2122-2133. [DOI: https://dx.doi.org/10.1109/ACCESS.2019.2961784]

34. Dablain, D.; Krawczyk, B.; Chawla, N.V. DeepSMOTE: Fusing Deep Learning and SMOTE for Imbalanced Data. IEEE Trans. Neural Networks Learn. Syst.; 2022; pp. 1-15. [DOI: https://dx.doi.org/10.1109/TNNLS.2021.3136503]

35. Bagui, S.; Li, K. Resampling imbalanced data for network intrusion detection datasets. J. Big Data; 2021; 8, 6. [DOI: https://dx.doi.org/10.1186/s40537-020-00390-x]

36. Choi, H.-S.; Jung, D.; Kim, S.; Yoon, S. Imbalanced Data Classification via Cooperative Interaction Between Classifier and Generator. IEEE Trans. Neural Networks Learn. Syst.; 2021; 33, pp. 3343-3356. [DOI: https://dx.doi.org/10.1109/TNNLS.2021.3052243] [PubMed: https://www.ncbi.nlm.nih.gov/pubmed/33531305]

37. Xie, X.; Liu, H.; Zeng, S.; Lin, L.; Li, W. A novel progressively undersampling method based on the density peaks sequence for imbalanced data. Knowl.-Based Syst.; 2020; 213, 106689. [DOI: https://dx.doi.org/10.1016/j.knosys.2020.106689]

38. Zheng, M.; Li, T.; Sun, L.; Wang, T.; Jie, B.; Yang, W.; Tang, M.; Lv, C. An automatic sampling ratio detection method based on genetic algorithm for imbalanced data classification. Knowl.-Based Syst.; 2021; 216, 106800. [DOI: https://dx.doi.org/10.1016/j.knosys.2021.106800]

39. Elyan, E.; Moreno-Garcia, C.F.; Jayne, C. CDSMOTE: Class decomposition and synthetic minority class oversampling technique for imbalanced-data classification. Neural Comput. Appl.; 2020; 33, pp. 2839-2851. [DOI: https://dx.doi.org/10.1007/s00521-020-05130-z]

40. Asniar,; Maulidevi, N.U.; Surendro, K. SMOTE-LOF for noise identification in imbalanced data classification. J. King Saud Univ. Comput. Inf. Sci.; 2021; 34, pp. 3413-3423. [DOI: https://dx.doi.org/10.1016/j.jksuci.2021.01.014]

41. Abdoli, M.; Akbari, M.; Shahrabi, J. Bagging Supervised Autoencoder Classifier for credit scoring. Expert Syst. Appl.; 2023; 213, 118991. [DOI: https://dx.doi.org/10.1016/j.eswa.2022.118991]

42. El Bakrawy, L.M.; Cifci, M.A.; Kausar, S.; Hussain, S.; Islam, A.; Alatas, B.; Desuky, A.S. A Modified Ant Lion Optimization Method and Its Application for Instance Reduction Problem in Balanced and Imbalanced Data. Axioms; 2022; 11, 95. [DOI: https://dx.doi.org/10.3390/axioms11030095]

43. Yang, M.; Wang, Z.; Li, Y.; Zhou, Y.; Li, D.; Du, W. Gravitation balanced multiple kernel learning for imbalanced classification. Neural Comput. Appl.; 2022; 34, pp. 13807-13823. [DOI: https://dx.doi.org/10.1007/s00521-022-07187-4]

44. Tanimoto, A.; Yamada, S.; Takenouchi, T.; Sugiyama, M.; Kashima, H. Improving imbalanced classification using near-miss instances. Expert Syst. Appl.; 2022; 201, 117130. [DOI: https://dx.doi.org/10.1016/j.eswa.2022.117130]

45. Thejas, G.S.; Hariprasad, Y.; Iyengar, S.; Sunitha, N.; Badrinath, P.; Chennupati, S. An extension of Synthetic Minority Oversampling Technique based on Kalman filter for imbalanced datasets. Mach. Learn. Appl.; 2022; 8, 100267. [DOI: https://dx.doi.org/10.1016/j.mlwa.2022.100267]

46. Wei, G.; Mu, W.; Song, Y.; Dou, J. An improved and random synthetic minority oversampling technique for imbalanced data. Knowl.-Based Syst.; 2022; 248, 108839. [DOI: https://dx.doi.org/10.1016/j.knosys.2022.108839]

47. Gao, Y.; Gao, L.; Li, X.; Cao, S. A Hierarchical Training-Convolutional Neural Network for Imbalanced Fault Diagnosis in Complex Equipment. IEEE Trans. Ind. Inform.; 2022; 18, pp. 8138-8145. [DOI: https://dx.doi.org/10.1109/TII.2022.3177662]

48. Mohammed, R.; Rawashdeh, J.; Abdullah, M. Machine Learning with Oversampling and Undersampling Techniques: Overview Study and Experimental Results. Proceedings of the 11th International Conference on Information and Communication Systems (ICICS); Irbid, Jordan, 7–9 April 2020; pp. 243-248. [DOI: https://dx.doi.org/10.1109/icics49469.2020.239556]

49. Li, W.; Chen, J.; Cao, J.; Ma, C.; Wang, J.; Cui, X.; Chen, P. EID-GAN: Generative Adversarial Nets for Extremely Imbalanced Data Augmentation. IEEE Trans. Ind. Inform.; 2022; 19, pp. 3208-3218. [DOI: https://dx.doi.org/10.1109/TII.2022.3182781]

50. Zieba, M.; Tomczak, J.M. Boosted SVM with active learning strategy for imbalanced data. Soft Comput.; 2014; 19, pp. 3357-3368. [DOI: https://dx.doi.org/10.1007/s00500-014-1407-5]

51. He, H.; Zhang, W.; Zhang, S. A novel ensemble method for credit scoring: Adaption of different imbalance ratios. Expert Syst. Appl.; 2018; 98, pp. 105-117. [DOI: https://dx.doi.org/10.1016/j.eswa.2018.01.012]

52. Li, D.-C.; Wang, S.-Y.; Huang, K.-C.; Tsai, T.-I. Learning class-imbalanced data with region-impurity synthetic minority oversampling technique. Inf. Sci.; 2022; 607, pp. 1391-1407. [DOI: https://dx.doi.org/10.1016/j.ins.2022.06.067]

53. Fernandez, A.; Garcia, S.; Herrera, F.; Chawla, N.V. SMOTE for Learning from Imbalanced Data: Progress and Challenges, Marking the 15-year Anniversary. J. Artif. Intell. Res.; 2018; 61, pp. 863-905. [DOI: https://dx.doi.org/10.1613/jair.1.11192]

54. Pereira, R.M.; Costa, Y.M.; Silla, C.N., Jr. MLTL: A multi-label approach for the Tomek Link undersampling algorithm. Neurocomputing; 2019; 383, pp. 95-105. [DOI: https://dx.doi.org/10.1016/j.neucom.2019.11.076]

55. Hernandez, J.; Carrasco-Ochoa, J.A.; Martínez-Trinidad, J.F. An Empirical Study of Oversampling and Undersampling for Instance Selection Methods on Imbalance Datasets. Progress in Pattern Recognition, Image Analysis, Computer Vision, and Applications, Proceedings of the 18th Iberoamerican Congress, CIARP 2013, Havana, Cuba, 20–23 November 2013; Springer: Berlin/Heidelberg, Germany, 2013; pp. 262-269. [DOI: https://dx.doi.org/10.1007/978-3-642-41822-8_33]

56. Kamei, Y.; Monden, A.; Matsumoto, S.; Kakimoto, T.; Matsumoto, K.-I. The effects of over and under sampling on fault-prone module detection. Proceedings of the First International Symposium on Empirical Software Engineering and Measurement (ESEM 2007); Madrid, Spain, 20–21 September 2007; pp. 196-204.

57. More, A. Survey of resampling techniques for improving classification performance in unbalanced datasets. arXiv; 2016; arXiv: 1608.06048

58. Liu, W.; Wang, Z.; Liu, X.; Zeng, N.; Liu, Y.; Alsaadi, F.E. A survey of deep neural network architectures and their applications. Neurocomputing; 2017; 234, pp. 11-26. [DOI: https://dx.doi.org/10.1016/j.neucom.2016.12.038]

59. Caterini, A.L.; Chang, D.E. Deep Neural Networks in a Mathematical Framework; Springer International Publishing: Berlin/Heidelberg, Germany, 2018; [DOI: https://dx.doi.org/10.1007/978-3-319-75304-1]

60. Pal, S.K.; Mitra, S. Multilayer Perceptron, Fuzzy Sets, Classification. IEEE Trans. Neural Netw.; 1992; 3, pp. 683-697. [DOI: https://dx.doi.org/10.1109/72.159058] [PubMed: https://www.ncbi.nlm.nih.gov/pubmed/18276468]

61. Guo, Y.; Du, G.-Q.; Shen, W.-Q.; Du, C.; He, P.-N.; Siuly, S. Automatic myocardial infarction detection in contrast echocardiography based on polar residual network. Comput. Methods Programs Biomed.; 2020; 198, 105791. [DOI: https://dx.doi.org/10.1016/j.cmpb.2020.105791]

62. Albawi, S.; Mohammed, T.A.; Al-Zawi, S. Understanding of a convolutional neural network. Proceedings of the International Conference on Engineering and Technology (ICET); Antalya, Turkey, 21–23 August 2017; pp. 1-6.

63. O’Shea, K.; Nash, R. An introduction to convolutional neural networks. arXiv; 2015; arXiv: 1511.08458

64. Lin, T.-Y.; Goyal, P.; Girshick, R.; He, K.; Dollár, P. Focal loss for dense object detection. Proceedings of the IEEE International Conference on Computer Vision; Venice, Italy, 22–29 October 2017; pp. 2980-2988.

65. Mulyanto, M.; Faisal, M.; Prakosa, S.W.; Leu, J.-S. Effectiveness of Focal Loss for Minority Classification in Network Intrusion Detection Systems. Symmetry; 2020; 13, 4. [DOI: https://dx.doi.org/10.3390/sym13010004]

66. Alcalá-Fdez, J.; Fernández, A.; Luengo, J.; Derrac, J.; García, S.; Sánchez, L.; Herrera, F. Keel data-mining software tool: Data set repository, integration of algorithms and experimental analysis framework. J. Mult. Valued Log. Soft Comput.; 2011; 17, pp. 1-36.

67. Joloudari, J.H.; Azizi, F.; Nematollahi, M.A.; Alizadehsani, R.; Hassannatajjeloudari, E.; Nodehi, I.; Mosavi, A. GSVMA: A Genetic Support Vector Machine ANOVA Method for CAD Diagnosis. Front. Cardiovasc. Med.; 2022; 8, 2178. [DOI: https://dx.doi.org/10.3389/fcvm.2021.760178]

68. Li, J.; Fong, S.; Zhuang, Y. Optimizing SMOTE by metaheuristics with neural network and decision tree. Proceedings of the 3rd International Symposium on Computational and Business Intelligence (ISCBI); Bali, Indonesia, 7–9 December 2015; pp. 26-32.

69. Chowdary, M.K.; Nguyen, T.N.; Hemanth, D.J. Deep learning-based facial emotion recognition for human–computer interaction applications. Neural Comput. Appl.; 2021; pp. 1-18. [DOI: https://dx.doi.org/10.1007/s00521-021-06012-8]

70. Narkhede, S. Understanding auc roc curve. Towards Data Sci.; 2018; 26, pp. 220-227.

71. Zhang, S.; Yuan, Y.; Yao, Z.; Wang, X.; Lei, Z. Improvement of the Performance of Models for Predicting Coronary Artery Disease Based on XGBoost Algorithm and Feature Processing Technology. Electronics; 2022; 11, 315. [DOI: https://dx.doi.org/10.3390/electronics11030315]

72. Alizadehsani, R.; Hosseini, M.J.; Sani, Z.A.; Ghandeharioun, A.; Boghrati, R. Diagnosis of coronary artery disease using cost-sensitive algorithms. Proceedings of the 12th International Conference on Data Mining Workshops; Brussels, Belgium, 10 December 2012; pp. 9-16.

73. Alizadehsani, R.; Habibi, J.; Hosseini, M.J.; Boghrati, R.; Ghandeharioun, A.; Bahadorian, B.; Sani, Z.A. Diagnosis of coronary artery disease using data mining techniques based on symptoms and ecg features. Eur. J. Sci. Res.; 2012; 82, pp. 542-553.

74. Alizadehsani, R.; Habibi, J.; Hosseini, M.J.; Mashayekhi, H.; Boghrati, R.; Ghandeharioun, A.; Bahadorian, B.; Sani, Z.A. A data mining approach for diagnosis of coronary artery disease. Comput. Methods Programs Biomed.; 2013; 111, pp. 52-61. [DOI: https://dx.doi.org/10.1016/j.cmpb.2013.03.004]

75. Babič, F.; Olejár, J.; Vantová, Z.; Paralič, J. Predictive and descriptive analysis for heart disease diagnosis. Proceedings of the 2017 Federated Conference on Computer Science and Information Systems (FedCSIS); Prague, Czech Republic, 3–6 September 2017; pp. 155-163.

76. Arabasadi, Z.; Alizadehsani, R.; Roshanzamir, M.; Moosaei, H.; Yarifard, A.A. Computer aided decision making for heart disease detection using hybrid neural network-Genetic algorithm. Comput. Methods Programs Biomed.; 2017; 141, pp. 19-26. [DOI: https://dx.doi.org/10.1016/j.cmpb.2017.01.004]

77. Li, H.; Wang, X.; Li, Y.; Qin, C.; Liu, C. Comparison between medical knowledge based and computer automated feature selection for detection of coronary artery disease using imbalanced data. Proceedings of the BIBE 2018, International Conference on Biological Information and Biomedical Engineering; Shanghai, China, 6–8 July 2018; pp. 1-4.

78. Abdar, M.; Acharya, U.R.; Sarrafzadegan, N.; Makarenkov, V. NE-nu-SVC: A New Nested Ensemble Clinical Decision Support System for Effective Diagnosis of Coronary Artery Disease. IEEE Access; 2019; 7, pp. 167605-167620. [DOI: https://dx.doi.org/10.1109/ACCESS.2019.2953920]

79. Abdar, M.; Książek, W.; Acharya, U.R.; Tan, R.-S.; Makarenkov, V.; Pławiak, P. A new machine learning technique for an accurate diagnosis of coronary artery disease. Comput. Methods Programs Biomed.; 2019; 179, 104992. [DOI: https://dx.doi.org/10.1016/j.cmpb.2019.104992]

80. Khan, Y.; Qamar, U.; Asad, M.; Zeb, B. Applying Feature Selection and Weight Optimization Techniques to Enhance Artificial Neural Network for Heart Disease Diagnosis. Intelligent Systems and Applications, Proceedings of the 2019 Intelligent Systems Conference (IntelliSys), London, UK, 5–6 September 2019; Springer: Berlin/Heidelberg, Germany, 2019; pp. 340-351. [DOI: https://dx.doi.org/10.1007/978-3-030-29516-5_26]

81. Kolukısa, B.; Hacılar, H.; Kuş, M.; Bakır-Güngör, B.; Aral, A.; Güngör, V.Ç. Diagnosis of coronary heart disease via classification algorithms and a new feature selection methodology. Int. J. Data Min. Sci.; 2019; 1, pp. 8-15.

82. Nasarian, E.; Abdar, M.; Fahami, M.A.; Alizadehsani, R.; Hussain, S.; Basiri, M.E.; Zomorodi-Moghadam, M.; Zhou, X.; Pławiak, P.; Acharya, U.R. et al. Association between work-related features and coronary artery disease: A heterogeneous hybrid feature selection integrated with balancing approach. Pattern Recognit. Lett.; 2020; 133, pp. 33-40. [DOI: https://dx.doi.org/10.1016/j.patrec.2020.02.010]

83. Shahid, A.H.; Singh, M. A Novel Approach for Coronary Artery Disease Diagnosis using Hybrid Particle Swarm Optimization based Emotional Neural Network. Biocybern. Biomed. Eng.; 2020; 40, pp. 1568-1585. [DOI: https://dx.doi.org/10.1016/j.bbe.2020.09.005]

84. Ghiasi, M.M.; Zendehboudi, S.; Mohsenipour, A.A. Decision tree-based diagnosis of coronary artery disease: CART model. Comput. Methods Programs Biomed.; 2020; 192, 105400. [DOI: https://dx.doi.org/10.1016/j.cmpb.2020.105400]

85. Joloudari, J.H.; Joloudari, E.H.; Saadatfar, H.; Ghasemigol, M.; Razavi, S.M.; Mosavi, A.; Nabipour, N.; Shamshirband, S.; Nadai, L. Coronary Artery Disease Diagnosis; Ranking the Significant Features Using a Random Trees Model. Int. J. Environ. Res. Public Health; 2020; 17, 731. [DOI: https://dx.doi.org/10.3390/ijerph17030731]

86. Zomorodi-Moghadam, M.; Abdar, M.; Davarzani, Z.; Zhou, X.; Pławiak, P.; Acharya, U. Hybrid particle swarm optimization for rule discovery in the diagnosis of coronary artery disease. Expert Syst.; 2019; 38, e12485. [DOI: https://dx.doi.org/10.1111/exsy.12485]

87. Ashish, L.; Kumar, S.; Yeligeti, S. Ischemic heart disease detection using support vector Machine and extreme gradient boosting method. Mater. Today Proc.; 2021; [DOI: https://dx.doi.org/10.1016/j.matpr.2021.01.715]

88. Gupta, A.; Kumar, R.; Arora, H.S.; Raman, B. C-CADZ: Computational intelligence system for coronary artery disease detection using Z-Alizadeh Sani dataset. Appl. Intell.; 2021; 52, pp. 2436-2464. [DOI: https://dx.doi.org/10.1007/s10489-021-02467-3]