Simulation-Based Optimization on the System-of-Systems Model via Model Transformation and Genetic Algorithm: A Case Study of Network-Centric Warfare

Abstract

Simulation of a system-of-systems (SoS) model, which consists of a combat model and a network model, has been used to analyze the performance of network-centric warfare in detail. However, finding the combat model parameters satisfying the required combat power using simulation can take a long time for two reasons: (1) the prolonged execution time per simulation run and (2) the enormous number of simulation runs. This paper proposes a simulation-based optimization method for the SoS-based simulation model to overcome these problems. The method consists of two processes: (1) the transformation of the SoS-based model into an integrated model using the neural network to reduce the execution time and (2) the optimization of the integrated model using the genetic algorithm with ranking and selection to decrease the number of simulation runs. The experimental result reveals that the proposed method significantly reduced the time for finding the optimal combat parameters with an acceptable level of accuracy.

This is an open access article distributed under the Creative Commons Attribution License, which permits unrestricted use, distribution, and reproduction in any medium, provided the original work is properly cited.

1. Introduction

In modern warfare, communication, responsible for the flow of information between combat entities, is a primary factor in attaining victory. That is, without proper communication, the combat commander cannot obtain detection information and forward appropriate attack orders to combat entities. Such a battlefield environment based on communication systems is called network-centric warfare (NCW) [1, 2]. Many researchers in the defense modeling and simulation (DM & S) domain have recognized the importance of NCW and have tried to analyze combat systems using war game simulators reflecting the effects of communication. In earlier studies, researchers depicted communication effects by using a simplified model or by assuming perfect communication in a combat model from the perspective of a standalone system [3, 4]. However, as the importance of communication and the need for a detailed depiction of the network system increase, recent studies have separated the network model from the combat model [5, 6]. Namely, they have implemented each model with dedicated modeling development tools and have interconnected the model through an interoperation standard, such as the high-level architecture (HLA) or the test and training enabling architecture from the perspective of the system of systems (SoS) [7–10].

In the SoS, the system means a set of interrelated components working together toward some common objective or purpose, and the SoS means a set of systems for a task that none of the systems can accomplish on its own [11, 12]. This SoS-based approach has a merit in that a collection of dedicated systems offers more functionality and performance than simply the sum of the constituent systems and the conduct of various analyses by using existing models developed using the specialized tools in each domain [13–15]. For example, the combat model, which was only able to analyze the combat power, can measure the combat power under the communication environment by combining with a network model, which only can describe the communication environment.

The SoS-based NCW simulation offers the advantage of enabling the conduct of the “what-if” analysis of the combat power against various combat scenarios with related parameters in the combat model, by reflecting detailed communication effects from the network model [16, 17]. However, the high complexity of the network model and the interface connecting the two models paradoxically increase the execution time per simulation trial [18–20]. Porche et al. raised the simulation overhead of the highly complex network model and then analyzed the combat system with the metamodel of the network model [21]. Miner et al. also brought up the prolonged execution time due to the network model when developing the SoS analysis toolset, and they constructed its metamodel [19]. Although some studies have conducted the abstraction of the SoS-based NCW simulation model, they have not entirely reflected the SoS property due to using data from the simulation of the standalone network model, not the SoS.

Furthermore, because the NCW simulation model is usually represented as a discrete-event simulation model with a stochastic property, a large number of replicated simulations are required to obtain a steady state of the combat power [22]. As a result, the “what-if” analysis based on the SoS-based NCW simulation becomes a time-consuming problem due to two challenges: (1) enormous total simulation runs due to a large number of experimental points and replicated simulations per point and (2) the long execution time per simulation run. When a decision-maker wants to find the parameters of the combat model satisfying the requirement operation capability (ROC) of the combat power using the SoS-based NCW simulation, a significant computational cost is required.

One way to reduce the enormous simulation runs is to apply the optimization method for the discrete-event stochastic simulation model [17, 23, 24]. However, this process also requires the considerable time because it takes a long time per run of the SoS-based simulation model. On the other hand, another approach to dealing with the long execution time per run is to transform the SoS-based simulation model into an integrated simulation model via the abstraction of the model [25–27]. Although this can reduce the time required to conduct the exploratory analysis, for example, a full search on the design space, the “what-if” analysis is also time-consuming because the integrated simulation model requires somewhat execution time [28, 29]. That is, independently using each approach to the SoS-based NCW simulation model has a limitation when it comes to solving the two drawbacks simultaneously.

This paper applies a simulation-based optimization method to the SoS-based NCW simulation model in the form of a hybrid approach to overcome the abovementioned challenges. It consists of two processes: (1) the transformation of the SoS-based NCW simulation model into the integrated simulation model via the model abstraction using the neural network [30, 31] and (2) the simulation-based optimization using the genetic algorithm (GA) with ranking and selection (R & S) [32, 33]. The former plays a role in reducing the execution time for the NCW simulation with an acceptable error; the latter plays a role in reducing the number of simulation runs required for finding the parameters of the combat model satisfying the given combat power in the entire design space. By integrating the merits of the two approaches, this paper finally facilitates simulation-based optimization in the SoS environment.

In our case study, we measured the accuracy and efficiency of the proposed optimization method for the SoS-based NCW simulation, which has already been validated. Consequently, the experimental results show that the first process reduced the execution time 113.31 times within the 5.6248% error range and the second process decreased the total number of simulation runs 19.76 times for obtaining the optimal solution with 95% accuracy. Considering the synergy effect, the proposed method ultimately reduced the execution time 2239 times within acceptable accuracy. Finally, our study provides an alternative method for finding the optimized parameters satisfying the given performance requirements in the enormous design space in the SoS.

This study is organized as follows. Section 2 describes the background and related work. Section 3 explains the proposed method. Section 4 discusses the experimental results from the NCW simulation. Finally, Section 5 concludes this study.

2. Background

2.1. System-of-Systems Stochastic Simulation Model for NCW

The NCW system consists of combat entities that are geographically or hierarchically dispersed and that share information through communication [34]. To describe the NCW system as depicted in Figure 1, modelers have developed a combat model for the military operation and a network model for communication effects separately from the perspective of the SoS, and they have enabled the information between two heterogeneous models to be shared through the interoperation middleware.

[figure omitted; refer to PDF]

In this structure, the network model reflects the communication effects on information delivery among entities of the combat model, which acquires the communication effects, for example, the packet delivery ratio (PDR) or end-to-end delay. The former refers to the ratio of the number of successfully delivered packets from the source to the destination node; the latter refers to the average time the packets take to arrive at the destination from the source node [35]. By reflecting these effects, the combat model simulates the combat power.

2.2. Optimization in the Stochastic Simulation Model

The concept of optimization in the stochastic simulation models can simply be depicted in Figure 2. Since the model cannot be expressed in an analytic closed form, conducting the optimization using the inverse model is impossible. Thereby, conducting repeated simulations for solving the optimization problem is inevitable [36]. To be specific, if we find an optimized parameter setting that maximizes the model’s performance among $k$ design alternatives (where design $x_i$ refers to a particular setting of the model parameters), the total $k\times h$ simulation runs are required when considering $h$ replications per design. To overcome the stochastic noise and to find the optimal design accurately, increasing $h$ and obtaining the accurate performance of each design (i.e., ${\overline{\mu }}_i$) are necessary. However, increasing $h$ causes efficiency problems in the optimization due to the increased cost of repeating the simulation as previously mentioned. The greater the number of design alternatives k, the worse this problem becomes.

[figure omitted; refer to PDF]

Many methods for solving the efficiency problem have been proposed. These methods can be classified into two categories according to the number of alternatives [37]. If $k$ is infinite or very large, classical metaheuristic search methods, such as simulated annealing [38], GA [39], and tabu search [40], can be considered efficient approaches. These methods reduce the required number of simulation runs by performing the simulations only for selected alternatives rather than for all alternatives as depicted in Figure 3(a). To accurately find optimal designs using these methods, selecting alternatives for the simulation, including such optimal designs, is important.

[figures omitted; refer to PDF]

On the other hand, if $k$ is finite and relatively small (specifically, all alternatives can be simulated more than five times), the ranking and selection (R & S) methods in statistics, such as OCBA [41], KN [42], and UE [43], can be considered efficient approaches. Unlike classical metaheuristic methods, statistical R & S performs simulations for all alternatives. It reduces the total number of simulation runs by appropriately adjusting $h$ in each design as indicated in Figure 3(b). That is, the aim of R & S is to select the optimal designs accurately by allocating the limited number of simulation runs effectively. Because the R & S performs the simulations for all designs, it is less likely to fall into a local optimization compared with metaheuristic methods that perform the simulations only for the selected alternatives, and it ensures that the discovered optimal design is the global optimum. However, it is clear that the R & S cannot be applied to the case of a large number of alternatives because the simulation should be performed for all alternatives at least more than five times.

3. Proposed Method

As illustrated in Figure 4, the objective of this study is to find the optimized input parameters of the combat model, which satisfy the given ROC corresponding to the combat model’s outputs (i.e., the combat power in the SoS-based simulation model). To find the parameters as efficiently as possible, this paper is aimed at minimizing the simulation cost and time due to a large number of simulation runs while retaining the accuracy. Figure 5 reveals the overall procedure of this proposed method for minimizing the simulation cost for the optimization problem, which consists of two processes: (1) transformation and (2) optimization.

[figure omitted; refer to PDF] [figure omitted; refer to PDF]

The transformation process transforms the stochastic simulation model into the integrated stochastic simulation model (upper part). For this, this paper abstracts the network model to the abstracted model. Owing to the simplified network model and the removal of the interoperation middleware, this process surely shortens the execution time per simulation run.

In the optimization process subsequently, an optimization algorithm specialized in the stochastic simulation model minimizes the number of simulation runs required for finding the optimized input in the entire design space (lower part). As a result of the two processes, the proposed method can reduce the total time for finding the parameters satisfying the required combat power. The following sections explain each process in detail.

3.1. Transformation Process

Figure 6 depicts the transformation process consisting of two phases: (1) data acquisition from the simulation of the SoS-based NCW stochastic model (right upper part) and (2) the abstracted network model construction using the neural network (right lower part). As mentioned in the prior section, the original network model plays a role in calculating the communication effects. Likewise, the abstracted model should be able to calculate the same effects. In addition, because the original network model is the discrete-event model, the abstracted network model should also be a discrete-event model that can participate in the discrete-event simulation.

[figure omitted; refer to PDF]

For the abstraction, data exchanged between the two models in the original SoS simulation should be collected and analyzed. From the C response message, the PDR and end-to-end delay can be approximated. The C request messages are used for calculating the interdeparture time (intDepT) of the messages. This paper uses the interdeparture time for the identification of the original network model [44, 45]. From the raw data, we can construct each data input and output set, {intDepT, PDR} and {intDepT, DELAY}, for training the neural network of two communication effects.

After the data acquisition, the second phase involves constructing the abstracted network model through the model hypothesis in phase 2.1 and the variable estimation in phase 2.2. In the model hypothesis phase, a discrete-event system specification (DEVS) atomic model is constructed and will participate in the discrete-event simulation with the combat model. To estimate the PDR and end-to-end delay in the variable estimation phase, the input and output sets are trained in the nonlinear autoregressive exogenous (NARX) neural network, which is suitable for representing the dynamic and nonlinear properties of the network model [46]. Finally, this paper constructs the integrated stochastic model (left lower part of Figure 6) by substituting the network model with the identified and abstracted network model from the SoS-based NCW stochastic model (left upper part of Figure 6).

Figure 7 represents the abstracted network model (ANM) based on the DEVS formalism. We hypothesize that the abstracted model, which an atomic DEVS model describes, is a probabilistic priority queue, where the DELAY associated with events are considered to be priorities by influencing the P_PDR in generating stochastic events in the mixed time [47]. The formal specification of ANM can be found in Specification 1.

[figure omitted; refer to PDF]

Specification 1

     $ANM=<X,Y,S,{\delta }_{\mathrm{ext}},{\delta }_{\mathrm{int}}^p,\lambda ,ta>$

$X=\left\{Crequest\right\}$;

$Y=\left\{Cresponse\right\}$;

$S=\left\{\left(s,q\right)\mid s\in \left\{WAIT,COMM,SEND\right\},q\in Q\right\}$

   where $Q$ is message queue;

${\delta }_{\mathrm{ext}}:\left(\left(WAIT,q\right),Crequest\right)\to \left(COMM,q^{\prime }\right)$

  $\left(\left(COMM,q\right),Crequest\right)\to \left(COMM,q^{\prime }\right)$

${\delta }_{\mathrm{int}}^p:\left(\left(COMM,q\right),P_{PDR}\right)\to \left(SEND,q\right)$;

  $\left(\left(COMM,q\right),1-P_{PDR}\right)\to \left(WAIT,q^{\prime }\right)$

   where Q is empty;

  $\left(\left(COMM,q\right),1-P_{PDR}\right)\to \left(COMM,q^{\prime }\right)$

   where Q is not empty;

  $\left(\left(SEND,q\right)\right)\to \left(COMM,q^{\prime }\right)$ where Q is not empty;

  $\left(\left(SEND,q\right)\right)\to \left(WAIT,q^{\prime }\right)$ where Q is empty;

$\lambda :\left(SEND,q\right)\to Cresponse$;

$ta:\left(WAIT,q\right)\to \infty$;

  $\left(COMM,q\right)\to minTA$;

  $\left(SEND,q\right)\to 0$;

$V_{ANM}=\left\{P_{PDR},DELAY\right\}$ is $ANM$’s variable;

    $P_{PDR}$ is the packet delivery ratio;

    $DELAY$ is the end-to-end delay;

$minTA$ = the minimum remaining time related $DELAY$ with among all messages in Q.

This model has three states: $\left(WAIT,q\right)$, $\left(COMM,q\right)$, and $\left(SEND,q\right)$. The first and second are the states of $q$ without and with messages received from the combat model; the third state is the state sending the message to the combat model. When receiving a message for reflecting the communication effects in $\left(WAIT,q\right)$ from ${\delta }_{\mathrm{ext}}$, this model calculates $DELAY$ and $P_{PDR}$ values using $calculateDelay$ and $calculatePDR$, which are trained via the NARX model and implemented as the functions. Then, the model stores the message with a scheduled time for output ($Cresponse$) and $P_{PDR}$ by adding the current time and $DELAY$ in $q$. Then, it updates $minTA$ and transits to $\left(COMM,q\right)$. After the $minTA$ time, this model conducts ${\delta }_{\mathrm{int}}^p$ to $\left(SEND,q\right)$ based on $P_{PDR}$ and generates the message to the combat model in $\left(SEND,q\right)$. Through such a model structure, this abstracted model can participate in the discrete-event simulation, which is impossible using only the neural network, and generates a different output according to the status of the adjacent model through the neural network.

3.2. Optimization Process

As mentioned previously, the optimization methods for the stochastic simulation model can be classified into the classical metaheuristic and R & S methods depending on the number of design alternatives $k$. To increase the efficiency of the optimization (i.e., to decrease the total number of simulation runs $T$ denoted as $k\times h$ in Figure 2), the metaheuristic methods reduce the $k$ when $k$ is large. On the other hand, the R & S method improves the efficiency by decreasing $h$ when $k$ is relatively small. Although the applicable case of each method is different, the efficiency of the optimization can be further increased by reducing $k$ and $h$ simultaneously if both methods can be used together. Therefore, in this section, we introduce a GA with R & S to improve the efficiency of the optimization for the SoS-based NCW stochastic simulation model.

The GA is a population-based search algorithm inspired by the process of natural selection [39]. Figure 8(a) represents the procedure of the GA. Each design alternative is encoded as a single chromosome represented as a series of binary bits to apply genetic operations. Since the GA finds the optimal design by repeatedly updating the population through evaluating the population with stochastic simulation and applying the genetic operations, its performance highly depends on the quality of the population. That is, the more the population contains optimal or similar designs, the more the GA can find the solution efficiently with fewer repetitions.

[figure omitted; refer to PDF]

To increase the quality of the population, two parent chromosomes selected stochastically based on the evaluated ranking revealed in Figure 8(a) should be the designs with higher performance than the other designs in the population. Accordingly, the evaluated ranking needs to be more accurate. However, estimating the exact performance of a chromosome via a large number of simulation replications for each chromosome (i.e., increasing $h$) reduces the GA’s efficiency by increasing $T$. As mentioned previously, the aim of R & S is to find optimal designs accurately by allocating limited simulation replications effectively to $k$ design alternatives. Since the size of the population is small enough to apply R & S (i.e., $k$ is small), applying R & S to the evaluation stage of the GA can improve the quality of the population via finding the ranking accurately without increasing $T$ as demonstrated in Figure 8(b). Consequently, the efficiency of GA can be increased [37, 48, 49].

Among various R & S algorithms, we apply the UEmr algorithm which is the most recently developed based on the uncertainty evaluation (UE) framework [43]. Algorithm 1 represents the UEmr algorithm (see [50] for detail). This algorithm focuses on finding good designs of which the performance is within the top-$m$ and identifying their rankings accurately. To maximize the accuracy under the limited number of simulation runs, the algorithm allocates further simulation runs $\mathrm{\Delta }$ sequentially based on the uncertainty. The uncertainty, the criterion for allocating $\mathrm{\Delta }$ for each design, is defined with the $p$ value of the statistical hypothesis test. It indicates the degree to which the observed simulation results of design, such as the sample mean and standard error, can be significant evidence for the correct selection of the design. A small value of the uncertainty close to zero means that the observed results are significant evidence, so the selection of the design can be statistically considered correct. On the other hand, a relatively larger value means that the results cannot be the evidence, so it is uncertain whether the selection of the design is correct. Depending on the meaning of the uncertainty, it is necessary to allocate more simulation runs to designs with a relatively higher value of the uncertainty, as shown in step 5 of Algorithm 1.

Algorithm 1 [50]: Find the top-$m$ designs out of $k$ designs including the performance ranking of the $m$ designs.

Parameters: $T\left(\le {kn}_0\right)$, $n_0$, $\mathrm{\Delta }\left(\ll T\right)$, $m\left(\le k\right)$

  INITIALIZATION:

1:   simulate $n_0$ times for each design $x_i$, $i\in \left\{1,\dots ,k\right\}$

2:   set $T\leftarrow T-{kn}_0$

  LOOP:

3:   while $T\ge 0$do

4:  calculate the uncertainty ${\omega }_i$ for each $x_i$ using the simulation results

5:  allocate $\mathrm{\Delta }$ to all designs according to ${\omega }_i$:

       $a_i/a_j={\omega }_i/{\omega }_j,$

    where $i,j\in \left\{1,\dots ,k\right\}$, $i\ne j$, and $$\mathrm{\Delta }=\sum _{i=1}^k{a}_i$$

6:  simulate $a_i$ times for each $x_i$ additionally

7:  update the simulation results

8:  set $T\leftarrow T-\mathrm{\Delta }$

9:   end while

  FINALIZATION:

10:   return the top-$m$ designs and their performance ranking based on the simulation results

Compared with the other similar R & S algorithms, such as UEm [23], OCBAm+ [48], OCBAm [51], and EOC-m [49], the UEmr can find the top-$m$ designs and their rankings with higher accuracy if the simulation model has a large amount of stochastic noise. In the case of complex models, such as the SoS-based NCW model covered in this paper, stochastic noise tends to be great due to its high complexity; thus, the UEmr property of high robustness to noise allows the GA to perform the optimization of such complex models more efficiently. To verify this, we applied the UEmr and the similar R & S algorithms to the evaluation stage of the GA and compared the improved efficiency for the three kinds of benchmark models of which the optimal performance value is zero. The comparative results revealed in Figure 9 demonstrate the improved efficiency of the GA with the UEmr. In particular, in the case of Schaffer’s model, in which the stochastic noise is relatively greater than that in the other models due to many similar peaks (where the standard GA did not even converge), the GA with UEmr showed the highest efficiency improvement. Algorithm 2 represents the proposed GA with the UEmr.

[figures omitted; refer to PDF]

Algorithm 2: Proposed GA with R & S.

Parameters:

 GA: $k$, $G_{\max }$, $e\left(\le m\right)$, $P_c$, $P_m$

 R & S (Algorithm 1): $T\left(\le {kn}_0\right)$, $n_0$, $\mathrm{\Delta }\left(\ll T\right)$, $m\left(\le k\right)$

  INITIALIZATION:

1:  choose $k$ chromosomes randomly as initial population

  LOOP:

2:  for $i=1$ to $G_{\max }$ do

   EVALUATION WITH R & S:

3:   evaluate and find elite $m$ chromosomes from the population with Algorithm 1 ($T$, $n_0$, $\mathrm{\Delta }$, $m$)

   ELITISM:

4:   put the best e chromosomes among the elite $m$ chromosomes into the emptied new population

   GENETIC OPERATIONS:

5:   do

6:    choose 2 parent chromosomes from the elite $m$ chromosomes with the linear ranking selection

7:    generate 2 child chromosomes from the 2 parents with the uniform crossover with $P_c$ rate

8:    mutate the 2 children with the simple bit inversion method with $P_m$ rate

9:    put the 2 children into the new population

10:   while size of the new population $\le k$

11: end for

  FINALIZATION:

12: return the best chromosome among the population

Algorithm 2 has various parameters, and they can be classified into the two groups of GA and R & S. Among the parameters for the GA, $k$ and $G_{\max }$ mean the size of the population and the number of generations, respectively. $P_c$ and $P_m$ refer to the crossover and mutation probabilities, respectively, used in the genetic operations. Parameter $e$ indicates the number of chromosomes for the elitism that guarantees that the quality of the population does not degrade as the generation progresses [52]. On the other hand, among the parameters for the R & S, $T$ is the total number of simulation runs that can be allocated to the $k$ designs in the population via Algorithm 1 from each generation. $n_0$ indicates the initial number of simulation runs for the $k$ designs to obtain the minimum simulation results for further allocation (as mentioned previously, due to this $n_0$, R & S cannot be applied when $k$ is large). $\mathrm{\Delta }$ means the additional number of replications to be distributed to the $k$ designs according to the evaluated uncertainties in each iteration of the loop. Finally, $m$ is the number of the top-$m$ designs selected by Algorithm 1.

In summary, Algorithm 2, which is a combination of the GA and R & S, can further increase the efficiency of the optimization for the stochastic simulation model by reducing $k$ and $h$ in Figure 2 simultaneously through applying both metaheuristic and R & S methods together. That is, applying R & S to the evaluation stage of the GA can improve the quality of the population by finding the good chromosomes and their rankings accurately under limited simulation runs. The experimental results in Section 4.2 demonstrate the improved efficiency of Algorithm 2 in the optimization of the SoS-based NCW stochastic simulation model.

4. Experimental Results

The objective of this section is to demonstrate how much the proposed method reduces the simulation costs while retaining accuracy when finding the parameters satisfying the required combat power. Here, the given required combat power is to maximize the loss-exchange ratio (LER), which refers to the ratio of the number of red casualties to the number of blue casualties. That is, the optimization problem in this section is finding the optimal parameters of NCW [53, 54] that maximize the LER using the SoS-based NCW simulation.

Figure 10 illustrates the scenario of the NCW simulation model that includes complex and hierarchical information exchange among combat entities [55, 56]. When the red force approaches, a blue force’s soldier detects it and hierarchically sends the detection information to company C2 through intelligence fusion (IF). Then, C2 performs threat evaluation and weapon assignment (TEWA) and transmits the results to indirect weapons for the attack. Afterward, when the red force’s combat power is below a certain level, it instructs the subordinate entities to begin close combat for the direct attack. In this situation, the information exchange (blue arrow) occurs through communication, and thereby, its performance influences the combat power.

[figure omitted; refer to PDF]

According to this scenario, the blue force executes a defense operation against the red force with three times the military strength in a 2 km × 2 km operation area. The combat model represents the army’s military logic at an infantry company level including 131 entities, implemented in the DEVSimHLA [57, 58]. The network model depicts the mobile ad hoc network (MANET) based on the destination sequenced distance vector (DSDV) routing protocol including 131 nodes corresponding to the entities, implemented in the ns-3 discrete-event network simulator [59–62]. The two models participate in the SoS-based NCW simulation using HLA-RTI, which manages simulation time and exchanges the data [63]. The environment for this case study is as follows. The combat model uses CPU I5-3550 (3.3 GHz), RAM (4 GB), and DEVSim++ v.3.1; the network model uses NS3 v.3.18. They used RTI 1.3-NG and progressed simulation time over 5000 sec. For the training, this paper uses the MATLAB neural network toolbox v.8.2.1 and MATLAB Coder v.2.7.

4.1. Experimental Design and Results: Transformation

For the first process of the proposed optimization method, we applied the transformation process of Section 3.1 to the SoS-based NCW simulation to reduce the simulation execution time. To acquire the data for the transformation, we conducted the SoS-based NCW simulation. Among the full factorial design spaces, we selected 34 experimental points for training using a Latin hypercube design [60] and conducted 30 replicated simulations against each point. According to the process in Figure 6, we constructed an abstracted network model with the NARX model, which consists of two input delays, two output delays, and 10 neurons in the hidden layer using the Levenberg-Marquardt algorithm. After the transformation, we chose an extra 17 test points at random to compare the experimental results from the SoS-based simulation with the integrated simulation with those from the replicated simulation.

From the perspective of accuracy, the absolute root-mean-square error (RMSE) of the LER indicates 0.0742, which means a relative 5.6248% error when considering the minimum and maximum values (0.8287 and 2.149, respectively). In addition, to analyze the speed, we compared the average simulation execution time and the average number of processed events per trial in the SoS-based and integrated simulation. The SoS-based simulation indicates 67.083 minutes and $3.0782e+08$ events, respectively, and the integrated simulation indicates 0.592 minutes and $2.5605e+06$ events, respectively. Assuming that the time for training is negligible, we could reduce the simulation execution time about 113.31 times by abstracting the network model with enormous events and removing the interoperation middleware.

4.2. Experimental Design and Results: Optimization

For the second process of the proposed optimization method, we applied the proposed algorithm (Algorithm 2) to the integrated model, which was developed in the previous process, to efficiently find the optimal parameter setting that maximizes the LER by reducing the number of simulation runs. A total of 16,384 design alternatives were generated depending on the combat model parameters in Table 1 (i.e., $x_i=\left[P_{\mathrm{C}\mathrm{I}\mathrm{F}},P_{\mathrm{C}\mathrm{T}\mathrm{E}},P_{\mathrm{C}\mathrm{W}\mathrm{A}},P_{\mathrm{O}\mathrm{C}},P_{\mathrm{P}\mathrm{I}\mathrm{F}},P_{\mathrm{S}\mathrm{I}\mathrm{F}},P_{\mathrm{I}\mathrm{W}\mathrm{T}}\right]$, and each design was encoded to a chromosome of 14 bits (i.e., two bits per parameter in Table 1). The parameters of GA in Algorithm 2 were set as follows according to the typical Dejong setting [64]: $k=50$, $G_{\max }=1000$, $e=15$, $P_{\mathrm{c}}=0.5$, and $P_{\mathrm{m}}=0.01$. The parameters of R & S were set as follows according to the efficient setting guideline [43]: $T=5000$, $n_0=10$, $\mathrm{\Delta }=50$, and $m=30$. As a result, we could find the optimal design that maximizes the LER, [20, 20, 20, OC4, 20, 20, type 4], among 16,384 alternatives very efficiently via applying Algorithm 2. Table 2 and Figure 11 demonstrate the improved efficiency of Algorithm 2 compared to the standard GA. Here, the accuracy implies the probability of correct selection [37] and was estimated over 1000 independent trials.

Table 1

Combat model parameters and their description.

Parameter name	Parameter level	Description
Company C2 intelligence fusion time ($P_{\mathrm{C}\mathrm{I}\mathrm{F}}$)	20, 40, 60, 80 (sec)	The time required for intelligence fusion in company C2
Company C2 threat evaluation time ($P_{\mathrm{C}\mathrm{T}\mathrm{E}}$)	20, 40, 60, 80 (sec)	The time required for threat evaluation in company C2
Company C2 weapon assignment time ($P_{\mathrm{C}\mathrm{W}\mathrm{A}}$)	20, 40, 60, 80 (sec)	The time required for weapon assignment in company C2
Operation criteria for close combat ($P_{\mathrm{O}\mathrm{C}}$)	OC 1, 2, 3, 4	The enemy’s required combat power to begin the close combat
Platoon C2 intelligence fusion time ($P_{\mathrm{P}\mathrm{I}\mathrm{F}}$)	20, 40, 60, 80 (sec)	The time required for intelligence fusion in platoon C2
Squad C2 intelligence fusion time ($P_{\mathrm{S}\mathrm{I}\mathrm{F}}$)	20, 40, 60, 80 (sec)	The time required for intelligence fusion in squad C2
Indirect weapon type ($P_{\mathrm{I}\mathrm{W}\mathrm{T}}$)	Types 1, 2, 3, 4	The types of indirect weapons

Table 2

The number of simulation runs required to achieve the specific accuracy for the standard GA and Algorithm 2.

Accuracy	Standard GA	Algo. 2 (GA + R & S)	Ratio
0.8	1,350,000	85,000	15.88
0.85	1,565,000	90,000	17.39
0.9	1,835,000	110,000	16.68
0.95	2,470,000	125,000	19.76
0.99	4,000,000	215,000	18.60

[figure omitted; refer to PDF]

Algorithm 2 could greatly reduce the total number of simulation runs required to find the optimal design. As shown in Table 2, to find the optimal design with 95% accuracy, the standard GA required 2,470,000 simulation runs (5000 × 494 generations), whereas Algorithm 2 used only 125,000 (5000 × 25 generations) runs. In the case of applying only R & S instead of GA to find the optimal design, it is inefficient compared with Algorithm 2 because 163,840 simulation runs are required even if just 10 initial replications per design are conducted for 16,384 designs. Namely, Algorithm 2, which decreases both $k$ and $h$ by combining GA with R & S, can perform the optimization on the integrated model more efficiently than GA and R & S that only reduce $k$ or $h$.

As mentioned previously, the improved efficiency of Algorithm 2 is due to the increased quality of the population via applying R & S that finds the good chromosomes and their rankings accurately under limited simulation runs. This effect can be more clearly identified by changing the parameter settings of the algorithm, as shown in Figure 11. In Figure 11, the setting P1 means the previously mentioned setting, whereas P2 and P3 are the settings in which $T$ and $e$ were changed from P1 (i.e., $\mathrm{P}1\left[T=5000,e=15\right]$, $\mathrm{P}2\left[T=2500,e=10\right]$, and $\mathrm{P}3\left[T=1500,e=5\right]$). Here, $T$ and $e$ are the parameters that have the greatest effect on the quality of the population. In the case of the standard GA, as $T$ and $e$ decrease, the quality of the population becomes poor due to the inaccurately evaluated rankings of chromosomes and the reduced effects of the elitism; thus, the efficiency is reduced greatly, as shown in Figure 11. On the other hand, the efficiency of Algorithm 2 does not decrease significantly because the algorithm finds relatively accurate ranking and good chromosomes via R & S. In addition, in the case where the stochastic noise is large due to many similar peaks as shown in Figure 12, the high robustness to noise of the applied UEmr further increases the efficiency of Algorithm 2.

[figure omitted; refer to PDF]

5. Conclusion

To solve the time-consuming problem occurring when trying to find the combat model parameters satisfying the ROC, for example, combat power, this paper has proposed a simulation-based optimization method for the SoS-based NCW stochastic simulation model in the form of a hybrid approach with two processes.

In the first process, to shorten the prolonged execution time per simulation run, the SoS-based model was transformed into an integrated model by abstracting the highly complex network model and removing the interoperation middleware. The transformation based on the neural network achieved a small level of error by maintaining the network model’s characteristics. In the second process, to decrease the number of simulation runs, the simulation-based optimization for the integrated model was conducted using the GA with R & S. The simulation-based optimization significantly reduced the number of simulation runs for finding the optimized parameters compared with the optimization using the GA or R & S alone.

The empirical analysis revealed that the first process reduced the simulation execution time by 113.31 times within a 5.6248% error range; the second process decreased the number of simulation runs by 19.76 times to find the optimal parameters with 95% accuracy. Taken together, the proposed method ultimately reduced the total time by 2239 times with acceptable accuracy. We expect that the proposed method will help to conduct the simulation-based optimization on the SoS-based model in various domains, such as the military domain.

Conflicts of Interest

The authors declare that they have no conflicts of interest.

Acknowledgments

This work was supported by the Brain Korea 21 Plus program through the National Research Foundation (NRF) of Korea funded by the Ministry of Education. The authors would like to thank the Editor, Prof. László T. Kóczy, and anonymous reviewers for their valuable comments and constructive suggestions.

References

[1] J. Moffat, Complexity Theory and Network Centric Warfare, 2010.

[2] G. Neuneck, "The revolution in military affairs: its driving forces, elements, and complexity," Complexity, vol. 14 no. 1, pp. 50-61, DOI: 10.1002/cplx.20236, 2008.

[3] J. H. Kim, I.-C. Moon, T. G. Kim, "New insight into doctrine via simulation interoperation of heterogeneous levels of models in battle experimentation," Simulation, vol. 88 no. 6, pp. 649-667, DOI: 10.1177/0037549711414773, 2012.

[4] A. Yang, H. A. Abbass, R. Sarker, "Land combat scenario planning: a multiobjective approach," Simulated Evolution and Learning, pp. 837-844, DOI: 10.1007/11903697_105, 2006.

[5] A. Tolk, Engineering Principles of Combat Modeling and Distributed Simulation,DOI: 10.1002/9781118180310, 2012.

[6] B. G. Kang, T. G. Kim, "Reconfigurable C3 simulation framework: interoperation between C2 and communication simulators," 2013 Winter Simulations Conference (WSC), pp. 2819-2830, DOI: 10.1109/WSC.2013.6721652, .

[7] HLA Working Group, "IEEE standard for modeling and simulation (M and S) high-level architecture (HLA)-framework and rules," ,DOI: 10.1109/IEEESTD.2010.5553440, 2000.

[8] US DoD, "TENA-the test and training enabling architecture," .

[9] C. E. Maldonado, N. A. Gómez Cruz, "The complexification of engineering," Complexity, vol. 17 no. 4,DOI: 10.1002/cplx.20395, 2012.

[10] K. H. Lee, J. H. Hong, T. G. Kim, "System of systems approach to formal modeling of CPS for simulation-based analysis," ETRI Journal, vol. 37 no. 1, pp. 175-185, DOI: 10.4218/etrij.15.0114.0863, 2015.

[11] B. S. Blanchard, W. J. Fabrycky, W. J. Fabrycky, Systems Engineering and Analysis, 1990.

[12] ISO, IEC/IEEE 15288: 2015, Systems and Software Engineering-Content of Systems and Software Life Cycle Process Information Products (Documentation), 2015.

[13] M. Jamshidi, Systems of Systems Engineering Principles and Applications, 2008.

[14] M. Jamshidi, System of Systems Engineering Innovations for the Twenty-First Century, 2011.

[15] S. W. Popper, S. C. Bankes, R. Callaway, D. DeLaurentis, System of Systems Symposium: Report on a Summer Conversation, 2004.

[16] S. H. Choi, K.-M. Seo, T. G. Kim, "Accelerated simulation of discrete event dynamic systems via a multi-fidelity modeling framework," Applied Sciences, vol. 7 no. 10, article 1056,DOI: 10.3390/app7101056, 2017.

[17] J. H. Hong, K. M. Seo, T. G. Kim, "Simulation-based optimization for design parameter exploration in hybrid system: a defense system example," Simulation, vol. 89 no. 3, pp. 362-380, DOI: 10.1177/0037549712466707, 2013.

[18] I. Porche, L. Jamison, T. Herbert, Framework for Measuring the Impact of C4ISR Technologies and Concepts on Warfighter Effectiveness Using High Resolution Simulation, 2004.

[19] N. E. Miner, B. P. Van Leeuwen, K. M. Welch, M. Donan Estill, M. A. Smith, H. D. Le, C. R. Lawton, "Evaluating communications system performance effects at a system of systems level," MILCOM 2012 - 2012 IEEE Military Communications Conference,DOI: 10.1109/MILCOM.2012.6415686, .

[20] B. D. Paz, J. A. Baer, "Communication effect server integration with OneSAF for mission level simulation," Proceedings of the 2008 Fall Simulation Interoperability Workshop, FIW 2008, pp. 271-276, .

[21] I. Porche, R. Isaac, W. Bradley, The Impact of Network Performance on Warfighter Effectiveness, 2006.

[22] A. M. Law, Simulation Modeling and Analysis, 2007.

[23] S. H. Choi, T. G. Kim, "Optimal subset selection of stochastic model using statistical hypothesis test," IEEE Transactions on Systems, Man, and Cybernetics: Systems, vol. 48 no. 4, pp. 557-564, DOI: 10.1109/TSMC.2016.2608982, 2018.

[24] S. H. Choi, T. G. Kim, "Pareto set selection for multiobjective stochastic simulation model," IEEE Transactions on Systems, Man, and Cybernetics: Systems,DOI: 10.1109/TSMC.2018.2846680, 2018.

[25] B. G. Kang, B. S. Kim, T. G. Kim, "Abstraction on network model under interoperable simulation environment," ECMS 2016 Proceedings edited by Thorsten Claus, Frank Herrmann, Michael Manitz, Oliver Rose, pp. 460-466, DOI: 10.7148/2016-0460, .

[26] B. G. Kang, K. M. Seo, T. G. Kim, "Machine learning-based discrete event dynamic surrogate model of communication system for simulating command, control, and communication system of systems," Simulation: Transactions of the Society for Modeling and Simulation International, 2018.

[27] B. G. Kang, K. M. Seo, T. G. Kim, "Communication analysis of network-centric warfare via transformation of system of systems model into integrated system model using neural network," Complexity, vol. 2018,DOI: 10.1155/2018/6201356, 2018.

[28] J. H. Bigelow, P. K. Davis, Implications for Model Validation of Multiresolution, Multiperspective Modeling (mRMPM) and Exploratory Analysis, 2003.

[29] A. Brooks, S. Bankes, B. Bennett, Weapon Mix and Exploratory Analysis a Case Study, 1997.

[30] Z. Xiao, L. Peng, Y. Chen, H. Liu, J. Wang, Y. Nie, "The dissolved oxygen prediction method based on neural network," Complexity, vol. 2017,DOI: 10.1155/2017/4967870, 2017.

[31] T. Gligorijević, Z. Ševarac, B. Milovanović, V. Đajić, M. Zdravković, S. Hinić, M. Arsić, M. Aleksić, "Follow-up and risk assessment in patients with myocardial infarction using artificial neural networks," Complexity, vol. 2017,DOI: 10.1155/2017/8953083, 2017.

[32] Y. Zhou, L. Zhou, Y. Wang, Z. Yang, J. Wu, "Application of multiple-population genetic algorithm in optimizing the train-set circulation plan problem," Complexity, vol. 2017,DOI: 10.1155/2017/3717654, 2017.

[33] A. Jain, N. S. Chaudhari, "An improved genetic algorithm for developing deterministic OTP key generator," Complexity, vol. 2017,DOI: 10.1155/2017/7436709, 2017.

[34] D. S. Alberts, J. J. Garstka, F. P. Stein, "Network centric warfare: developing and leveraging information superiority," 2000.

[35] M. F. Khan, E. A. Felemban, S. Qaisar, S. Ali, "Performance analysis on packet delivery ratio and end-to-end delay of different network topologies in wireless sensor networks (WSNs)," 2013 IEEE 9th International Conference on Mobile Ad-hoc and Sensor Networks, pp. 324-329, DOI: 10.1109/msn.2013.74, .

[36] M. C. Fu, "Feature article: optimization for simulation: theory vs. practice," Informs Journal on Computing, vol. 14 no. 3, pp. 192-215, DOI: 10.1287/ijoc.14.3.192.113, 2002.

[37] C. H. Chen, L. H. Lee, "Stochastic simulation optimization: an optimal computing budget allocation," World Scientific, vol. 1, 2011.

[38] S. Kirkpatrick, C. D. Gelatt, M. P. Vecchi, "Optimization by simulated annealing," Science, vol. 220 no. 4598, pp. 671-680, DOI: 10.1126/science.220.4598.671, 1983.

[39] J. H. Holland, Adaptation in Natural and Artificial Systems, 1975.

[40] F. Glover, "Tabu search, part I," ORSA Journal on Computing, vol. 1 no. 3, pp. 190-206, DOI: 10.1287/ijoc.1.3.190, 1989.

[41] C. H. Chen, J. Lin, E. Yücesan, S. E. Chick, "Simulation budget allocation for further enhancing the efficiency of ordinal optimization," Discrete Event Dynamic Systems, vol. 10 no. 3, pp. 251-270, DOI: 10.1023/A:1008349927281, 2000.

[42] S. H. Kim, B. L. Nelson, "On the asymptotic validity of fully sequential selection procedures for steady-state simulation," Operations Research, vol. 54 no. 3, pp. 475-488, DOI: 10.1287/opre.1060.0281, 2006.

[43] S. H. Choi, T. G. Kim, "Efficient ranking and selection for stochastic simulation model based on hypothesis test," IEEE Transactions on Systems, Man, and Cybernetics: Systems, vol. 48 no. 9, pp. 1555-1565, DOI: 10.1109/TSMC.2017.2679192, 2018.

[44] K. Worden, C. X. Wong, U. Parlitz, A. Hornstein, D. Engster, T. Tjahjowidodo, F. al-Bender, D. D. Rizos, S. D. Fassois, "Identification of pre-sliding and sliding friction dynamics: grey box and black-box models," Mechanical Systems and Signal Processing, vol. 21 no. 1, pp. 514-534, DOI: 10.1016/j.ymssp.2005.09.004, 2007.

[45] P. Amani, M. Kihl, A. Robertsson, "NARX-based multi-step ahead response time prediction for database servers," 2011 11th International Conference on Intelligent Systems Design and Applications, pp. 813-818, DOI: 10.1109/isda.2011.6121757, .

[46] S. Salivahanan, A. Vallavaraj, C. Gnanapriya, Digital Signal Processing, 2001.

[47] C. M. Seo, B. P. Zeigler, D. H. Kim, K. Duncan, "Integrating web-based simulation on IT systems with finite probabilistic DEVS," Proceedings of the Symposium on Theory of Modeling & Simulation: DEVS Integrative M and S Symposium, TMS 2015, pp. 173-180, .

[48] S. Zhang, L. H. Lee, E. P. Chew, J. Xu, C. H. Chen, "A simulation budget allocation procedure for enhancing the efficiency of optimal subset selection," IEEE Transactions on Automatic Control, vol. 61 no. 1, pp. 62-75, DOI: 10.1109/TAC.2015.2423832, 2016.

[49] S. Gao, W. Chen, "Efficient subset selection for the expected opportunity cost," Automatica, vol. 59, pp. 19-26, DOI: 10.1016/j.automatica.2015.06.005, 2015.

[50] S. H. Choi, T. G. Kim, "A heuristic approach for selecting best-subset including ranking within the subset," IEEE Transactions on Systems, Man, and Cybernetics: Systems, 2018.

[51] C. H. Chen, D. He, M. Fu, L. H. Lee, "Efficient simulation budget allocation for selecting an optimal subset," Informs Journal on Computing, vol. 20 no. 4, pp. 579-595, DOI: 10.1287/ijoc.1080.0268, 2008.

[52] S. Baluja, R. Caruana, "Removing the genetics from the standard genetic algorithm," Machine Learning Proceedings 1995: Proceedings of the Twelfth International Conference on Machine Learning, pp. 38-46, DOI: 10.1016/b978-1-55860-377-6.50014-1, .

[53] M. Phythian, Understanding the Intelligence Cycle, 2013.

[54] J. Marshall, L. M. ER, J. Tapp, A. T. Diotte, D. Quigley, J. Givens, K. Reszka, C. Williams, M. Bonnett, An Analysis of the Open Architecture Warfare System Domain Model for Surface Time Critical Targets, 2007.

[55] A. H. Dekker, C4ISR Architectures, Social Network Analysis and the FINC Methodology: An Experiment in Military Organisational Structure, 2002.

[56] I. C. Moon, K. M. Carley, T. G. Kim, Modeling and Simulating Command and Control: For Organizations Under Extreme Situations, 2013.

[57] B. P. Zeigler, T. G. Kim, H. Praehofer, Theory of Modeling and Simulation Integrating Discrete Event and Continuous Complex Dynamic Systems, 2000.

[58] T. G. Kim, C. H. Sung, S.-Y. Hong, J. H. Hong, C. B. Choi, J. H. Kim, K. M. Seo, J. W. Bae, "DEVSim++ toolset for defense modeling and simulation and interoperation," The Journal of Defense Modeling and Simulation: Applications, Methodology, Technology, vol. 8 no. 3, pp. 129-142, DOI: 10.1177/1548512910389203, 2011.

[59] S. Kaur, S. kaur, C. Sharma, "An overview of mobile ad hoc network: application, challenges and comparison of routing protocols," IOSR Journal of Computer Engineering, vol. 11 no. 5,DOI: 10.9790/0661-1150711, 2013.

[60] C. Rajabhushanam, A. Kathirvel, "Survey of wireless MANET application in battlefield operations," International Journal of Advanced Computer Science and Applications, vol. 2 no. 1, pp. 50-58, DOI: 10.14569/ijacsa.2011.020108, 2011.

[61] T. R. Henderson, M. Lacage, G. F. Riley, C. Dowell, J. Kopena, "Network simulations with the ns-3 simulator," SIGCOMM Demonstration, .

[62] J. Pan, R. Jain, A Survey of Network Simulation Tools: Current Status and Future Developments, 2008.

[63] M.-W. Yoo, C. Choi, T. G. Kim, "High-level architecture service management for the interoperation of federations," Simulation, vol. 91 no. 6, pp. 566-590, DOI: 10.1177/0037549715584403, 2015.

[64] K. A. DeJong, W. M. Spears, "An analysis of the interacting roles of population size and crossover in genetic algorithms," Parallel Problem Solving from Nature. PPSN 1990. Lecture Notes in Computer Science, pp. 38-47, DOI: 10.1007/BFb0029729, .