MULTIMOORA under Interval-Valued Neutrosophic Sets as the Basis for the Quantitative Heuristic Evaluation Methodology HEBIN

Abstract

During the last decade, researchers put a lot of effort into the development of the multi-criteria decision methods (MCDM) capable of dealing with the uncertainty and vagueness of the initial information. MCDM approaches that work under the environment of the interval-valued neutrosophic sets (IVNS) demonstrate credibility for the analysis of different opinions as well as for the inconsistency of the criteria evaluation data. The novel multicriteria decision-making approach MULTIMOORA-IVNS (multi-objective optimisation by ratio analysis under interval-valued neutrosophic sets) is presented in this paper. A novel heuristic evaluation methodology HEBIN (heuristic evaluation based on interval numbers) that exploits MULTIMOORA-IVNS for the processing of the evaluation results is also presented in this research. HEBIN is able to increase the accuracy of the checklists-based heuristic evaluation and to diminish the impact of the inconsistencies caused by the evaluators. A comparison of six e-commerce websites is introduced to reveal the practicalities of the proposed multicriteria decision-making application.

1. Introduction Multi-criteria decision making (MCDM) theory is intensively investigated for both the theoretical and implementation aspects. Since there are many real-life applications where the decision information cannot be rigorously represented due to its incompleteness, indeterminacy, and inconsistency, researchers are constantly looking for the novel mathematical modelling techniques that can be applied to deal with this kind of challenge.

The pioneering ideas to deal with non-rigid boundaries of decision information was proposed by Zadeh [1], who introduced the fuzzy set concept. By this theory, each object of the universe is described by the single relatively graded membership. Atanassov [2] extended traditional fuzzy sets formulation by incorporating the degree of hesitation into the decision-making and named this extension as the intuitionistic fuzzy sets (IFS). Since the IFS theory requires to keep the sum of the membership and non-membership degrees in the closed interval [0, 1], it also raises some limitations for the IFS applications. Therefore, the q-rung orthopair fuzzy sets were proposed as the augmentation of the intuitionistic fuzzy sets and Pythagorean fuzzy sets [3]. The q-rung orthopair fuzzy sets are governed by the condition that the summation of q^th power of the membership function and the q^th power of the non-membership grade that are limited in the interval [0, 1]. These and other extensions of the fuzzy sets were proposed by researchers for the implementation into the various MCDM problems [4,5,6,7].

Since fuzzy sets could not take into consideration all types of uncertainties that emerge in the construction of the mathematical models developed for the solutions of real-life problems, Neutrosophic Sets (NS) were introduced by Smarandache [8,9]. The essence of the NS theory lays in the addition of the parameter named as the “knowledge of neural thought”. The inclusions of this independent “neutral” parameter distinguish neutrosophic sets from other fuzzy set theories. In fact, neutrosophic sets can be considered as a generalisation of the other fuzzy sets, that provide better possibilities for the modelling of the uncertainty and vagueness of decision-making information [10,11]. In the theory of the neutrosophic sets, objects of the universe are exhibit by three characteristics: the degree of the truth (T), a degree of the indeterminacy (I) and a degree of the falsity (F) [8].

Recently, neutrosophic sets were proposed to be effective for multi-criteria decision-making problems in different domains [12,13,14]. However, most of these approaches used single-valued numbers for the construction of the decision matrix and tolerated inconsistencies that might arise due to the subjectivity of the evaluators’ that assess the alternatives.

Usage of interval numbers (IN) might be involved when there is a need to provide information as intervals instead of the single-valued numbers. Interval-valued neutrosophic sets (IVNS) was introduced by Wang et al. [15] as the appropriate way to represent uncertain, incomplete, imprecise, and inconsistent information. Since IVNS shows greater flexibility and precision than single-valued neutrosophic sets [16], IVNS applications became the object of interest for many researchers. The credibility of the interval-valued neutrosophic sets (IVNS) was demonstrated by [17,18,19].

Recently, researchers apply the theory of neutrosophic sets to produce different extensions of MULTIMOORA [20] approach. Liang et al. [21] carried out MULTIMOORA extension referred to Linguistic Neutrosophic Numbers. Tian et al. [22] proposed simplified neutrosophic linguistic MULTIMOORA version and Zavadskas et al. [14] announced the single-valued neutrosophic MULTIMOORA (MULTIMOORA–SVNS). For the best of our knowledge, MULTIMOORA modification based on interval-valued neutrosophic sets is still not developed.

The novel approach, namely MULTIMOORA-IVNS (multi-objective optimisation by ratio analysis under interval-valued neutrosophic sets), will be presented in this paper. The original quantitative heuristic evaluation methodology HEBIN will also be offered in this paper as the practical application of the MULTIMOORA-IVNS. HEBIN application for six international e-commerce websites will be presented to reveal the practicalities of HEBIN and MULTIMOORA-IVNS. 2. MULTIMOORA under Interval-Valued Neutrosophic Sets

MULTIMOORA is the updated form of the multi-objective optimisation by ratio analysis (MOORA) [20]. Since MULTIMOORA exploits the vector normalisation technique and three subordinate ranking methods (ratio system, reference point approach, and full multiplicative form) to produce relative rankings of numerous alternatives, it provides more robust results than those MCDM methods, that employ a single utility function. Moreover, MULTIMOORA includes simple mathematics, low computational time, straightforwardness for decision-makers and ranking aggregation tools to present ranking of the alternatives [23].

2.1. Interval-Valued Neutrosophic Sets

In this section, a short introduction of the main statements related to the neutrosophic sets and the general properties of the interval-valued neutrosophic set (IVNS) that have been proposed by [24] will be presented.

Definition 1.

There is space X of the certain objects where the separate generic elementsx∈X.An interval-valued neutrosophic set (IVNS)N⊂Xhas the form of

N=〈x,_TNx,_INx,_FNx〉:x∈X

where_TNx:X→0, 1,_INx:X→0, 1and_FNx:X→0, 1with0≤_TNx+_INx+_FNx≤3or allx∈X.The variables_TAx,_NAxand_FAxdefine truth-membership degree function, the indeterminacy-membership degree function and the falsity-membership degree function of x to N, respectively. For the case of the interval neutrosophic set, these functions must be described as_TNx=inf_TNx,sup_TNx⊆0, 1 , _INx=inf_INx,sup_INx⊆0, 1,_FNx=inf_FNx,sup_FNx⊆0, 1and the sum of these functions satisfy the condition0≤sup_TNx+sup_INx+sup_FNx≤3.

Definition 2.

If_N1=〈inf_TN1,sup_TN1,inf_IN1x,sup_IN1,inf_FN1x,sup_FN1〉and_N2=〈inf_TN2x,sup_TN2,inf_IN2x,sup_IN2,inf_FN2x,sup_FN2〉are two interval-valued neutrosophic numbers (IVNN), then_N1is contained in the other neutrosophic element_N2,_N1⊆_N2if and only if:

inf _TN1≤inf _TN2,sup_TN1≤sup _TN2,inf _IN1≥inf _IN2,sup _IN1≥sup _IN2,inf _FN1≥inf _FN2,sup_FN1≥sup _FN2, for any x∈X

Definition 3.

Two IVNNs_N1and_N2are equal, described as_N1=_N2, if and only if_N1⊆_N2, and_N1⊇_N2.

Definition 4.

Comparison of the interval-valued neutrosophic numbers is completed employing the score, accuracy and certainty functions. For the interval-valued neutrosophic number_N1=〈inf_TN1x,sup_TN1,inf_IN1x,sup_IN1,inf_FN1x,sup_FN1〉the mentioned functions are of the form

s_N1=inf_TN1+1−sup_IN1+1−sup_FN1,sup_TN1+1−inf_IN1+1−inf_FN1a_N1=min{inf_TN1−inf_FN1,sup_TN1−sup_FN1},max{inf_TN1−inf_FN1,sup_TN1−sup_FN1}c_N1=inf_TN1,sup_TN1

wheres_N1,a_N1andc_N1means the score, accuracy and certainty functions of the IVNN_N1, respectively.

Definition 5.

If_N1and_N2are two interval-valued neutrosophic numbers, then their determination should be completed in the following way:

• Ifps_N1≥s_N2>0.5, then_N1is greater than_N2or_N1is superior to_N2and this fact can be represented as_N1≻_N2.

• Ifps_N1≥s_N2=0.5andpa_N1≥a_N2=0.5, then_N1is greater than_N2or_N1is superior to_N2and this fact can be represented as_N1≻_N2.

• Ifps_N1≥s_N2=0.5,pa_N1≥a_N2=0.5andpc_N1≥c_N2=0.5, then_N1is greater than_N2or_N1is superior to_N2and this fact must be represented as_N1≻_N2.

• Ifps_N1≥s_N2=0.5,pa_N1≥a_N2=0.5andpc_N1≥c_N2=0.5, then_N1is equal to_N2or_N1is indifferent to_N2and this fact can be represented as_N1~_N2.

Definition 6.

The degree of the possibility of the score function is determined by the following expression:

ps_N1≥s_N2=max1−maxsup(s_N2)−inf(s_N1)_lN1+_lN2,0,0

where_lN1=sup(s_N1)−inf(s_N1)and_lN2=sup(s_N2)−inf(s_N2). The degrees of the possibility for the accuracy and certainty functions are calculated in the respective approach.

Definition 7.If we consider two IVNNs

_N1=〈inf_TN1,sup_TN1, inf_IN1,sup_IN1, inf_FN1,sup_FN1〉,_N1=〈inf_TN1,sup_TN1, inf_IN1,sup_IN1, inf_FN1,sup_FN1〉λ>0.

The operations for IVNNs can be expressed as follows:

λ_N1=〈1−^{1−inf_TN1λ}, 1−^{1−sup_TN1λ}, ^inf_IN1λ, ^sup_IN1λ, ^inf_FN1λ, ^inf_FN1λ〉

_N1+_N2= =〈inf_TN1+inf_TN2−inf_TN1⋅inf_TN2, sup_TN1+sup_TN2−sup_TN1⋅sup_TN2,inf_IN1⋅inf_IN2,sup_IN1⋅sup_IN2, inf_FN1⋅inf_FN2, sup_FN1⋅sup_FN2〉

_N1⋅_N2 == 〈[(inf_TN1⋅inf_TN2), (sup_TN1⋅sup_TN2)],[(inf_IN1 + inf_IN2 − inf_IN1⋅inf_IN2), (sup_IN1 + sup_IN2 − sup_IN1⋅sup_IN2)],[(inf_FN1 + inf_FN2 − inf_FN1⋅inf_FN2), (sup_FN1 + sup_FN2 − sup_FN1⋅sup_FN2)]〉

The distance measure between two interval-valued neutrosophic numbers is described by the expression:

D(_(xN*)1,_{xN* )2}==16^{inf_tN1−inf_tN22}+^{sup_tN1−sup_tN22}+^{inf_iN1−inf_iN22}+^{sup_iN1−sup_iN22}+^{inf_fN1−inf_fN22}+^{sup_fN1−sup_fN22}.

2.2. MULTIMOORA—IVNS Approach

The essence of the novel approach MULTIMOORA-IVNS consists of the operational functionality of interval-valued neutrosophic sets and crisp MULTIMOORA extensions described by [20].

Step 1. The initial step in the multicriteria decision-making methods is the construction of the initial decision matrix X, where elements_xijare interval numbers corresponding to the^ithcriteria of^jthalternative. The normalisation of the decision matrix elements is done applying the function, that was specifically developed for appropriate estimation of the certain features of the neutrosophic sets and interval-valued numbers.

inf_xij*=inf_xijmax i_xijm, sup_xij*=sup_xijmax i_xijm

The proposed normalisation function ensures better stability and resolution range for the proposed MULTIMOORA–IVNS approach.

Step 2: The neutrosophication for the elements of the decision matrix. The members of the interval values are converted into interval-valued neutrosophic numbers applying the standard modification rates as in [14].

Step 3: Assembly of the neutrosophic decision matrix consisting of the elements_xn*ij.

Step 4: The first target of interval-valued neutrosophic MULTIMOORA proposal can be described as:

_Qj=∑i=1g_{wi xn*ij}+^{∑i=g+1n_{wi xn*ij}c}

where g elements match members of beneficial criteria,n−gmatch to members of non-beneficial criteria. The second component in Equation (12) is constructed applying supplementing part of the interval-valued neutrosophic member, which can be described in the expression:

^_xn1*c=〈inf_fn1,sup_fn1,1−sup_in1,1−inf_in1,inf_tn1,sup_tn1〉

Step 5: Calculation of the second objective of interval-valued neutrosophic MULTIMOORA approach. The second objective is established taking into account deviation from the reference point and the Min-Max metric of Tchebycheff norm

minjmaxi D_ri−_{wi xn*ij}

The reference point is calculated for the case of the beneficial criteria by the expression:

_ri=maxi_{wi xn*ij}

In the case of the non-beneficial criteria_riis defined as:

_ri=mini_{wi xn*ij}

The matching of the interval-valued neutrosophic members is done by applying the degree of the possibility of the score function as followed in Definitions (6) and (7).

Step 6: Calculation of the third objective of interval-valued neutrosophic MULTIMOORA expression. Full multiplicities should be used for the third objective, which implements the purely multiplicative utility function for the criteria to be maximised as well as for the criteria to be minimised. Consequently, for each analysed alternative must be assembled the common utility, which must be described:

_Uj=S_AjS_Bj

Here,_Ajand_Bjcomponents are calculated as

_Aj=∏i=1g_{wi xn*ij}, _Bj=∏j=g+1n_{wi xn*ij}

The product of maximised criteria of^jthalternative represented by the first component_Aj. The product of minimized of criteria of^jthalternative described by the second component_Bj.

Step 7: The final summarization of first, second and third goals of MULTIMOORA-IVNS is completed within the dominance theory framework [20].

3. Quantitative Heuristic Evaluation Methodology HEBIN

Heuristic evaluation [25] is a widely used website inspection method devoted to examining interfaces via the recommendations grounded on the user-centred design principles identified as heuristics [26]. Depending on the selected procedure, HE technique may provide qualitative or quantitative results. While qualitative heuristic evaluation (QLHE) brings extensive information about the quality of the single interface, quantitative heuristic evaluation (QNHE) provides numerical data mandatory for the comparison of the alternatives. However, quantitative heuristic evaluation is a challenging task since neither unified methodology on how to do it is presented for the current day.

González et al. [27] stated that results of the QNHE depend on the three main components: (I) the characteristics of the evaluators; (II) the set of the domain orientated heuristics and sub-heuristics and (III) the mathematical model that is chosen to process data. Comprehensive checklists (questioners) where heuristics are divided into the sub-heuristics are an important part of the QNHE [28]. The authors of this article compared several studies that employ checklist based QNHE to revealed differences in their applicability (Table 1).

It can be seen in Table 1, that different sets of heuristics and sub-heuristics can be used for the QNHE. The amount and the experience of the evaluators participating in the experiments also differs. It is well known that inconsistencies related to the diverse expertise, culture, gender, age, attention and information processing capacities of the evaluators strongly affect results of the HE [32]. Irregular understanding of the predefined heuristics raises additional challenges in the heuristics-based decision making. However, the biggest struggles of the QNHE are associated with the selection of the mathematical model.

Usability index, which represents the total number of usability problems found on a website, divided by the total number of pages investigated on the site, was presented in [30]. The number of websites with the violated heuristic divided by the total number of analysed websites was calculated to compare the quality of the museum websites [29]. Shayganmeh et al. [31] stressed that indices (heuristics) described by indicators (sub-heuristics) are able to evaluate wider dimensions of the e-services websites and proposed to employ MCDM theory for the checklist-based comparison of the websites. PROMETHEE [33] was suggested to rank indicators, and Analytical Hierarchy Process [34] was proposed to weight indices. The final readiness values were obtained, adding products of indexes weights to the single average indicator readiness value.

Authors of this article believe that MCDM methods are an appropriate way to compare different interfaces based on the data collected from the checklist based heuristic evaluation. Therefore, in this paper, we decided to exploit the advantages of the interval numbers for the MCDM based quantitative heuristic evaluation. This novel methodology will be presented later in this section. 3.1. Heuristic Evaluation and the Inconsistencies of the Judgements

Traditionally heuristic evaluation is understood as the expert-based website inspection technique. According to Nielsen et al. [25], HE requires 3–5 evaluators to assess interfaces. HE provides the most reliable results when each of the experts works separately, but at the end of the experiment gathers together to reach a consensus on the evaluation results. If there is a possibility to bring all the team members on board, the probability of having a decision that everyone likes, respects, and supports increases. Nevertheless, there is always a possibility that the desire to reach an agreement might cause people to ignore some of the findings and to put aside insights that may derail the consensus decision. This situation has come to be known as the evaluator effect and has been well-documented by [35].

Ideally, heuristic evaluation should be performed by five usability experts having a deep understanding of the chosen heuristic set and the experience in the application domain. In practice, small companies often do not have a sufficient budget to hire usability experts; therefore, the need for the HE methodologies that can be performed by novice evaluators is getting increased attention. The term “novice evaluators” can be understood as the professionals that do not have enough knowledge on the user experience and possibly participate in the heuristic evaluation for the first time [36]. For such cases, checklist-based HE might be a beneficial approach to reduce misunderstanding related to the inconsistent interpretation of the heuristics. However, checklist-based criteria (heuristics) assessments are not able to remove all the inaccuracies raised by the differences of the evaluators. Therefore, MULTIMOORA-IVNS is proposed in this study as the mathematical model for the analysis of QNHE. The novel QNHE methodology HEBIN (Heuristic evaluation based on interval numbers) that exploit MULTIMOORA-IVNS for decision-making is also presented in this paper.

3.2. HEBIN Methodology

Heuristic evaluation based on interval numbers (HEBIN) methodology consists of 7 stages, each of which is briefly described in Figure 1.

When the novice evaluators or the usability experts are hired for the experiment, the short briefing session, where the goal of the research, methodology and the chosen heuristics set explained by sub-heuristics, should be organised. Each of the evaluators is asked to work individually. The final estimate for each of the heuristics is calculated as the sum of the points assigned to the corresponding sub-heuristics. If five evaluators are hired for the experiment, five separate reports of the HE should be prepared for each of the alternatives. As soon as it is done, the collected data can be used for the construction of the initial decision matrix X consisting of the values_xna:

X=_x11x12…_x1axn1xn2…_xna⋮_xN1⋮_xN2⋱⋯⋮_xNA

Here,a=1, 2, …Adenotes the number of the analysed alternatives andn=1, 2, …Ndenote the number of the heuristics. Value_xnafor each alternative a and the heuristic H_n has to be determined as the intervalmin_Hna;max_Hna ,wheremin_Hnais the lowest estimate of the heuristics H_n, andmax_Hnais the highest estimate of the heuristics H_n among all five evaluators that presented their estimates for the heuristic H_n. In this way, the inconsistencies caused by the experience of the evaluators can be recorded for further data processing. We propose to employ MULTIMOORA-IVNS as the appropriate approach to deal with the uncertainty and inconsistencies of the collected data.

Heuristic evaluation based on interval numbers (HEBIN) exploits the different opinions of the evaluators and does not seek consensus on the valuation results. There is only one requirement for the evaluators. All evaluators must use the same set of heuristics and sub-heuristics for the assessment of the alternatives. 4. HEBIN Application for the Comparison of E-Commerce Websites

Over the past few years, e-commerce has become an irreplaceable part of the international retail system. Global data platform www.statista.com shows that the total number of people purchasing goods and services online reached 1.92 billion customers in 2019th. In the same period, the total annual sales revenue from the e-commerce market topped 3.5 trillion U.S. dollars. Since we are living in the global industry and internet users can freely choose electronic shops (e-shops) where they would like to purchase, neither of the online business can be prosperous without the periodical appraisal of the e-commerce websites they own. In this context, analysis of the competitive environment is becoming especially important for the success of the online businesses. The competitor benchmarking allows business owners to identify the advantages and disadvantages of the solutions they provide, gives an understanding of the features, functions and design decisions successfully acting in the rival e-shops.

However, it is still a great challenge to judge and compare the quality of different electronic shops, since both the functional and non-functional requirements should be assessed to make reliable decisions on the quality of e-commerce websites. Even though functionality, security, privacy, accessibility and reliability are still traditionally recognised as the significant criteria affecting the value of the online shops [37]; trustworthiness, personalisation, navigation and customer support are slowly becoming the decisive factors for the customers’ willingness to buy [38,39]. While non-functional requirements like user experience have the positive impact for the quality of the electronic shops and the negative effect on the uncertainty of the assessment information [40,41], specific checklists capable of collecting data on the user experience of the websites should be chosen for the competitor benchmarking.

Quinones and Rusu [26] made a review of the studies where various sets of domain-orientated heuristics were offered. Research presented by Bonastre and Granollers [42] was the only one appraising the user experience of e-commerce websites. The checklist presented in [42] consists of 64 questions divided into six stages of online purchasing: (1) need recognition and problem awareness, (2) information search, (3) purchase decision-making, (4) transaction, (5) post-sales behaviour and (6) other factors that affect the user experience. Since these stages of the purchasing process cannot be directly mapped with the heuristics representing service quality, system quality and information quality [43,44], based on it we composed a new checklist dedicated to assessing trust, response time, reliability, responsiveness, empathy, timeliness, accuracy, navigability and accessibility of e-commerce websites. Nine criteria that we analysed as heuristic were proposed by Nilashi in [45]. The novel checklist that consists of 9 heuristics and 82 sub-heuristics is presented in Table A1 in Appendix A.

Three different scales are proposed to assess sub-heuristics. Most of the sub-heuristics can be measured in a two-point scale, where 0 means “No”, 1— “Partly yes”, 2— “Yes”. Since reputation is a critical aspect of any online business, the sub-heuristic TR1 has the 5-point evaluation scale. Checklist items that describe accessibility (AC) issues are the only ones that require an additional tool for the assessment of the sub-heuristics. In the study case presented in this paper, evaluators were recommended to use https://www.webpagetest.org to measure webpage size and the loading time.

4.1. Weighting of the Heuristics

Criteria weighting is an important part of any multicriteria decision-making process. Direct and indirect weighting approaches can be applied for the criteria weighting. When indirect methods are applied, criteria weights are derived from mathematical modelling, whereas in the direct methods, the decision-makers compare criteria directly, via a chosen ratio scale. Direct weighting (DW) techniques like the SWING [46], SMARTS [47], SMARTER [47], point allocation [48], direct rating [48], or the VASMA weighting [49] were recently applied in various MCDM tasks [49,50,51].

SMARTS methodology was chosen for the heuristics weighting in HEBIN methodology. Ten external experts working with online shopping were asked to participate in the experiment. A matrix constructed of nine visual analogue scales (VAS) with the endpoints meaning “Not important” (numeric value 1) and “Very important” (100) was printed and presented for each of the evaluators to simplify the preference elicitation process. The distance between the tick marks of the VAS scales was determined to 5.

At the beginning of the meeting, all ten decision-makers (DM) agreed that Trust (TR) is the most important aspect of the e-commerce business. Also, they decided that all nine heuristics involved in the evaluation procedure have a significant impact on the quality of the electronic shop. Therefore, 50 was determined as the minimum value that can be given to assess the importance of heuristics. Then, all ten DMs individually ranked the heuristics according to their importance to the quality of the e-commerce websites. SMARTS weights provided by the DMs are provided in Table 2, and their normalised values are shown in Table 3.

It was also determined that online shops of the highest quality are those where the maximum number of points is given for heuristics trust (TR), reliability (RE), customer support (CS)_, empathy (EM), ease of site navigation (SN) and the accuracy of information (AI). The minimum number of points should be assigned for heuristics System response time (RT), the number of accessibility issues (AC) and the amount of outdated content (OC). 4.2. Data Collection and Construction of the Decision Matrices Since HEBIN is designed as the methodology that can be used by both the experts and the novice evaluators, 30 persons with different online shopping experiences were asked to assess the quality of the chosen e-commerce websites. Specifically, 5 UX experts, 5 IT professionals, 5 middle-aged persons (who do not work in IT industries) and 15 multimedia students participated in this study. Six global e-commerce websites (A1. Amazon.com, A2. Walmart.com, A3. Rakuten.com, A4. Ebay.com, A5. Aliexpress.com, A6. BestBuy.com) were analysed in the experiment, which was completed in January of 2019. Each of the participants assessed all the alternatives individually and then sent the prefilled questioners to the organizers of this study. When all the appraisals were collected, we analysed these responses as six different experiments designed to determine how HEBIN responds to the HE performed by different target groups (15 students were randomly divided into three groups of 5 people).

Although all the participants used the same checklist to judge the websites, individual assessments of the heuristics diverged. While the dispersion of the judgements gathered the UX experts was noticeably small, judgements collected from the novice evaluators were much more diverse. For instance, Trust (TR) of the alternative A5 got 17–18 points from UX experts; 14–18 points from IT professionals; 14–21 points from the persons who do not work in IT industries; 13–20 points from the first group of students; 10–19 from the second group of students and 10–20 points from the third group of students. Such inconsistency in the HE results might have a significant effect on the final rankings of the analysed alternatives. Therefore, six separate decision matrices X were constructed for each of the target groups. The decision matrix for expert based judgements is presented in Table 4. Another five matrices were generated in the same manner.

When the decision matrix X was constructed (Table 4), and the importance of the heuristics (weights) was determined (Table 3), the novel multicriteria decision-making approach MULTIMOORA-IVNS was applied to determine the final ranks of the alternatives. Elements of the initial decision matrix X calculated after the normalisation and the neutrosophication are presented in Table 5. The normalisation function that was applied is presented in Equation (10).

The first target (the interval-valued neutrosophic ratio system objective) was calculated by means of the Equations (11) and (12). Rankings for the first objective of the MULTIMOORA-IVNS are presented in Table 6.

The second objective of the neutrosophic MULTIMOORA approach was calculated as the deviation from the reference point and the min-max matrix. Equations (13)–(15) were applied to get the scores of the second objective:

maxi D_ri−_{wi xn*ij} =^{0.97070.98000.98670.97880.96260.9694T}

The third objective of MULTIMOORA –IVNS approach is presented as the matrix U, which is calculated by the Equation (16), where A_j is the product of criteria of the^jthalternative to be maximized and_Bj corresponds to the product of criteria of the^jth alternative to be minimized (Table 7).

Finally, the dominance theory was applied for the summarisation of all three objectives. The final ranks of the six international e-commerce websites are presented in Table 8.

Analogous calculations were done for each of the six decision matrices constructed from the data of the experiment. The final ranks of the analysed websites determined separately for each of the target groups are provided in Figure 2.

It can be seen that alternative A5 (Aliexpress.com) was recognized as the leader among the IT professionals and UX experts. A1 (Amazon.com) was identified as the best website for the professionals who do not work in IT industries, and A4 (Ebay.com) was detected as the best website for all three groups of multimedia students. However, it must be mentioned that the presented study was performed at the beginning of 2019, and currently, the quality of these websites might be altered. 5. Results and Discussion

Comparison of the MULTIMOORA–IVNS and MULTIMOORA–SVNS [14] was completed to analyse the credibility of the interval-valued neutrosophic sets. Since MULTIMOORA-SVNS works with single-valued numbers, the new decision-making matrices X’ were constructed, where intervalsmin_Hna;max_Hnawere converted to the single-valued numbers_xna’by a formula:

_xna’=int min_Hna+max_Hna2,

wheremin_Hnais the lowest estimate of the heuristics H_n andmax_Hna is the highest estimate of the heuristics H_n among all five evaluators that assessed the alternative a. The example of such a decision matrix constructed from the judgements of UX experts is presented in Table 9.

Analogously, decision matrices were constructed for the rest of the five target groups. Then, MULTIMOORA-SVNS approach [14] was applied for the ranking of the alternatives. Rankings calculated by MULTIMOORA-SVNS are presented in Figure 3.

Comparison of the MULTIMOORA–IVNS and MULTIMOORA–SVNS also disclosed that MULTIMOORA-IVNS provides high stability among the rankings calculated for all three groups of multimedia students (Figure 2). Such stability cannot be seen when MULTIMOORA–SVNS is applied (Figure 3). This finding suggests that interval-valued neutrosophic sets should be chosen when decision-makers are trying to understand how alternatives are ranked in the target group where assessors have a similar experience on the analysed topic. However, more studies should be performed to approve or negate this finding.

Additionally, the comparison of four different multicriteria decision-making approaches was completed for the sensitivity analysis. MULTIMOORA–IVNS, MULTIMOORA-SVNS [14], WASPAS-SVNS [52], and Crisp PROMETHEE [53] were applied for the comparison of rankings based on the data provided by UX experts (Table 4 and Table 9). The results presented in Table 10 displays high consistency in the alternative ranking regardless of the chosen MCDM method. This shows the reliability of MULTIMOORA–IVNS and also implies that the checklist presented in A1 was appropriately constructed for the assessment of the e-commerce websites.

6. Conclusions The novel multicriteria decision-making approach MULTIMOORA-IVNS (multi-objective optimisation by ratio analysis under interval-valued neutrosophic sets) was presented in this paper. The original quantitative heuristic evaluation methodology HEBIN that exploit IVNS theory was also presented in this paper. HEBIN under MULTIMOORA-IVNS is an easy-to-use approach that exploits the advantages of the interval-valued neutrosophic sets and reduces biases and instabilities caused by novice evaluators. In this study, HEBIN was used to assess the quality of the six international e-commerce websites. A comparison of the results provided by MULTIMOORA–IVNS and MULTIMOORA-SVNS revealed that MULTIMOORA-IVNS is a reliable MCDM approach, which shows its credibility when the distribution of the opinions in the group of the evaluators is growing.

Research Object	Museum Websites [29]	University Websites [30]	E-services of Websites [31]
Amount of heuristics	10	4	9
Amount of sub-heuristics	10	34	74
Amount of evaluators	5 field experts	2 usability specialists and 3 web experts.	80 experts from the IT and e-services domain
Amount of alternatives	47	3	21
Comparison metrics	Number of websites with the violated heuristic divided by the total number of websites	A total number of usability problems divided by the total number of pages investigated on the site.	Readiness index where indices are weighted by AHP and indicators are ranked by PROMETHEE.

Heuristic	DM1	DM2	DM3	DM4	DM5	DM6	DM7	DM8	DM9	DM10	Average Weight
TR	100	100	100	100	100	100	100	100	100	100	100.0
RE	80	75	85	85	65	80	90	85	75	65	78.5
CS	75	65	75	75	80	70	80	75	85	90	77.0
EM	70	70	80	70	70	75	75	80	90	85	76.5
SN	65	50	60	55	85	55	60	55	65	60	61.0
RT	90	80	90	80	75	95	85	90	70	75	83.0
AC	55	55	55	50	50	90	55	60	55	55	58.0
AI	85	60	65	65	55	60	65	65	80	70	67.0
OC	60	85	70	60	60	65	70	70	60	80	68.0

Heuristic	Optimum	Number of Sub-Heuristics	SMARTS Weight	Normalized Weight
Trust (TR)	MAX	9	100	0.149
Reliability (RE)	MAX	24	78.5	0.117
Customer support (CS)	MAX	10	77	0.115
Empathy (EM)	MAX	12	76.5	0.114
Ease of site navigation (SN)	MAX	10	61	0.091
System response time (RT)	MIN	4	83	0.124
Number of accessibility issues (AC)	MIN	3	58	0.087
Accuracy of information (AI)	MAX	7	67	0.100
Amount of outdated content (OC)	MIN	3	68	0.102
Total		82		1.000

Heuristic	Optimum	A1	A2	A3	A4	A5	A6
TR	MAX	[18; 19]	[14; 16]	[6; 7]	[15; 17]	[17; 18]	[13; 15]
RE	MAX	[29; 30]	[35; 36]	[26; 29]	[41; 42]	[41; 43]	[32; 34]
CS	MAX	[11; 13]	[8; 9]	[5; 6]	[8; 10]	[14; 17]	[12; 13]
EM	MAX	[23; 26]	[18; 19]	[13; 15]	[16; 17]	[24; 28]	[17; 18]
SN	MAX	[14; 16]	[14; 15]	[9; 11]	[18; 20]	[17; 18]	[14; 16]
RT	MIN	[16; 18]	[17; 19]	[66; 68]	[9; 10]	[11; 12]	[18; 20]
AC	MIN	[2; 3]	[0; 1]	[2; 3]	[2; 3]	[1; 3]	[0; 1]
AI	MAX	[12; 14]	[8; 10]	[7; 9]	[13; 14]	[12; 14]	[11; 13]
OC	MIN	[0; 1]	[1; 2]	[2; 3]	[0; 1]	[0; 1]	[2; 3]

Heuristic/Optimum	Alternatives
	A1	A2
TR max	{[0.0621, 0,0664], [0.9443, 0.9483], [0.9336, 0.9379]}	{[0.0462, 0.0540], [0.9559, 0.9633], [0.9460, 0.9538]}
RE max	{[0.0343, 0.0356], [0.9718, 0.9731], [0.9644, 0.9657]}	{[0.0427, 0.0441], [0.9639, 0.9652], [0.9559 0.9573]}
CS max	{[0.0287, 0.0347], [0.9726,0.9782], [0.9653, 0.9713]}	{[0.0202, 0.0230], [0.9826, 0.9839], [0.9770, 0.9798]}
EM max	{[0.0455, 0.0528], [0.9556, 0.9625], [0.9472, 0.9545]}	{[0,0341, 0.0363], [0.9711, 0.9732], [0.9637, 0.9659]}
SN max	{[0.0303, 0.0355], [0.9708, 0.9757], [0.9645, 0.9697]}	{[0.0303, 0.0329], [0.9733, 0.9757], [0.9671, 0.9697]}
RT min	{[0.0125, 0.0141, [0.9865, 0.9875], [0.9859, 0.9875]}	{[0.0133, 0.0149], [0.9860, 0.9869], [0.9851, 0.9867]}
AC min	{[0.0274, 0.0448], [0.9614, 0.9783], [0.9552, 0.9726]}	{[0.0061, 0.0127], [0.9891, 0.9939], [0.9873, 0.9939]}
AI max	{[0.0423, 0.0513], [0.9559, 0.9649], [0.9487, 0.9577]}	{[0.0263, 0.0340], [0.9726, 0.9799], [0.9660, 0.9737]}
OC min	{[0.0072, 0.0148], [0.9873, 0.9928], [0.9852, 0.9928]}	{[0.0148, 0.0319], [0.9747, 0.9873], [0.9681, 0.9852]}
	A3	A4
TR max	{[0.0183, 0.0216], [0.9815, 0.9831], [0.9784, 0.9817]}	{[0.0501, 0.0580], [0.9521, 0.9521], [0.9420, 0.9499]}
RE max	{[0.0303, 0.0343], [0.9731, 0.9769], [0.9657, 0.9697]}	{[0.0516, 0.0532], [0.9553, 0.9568], [0.9468, 0.9484]}
CS max	{[0.0123, 0.0149], [0.9866, 0.9879], [0.9851, 0.9877]}	{[0.0202, 0.0258], [0.9810, 0.9839], [0.9742, 0.9798]}
EM max	{[0.0237, 0.0277], [0.9792, 0.9823], [0.9723, 0.9763]}	{[0.0319, 0.0341], [0.9732, 0.9752], [0.9659, 0.9681]}
SN max	{[0.0184, 0.0230], [0.9826, 0.9861], [0.9770, 0.9816]}	{[0.0410, 0.0469], [0.9597, 0.9657], [0.9531, 0.9590]}
RT min	{[0.0607, 0.0631], [0.9457, 0.9486], [0.9369, 0.9393]}	{[0.0069, 0.0077], [0.9923, 0.9931], [0.9923, 0.9931]}
AC min	{[0.0274, 0.0448], [0.9614, 0.9783], [0.9552, 0.9726]}	{[0.0274, 0.0448], [0.9614, 0.9783], [0.9552, 0.9726]}
AI max	{[0.0226, 0.0301], [0.9763, 0.9834], [0.9699, 0.9774]}	{[0.0467, 0.0513], [0.9559, 0.9608], [0.9487, 0.9533]}
OC min	{[0.0319, 0.0521], [0.9552, 0.9747], [0.9479, 0.9681]}	{[0.0072, 0.0148], [0.9873, 0.9928], [0.9852, 0.9928]}
	A5	A6
TR max	{[0.0580, 0.0621], [0.9483, 0.9521], [0.9379, 0.9420]}	{[0.0425, 0.0501], [0.9596, 0.9521], [0.9499, 0.9575]}
RE max	{[0.0516, 0.0548], [0.9539, 0.9568], [0.9452, 0.9484]}	{[0.0384, 0.0412], [0.9666, 0.9568], [0.9588, 0.9616]}
CS max	{[0.0379, 0.0477], [0.9604, 0.9696], [0.9523, 0.9621]}	{[0.0317, 0.0347], [0.9726, 0.9696], [0.9653, 0.9683]}
EM max	{[0.0479, 0.0580], [0.9500, 0.9602], [0.9420, 0.9521]}	{[0.0319, 0.0341], [0.9732, 0.9602], [0.9659, 0.9681]}
SN max	{[0.0382, 0.0410], [0.9657, 0.9683], [0.9590, 0.9618]}	{[0.0303, 0.0355], [0.9708, 0.9683], [0.9645, 0.9697]}
RT min	{[0.0084, 0.0092], [0.9908, 0.9916], [0.9908, 0.9916]}	{[0.0141, 0.0158], [0.9856,0. 9916], [0.9842, 0.9859]}
AC min	{[0.0127, 0.0448], [0.9614, 0.9891], [0.9552, 0.9873]}	{[0.0061, 0.0127], [0.9891, 0.9891], [0.9873, 0.9939]}
AI max	{[0.0423, 0.0513], [0.9559, 0.9649], [0.9487, 0.9577]}	{[0.0381, 0.0467], [0.9608, 0.9649], [0.9533, 0.9619]}
OC min	{[0.0072, 0.0148], [0.9873, 0.9928], [0.9852, 0.9924]}	{[0.0319, 0.0521], [0.9552, 0.9928], [0.9479, 0.9681]}

Alternative	Qi	S(Qi)	Rank
A1	(0.9376; 0.9590; 0.0394; 0.0592; 0.0410; 0.0624)	(2.8160; 2.8786)	4
A2	(0.9436; 0.9649; 0.0353; 0.0514; 0.0351; 0.0564)	(2.8358; 2.8944)	3
A3	(0.8226; 0.8579; 0.1233; 0.1588; 0.1421; 0.1774)	(2.4864; 2.5925)	6
A4	(0.9456; 0.9660; 0.0319; 0.0512; 0.0340; 0.0544)	(2.8400; 2.9001)	2
A5	(0.9458; 0.9754; 0.0249; 0.0515; 0.0246; 0.0542)	(2.8402; 2.9258)	1
A6	(0.9280; 0.9520; 0.0451; 0.0670; 0.0480; 0.0720)	(2.7891; 2.8589)	5

Alternative	S(Aj)	S(Bj)	Uj	Rank
A1	(0.0810; 0.1827) × 10−7	(0.0044; 0.0160) × 10−3	(0.0005; 0.0042)	3
A2	(0.0244; 0.0503) × 10−7	(0.0022; 0.0099) × 10−3	(0.0005; 0.0042)	5
A3	(0.0015; 0.0048) × 10−7	(0.0812; 0.2412) × 10−3	(0.00005; 0.00007)	6
A4	(0.0733; 0.1499) × 10−7	(0.0024; 0.0088) × 10−3	(0.0008; 0.0062)	4
A5	(0.2033; 0.4676) × 10−7	(0.0014; 0.0107) × 10−3	(0.0019; 0.0327)	1
A6	(0.0428; 0.0924) × 10−7	(0.0049; 0.0175) × 10−3	(0.0002; 0.0019)	2

Alternative	The Interval-Valued Neutrosophic Ratio System	The Neutrosophic Reference Point	The Neutrosophic Full Multiplicative Form	Final Rank
A1	4	3	3	3
A2	3	4	5	4
A3	6	6	6	6
A4	2	2	4	2
A5	1	1	1	1
A6	5	5	2	5

Heuristic ID	Optimum	A1	A2	A3	A4	A5	A6
TR	MAX	19	15	7	16	18	14
RE	MAX	30	36	28	42	42	33
CS	MAX	12	9	6	9	16	13
EM	MAX	25	19	14	17	26	18
SN	MAX	15	15	10	19	18	15
RT	MIN	17	18	67	10	12	19
AC	MIN	3	1	3	3	2	1
AI	MAX	19	9	8	14	13	12
OC	MIN	1	2	3	1	1	3

Alternative	Proposed Method	MULTIMOORA-SVNS	WASPAS-SVNS	Crisp PROMETHEE
A1	3	3	3	3
A2	4	4	4	4
A3	6	6	6	6
A4	2	2	2	2
A5	1	1	1	1
A6	5	5	5	5

Author Contributions

Conceptualisation, I.L., E.K.Z., R.B. and A.U.; methodology, I.L., E.K.Z., R.B. and A.U.; software, I.L. and R.B.; validation, I.L., E.K.Z., R.B. and A.U; formal analysis, I.L. and R.B.; investigation, I.L., R.B., E.K.Z. and A.U.; resources, I.L. and A.U; data curation, I.L. and R.B.; writing-original draft preparation, I.L, R.B. and A.U.; writing-review and editing, I.L., E.K.Z., R.B. and A.U.; visualisation, I.L.; supervision, R.B and E.K.Z.; project administration, E.K.Z., I.L. and R.B. All authors have read and agreed to the published version of the manuscript.

Funding

This research received no external funding.

Institutional Review Board Statement

Not applicable.

Informed Consent Statement

Not applicable.

Data Availability Statement

Data sharing not applicable.

Conflicts of Interest

The authors declare no conflict of interest.

Appendix A

Table

Table A1.Heuristics and the sub-heuristics for the evaluation of e-commerce website quality.

Table A1.Heuristics and the sub-heuristics for the evaluation of e-commerce website quality.

ID	Heuristics/Sub-Heuristics	Evaluation Scale
Trust (TR)
TR1	Does the website reputation create trust?	1-5
TR2	Are there product-related ratings and reviews?	0-2
TR3	Is there an opportunity to comment or react to other reviews?	0-2
TR4	Are the buyers allowed to indicate the usefulness of other reviews?	0-2
TR5	Is the connection secure?	0-2
TR6	Does the website show Security Certificates provided by external companies?	0-2
TR7	Are there any trust logos associated with the shipment and payment?	0-2
TR8	Does the website give information about the company or presents a link to it?	0-2
TR9	Is there a Privacy Policy available from all the pages?	0-2
Reliability (RE)
RE1	Does the appearance of the website look safe and reliable?	0-2
RE2	Do the product page layout and design assist in information understanding?	0-2
RE3	Are there integrated tools that help to compare different products?	0-2
RE4	Is the interface's style consistent?	0-2
RE5	Is there enough information about product availability in stock?	0-2
RE6	Do the product pages show the number of products already sold?	0-2
RE7	Does the website provide information about the countries where the shipments are allowed?	0-2
RE8	Are there enough options for the delivery of the order?	0-2
RE9	Are the delivery dates of the separate goods or the total order provided?	0-2
RE10	Are there enough payment options provided on the website?	0-2
RE11	Are the available payment methods shown in every product page?	0-2
RE12	Is there the possibility to return the products?	0-2
RE13	Is the return or exchange policy available on the website?	0-2
RE14	Is there a shopping cart accessible from all the pages?	0-2
RE15	Is it easy to modify the number of products in the shopping cart?	0-2
RE16	Are the additional charges (taxes and shipping costs) shown as soon as possible?	0-2
RE17	Is there a possibility to purchase goods without registration?	0-2
RE18	If the registration is necessary, is the process quick and require only the fundamental information?	0-2
RE19	Is the button confirming the purchase clearly visible in the interface?	0-2
RE20	Is the checkout process divided into logical and easy understandable steps?	0-2
RE22	Is the progress indicator shown in the checkout process?	0-2
RE23	Is it possible to track the status of the orders?	0-2
RE24	Is there a possibility for the registered users to modify or cancel the order?	0-2
Customer support (CS)
CS1	Has the website a Help Center or specific area devoted to Frequently Asked Questions?	0-2
CS2	Has the website any Intelligent Agents that assist in the purchasing process?	0-2
CS3	Is the customer support available 24/7?	0-2
CS4	Is the customer support available with and without login?	0-2
CS5	Does the website provide different ways to contact the company?	0-2
CS6	Does the website provide distinct contacts for the different types of questions?	0-2
CS7	Does the website support different scenarios for the order completion?	0-2
CS8	Does the company respond to comments and concerns expressed by customers?	0-2
CS9	Does the website send an email to confirm the order?	0-2
CS10	Is the error messages clear and informative?	0-2
Empathy (EM)
EM1	Does the website look innovative and attractive?	0-2
EM2	Does the company care about customers opinions?	0-2
EM3	Does the website personalise contact with the customer?	0-2
EM4	Is there a possibility to choose the currency in which the prices are shown?	0-2
EM5	Is there a possibility to choose a language in which the page is shown?	0-2
EM6	Does the website use appropriate multimedia to draw customers attention?	0-2
EM7	Are new products, discounts or special offers properly advertised?	0-2
EM8	Does the website offer recommendations for other products?	0-2
EM9	Are the recommendations related to the selected product?	0-2
EM10	Does the website provide a Wishlist?	0-2
EM11	Can customer add items to the Wishlist without registration?	0-2
EM12	Does the website provide an opportunity to become a VIP customer?	0-2
Ease of site navigation (SN)
SN1	Is the hierarchy of categories well-organised and help to find the products?	0-2
SN2	Is the navigation obvious enough in the related sections?	0-2
SN3	Are the titles of sub-pages appropriate and descriptive?	0-2
SN4	Do the pages and sub-pages support orientation control tools?	0-2
SN5	Are the call to action buttons clearly visible on the website?	0-2
SN6	Are the appropriate filters provided in the Category pages?	0-2
SN7	Does the website provide a search box to find the products and the information?	0-2
SN8	Has the website the additional possibilities to elaborate search results by features, categories, etc.?	0-2
SN9	Does the search engine deliver expected results?	0-2
SN10	Do all links work properly?	0-2
System response time (RT)
RT1	How long does it take to launch the homepage of the website?	Seconds
RT2	What is the homepage download size?	MB
RT3	How long does it take to launch the product page of the website?	Seconds
RT4	What is the product page download size?	MB
Number of accessibility issues (AC)
AC1	Are there any difficulties to open the website on the computer screen?	0-2
AC2	Are there any issues to see the website on mobile phones?	0-2
AC3	Are there any issues, that makes it difficult to use the site for persons with disabilities?	0-2
Accuracy of information (AI)
AI1	Is the content based on the users' needs instead of being based around the product description?	0-2
AI2	Is there enough information about products and services?	0-2
AI3	Is there enough information about the purchasing process?	0-2
AI4	Is the information about the products accurate and convincing?	0-2
AI5	Is the information about the products free of spelling errors?	0-2
AI6	Does the website use appropriate multimedia to describe goods and services?	0-2
AI7	Are the pictures correctly shown in an appropriate quality?	0-2
Amount of outdated content (OC)
OC1	Is there a big difference between the current year and the website update year shown on the website?	0-2
OC2	Are the latest comments about the distinct products or the whole website obsoleted?	0-2
OC3	Are unavailable or sold-out items shown to the customer?	0-2