1. Introduction
In the last 50 years, many authors have suggested project selection approaches for different topics such as healthcare, construction, and the public sector. One of the most relevant topics is R&D, with many relevant articles indexed in scientific databases. Thore [1] suggests that, by the end of the 20th century, as a consequence of the unbridled increase of communication and information technology, a new economy has arisen, recognizing it as the Knowledge economy. The drivers of this latest economy are the R&D projects. That is why R&D managers constantly need to develop systems and procedures, which will improve the likelihood of success of their business. According to UNESCO Institute for Statistics (UIS), the yearly global spending on R&D projects reached a record of almost US$ 1.7 trillion in 2018 [2]. While poor and middle-income countries invest in R&D less than developed countries, the numbers are bettering in some cases [3]. During the COVID-19 pandemic, R&D investment may increase even more, at least for the largest R&D spenders, who tend to increase R&D investment during financially struggling times, such as the worldwide financial crisis in 2008 [4].
In Brazil, for example, organizations have invested increasing amounts of money in R&D over the last two decades, despite the negative gradient in the last two years. The US$ 21 billion they disbursed in 2016 is almost six times the amount of that in 2000. Slightly more than half of these values come from public sources, which makes the Brazilian government the main engine for R&D and innovation in the country [5]. This proportion is quite different in developed countries such as Germany, Japan, and the United States, where during standard times less than 30% of the total invested in R&D comes from the public purse [6].
Regardless of who controls the company, they will consolidate R&D investments mainly by implementing projects, whose selection occurs according to their alignment to the organization’s goals [7]. However, the associated risks in performing R&D projects have proved to have a significant impact since the choice of unsuitable projects may result in a consequential loss of financial and human resources [8,9]. In this case, the impact of corporate strategy is usually perceived correctly in the selection of R&D projects [10]. Thus, joining all projects with the organization’s strategic direction is crucial to utilizing the resources better [11,12,13]. We know this practice as R&D Project Portfolio Selection (PPS), a branch of the Project Portfolio Management (PPM) field of study. R&D PPS is commonly a multicriteria process, with model and criteria selection processes as two of its main steps [14].
The decision-making process in R&D PPS is similar to decision-making in other domains. Therefore, the decision-making frameworks do not conceptually change depending on the portfolios’ characteristics and application domain. However, the used selection methods have changed, and many scientific papers address diverse methodologies for R&D PPS. The main differences and challenges in R&D PPS are (a) the expenditure in projects expresses sizable investments; (b) the enterprises make those investments in their future; thus, (c) the projects need be tied to the corporate strategy; and (d) the R&D projects returns have extended lead times, are risky and multidimensional; (e) the environment is rough, and the results changeable [15,16,17]. These unique characteristics make it challenging to perform suitable or optimal decisions.
On the other hand, R&D PPS still have difficulties that are shared by PPS in other fields. Commonly, the selection process may consider:
A big portfolio, with several projects [18,19,20];
Qualitative and quantitative data [21,22];
Uncertainty generated by imprecise information [23,24,25];
Uncertainty generated by limited data [26,27,28];
Multiple interdependent and/or conflicting criteria (attributes and/or goals) [29];
Interdependence and interrelation among projects [30,31,32];
Mutually exclusive projects or cannibalization [9,33,34,35,36];
Resource constraints [37,38];
The optimal schedule [39,40,41];
Human resource allocation [42];
MCDM methods are recurrent on R&D PPS by assisting the decision-makers in ranking and choosing the most suitable alternative based on several conflicting criteria. Regarding the methods used in PPS, MCDM methods are the highly scientifically investigated approaches. MCDM holds the decision-makers in ranking or electing the best alternatives based on numerous, sometimes conflicting, criteria. They cover from plain [43] to elaborate approaches [17], from typical [44] to singular [45], and from single [16] to combined ones [46].
The importance of the R&D PPS subject, to the best of our knowledge the literature gap remains in contributing a broad picture of the role of MCDM methods in R&D PPS through classifying, comparing, and analyzing the various MCDM approaches used. We aim to explore the area and provide a state-of-the-art reference of MCDM-based R&D PPS. This paper aims to systematically collect and analyze papers published on the subject from 1970 to 2020 and are available through two best-known databases available: Scopus and Web of Science Core Collection. Recently, similar approaches have been used in other fields of study [47,48].
The main innovations of this paper are in the extension of the explorations and analysis performed, and on the perspectives put on the R&D PPS field of study. We also position our investigation in the light of other up-to-date studies. For instance, Afshari [49] highlights the importance of the criteria used during project selection processes and advocates for the creation of systematic frameworks that may help decision makers on finding the most suitable criteria. Almeida et al. [50] indicate the need for scientific material that assist researchers on selecting MCDM methods for a given application. Souza et al. [51] discuss the importance of knowing the R&D PPS ambience to facilitate finding similar approaches that have already been developed. Considering this, the research questions addressed in this review are:
RQ1: Methods. Which MCDM methods are used in R&D PPS? Which is the nature of their alternatives? Are the methods used as individual or integrated approaches? What are the most frequently used MADM (Multi-attribute decision making) and MODM (Multi-objective decision making) methods? How has the usage of those methods changed with time? How are they used? How do they consider uncertainty?
RQ2: Portfolios. How big are they? Which application domains are the most explored? Which software, solvers or programming languages are employed in the selection process? How much attention do the papers give to the criteria used?
RQ3: Research Field. Can the publication timeline be split into periods of theory intensification? Which are the most cited articles? Who are the top authors? Where are they from?
RQ4: Whole data. Which data are correlated? Which conclusions can be made by looking at those correlations?
RQ5: Criteria used. Which criteria are used by the authors? Are all those criteria expressing different perspectives? Or could they be summarized into a smaller list of criteria?
RQ6: Research opportunities and trends. Which extensions of previous works could be done? Which research opportunities could be explored?
In this work, we attempt to answer those questions through a full extent mapping of approach’s natures, integration’s approach, papers correlations analysis, software’s types, portfolio’s size, uncertainty models, list and explain the criteria and bibliometric analysis. Hopefully, these findings of our analysis will be beneficial to the community of academics and practitioners in R&D PPS.
The remainder of this paper is arranged as follows: Section 2 introduces the methodology and presents the foundations underlying the next three sections and systematically reviews the literature on MCDM-based R&D PPS. We present a timeline of the utilization of the main methods and perform a broad classification of all MCDM methods used in the last 50 years. We categorized them according to the nature of alternatives (MADM—Multi-attribute decision making—and MODM—Multi-objective decision making), type of integration approach (Individual and Integrated), the MCDM method itself and types of uncertainty (Deterministic, Probabilistic and Fuzzy). The portfolios’ classification follows the application domain and number of projects. In Section 3, we present the criteria used by the SLR papers. We also propose a smaller list of criteria that, at least for the main five Brazilian R&D public organizations, represent the perspectives which are explored by the 263 criteria proposed in the literature. We present a bibliometric analysis of the considered papers in Section 4, which provides trends on the topic and information about papers, authors, and their countries. A short Section 5 highlights opportunities in MCDM-based R&D PPS. Finally, Section 6 presents the conclusions.
2. Methodology
Our paper reports a comprehensive literature review about MCDM-based R&D PPS. The research methodology follows the recommendations of Rowley and Slack [52] on how to conduct a systematic literature review, and the PRISMA framework [53] (Preferred Reporting Items for Systematic Reviews and Meta-analyses). We also based our work on the frameworks proposed by Jahangirian et al. [54], Diaby et al. [55]. In addition, we adopted and expanded it to pre-search steps, as shown in Figure 1 and the Figure A1 in the Appendix. The purpose of our work is also similar to the ones conducted by Harrison et al. [47] and Pourhabibi et al. [48] that recently proposed an SLR of portfolio optimization for defense applications. Nevertheless, we addressed it to a different field of study, and we applied different researches tools, such as the online application Parsifal® as a checklist to guide the SLR process, and we also used the Prisma checklist and the Prisma abstract checklist, as shown in the supplementary materials. The PRISMA statement intends to assist authors in improving the reporting of systematic literature reviews through a 27-item checklist [53].
Firstly, we perform an exploratory search to obtain the most cited articles on MCDM bibliometric analysis and literature reviews [56,57,58,59,60,61,62,63,64,65,66]. We got from these articles many domains and several keywords associated with the MCDM searches. These keywords are acronyms, synonyms, and equivalents words to MCDM and its highly cited methods. Afterward, we combined them with keywords related to R&D and PPS. Thus, we used a total of 134 keywords that gave a total of 2604 Boolean combinations to find articles related to MCDM-based approaches in R&D PPS. The articles were found according to their title, abstract, and article keywords. The articles are filtered according to Table 1. The searching keywords, which were taken from [67], can be seen in Table 2.
We performed this search in the two main widespread databases available: Scopus®, the largest multidisciplinary database, including approximately 15,000 peer-reviewed journals and over 4000 publishers [54]; and Web of Science® Core Collection, a database that includes around 10,000 peer-reviewed journals and it was for years the only citation database covering all scientific research domains [68]. However, more articles can be found outside of those databases; the scope of this paper was restricted to only papers available in those two. The search started in 1 January 2019, and the last update finished in 26 January 2021. A total of 314 results from 1970 (the year when the first article is dated) to 2020 could be found. From those, we have considered only non-duplicated articles in English and peer-reviewed published journals for the next steps. Then, we performed three screening steps. Firstly, the titles were analyzed, and we rejected articles that did not suit the scope of this work. Afterward, articles were rejected based on their abstracts and subsequently on the full text. A total of 66 articles were finally selected (see Appendix on Table A1 and Table A2). Table 1 shows the inclusion and exclusion criteria as well as the corresponding screening steps. Every step also considers the inclusion and exclusion criteria of subsequent steps. For pattern, authors 1 and 2 independently reviewed all the studies, then reached a final database consensus, as shown in Figure 1. Moreover, as an example of the selection process, although the studies of Huang and Chu [69], and Souza et al. [51] might appear to match the inclusion criteria, we excluded them because they did not focus on R&D subject.
MCDM approaches classification variation as the classification criteria. Concerning the nature of the alternatives, they are Multi-Objective Decision-Making (MODM) and Multi-Attribute Decision-Making (MADM), or an aggregate of both (see Figure 2). MODM methods have no predetermined options, and the optimal option selection depends on an infinite and continuous number of circumstances subjected to a set of constraints. Generally, MODM methods include mathematical approaches, i.e., integer linear programming, goal programming, linear programming, integer non-linear programming, multi-objective programming [58]. On the other hand, MADM methods deal with a discrete and finite number of options designated by a predetermined set of criteria. Thus their main task is to achieve a reasonable selection, assessment, and grading among the viable possibilities [66]. AHP (Analytical Hierarchy Process) and TOPSIS (Technique for Order Preference by Similarity to Ideal Solution) are relevant examples of MADM methods. Therefore, some authors refer to MODM and MADM problems as continuous and discrete problems, respectively [59]. Notwithstanding, PPS problems use knapsack problems solutions since the latter involves just discrete input data for each project. Thus, MODM methods are commonly constrained to work with discrete alternatives. Therefore, the classification in MADM and MODM considers the widespread application of the method. In this work, we follow the classification made by Chai et al. [70], that includes DEA as an MODM method.
According to their methodology, MCDM methods classification follows the individual methodology approach and integrated methodology approach, which depend on the number of methods integrated into them (see Figure 3) [57]. The work of Meadeand Presley [16] is a pertinent example of an individual methodology approach using ANP (Analytical Network Process), a MADM method. On the other hand, Bard et al. [44] introduce 0–1 integer programming as an individual MODM approach method to select the optimal project portfolio. Liberatore [15] shows how to integrate MADM and MODM methods by coupling AHP and 0–1 integer linear programming into an integrated approach. The proportion between the type of integration approach also seems to be constant over all periods analyzed. However, the integrated methods have changed: today, we increasingly integrate two or more different MADM methods (MADM-MADM integration). Indeed, until 1995 there were only MADM-MODM and MODM-MODM integrations. Another possible analysis is that papers addressing more than one model or comparing models do not appear too often as in the past. For instance, the last paper addressing both individual and integrated approaches dates 1988. It also reflects on the greater acceptance for specific articles today, rather than the generalist ones.
It is worth mentioning that the usage of MODM methods as Individual Approaches and Integrated Approaches presents a moderate positive Pearson correlation coefficient (0.54) and a moderate negative correlation coefficient (−0.52), respectively. Generally, those articles employ only a few criteria, which are used as objective functions or constraints to the problem. MADM methods have a leading role in integrated approaches: criteria weight. In this case, a considerable number of papers do not even explain the criteria used, which is also pointed out by the moderate negative correlation (−0.51) between Linear Programming (So far, the most frequently used MODM method) and the presence of explained criteria on the paper. The information presented by Figure 3 is desegregated by year in Figure 4. It makes it visually easy to observe the increasing in use of integrated approaches in the last two decades, specially during the last five years.
Observe that the pattern regarding the nature of alternatives changes over the years, Figure 5. From 1970 to 1995, the only methods used were MODM and MADM/MODM. That period coincides with the publication of the first PMBoK (Project Management Body of Knowledge), in 1996 [71], and will define here the first period of theory intensification. There appeared many forms of 0–1 integer programming in several articles [13,15,31,42,44,72]. In the same period, AHP was the most integrated MADM method, exclusively with 0–1 integer linear programming and by Liberatore [12,13,15]. From 1996 to the present, the second period of theory intensification saw the emergence of individual MADM, and integrated MADM-MADM approaches. AHP and its variations were the most frequently used individual methods [73,74,75,76], followed by ANP [16,29] and ROA (Real Option Analysis) [22,23,25]. Regarding Integrated MADM-MADM approaches, there is a variety of combinations with commonly used methods, such as AHP and DEA [10,43]; TOPSIS [21,77,78]; and DEMATEL (Decision Making Trial and Evaluation Laboratory) [46,79,80]. Figure 6 and Figure 7 introduce the most frequently used MADM and MODM methods, respectively. Notice that Figure 6 shows only the most frequently used methods; other methods correspond for 14.9% of the total. The meaning of the main MADM methods acronyms are in Table 3, as well as their first reference on literature.
Among all MCDM methods for R&D PPS, AHP is the most relevant one, appearing in 15 papers. From those papers, 14 have their scopes presented and properly explained by Souza et al. [51], which presented an integration of Fuzzy-AHP Extent Analysis and Fuzzy-DEMATEL to criteria selection in R&D. The only paper not covered by Souza et al. is the one from Samanlioglu et al. [94], that presented an integration of hesitant F-AHP and hesitant F-VIKOR, in order to evaluate and rank innovation projects. In this case, hesitant F-AHP is applied to get fuzzy evaluation criteria weights, and hesitant F-VIKOR is applied to rank innovation project options. For completeness, we will not reproduce here the timeline of AHP in R&D Project Selection. We will focus on the other methods that are not explained by Souza et al. [51].
Another MCDM method similar do AHP is ANP, which appears in 8.8% of the articles. Mohanty et al. [95] show a utilization of fuzzy ANP (analytic network process) accompanying with fuzzy cost examination in selecting R&D projects, aiming to overwhelm the uncertainty in the preferences. The approach is interactive and built on two sets of critical factors. Initially, they screened projects to check if they were acceptable and reasonably progressing toward completion. Then, the projects which failed the test are finished, and the projects remaining are weighed with candidate projects to determine which one should be included in the portfolio. Meade and Presley [16] discussed the use of the Analytic Network Process (ANP) and presented a generic model based on many factors and criteria to support different situations. Jung and Seo [29] explored the analytic network process (ANP) approach applied for the evaluation of R&D projects that are components of programs with complex objectives. Jeng and Huang [79] proposed a decision model for assessing a project portfolio at the beginning initiation stage, including a modified Delphi method (MDM), decision-making trial and evaluation laboratory (DEMATEL) method, and analytic network process (ANP). Mohaghar et al. [21] presented an integrated fuzzy approach, with Fuzzy-ANP and Fuzzy-TOPSIS, for selecting R&D projects. DEMATEL and ANP are moderately correlated in the articles (+0.55). It is justified by using the influence matrix given by DEMATEL as an input to ANP or Fuzzy-ANP.
As frequently as ANP, ROA (Real Option Analysis) is also well-used to select R&D Project Portfolios, appearing in 8.8% of the papers. Its first usage in the topic dates 2006 [25], remaining used until 2014 [23]. It is mainly used as a side method in integrated approaches along with other MADM or MODM Methods. It is interesting to mention that all applications of this method are given in the fuzzy environment, with the most realistic option valuation given by a fuzzy pay-off method [24,96,97].
TOPSIS/Fuzzy-TOPSIS appears five times and DELPHI method and CBA/Fuzzy-CBA (Cost-Benefit Analysis) were used four times each, followed by DEMATEL, Scoring/Fuzzy-Scoring methods and Grey Theory (three times each) and then MAUT (Multi-Attribute Utility Theory), BSC (Balanced Scorecard) and Stochastic Dominance (two times each). Other methods with one apparition each sum up for 14,9% of the MADM applications. In the case of CBA, it mainly appears as an auxiliary method in Integrated Approaches and shows a strong positive correlation (+0.86) with articles that introduce more than one approach to select project portfolios. A similar side role is performed by BCG (Boston Consulting Group) Matrix, BSC, and Scoring methods.
We can classify the MCDM methods in many ways [50]. A traditional classification divides them into unique criteria of synthesis methods, outranking methods, and interactive methods. We may also classify it into compensatory or non-compensatory methods. In this case, the preference relation will be compensatory if there are trade-offs among criteria and non-compensatory otherwise. Generally, the unique criterion of synthesis methods is also compensatory.
On the other hand, outranking methods use non compensatory rationality. Interactive methods cover the whole spectrum of MODM methods. If we look only at MADM methods, the most classic ones can fit into four categories [98].
-
Multi-criteria value functions. The methods are commonly based on a value function, obtained by weighted summation or weighted multiplication. The criteria weights are non-negative and sum to 1. If weighted multiplication is used then criteria will be non compensatory, where a zero score on any individual value will result in an overall zero performance score.
-
Outranking approaches. Those methods generally involve the identification of every pair of decision options i and giving pairs in total. Outranking approaches also apply some utility function, containing criteria weights.
-
Distance to ideal point methods. These methods calculate ideal and anti-ideal values for the criteria. Then, decision options that are closest to an ideal solution are preferred, while decision options closest to the anti-ideal solutions are avoided. The concept of Euclidean distance are normally adopted.
-
Pairwise comparison methods. These methods compare each unique pair’s criteria and alternatives, giving comparisons. The comparisons are made between criteria and also between decision options.
We compare those categories in Table 4, with advantages and disadvantages [99]. Notice that Table 4 lists only the methods used by the articles considered in this literature review. Thus, well known outranking methods, such as ELECTRE (in French: ELimination Et Choix Traduisant la REalité—in English: Elimination and Choice Expressing Reality) and PROMETHEE (Preference Ranking Organization Method for Enriched Evaluation) families do not appear in R&D PPS context, which may be interesting in some occasions. Other not used and well know Distance to ideal point method is CP (Compromise Programming).
It is worth mentioning that only a few articles give proper explanations of why they have chosen a specific MCDM method in their R&D PPS context [26,100]. In fact, to the best of our experience, there is no framework available in the literature that helps researchers select the best methods in each PPS case and other MCDM applications [50].
The classification used in Figure 7 is the same used by Chai et al. [70]. Linear programming is the most frequently used MODM method with 17 appearances. Subsequently, 10 of them correspond to integer approaches and four to mixed-integer approaches. Multiobjective programming is the second most frequently used MODM method, appearing in nine papers. From those, seven are integer approaches, and only one has used non-linear data. DEA appears in five papers, followed by Non-Linear programming, used in four papers, with two integer approaches and one mixed-integer approach. Stochastic and Goal Programming appeared twice each. Goal programming, Linear and Non-Linear models appeared together.
Primarily associated with AHP, mathematical models are also conventional approaches to select R&D projects. The 0–1 integer programming is the most frequently used one, appearing in 17 articles. Several other relevant papers use mathematical models. For instance, Wang and Hwang [17] formulate a fuzzy 0–1 integer programming model that can both manage uncertain and flexible parameters to define the optimal project portfolio. Bard et al. [44] employed 0–1 integer programming to evaluate both active and prospective R&D projects. They considered the full range of organizational, environmental, and technical concerns. Stummer and Heidenberger [32] describe a three-phase approach to support managers in reaching the most attractive project portfolio. First, it identifies worthy project proposals for further evaluation, keeping the number of projects within a manageable size for entering the subsequent phase. Second, a multiobjective integer linear programming model determines the solution space of all efficient portfolios. Third, it aims to find a portfolio that fits the decision-maker notions. Carlsson et al. [101] developed the trapezoidal fuzzy numbers estimation of future cash flows to value the alternatives on R&D projects. They presented a fuzzy mixed integer programming model and discussed how to use their methodology to support optimal R&D project selection in a corporate environment. Czajkowski and Jones [31] proposes a decision support modeling framework for multi-project technology planning and project selection using 0–1 integer programming in which technical and explicitly assess the benefit interactions. Sun and Ma [41] developed and applied a heuristic packing-multiple boxes (PMB, or multi-knapsacks-model) model, based on several 0–1 integer programming methods, to pronounce both selecting and scheduling R&D projects.
Data Envelopment Analysis (DEA) appeared in 8% of the papers. Besides the work of Rabbani et al. [36], two more articles use DEA among their methods. Eilat et al. [33] developed an extended version of Data Envelopment Analysis (DEA) by integrating a balanced scorecard and the DEA itself for R&D project evaluation. Oral et al. [102] proposes a methodology for evaluating and selecting R&D projects. While the evaluation process relies on the DEA method, the selection process uses an ordinal scale throughout a method that uses model-based outranking.
A genetic algorithm is the main metaheuristic used to solve the mathematical models, appearing in 5% of the articles. Bhattacharyya et al. [30] presented a fuzzy multiobjective programming approach to aid the decision-makers in dealing with interdependences and uncertainty in R&D project selection. They presented a case study to demonstrate the proposed method where the solution is provided by a genetic algorithm (GA) and multiple objective genetic algorithms (MOGA). Eshlaghy and Razi [34] proposed a new approach of outranking relation in MCDA methods together with a data mining method for clustering and ranking the best R&D projects in a portfolio, presenting then a two-phase decision model for project portfolio selection problems. In the first phase, the clustering R&D projects in a portfolio with the most suitable projects specification combination uses a k-means algorithm. In the second phase, grey relational analysis (GRA) selects and evaluates the most efficient project in any cluster. Finally, a genetic algorithm (GA) calculates the Pareto front rank. Stewart [28] presented a solution for the project portfolio optimization problem using multiobjective programming and genetic algorithm.
For Malczewski [59], the classification of the methods should also follow the uncertainty related to the variables (see Figure 8). Thus, the decision-maker has to have a perfect comprehension of the decision environment for deterministic decision-making. If not, the probabilistic indication (lack of information) or fuzzy process (imprecision of semantic meanings). Interestingly, only deterministic approaches are the most frequently used ones since the R&D environment is turbulent and the results uncertain by nature.
Another feature of MCDM approaches is the full range of decision environments that have been employed over the last years [59]. Figure 9 highlights the main R&D PPS applications areas of the MCDM approaches.
Another interesting piece of information about the papers is the number of projects considered in the model. If the number of projects is excessively large, using pairwise methods, such as AHP, ANP, and DEMATEL, is not as good as to compare projects [12,15,103]. Another issue is that the largest the number of projects is, the higher the influence of uncertainty over the results will be. Thus fuzzy or stochastic approaches would be considered [9,26]. Figure 10 shows us the most common sizes of R&D project portfolios.
In Figure 11, we can observe that from 1970 to 1995, medium-sized portfolios accounted for 67% of all portfolios analyzed in the papers, against 22% of small-sized portfolios. However, from 1996 to the present small-sized portfolios has doubled its occurrence, representing 45% of all case studies addressed by the papers. This variation connects to the employment of MADM methods today, specially pairwise comparison methods, such as AHP and ANP. That also links to the more significant offer of software that facilitates the usage of MADM methods. Big-sized portfolios still represent fewer cases and may configure an opportunity to be explored by future papers, since it may represent the reality of many modern companies with big data-sets. In the case of big-sized portfolios, all papers use traditional MCDM approaches, except for Wei et al. [104], that used correlation analysis as an objective method.
In some articles, researchers have used software, solvers, or programming languages to implement MCDM methods in R&D PPS. Figure 12 illustrates the most frequently used computational approaches to solve PPS problems. Complementary to the data of Figure 12, Excel is the most frequently used spreadsheet software (8% of all computational approaches), Lingo/Lindo is the most frequently used solver (16% of all computational approaches), followed by Cplex (3% of all computational approaches). In the case of dedicated software, Expert Choice is the most frequently used one (8 % of all computational approaches). Some old software did have continuous use in the past, such as Lotus 1-2-3 and Steuer’s ADBase, both with three appearances each. Regarding programming languages, there are three no specified appearances, while Fortran, Pascal, and C++ appeared twice each one.
Notice that non-mathematical or easy-to-use models are not presented by the articles. In fact, this is a research opportunity for all PPS fields. According to Schiffels et al. [105], other companies seldom replicate quantitative approaches, and the same to black-box models in terms of acceptance by firms. Thus, managers frequently rely on simple decision rules, since easy-to-use approaches are not available. This opens a wide field of exploration, especially for small-profitable R&D companies, that cannot afford customized solutions.
Another relevant piece of information regards the criteria used by the approaches. Only 23 (35%) out of 65 articles explain the criteria used, while 42 articles (65%) do not explain the criteria used. Selecting and understanding the criteria used is a critical step in project portfolio selection and should not be avoided in real-world applications [49]. In fact, from all SLR articles, only Huang and Chu [69] and Souza et al. [51] propose methodologies for criteria selection in R&D PPS. In the first case, they present a Fuzzy-ANP for the Chinese government. In the second case, there is the integration between Fuzzy-AHP Extent Analysis and Fuzzy-based DEMATEL to find the best decision criteria in the case of the leading Brazilian R&D energy organization. This fact highlights potential research opportunities for future works and will be discussed with other opportunities later in this paper.
All classifications performed in this literature review are summarized by Figure 13 and we defined the methods and the applications in Table 5. Table 6, Table 7 and Table 8 present the articles that fall in each category.
3. The Criteria Used in R&D PPS
In these 66 articles, authors 1 and 2 extracted the criteria through spreadsheet support and independent collection; after that, authors 3 and 4 revised and validated the theoretical list of criteria. We found 263 criteria employed in different R&D PPS contexts. The list is extensive, varying from commonly used criteria, such as net present value and value of return [36,44], to more different criteria, such as reputability of the project manager or the number of papers that could result from the research [10,74]. We listed all criteria references used, which are also available in the supplementary file.
A criterion expresses a perspective, and each perspective is instantiated through multiple criteria [43]. The selection of each criterion has to be precise in order to avoid duplication, overlapping, and misalignment with the organization’s strategic goals. Thus, the selected criteria must be representative, significant, and indispensable for the project selection process [117]. In MCDM models, criteria may also appear as synonymous with attributes, objectives, or variables, which can be measured by quantitative/factual or qualitative/intangible data.
However, yet the 263 criteria are different in name and represented by distinct metrics, they express lesser scenario perspectives. Thus, in this paper, we summarize those 263 criteria in 23 criteria with broader scopes (or perspectives).
The grouping method is an adaptation of the one proposed by Jiro [118] and used in many grouping approaches [119,120]: the Affinity Diagram (also known as KJ Method, named after its author, Kawakita Jiro). By using the Affinity Diagram, we group criteria based on their natural relationships, obtained through brainstorming. Ten experts in R&D PPS acted in this process, being the steps described as follows:
Step 1: The criteria split in 263 digital cards, and a document referencing and explaining each one.
Step 2: Three experts received a third of the cards. Then the cards were placed in groups of affinity.
Step 3: The experts then accessed the formed groups. They could move the cards among groups by arguing with the others.
Step 4: A consensus occurred just when all three experts have stopped moving the cards.
Step 5: 27 groups of criteria were obtained, named, and described by the experts by using references from the 263 initial criteria.
Step 6: The next step was to verify this list. The verification intended to evaluate the internal consistency of the list of criteria. By consistency, we understand the lack of internal contradiction or intersection among the criteria. The verification step took into account the opinion of seven experts in R&D PPS. During the interview, they were asked to confront the actual list of criteria used by their organization with the 27 criteria proposed, and check if these new criteria could replace those already used. Their opinion consolidated within a single list of suggested of eventual modifications.
Step 7: We formed a new list consisting of 23 criteria that all ten experts accepted all changes.
Mathematical and computational grouping/clustering approaches were not employed, such as K-Means Method and Hierarchical Clustering Algorithms. They would require great effort from the experts to perform quantitative or qualitative judgements to each criterion, due to the number of criteria and the non-existence of clustering attributes that would cover all criteria. It also would happen with clustering approaches based on graphs, such as spectral clustering. For instance, directly creating a symmetric adjacency matrix (263 × 263) would require 34,453 comparison among criteria, which is not reasonable to be manually handled.
The new list of criteria is shown on Table 9 and Table 10. The numbers of papers that used the criteria appear in the Utilization column. We present the profiles of the ten experts in Table 11.
As complement to the list of experts, a short description of all five organizations are given bellow:
CNPq: The National Council for Scientific and Technological Development (CNPq) is a governmental agency belonging to the Brazilian Minister of Science, Technology, Innovations, and Communications (MCTIC). CNPq founded in 1951 with the main function of promoting scientific and technological research over the country. In 2015, CNPq invested US$ 623 million in R&D. From those, 87% was dedicated to research grants in Brazil and to Brazilians abroad.
FINEP: The Financing Institution of Research and Innovation (FINEP) is a Brazilian public organization attached to the MCTIC. Created in 1967 to promote innovation and R&D in Brazilian companies, universities, and public institutions. In 2018, FINEP invested US$ 250 million in innovative initiatives. Several Brazilian agencies rely on FINEP resources to sponsor R&D projects, such as BNDES and CNPq.
ANEEL: The Brazilian Electricity and Regulatory Agency (ANEEL) is an autarky established in 1997 below a special regime joined to the Brazilian Ministry of Mines and Energy. Its goal is to control the Brazilian electric sector. At the end of 2018, ANEEL approved a budget for energetic development in 2019 of US$ 5.2 billion. From 2008 to 2017, ANEEL produced around US$ 1.2 billion to finance R&D projects in the electricity sector and effectively utilized around 89%.
BNDES: The Brazilian Development set (BNDES) started its activities in 1952, and today is one of the largest development sets in the world. It is the Brazilian federal government’s largest instrument to finance long-term projects in all economic segments. Along with companies and public organizations, BNDES makes specific portfolio selections to promote innovation and national research and development. The last updated project portfolio selections available in its website add together resources up to US$ 4 billion.
ANP: The National Agency of Petroleum, Natural Gas, and Biofuels (ANP) was created in 1997 and is responsible for regulating the Brazilian activities in Petroleum, Natural Gas, and Biofuels. It is an autarky linked to the Brazilian Ministry of Mines and Energy. From 1998 to 2018, the agency has invested US$ 4 billion in R&D projects. Only in 2018, the total investment sum up US$ 147 million. Other organizations implement all ANP R&D investments in the segment. Petrobras, the 256th worldwide most innovative organization [4], was responsible for 74% of those investments in 2018.
4. Bibliometric Analysis
Based on the collected papers on MCDM methods employed in R&D PPS, in this section, we conducted a bibliometric analysis of the investigated papers. The data collected produce quantitative information about publications per year, the main authors, most productive countries, highly-cited papers, and leading journals on the topic. This bibliometric analysis intends to discover possible research trends and provide researchers and other practitioners a picture of the field. It is not intended to be a standard bibliometric analysis of the field, and we consider only the 66 previously discussed papers in our analysis.
Figure 14 compares the number of published articles on MCDM in R&D PPS over the years. We can graphically observe a warm-up in the field over 1982–1996 and a general increasing tendency in the number of published articles from 2000 to the present. We can also observe a statistically notable rise in the number of published papers from 1970 to 2019. It is expressed by the overall Person correlation coefficient ( and , for ). By analyzing the publication timeline in Figure 4, Figure 5 and Figure 11, the publication of the last three years, there is a tendency of concentration in MADM, integrated methodology approach, and small-size portfolio.
Figure 15 shows the top-ten first authors in R&D PPS. The numbers are presented in terms of yearly citations average.
Additionally, over the same period, the authors publishing from the United States have been the most productive ones as they have contributed to 19 out of the 66 papers published in the period. Other countries, such as Turkey, Taiwan, China, Iran, India, and Finland, also contributed significantly. In Figure 16, we have displayed only countries with four contributions or more. Other countries contributed with 15 papers, which represents around 25% of all publications. Nevertheless, if we consider the country region, the most productive region was Asia with 47% of the studies followed by America with 34.8%, the other regions contribute 9.1% or less.
In Table 12, the top ten papers are ordered by their total citations. Together, they sum 16% of all published articles, but contribute to 59% of all citations MCDM methods applied to R&D PPS. There is an indication of the number of citations per year, and the citation numbers collected from the database where the paper is available if the paper is available in both Web of Science and Scopus.
Observe that ranking articles based upon the total citation does not always match the average citation ranking. All top-cited papers appeared at least ten years ago. Typically, an influential paper establishes many citations after a while, such as Liang [100]. Over the review time frame, Figure 17 shows the leading journals on the R&D PPS MCDM applications. We highlight the first three journals. With 11 (17%) papers, the IEEE Transactions on Engineering Management contributed more, and the second was the European Journal of Operational Research with 5 (8%) papers, and the third was Omega with 3 (5%) papers.
5. Opportunities and New Paths
We also present opportunities that could be explored by researchers and practitioners of R&D PPS in their future works. Formulating research question is an appropriate way to highlight and guide future research, while preventing researchers from pursuing unnecessary and no longer used directions [121]. Thus, the formulation of clear research questions, derived from data and insights obtained from the papers in the literature review, is a guide for future work. The research questions divisions appear in two groups: research questions presented by recent articles (last three years) as opportunities for future works (last three years), and research questions that could not be answered by the articles considered in the SLR (see, Table 13). In the case of this last type of questions, all of them were prior discussed in their correspondent topics.
6. Conclusions
Many companies rely on R&D in order to have chances of standing in the market. However, since the resources are limited, only a few project proposals will integrate the company’s project portfolio. In this context, MCDM methods have risen as essential tools to help decision-makers select R&D projects. Hence, in this paper, we attended several objectives whose summary of findings we present in Table 14.
This paper has reviewed the existing research literature in MCDM-based R&D PPS in order to guide the community of academics and contribute to knowledge accumulation and creation concerning the usage of MCDM methods in R&D PPS. It also is a good starter reading for those that are taking the first steps in the topic. For this purpose, we summarized all approaches to MCDM used on the papers in R&D PPS from 1970 to 2020. We classified them according to their methods, nature, integration approach, and uncertainty related to the variables. There is a categorization of portfolios according to the application domain and size. Opportunities and possibilities of future works were discussed, and bibliometric analysis of the papers was also performed.
From 1970 to 2020, MADM and MODM methods emerged in similar proportions. However, MADM methods tend to be more used nowadays than in the past. Among MADM methods, AHP (23%) is the most employed one, followed by ANP (9%), ROA (9%), TOPSIS (8%), and DELPHI methods (6%). Linear integer programming (28%) is the most applied MODM approach. According to the uncertainty related to the variables, deterministic decision making is preferred over probabilistic and fuzzy approaches, appearing in 49% of all papers. Pharmaceutical and Electronic Electricity R&D domains are the most investigated ones, and each one is studied in 13% of the papers. Programming Language (21%) is the most commonly used way to solve R&D PPS problems, and the most studied portfolios (52%) are medium-sized portfolios.
Fuzzy sets are used in 27% of the papers. Despite the popularity of triangular fuzzy numbers among MCDM practitioners, in R&D PPS other membership functions and fuzzy environments are preferred. However, all fuzzy set applications are used as a way to tackle imprecise information, mainly originated from the use of verbal scales when using MADM approaches, and to facilitate the aggregation of experts judgments.
Likewise, from the SLR, we have collected 263 criteria used by the 66 articles about MCDM-based R&D PPS, published from 1970 to 2020. Those criteria were condensed into a shorter list of criteria, each one representing a different perspective. The whole process was conducted with the assistance of experts from five Brazilian R&D organizations that together manage portfolios valued around US$ 5 billion a year, which accounts for 38% of all Brazilian annual expenditures in R&D projects.
From the bibliometric analysis, we have also obtained relevant results. Responsible for 41% of all papers, the United States is the country with the most significant quantity of contributions to MCDM-based R&D PPS. However, Asia is the most active region in R&D PPS, holding 45% of all papers, followed by Americas with 35%. The top ten most-cited papers account for 59% of all citations in the area, and IEEE Transactions on Engineering Management is the journal publishing most of the papers (18%).
Limitations still need to be overcome in future works. For example, further research should consider scientific databases other than the Web of Science and Scopus, and although the extent Boolean combination was high, we may have missed some papers or important conference papers of R&D PPS. Snowball techniques are also suitable, as they may enlarge the number of papers used in the SLR/BA. The literature review we presented also did not investigate any methods other than MCDM methods, such as data-driven and logic-based approaches. Those also seem viable possibilities for future expansions of the scope of the work we presented.
In this research, all authors contribute in some way. D.G.B.d.S. and E.A.d.S. screened each record and were responsible for almost all investigation, writing, and original draft preparation. N.Y.S. and C.E.S.d.S. did the conceptualization, validation, and supervision. All authors have read and agreed to the published version of the manuscript.
This work was supported in part by the National Council for Scientific and Technological Development (CNPq) of Brazil, and in part by the Research Supporting Foundation of Minas Gerais State (FAPEMIG) of Brazil, and in part by the São Paulo Reaserch Foundation (FAPESP) of Brazil.
Not applicable.
Not applicable.
The data is contained within the article and addressed in
Dedication: this paper is in loving memory of Fábio Carneiro Mokarzel (1941–2021).
All authors have approved the manuscript and agree with its submission. Furthermore, we confirm that this manuscript has not been previously published and is not considered for publication in any other journal. Therefore, no conflict of interest needs to be disclosed.
The following abbreviations are used in this manuscript:
| AHP | Analytic Hierarchy Process |
| ANEEL | Brazilian Electricity and Regulatory Agency (in Portuguese: Agência Nacional de Energia Elétrica) |
| ANP | Analytic Network Process Or National Agency of Petroleum, Natural Gas and Bio-fuels (in Portuguese: Agência Nacional do Petróleo, Gás Natural e Biocombustíveis) |
| ARA | Additive Ration Assessment |
| BCG | Boston Consulting Group |
| BNDES | Brazilian Development Bank (in Portuguese: Banco Nacional de Desenvolvimento Econômico e Social) |
| BSC | Balanced Scorecard |
| CBA | Cost-Benefit Analysis |
| CBR | Case Based Reasoning |
| CFPR | Consistent Fuzzy Preference Relations |
| CNPq | National Council for Scientific and Technological Development (in Portuguese: Conselho Nacional de Desenvolvimento Científico e Tecnológico) |
| COPRAS | Complex Proportional Assessment |
| CP | Compromise Programming |
| DEA | Data Envelopment Analysis |
| DEMATEL | Decision-Making Trial and Evaluation Laboratory |
| ELECTRE | ELimination and Choice Expressing Reality (in French: ELimination Et Choix Traduisant la REalité) |
| EVAMIX | Evaluation of Mixed Data |
| FINEP | Financing Institution of Research and Innovation (in Portuguese: Financiadora de Inovação e Pesquisa) |
| GA | Genetic Algorithm |
| GRA | Gray Relational Analysis |
| LINMAP | Linear Programming Technique for Multidimensional Analysis and Preference |
| MADA | Multi-Attribute Decision Analysis or Multi-Attribute Decision Aiding |
| MADM | Multi-Attribute Decision Making |
| MAUT | Multi-Attribute Utility Theory |
| MAVT | Multi-Attribute Value Theory |
| MCDA | Multi-Criteria Decision Analysis or Multi-Criteria Decision Aiding |
| MCDM | Multi-Criteria Decision Making |
| MCTIC | Brazilian Minister of Science, Technology, Innovations and Communication (in Portuguese: Ministério da Ciência, Tecnologia, Inovações e Comunicações) |
| MME | Brazilian Ministry of Mines and Energy (in Portuguese: Ministério de Minas e Energia) |
| MODA | Multi-Objective Decision Analysis or Multi-Objective Decision Aiding |
| MODM | Multi-Objective Decision Making |
| MOGA | Multiple objective genetic algorithm |
| MOORA | Multi-Objective Optimization on the basis of Ration Analysis |
| MULTIMOORA | Multiplicative form with Multi-Objective Optimization on the basis of Ration Analysis |
| NAIADE | Novel Approach to Imprecise Assessment and Decision Environments |
| PMBOK | Project Management Body of Knowledge |
| PPM | Project Portfolio Management |
| PPS | Project Portfolio Selection |
| PROMETHEE | Preference Ranking Organization Method for Enrichment of Evaluations |
| REMBRANDT | Ratio Estimation in Magnitudes or Decibels to Rate Alternatives which are Non-Dominated |
| R&D | Research and Development |
| ROA | Real Options Analysis |
| SAW | Simple Additive Weighting |
| SLR | Systematic Literature Review |
| SMART | Simple Multi-Attribute Rating Technique |
| SWARA | Step-wise Weight Assessment Ration Analysis |
| TOPSIS | Technique for Order Preference by Similarity to Ideal Solution |
| UIS | UNESCO Institute for Statistics |
| UTADIS | Utilities Additives Discriminates |
| VIKOR | Multi-criteria Optimization and Compromise Solution (in Serbian: VIseKriterijumska Optimizacija I Kompro-misno Resenj) |
| WASPAS | Weighted Aggregated Sum Product Assessment |
| WPM | Weighted Product Method |
| WSM | Weighted Sum Method |
Publisher’s Note: MDPI stays neutral with regard to jurisdictional claims in published maps and institutional affiliations.
Inclusion and exclusion criteria.
| Step | Inclusion Criteria | Exclusion Criteria |
|---|---|---|
| Title screening | Articles related or that could be related to project portfolio in general. Since the title briefly introduce the main topic of the article, many works that performs MCDM-based R&D PPS as a secondary topic could fall out the SLR. | Papers that do not present approaches or cases related to project portfolio were left out. This is the case of articles introducing MCDM methods to general applications, other fields of study or other subjects inside the big area of project management, such as expert assessment, market assessment and performance evaluation of already concluded projects. |
| Abstract screening | Articles related to R&D PPS in general. Since some abstracts do not present the methods and approaches employed, we have decided to check this information later. | Additionally to the exclusion criteria performed in the first step, were left out the SLR articles that are addressing PPS to other areas rather than R&D, or are selecting other elements rather than projects, such as technology, suppliers, products and others. |
| Text screening | Only articles that present MCDM-based approaches to select R&D projects. | Additionally to the exclusion criteria performed in the two previous steps, articles addressing mono-criteria project selection or that do not find a set of optimal or recommended projects were left out. |
Strings used to perform the search.
| Boolean Combination: (MCDM OR MADM OR MODM OR Methods) AND PPS AND R&D | |
|---|---|
| MCDM | “MCDM” OR “multicriteria decision making” OR “multi-criteria decision making” OR “multi criteria decision making” OR “multiplecriteria decision making” OR “multiplecriteria decision making” OR “multiple criteria decision making” OR “MCDA” OR “multicriteria decision analysis” OR “multi-criteria decision analysis” OR “multi criteria decision analysis” OR “multiplecriteria decision analysis” OR “multiple-criteria decision analysis” OR “multiple criteria decision analysis” OR “multicriteria decision aiding” OR “multi-criteria decision aiding” OR “multi criteria decision aiding” OR “multiplecriteria decision aiding” OR “multiple-criteria decision aiding” OR “multiple criteria decision aiding” |
| MADM | “MADM” OR “multiattribute decision making” OR “multi-attribute decision making” OR “multi attribute decision making” OR “multipleattribute decision making” OR “multiple-attribute decision making” OR “multiple attribute decision making” OR “MADA” OR “multiattribute decision analysis” OR “multi-attribute decision analysis” OR “multi attribute decision analysis” OR “multipleattribute decision analysis” OR “multiple-attribute decision analysis” OR “multiple attribute decision analysis” OR “multiattribute decision aiding” OR “multi-attribute decision aiding” OR “multi attribute decision aiding” OR “multipleattribute decision aiding” OR “multiple-attribute decision aiding” OR “multiple attribute decision aiding” |
| MODM | “MODM” OR “multiobjective decision making” OR “multi-objective decision making” OR “multi objective decision making” OR “multipleobjective decision making” OR “multiple-objective decision making” OR “multiple objective decision making” OR “MODA” OR “multiobjective decision analysis” OR “multi-objective decision analysis” OR “multi objective decision analysis” OR “multipleobjective decision analysis” OR “multiple-objective decision analysis” OR “multiple objective decision analysis” OR “multiobjective decision aiding” OR “multi-objective decision aiding” OR “multi objective decision aiding” OR “multipleobjective decision aiding” OR “multiple-objective decision aiding” OR “multiple objective decision aiding” |
| Methods | “Simple Additive Weighting” OR “Additive Ration Assessment” OR “SWARA” OR “Step-wiseWeight Assessment Ration Analysis” OR “TOPSIS” OR “Technique for Order of Preference by Similarity to Ideal Solution” OR “ELECTRE” OR “Elimination et Choix Traduisant la Réalité” OR “Elimination and Choice Expressing REality” OR “LINMAP” OR “Linear Programming Technique for Multidimensional Analysis and Preference” OR “AHP” OR “Analytic Hierarchy Process” OR “ANP” OR “Analytic Network Process” OR “PROMETHEE” OR “The Preference Ranking Organization Method for Enrichment of Evaluations” OR “MOORA” OR “Multi-Objective Optimization on the basis of Ration Analysis” OR “MULTIMOORA” OR “Multiplicative form with Multi-Objective Optimization on the basis of Ration Analysis” OR “DEA” OR “Data Envelopment Analysis” OR “VIKOR” OR “Visekriterijumska optimizacija i Kompromisno Resenje” OR “Multicriteria Optimization and Compromise Solution” OR “COPRAS” OR “Complex Proportional Assessment” OR “EVAMIX” OR “Evaluation of Mixed Data” OR “DEMATEL” OR “Decision-Making trial and Evaluation Laboratory” OR “WASPAS” OR “Weighted Aggregated Sum Product Assessment” OR “WSM” OR “Weighted Sum Method” OR “WPM” OR “Weighted Product Method” OR “Compromise Programming” OR “MAUT” OR “Multi-Attribute Utility Theory” OR “CBR” OR “Case Based Reasoning” OR “Genetic Algorithm” OR “SMART” OR “Simple Multi-Attribute Rating Technique” OR “MAVT” OR “Multi-Attribute Value Theory” OR “REMBRANDT” OR “Ratio Estimation in Magnitudes” OR “Decibels to Rate Alternatives which are Non- Dominated” OR “NAIADE” OR “Novel Approach to Imprecise Assessment and Decision Environments” OR “Linear Programming” OR “Non-Linear Programming” OR “Non Linear Programming” OR “Multi-Objective Programming” OR “Multi Objective Programming” OR “Multiobjective programming” OR “Goal Programming” OR “Integer Linear Programming” OR “Integer Non-Linear Programming” OR “Integer Non Linear Programming” OR “Integer Programming” |
| PPS | “Project Selection” OR “Project Evaluation” OR “Project Portfolio Selection” OR “Project Portfolio Evaluation” OR “Project Portfolio” OR “Project Portfolio Management” |
| R&D | “Research and Development” OR “Research & Development” OR “R&D” OR “R and D” OR “RnD” OR “R n D” OR “R & D” |
Source: Exactly as proposed by [
MADM methods acronyms, meanings and first references.
| Acronym | Method | First Reference | Year |
|---|---|---|---|
| AHP | Analytic Hierarchy Process | Saaty [ |
1980 |
| ANP | Analytic Network Process | Saaty [ |
2001 |
| BCG Matrix | Boston Consulting Group Matrix | Boston Consulting Group [ |
1970 |
| BSC | Balanced Scorecard | Norton and Kaplan [ |
1999 |
| CBA | Cost-Benefit Analysis | Mishan and Eauston [ |
1976 |
| COPRAS | Complex Proportional Assessment | Zavadskas et al. [ |
1994 |
| DEMATEL | Decision-Making trial and Evaluation Laboratory | Gabus and Fontela [ |
1973 |
| ELECTRE | French: Elimination et Choix Traduisant la Réalité (Elimination and Choice Expressing Reality) | Benayoun amd Sussman [ |
1966 |
| MAUT | Multi-Attribute Utility Theory | Keeney and Raiffa [ |
1976 |
| PROMETHEE | Preference Ranking Organization Method for Enrichment of Evaluations | Brans and Vincke [ |
1985 |
| ROA | Real Options Analysis | Trigeorgis [ |
1995 |
| TOPSIS | Technique for Order of Preference by Similarity to Ideal Solution | Hwang [ |
1981 |
| VIKOR | Serbian: Visekriterijumska optimizacija i Kompromisno Resenje (Multicriteria Optimization and Compromise Solution) | Opricovic [ |
2002 |
Main categories of MADM methods.
| Categories | Methods | Advantages | Disadvantages | Utilization |
|---|---|---|---|---|
| Multi-criteria value functions | Scoring, MAUT, CBA | Can incorporate preferences. The results are easy to understand. Some approaches are simple. | The preferences need to be precise. A lot of input is needed. | 13.2% |
| Outranking approaches | One novel model presented | It may take uncertainty and vagueness into account. Quantitative criteria may assume preference thresholds. | Do not weight the criteria in a systematic way. The outcome may be difficult do explain, since strengths and alternative are not directly identified. | 1.5% |
| Distance to ideal point | TOPSIS, VIKOR | Easy to use and program. The number and programming efforts of the steps remain the same regardless of the number of criteria and alternatives. | Its difficult to weight and keep consistency of judgement. Qualitative criteria are not easily handled. | 10.3% |
| Pairwise comparisons | AHP, ANP, DEMATEL | Not data intensive. Easy to use. Can easily handle with qualitative criteria. | Not recommended when there are several criteria and/or several alternatives, which should not be split into smaller comparison matrices. Depending on the number of comparisons, the process may be tiring and lead to inconsistency. | 35.3% |
Map description.
| Main Information | Description |
|---|---|
| Integration Approach | Individual when was used only one type of MCDM method, or Integrated when as used two or more methods. |
| Nature | We classify the methods into MADM, MODM, and integrated MADM/MODM approaches. MADM refers to methods that deal with a discrete and finite number of options designated by a predetermined set of criteria. MODM methods have no predetermined options, and the optimal option selection depends on an infinite and continuous number of circumstances subjected to a set of constraints. |
| Software | The classification points out the technology used to implement the R&D PPS methods: dedicated software, solvers, spreadsheet software, or algorithm/programming languages. |
| Uncertainty | The types of uncertainty were: Deterministic (when uncertainty is not considered), Probabilistic (when uncertainty is related to the lack of information), and Fuzzy (when their is imprecision of semantic meanings). Fuzzy numbers may be associated to Deterministic or, in rare cases, to Probabilistic approaches. |
| Portfolio sizes | We classified the portfolio sizes into Big (under 100 projects), Medium (between 9 and 99 projects), and Small (fewer than 8 projects). |
| Criteria explanation | We point out the papers that described the criteria used. |
| Application domain | Describe the decision environment where the R&D PPS is performed. |
MCDM-based R&D PPS articles: number of projects, methods and application domains—Part 1/3.
| Author | Number of Projects | Methods | Application Domain |
|---|---|---|---|
| Bell and Read [ |
40/12 and 22 | Linear Programming | Eletronic/Electricity and Chemical |
| Taylor et al. [ |
7 | Non-Linear Integer Goal Programming | Textile |
| Madey and Dean [ |
50 | MAUT, Mixed-Integer Non-Linear Programming, Multi-objective Programming, Preemptive Goal Programming | Aerospacial |
| Czajkowski and Jones [ |
25 | DELPHI, Integer Linear Programming | Spacial |
| Liberatore [ |
27 | AHP, CBA, Scoring, MAUT, Integer Linear Goal Programing | Chemical |
| Liberatore [ |
27 | AHP, CBA, Integer Linear Programming | Chemical |
| Bard et al. [ |
10 | Integer Linear Programming | Eletronic/Electricity |
| Liberatore [ |
24 | AHP, CBA, Integer Linear Programming | Industrial |
| Ringuest and Graves [ |
4 | DELPHI, Multi-objective Linear Programming, Goal Programming | Non-Specified |
| Ringuest and Graves [ |
4 | Multi-objective Linear Programming | Non-Specified |
| Oral et al. [ |
37 | DELPHI, Model-based Outranking Method, DEA | Metallurgy |
| Stewart [ |
20, 50 and 150/250 | Non-linear Programming | Eletronic/Electricity |
| Graves and Ringuest [ |
4 | Multi-objective Linear Programming | Non-Specified |
| Heidenberger [ |
2 | Mixed-Integer Linear Programming | Non-Specified |
| Henig and Katz [ |
5 | Not-specified MADM method | Biotechnology |
| Beaujon et al. [ |
400 | Integer Linear Programming | Automotive |
| Meade and Presley [ |
2 | ANP | Information Technology |
| Hsu et al. [ |
12 | Fuzzy-AHP | Industrial |
| Stummer and Heidenberger [ |
10 and 30 | Multi-objective Integer Linear Programming | Industrial |
| Kumar [ |
6 | AHP | Research |
| Ringuest et al. [ |
5 and 30 | Mean-Gini Analysis, Non-Linear Programming, Stochastic Dominance | Pharmaceutical |
| Gustafsson and Salo [ |
1000 and 200 | Multi-objective Mixed-Integer Linear Programming | Non-Specified |
| Mohanty et al. [ |
3 | Fuzzy-ANP; Fuzzy Cost Analysis | Metallurgy |
| Ringuest et al. [ |
5 and 30 | Mean-Gini Analysis, Linear Programming | Pharmaceutical |
| Sun and Ma [ |
8 | Integer Linear Programming | Non-Specified |
| Wang et al. [ |
Non-Specified | AHP, Fuzzy-Scoring | Information Technology |
| Karsak [ |
6 | Fuzzy Integer Non-Linear Programming, ROA | Information Technology |
| Rabbani et al. [ |
10 | AHP, Integer Linear Programming | Telecommunications |
| Carlsson et al. [ |
Non-Specified | Fuzzy Mixed-Integer Linear Programming | Non-Specified |
MCDM-based R&D PPS articles: number of projects, methods and application domains—Part 2/3.
| Author | Number of Projects | Methods | Application Domain |
|---|---|---|---|
| Medaglia et al. [ |
4 | Multi-objective Stochastic Linear Programming (Solved by Evolutionary Algorithm) | Non-Specified |
| Shin et al. [ |
5 | AHP | Nuclear |
| Wang and Hwang [ |
20 | Fuzzy Integer Linear Programming, Fuzzy-ROA (Compound Options) | Pharmaceutical |
| Bitman and Sharif [ |
Non-Specified | AHP, Scoring, BCG Matrix, BSC, DEA | Non-Specified |
| Conka et al. [ |
14 | AHP, DEA, VTA | Non-Specified |
| Eilat et al. [ |
60/50 | BSC, DEA, Linear Programming | Industrial |
| Fang et al. [ |
3 | Mixed-Integer Stochastic Linear Programming | Non-Specified |
| Imoto et al. [ |
18 | AHP, Fuzzy-Regression Analysis, PCA, Fuzzy Multi-objective Integer Linear Programming (Solved by Genetic Algorithm) | Metallurgy |
| Tolga et al. [ |
6 | Fuzzy-AHP, Fuzzy-ROA | Eletronic/Electricity |
| Wu et al. [ |
37 | Nash bargaining game | Non-Specified |
| Jung and Seo [ |
14 | ANP | Government Sponsored |
| Bhattacharyya et al. [ |
6 | Fuzzy Multi-objective Integer Linear Programming (Solved by Genetic Algorithm) | Civil, Mechanical, and others |
| Eckhause et al. [ |
2 and 3 | Integer Linear Programming | Non-Specified |
| Hassanzadeh et al. [ |
20 | Fuzzy Pay-off Method (ROA context), Fuzzy Integer Linear Programming | Pharmaceutical |
| Hassanzadeh et al. [ |
20 | Fuzzy Pay-off Method (ROA context), Fuzzy Integer Linear Programming | Pharmaceutical |
| Mohaghar et al. [ |
4 | Fuzzy-ANP, Fuzzy-TOPSIS | Manufacturing |
| Oral [ |
Non-Specified | E-DEA (Self-Efficiency DEA and Cross-Efficiency DEA) | Non-Specified |
| Collan and Luukka [ |
20 | Fuzzy-TOPSIS, Fuzzy pay-off method (ROA context) | Pharmaceutical |
| Hassanzadeh et al. [ |
14 | Multi-objective Integer Linear Programming | Information Technology |
| Bhattacharyya [ |
5 | Grey Theory Sets | Civil, Mechanical and others |
| Collan et al. [ |
20 | Fuzzy-TOPSIS | Pharmaceutical |
| Eshlaghy [ |
20 | Grey Theory, Clustering Method (Solved by GA and K-Means) | Non-Specified |
| Jeng and Huang [ |
5 | ANP, DELPHI, DEMATEL | Eletronic/Electricity |
| Karaveg et al. [ |
45 | TOPSIS, SEM | Agriculture, Innovation, Textile and others |
| Arratia et al. [ |
1500 | Mixed-integer Linear Programming | Private/Public Sector |
| Heydari T et al. [ |
6 | Non-Linear Integer Goal Programming | Non-Specified |
| Stewart [ |
Non-Specified | Multi-objective Non-linear Programming (Solved by Reference Point, Genetic Algorithm, NIMBUS) | Non-Specified |
MCDM-based R&D PPS articles: number of projects, methods and application domains—Part 3/3.
| Author | Number of Projects | Methods | Application Domain |
|---|---|---|---|
| Cheng et al. [ |
5 | ANP, DEMATEL, COPRAS-G, Fuzzy Grey Relations | Eletronic/Electricity |
| Karasakal [ |
60 | UTADIS (DEA based), AHP | Government Sponsored |
| Marcondes et al. [ |
10 | Mean-Gini Analysis, Stochastic Dominance | Non-Specified |
| Montajabiha et al. [ |
50 | Mixed-Integer Linear Programming | Pharmaceutical |
| Liang et al. [ |
6 | TOPSIS, Pythagorean Fuzzy Theory | Private/Public Sector |
| Storch de Gracia et al. [ |
5 | AHP | Energy |
| Wei et al. [ |
100 | Correlation Analysis; Multi-objective Non-Linear Integer Programming | Military |
| Samanlioglu et al. [ |
5 | Fuzzy-AHP; Fuzzy-VIKOR | Innovation |
| Aghdaie et al. [ |
4 | SWARA; COPRAS | Electricity/Mechanical/ |
| Yalcin et al. [ |
Non-Specified | Fuzzy-TOPSIS; Fuzzy-DEMATEL | Non-Specified |
Theoretical list of criteria—1/2.
| # | Criteria | Description | Utilization |
|---|---|---|---|
| 1 | Commercial and Market Risk (CMR) | It refers, in general, to the ambiguity of a project to generate commercial success [ |
3 (5%) |
| 2 | Competitiveness and Partnership (COP) | Measures a project’s potential to improve the company’s share in the market more than its competitors. It can be accomplished, for example, by using Science and Technology (S&T) policies or with the development, use, and commercialization of proprietary technology [ |
10 (16%) |
| 3 | Corporate Image (COI) | Describes the potential of a project to enhance the company’s visibility to society or with a specific company or with an economic segment. Some authors, like Liberatore [ |
5 (8%) |
| 4 | Environmental Impact (ENI) | Measures the capacity of a project to produce any environmental benefit [ |
5 (8%) |
| 5 | Extendibility (EXT) | It is related to a project’s potential to improve the company’s growth by adding new components or integrating the project with other public policies. It can be estimated, for illustration, by the applicability of a project results in other products and processes [ |
9 (15%) |
| 6 | External Environment Income (EEI) | It considers all factors and criteria that are not within the company and beyond its control, such as the existence of competitors [ |
28 (46%) |
| 7 | Financial Benefit (FIB) | It expresses the organization’s financial return of the project and can be measured by different indicators, such as net present value (NPV) [ |
41 (67%) |
| 8 | Financial Income (FII) | Relates to all financial resources required to implement the project, and they can be measured in terms of cost, budget, cash flow, total investment, and other metrics [ |
48 (79%) |
| 9 | Impact in Human Development (IHD) | Associates to any criteria correlated to the development and training of human resources [ |
9 (15%) |
| 10 | Internal Environment Income (IEI) | Includes the criteria connected with factors inside an organization, like workplace safety and manufacturing capability [ |
20 (33%) |
| 11 | Market Potential and Attractiveness (MPA) | Includes criteria exclusively related to the market and the receptivity by the market to the outcomes of the project [ |
3 (5%) |
| 12 | Material Resources (MAR) | Includes the criteria associated with resources that will be consumed, like raw material and energy [ |
38 (62%) |
| 13 | Non-Financial Benefit (NFB) | Expresses the non-financial gains of the project to an organizational, such as patents [ |
38 (62%) |
| 14 | Organizational Requirements (ORR) | Includes the criteria imposed by the organization, like the R&D objective, priority, congruence, and importance [ |
17 (28%) |
| 15 | Quality Requirements (QTR) | It brings together all the criteria that can interfere with the overall quality of the project, such as customer feedback, customer satisfaction, and quality proposal [ |
21 (34%) |
Theoretical list of criteria—2/2.
| # | Criteria | Description | Utilization |
|---|---|---|---|
| 16 | Scope Risk (SCR) | It measures the project’s results probability of staying outside its scope after the conclusion. Consequently, it can be associated with the risk of delay [ |
21 (34%) |
| 17 | Social Impact (SOI) | Measures the project’s capacity to create social benefit [ |
7 (11%) |
| 18 | Strategic Fitness (STF) | Measures the project’s capacity to meet the strategic goals of the company. It can be also described as strategic fit [ |
12 (20%) |
| 19 | Technical Contribution and Innovativeness | Indicates the project’s potential to introduce new approaches to accomplish new technologies [ |
16 (26%) |
| 20 | Technical Issues and Constraints (TIC) | This criterion is related to the leading technologies used in the project and their impact or possible associated problems. So, the criteria can be exemplified as technological connections. [ |
9 (15%) |
| 21 | Technical Risk (TER) | It is generally related to the uncertainty associated with the technology or the probability of occurrence of technical problems [ |
4 (7%) |
| 22 | Timing Requirements (TIR) | It is associated with all criteria connecting with a time dimension, such as timing, project completion time, and time to market [ |
15 (25%) |
| 23 | Work Resources (WOR) | This criterion comprises the resources that will be utilized, such as labor and their necessary knowledge and experience [ |
40 (66%) |
Experts, their qualifications and tasks.
| # | Experience in PPS in the Organization | Experience in PPS | Higher Education Degree | Participation as | Organization |
|---|---|---|---|---|---|
| E1 | 5 years | 5 years | M. Sc. | Project manager | ANP |
| E2 | 7 years | 8 years | M. Sc. | Project manager | ANP |
| E3 | 3 years | 3 years | Doctorate | Decision maker | CNPq |
| E4 | 9 years | 9 years | Doctorate | Decision maker | CNPq |
| E5 | 3 years | 10 years | Doctorate | Decision maker | FINEP |
| E6 | Decision maker | FINEP | |||
| E7 | 17 years | 17 years | Doctorate | Decision maker | ANEEL |
| E8 | 3 years | 10 years | Doctorate | Decision maker | ANEEL |
| E9 | 3 years | 5 years | Doctorate | Board Director Member (Responsible for implementing guidelines for R&D PPS) | ANEEL |
| E10 | 3 years | 10 years | Doctorate | Project manager | BNDES |
Most cited papers in MCDM-based R&D PPS from 1970 to 2019.
| Papers | Tota Citations | Citation per Year | Database |
|---|---|---|---|
| Meade and Presley [ |
327 | 19.24 | WoS/Scopus |
| Liberatore [ |
141 | 4.41 | WoS/Scopus |
| Wang and Hwang [ |
136 | 11.33 | WoS/Scopus |
| Eilat et al. [ |
132 | 12.00 | WoS/Scopus |
| Oral et al. [ |
131 | 4.68 | WoS |
| Stummer and Heidenberger [ |
123 | 7.69 | WoS/Scopus |
| Mohanty et al. [ |
122 | 8.71 | WoS |
| Carlsoon et al. [ |
102 | 8.50 | WoS/Scopus |
| Bard et al. [ |
73 | 2.35 | Scopus |
| Medaglia et al. [ |
69 | 5.75 | WoS |
Research questions to guide further research.
| Research Questions That Could Not Be Answered by The articles Considered in the SLR | |
|---|---|
| 1 | How historical data from previously executed project portfolios could be used to better select new projects? None of the presented models incorporate data-driven decision making approaches to their frameworks. Nonetheless, several classification and regression approaches, such as Random Forest and Support Vector Machines/Regressors could be used to enhance the selected portfolio. |
| 2 | Which MCDM approaches are more suitable to select project portfolios from several project proposals? Large portfolios are representative in many countries, especially developing countries, where R&D investments highly depend on public and governmental agencies through calls, which normally receive several project proposals. The R&D PPS field does not seem to be already impacted by big data. |
| 3 | Which are the best MCDM approaches to small-profitable R&D companies? The MCDM approaches proposed in the literature are, sometimes, far from the reality of many companies, that do not have the personnel to use them, nor sufficient money to provide software running those approaches [ |
| 4 | How to define, a priori, criteria for selecting projects? Only two articles from the performed SLR present methodologies for criteria selection [ |
| 5 | How to employ outranking MADM methods (such as PROMETHEE and ELECTRE) in the R&D PPS context? There is no result in literature for these methods when applied to R&D PPS in general. However, uncertainty, vagueness and preference thresholds are still characteristics of several R&D project portfolios. |
| 6 | How to select the best method for each R&D PPS application? Several articles do not explain why they have chosen some methods to PPS among all possible options. When this is the case, they mainly use usage frequency, which is the case of standalone AHP applications [ |
| Research Questions Presented by Recent Articles as Opportunities for Future Works | |
| 1 | Is the three-way decisions-based ideal solution, proposed by Liang et al. [ |
| 2 | Cheng et al. [ |
| 3 | How does the method proposed by Karasakal and Aker [ |
| 4 | How to consider resource constraints, interrelations, or mutual-exclusion to select projects? This approach appeared in Marcondes et al. [ |
| 5 | How would the multi-objective mixed-integer linear programming, proposed by Arratia et al. [ |
Summary of findings.
| Research Questions | Findings |
|---|---|
| RQ1: Methods. | We presented the nature of the alternatives in |
| RQ2: Portfolios. | Medium-sized portfolios stand for 67% of all portfolios analyzed in the papers, small-sized portfolios 22%, and big-sized portfolios 11%. The pharmaceutical application is the most explored domain, as shown in |
| RQ3: Research Field. | |
| RQ4: Whole data. | The correlations analysis showed that articles using MADM approaches tend to explain better the criteria used, While MODM focuses more on the model itself and leaves this kind of detail (explaining the criteria, for example) unattended. We highlighted this by the moderate negative correlation (−0.51) between linear programming (So far, the most frequently used MODM method) and the presence of explained criteria on the paper. Some MCDM methods also seem to be primarily used as supporting methods, which is the case of CBA, which presents a +0.86 correlation coefficient with the usage of Integrated Approaches. Similarly, DEMATEL is used as an auxiliary to ANP. |
| RQ5: Criteria used. | We presented the theoretical list of criteria comprised of 23 criteria in |
| RQ6: Opportunities and trends. | We explored the opportunities and new paths in sub |
Supplementary Materials
The following materials are available in the supplementary data, as bellow: Availability of data, code and other materials click
References
1. Thore, A. Technology Commercialization: DEA and Related Analytical Methods for Evaluating the Use and Implementation of Technical Innovation; Kluwer Academic Publishers: Dordrecht, The Netherlands, 2002.
2. UNESCO Institute for Statistics (UIS). How Much Your Country Invest in R&D?. 2018; Available online: http://uis.unesco.org/ (accessed on 12 October 2021).
3. Guimón de Ros, J.; Agapitova, N. Why should governments of developing countries invest in R&D and innovation?. Afr. J. Bus. Manag.; 2013; 7, pp. 899-905.
4. Strategy& Part of PWC Network. How The Global Innovation 1000 Study. 2018; Available online: https://www.strategyand.pwc.com/gx/en/insights/innovation1000.html (accessed on 12 October 2021).
5. MCTIC. Ministério da Ciência, Tecnologia, Inovação e Comunicações: Indicadores Nacionais de Ciência, Tecnologia e Inovação. Ministério da Ciência, Tecnologia e Inovações. Technical Report. 2018; Available online: https://www.gov.br/mcti/pt-br (accessed on 12 October 2021).
6. IPEA. Radar: Tecnologia, Produção e Comércio Exterior (Nº 48). Technical Report. 2016; Available online: https://www.ipea.gov.br/ (accessed on 12 October 2021).
7. Project Management Institute. The Standard for Portfolio Management; Project Management Institute, Inc.: Newton Square, PA, USA, 2013.
8. Jafarizadeh, B.; Ramazani Khorshid-Doust, R. A method of project selection based on capital asset pricing theories in a framework of mean—Semideviation behavior. Int. J. Proj. Manag.; 2008; 26, pp. 612-619. [DOI: https://dx.doi.org/10.1016/j.ijproman.2007.09.004]
9. Montajabiha, M.; Arshadi Khamseh, A.; Afshar-Nadjafi, B. A robust algorithm for project portfolio selection problem using real options valuation. Int. J. Manag. Proj. Bus.; 2017; 10, pp. 386-403. [DOI: https://dx.doi.org/10.1108/IJMPB-12-2015-0114]
10. Conka, T.; Vayvay, O.; Sennaroglu, B. A combined decision model for R&D project portfolio selection. Int. J. Bus. Innov. Res.; 2008; 2, pp. 190-202.
11. Graves, S.; Ringuest, J. Choosing the best solution in an R&D project selection problem with multiple objectives. J. High Technol. Manag. Res.; 1992; 3, pp. 213-224. [DOI: https://dx.doi.org/10.1016/1047-8310(92)90012-Q]
12. Liberatore, M. An expert support system for R&D project selection. Math. Comput. Model.; 1988; 11, pp. 260-265. [DOI: https://dx.doi.org/10.1016/0895-7177(88)90495-5]
13. Liberatore, M. R&D project selection. Telemat. Inform.; 1986; 3, pp. 289-300. [DOI: https://dx.doi.org/10.1016/S0736-5853(86)80065-X]
14. Tavana, M.; Keramatpour, M.; Santos-Arteaga, F.J.; Ghorbaniane, E. A fuzzy hybrid project portfolio selection method using data envelopment analysis, TOPSIS and integer programming. Expert Syst. Appl.; 2015; 42, pp. 8432-8444. [DOI: https://dx.doi.org/10.1016/j.eswa.2015.06.057]
15. Liberatore, M.J. Extension of the Analytic Hierarchy Process for Industrial R&D Project Selection and Resource Allocation. IEEE Trans. Eng. Manag.; 1987; EM-34, pp. 12-18.
16. Meade, L.; Presley, A. R&D project selection using the analytic network process. IEEE Trans. Eng. Manag.; 2002; 49, pp. 59-66. [DOI: https://dx.doi.org/10.1109/17.985748]
17. Wang, J.; Hwang, W.L. A fuzzy set approach for R&D portfolio selection using a real options valuation model. Omega; 2007; 35, pp. 247-257. [DOI: https://dx.doi.org/10.1016/j.omega.2005.06.002]
18. Arratia, M.; López, I.; Schaeffer, S.; Cruz-Reyes, L. Static R&D project portfolio selection in public organizations. Decis. Support Syst.; 2016; 84, pp. 53-63. [DOI: https://dx.doi.org/10.1016/j.dss.2016.01.006]
19. Gustafsson, J.; Salo, A. Contingent portfolio programming for the management of risky projects. Oper. Res.; 2005; 53, pp. 946-956. [DOI: https://dx.doi.org/10.1287/opre.1050.0225]
20. Stewart, T. A multi-criteria decision support system for r&d project selection. J. Oper. Res. Soc.; 1991; 42, pp. 17-26. [DOI: https://dx.doi.org/10.1057/jors.1991.3]
21. Mohaghar, A.; Fathi, M.; Faghih, A.; Turkayesh, M. An integrated approach of Fuzzy ANP and Fuzzy TOPSIS for R&D project selection: A case study. Aust. J. Basic Appl. Sci.; 2012; 6, pp. 66-75.
22. Tolga, A.; Kahraman, C. Fuzzy multiattribute evaluation of R&D projects using a real options valuation model. Int. J. Intell. Syst.; 2008; 23, pp. 1153-1176. [DOI: https://dx.doi.org/10.1002/int.20312]
23. Collan, M.; Luukka, P. Evaluating R&D projects as investments by using an overall ranking from four new fuzzy similarity measure-based topsis variants. IEEE Trans. Fuzzy Syst.; 2014; 22, pp. 505-515. [DOI: https://dx.doi.org/10.1109/TFUZZ.2013.2260758]
24. Hassanzadeh, F.; Nemati, H.; Sun, M. Robust optimization for interactive multiobjective programming with imprecise information applied to R&D project portfolio selection. Eur. J. Oper. Res.; 2014; 238, pp. 41-53. [DOI: https://dx.doi.org/10.1016/j.ejor.2014.03.023]
25. Karsak, E. A generalized fuzzy optimization framework for R&D project selection using real options valuation. Comput. Sci. Its Appl.; 2006; 3982, pp. 918-927.
26. Barucke Marcondes, G.A.; Leme, R.C.; Leme, M.d.S.; Sanches da Silva, C.E. Using mean-Gini and stochastic dominance to choose project portfolios with parameter uncertainty. Eng. Econ.; 2017; 62, pp. 33-53. [DOI: https://dx.doi.org/10.1080/0013791X.2016.1176283]
27. Ringuest, J.; Graves, S.; Case, R. Mean-Gini analysis in R&D portfolio selection. Eur. J. Oper. Res.; 2004; 154, pp. 157-169. [DOI: https://dx.doi.org/10.1016/S0377-2217(02)00708-7]
28. Stewart, T. Multiple objective project portfolio selection based on reference points. J. Bus. Econ.; 2016; 86, pp. 23-33. [DOI: https://dx.doi.org/10.1007/s11573-015-0797-x]
29. Jung, U.; Seo, D. An ANP approach for R&D project evaluation based on interdependencies between research objectives and evaluation criteria. Decis. Support Syst.; 2010; 49, pp. 335-342. [DOI: https://dx.doi.org/10.1016/j.dss.2010.04.005]
30. Bhattacharyya, R.; Kumar, P.; Kar, S. Fuzzy R&D portfolio selection of interdependent projects. Comput. Math. Appl.; 2011; 62, pp. 3857-3870. [DOI: https://dx.doi.org/10.1016/j.camwa.2011.09.036]
31. Czajkowski, A.F.; Jones, S. Selecting Interrelated R&D projects in Space Technology Planning. IEEE Trans. Eng. Manag.; 1986; EM-33, pp. 17-24.
32. Stummer, C.; Heidenberger, K. Interactive R&D portfolio analysis with project interdependencies and time profiles of multiple objectives. IEEE Trans. Eng. Manag.; 2003; 50, pp. 175-183. [DOI: https://dx.doi.org/10.1109/TEM.2003.810819]
33. Eilat, H.; Golany, B.; Shtub, A. R&D project evaluation: An integrated DEA and balanced scorecard approach. Omega; 2008; 36, pp. 895-912. [DOI: https://dx.doi.org/10.1016/j.omega.2006.05.002]
34. Eshlaghy, A.; Razi, F. A hybrid grey-based k-means and genetic algorithm for project selection. Int. J. Bus. Inf. Syst.; 2015; 18, pp. 141-159. [DOI: https://dx.doi.org/10.1504/IJBIS.2015.067262]
35. Oral, M. Action research contextualizes DEA in a multi-organizational decision-making process. Expert Syst. Appl.; 2012; 39, pp. 6503-6513. [DOI: https://dx.doi.org/10.1016/j.eswa.2011.12.054]
36. Rabbani, M.; Tavakkoli-Moghaddam, R.; Jolai, F.; Ghorbani, H. A comprehensive model for r and d project portfolio selection with zero-one linear goal-programming (research note). Int. J.-Eng.-Trans. A Basics; 2006; 19, pp. 55-66.
37. Heidenberger, K. Dynamic project selection and funding under risk: A decision tree based MILP approach. Eur. J. Oper. Res.; 1996; 95, pp. 284-298. [DOI: https://dx.doi.org/10.1016/0377-2217(95)00259-6]
38. Medaglia, A.L.; Graves, S.B.; Ringuest, J.L. A multiobjective evolutionary approach for linearly constrained project selection under uncertainty. Eur. J. Oper. Res.; 2007; 179, pp. 869-894. [DOI: https://dx.doi.org/10.1016/j.ejor.2005.03.068]
39. Heydari, T.; Seyed Hosseini, S.M.; Makui, A. Developing and solving an one-zero non-linear goal programming model to R and D portfolio project selection with interactions between projects. Int. Bus. Manag.; 2016; 10, pp. 4516-4521.
40. Imoto, S.; Yabuuchi, Y.; Watada, J. Fuzzy regression model of R&D project evaluation. Appl. Soft Comput. J.; 2008; 8, pp. 1266-1273. [DOI: https://dx.doi.org/10.1016/j.asoc.2007.02.024]
41. Sun, H.; Ma, T. A packing-multiple-boxes model for R&D project selection and scheduling. Technovation; 2005; 25, pp. 1355-1361. [DOI: https://dx.doi.org/10.1016/j.technovation.2004.07.010]
42. Taylor, B.; Moore, L.; Clayton, E. R and D Project Selection and Manpower Allocation with Integer Non-Linear Goal Programming. Manag. Sci.; 1982; 28, pp. 1149-1158. [DOI: https://dx.doi.org/10.1287/mnsc.28.10.1149]
43. Bitman, W.R.; Sharif, N. A conceptual framework for ranking R&D projects. IEEE Trans. Eng. Manag.; 2008; 55, pp. 267-278.
44. Bard, J.; Balachandra, R.; Kaufmann, P. An Interactive Approach to R&D Project Selection and Termination. IEEE Trans. Eng. Manag.; 1988; 35, pp. 139-146. [DOI: https://dx.doi.org/10.1109/17.7433]
45. Wu, J.; Liang, L.; Yang, F.; Yan, H. Bargaining game model in the evaluation of decision making units. Expert Syst. Appl.; 2009; 36, pp. 4357-4362. [DOI: https://dx.doi.org/10.1016/j.eswa.2008.05.001]
46. Cheng, C.H.; Liou, J.; Chiu, C.Y. A consistent fuzzy preference relations based ANP model for R&D project selection. Sustainability; 2017; 9, 1352.
47. Harrison, K.R.; Elsayed, S.; Garanovich, I.; Weir, T.; Galister, M.; Boswell, S.; Taylor, R.; Sarker, R. Portfolio Optimization for Defence Applications. IEEE Access; 2020; 8, pp. 60152-60178. [DOI: https://dx.doi.org/10.1109/ACCESS.2020.2983141]
48. Pourhabibi, T.; Ong, K.L.; Kam, B.H.; Boo, Y.L. Fraud detection: A systematic literature review of graph-based anomaly detection approaches. Decis. Support Syst.; 2020; 133, 113303. [DOI: https://dx.doi.org/10.1016/j.dss.2020.113303]
49. Reza Afshari, A. Selection of construction project manager by using Delphi and fuzzy linguistic decision making. J. Intell. Fuzzy Syst.; 2015; 28, pp. 2827-2838. [DOI: https://dx.doi.org/10.3233/IFS-151562]
50. de Almeida, A.T.; Geiger, M.J.; Costa Morais, D. Challenges in multicriteria decision methods. IMA J. Manag. Math.; 2018; 29, pp. 247-252. [DOI: https://dx.doi.org/10.1093/imaman/dpy005]
51. Souza, D.G.; Silva, C.E.; Soma, N.Y. Selecting Projects on the Brazilian R&D Energy Sector: A Fuzzy-Based Approach for Criteria Selection. IEEE Access; 2020; 8, pp. 50209-50226.
52. Rowley, J.; Slack, F. Conducting a literature review. Manag. Res. News; 2004; 27, pp. 31-39. [DOI: https://dx.doi.org/10.1108/01409170410784185]
53. Moher, D.; Liberati, A.; Tetzlaff, J.; Altman, D.G.; Group, P. Preferred reporting items for systematic reviews and meta-analyses: The PRISMA statement. PLoS Med.; 2009; 6, e1000097. [DOI: https://dx.doi.org/10.1371/journal.pmed.1000097]
54. Jahangirian, M.; Eldabi, T.; Naseer, A.; Stergioulas, L.K.; Young, T. Simulation in manufacturing and business: A review. Eur. J. Oper. Res.; 2010; 203, pp. 1-13. [DOI: https://dx.doi.org/10.1016/j.ejor.2009.06.004]
55. Diaby, V.; Campbell, K.; Goeree, R. Multi-criteria decision analysis (MCDA) in health care: A bibliometric analysis. Oper. Res. Health Care; 2013; 2, pp. 20-24. [DOI: https://dx.doi.org/10.1016/j.orhc.2013.03.001]
56. Ananda, J.; Herath, G. A critical review of multi-criteria decision making methods with special reference to forest management and planning. Ecol. Econ.; 2009; 68, pp. 2535-2548. [DOI: https://dx.doi.org/10.1016/j.ecolecon.2009.05.010]
57. Govindan, K.; Rajendran, S.; Sarkis, J.; Murugesan, P. Multi criteria decision making approaches for green supplier evaluation and selection: A literature review. J. Clean. Prod.; 2015; 98, pp. 66-83. [DOI: https://dx.doi.org/10.1016/j.jclepro.2013.06.046]
58. Ho, W.; Xu, X.; Dey, P.K. Multi-criteria decision making approaches for supplier evaluation and selection: A literature review. Eur. J. Oper. Res.; 2010; 202, pp. 16-24. [DOI: https://dx.doi.org/10.1016/j.ejor.2009.05.009]
59. Malczewski, J. GIS-based multicriteria decision analysis: A survey of the literature. Int. J. Geogr. Inf. Sci.; 2006; 20, pp. 703-726. [DOI: https://dx.doi.org/10.1080/13658810600661508]
60. Mardani, A.; Zavadskas, E.K.; Khalifah, Z.; Jusoh, A.; Nor, K.M.D. Multiple criteria decision-making techniques in transportation systems: A systematic review of the state of the art literature. Transport; 2016; 31, pp. 359-385. [DOI: https://dx.doi.org/10.3846/16484142.2015.1121517]
61. Mendoza, G.; Martins, H. Multi-criteria decision analysis in natural resource management: A critical review of methods and new modelling paradigms. For. Ecol. Manag.; 2006; 230, pp. 1-22. [DOI: https://dx.doi.org/10.1016/j.foreco.2006.03.023]
62. Pohekar, S.; Ramachandran, M. Application of multi-criteria decision making to sustainable energy planning—A review. Renew. Sustain. Energy Rev.; 2004; 8, pp. 365-381. [DOI: https://dx.doi.org/10.1016/j.rser.2003.12.007]
63. Ribeiro, R.A. Fuzzy multiple attribute decision making: A review and new preference elicitation techniques. Fuzzy Sets Syst.; 1996; 78, pp. 155-181. [DOI: https://dx.doi.org/10.1016/0165-0114(95)00166-2]
64. Stewart, T. A critical survey on the status of multiple criteria decision making theory and practice. Omega; 1992; 20, pp. 569-586. [DOI: https://dx.doi.org/10.1016/0305-0483(92)90003-P]
65. Wang, J.J.; Jing, Y.Y.; Zhang, C.F.; Zhao, J.H. Review on multi-criteria decision analysis aid in sustainable energy decision-making. Renew. Sustain. Energy Rev.; 2009; 13, pp. 2263-2278. [DOI: https://dx.doi.org/10.1016/j.rser.2009.06.021]
66. Zavadskas, E.K.; Turskis, Z.; Kildienė, S. State of art surveys of overviews on MCDM/MADM methods. Technol. Econ. Dev. Econ.; 2014; 20, pp. 165-179. [DOI: https://dx.doi.org/10.3846/20294913.2014.892037]
67. Souza, D.G.B. R&D Project Selection: Which Criteria Should We Use?. Ph.D. Thesis; Federal University of Itajuba: Itajubá, Brazil, 2019.
68. Chadegani, A.A.; Salehi, H.; Yunus, M.M.; Farhadi, H.; Fooladi, M.; Farhadi, M.; Ebrahim, N.A. A Comparison between Two Main Academic Literature Collections: Web of Science and Scopus Databases. Asian Soc. Sci.; 2013; 9, pp. 18-26. [DOI: https://dx.doi.org/10.5539/ass.v9n5p18]
69. Huang, C.C.; Chu, P.Y. Using the fuzzy analytic network process for selecting technology R&D projects. Int. J. Technol. Manag.; 2011; 53, pp. 89-115.
70. Chai, J.; Liu, J.N.; Ngai, E.W. Application of decision-making techniques in supplier selection: A systematic review of literature. Expert Syst. Appl.; 2013; 40, pp. 3872-3885. [DOI: https://dx.doi.org/10.1016/j.eswa.2012.12.040]
71. Committee, P.S. A Guide to the Project Management Body of Knowledge; PMI Publications: Newtown Square, PA, USA, 1996.
72. Madey, G.R.; Dean, B.V. Strategic Planning for Investment in R&D usiong decision analysis and mathematical programming. IEEE Trans. Eng. Manag.; 1985; EM-32, pp. 84-90.
73. Hsu, Y.G.; Tzeng, G.H.; Shyu, J. Fuzzy multiple criteria selection of government-sponsored frontier technology R&D projects. R D Manag.; 2003; 33, pp. 539-551.
74. Kumar, S. AHP-based formal system for R&D project evaluation. J. Sci. Ind. Res.; 2004; 63, pp. 888-896.
75. Shin, C.O.; Yoo, S.H.; Kwak, S.J. Applying the analytic hierarchy process to evaluation of the national nuclear R&D projects: The case of Korea. Prog. Nucl. Energy; 2007; 49, pp. 375-384. [DOI: https://dx.doi.org/10.1016/j.pnucene.2007.03.001]
76. Wang, K.; Wang, C.; Hu, C. Analytic hierarchy process with fuzzy scoring in evaluating multidisciplinary R&D projects in China. IEEE Trans. Eng. Manag.; 2005; 52, pp. 119-129. [DOI: https://dx.doi.org/10.1109/TEM.2004.839964]
77. Collan, M.; Fedrizzi, M.; Luukka, P. New Closeness Coefficients for Fuzzy Similarity Based Fuzzy TOPSIS: An Approach Combining Fuzzy Entropy and Multidistance. Adv. Fuzzy Syst.; 2015; 2015, [DOI: https://dx.doi.org/10.1155/2015/251646]
78. Karaveg, C.; Thawesaengskulthai, N.; Chandrachai, A. A combined technique using SEM and TOPSIS for the commercialization capability of R & D project evaluation. Decis. Sci. Lett.; 2015; 4, pp. 379-396.
79. Jeng, D.; Huang, K.H. Strategic project portfolio selection for national research institutes. J. Bus. Res.; 2015; 68, pp. 2305-2311. [DOI: https://dx.doi.org/10.1016/j.jbusres.2015.06.016]
80. Yalcin, A.S.; Kilic, H.S.; Guler, E. Research and development project selection via IF-DEMATEL and IF-TOPSIS. Advances in Intelligent Systems and Computing; Springer: Berlin, Germany, 2020; pp. 625-633.
81. Wind, Y.; Saaty, T.L. Marketing applications of the analytic hierarchy process. Manag. Sci.; 1980; 26, pp. 641-658. [DOI: https://dx.doi.org/10.1287/mnsc.26.7.641]
82. Saaty, T.L. Decision making with the analytic network process (ANP) and its super decisions software: The national missile defense (NMD) example. Proceedings of the ISAHP 2001; Berne, Switzerland, 2–4 August 2001; pp. 2-4.
83. Group, B.C. Perspectives on Experience; Boston Consulting Group: Boston, MA, USA, 1970.
84. Norton, D.P.; Kaplan, R. The Balanced Scorecard: Translating Strategy into Action; Harvard Business School Press: Boston, MA, USA, 1999.
85. Mishan, E.J.; Euston, Q. Cost-Benefit Analysis; Number 658.1554 M678 Praeger: New York, NY, USA, London, UK, 1976.
86. Zavadskas, E.K.; Kaklauskas, A.; Sarka, V. The new method of multicriteria complex proportional assessment of projects. Technol. Econ. Dev. Econ.; 1994; 1, pp. 131-139.
87. Gabus, A.; Fontela, E. Perceptions of the World Problematique: Communication Procedure, Communicating with Those Bearing Collective Responsibility; DEMATEL Report No. 1 Battelle Geneva Research Center: Geneva, Switzerland, 1973.
88. Benayoun, R.; Roy, B.; Sussman, B. ELECTRE: Une méthode pour Guider le Choix en Présence de Points de vue Multiples; SEMA: Montrouge, France, 1996.
89. Keeney, R.L.; Raiffa, H. Decisions with Multiple Objectives: Preferences and Value Trade-Offs; Cambridge University Press: Cambridge, UK, 1993.
90. Brans, J.P.; Vincke, P. Note—A Preference Ranking Organisation Method: (The PROMETHEE Method for Multiple Criteria Decision-Making). Manag. Sci.; 1985; 31, pp. 647-656. [DOI: https://dx.doi.org/10.1287/mnsc.31.6.647]
91. Trigeorgis, L. Real Options in Capital Investment: Models, Strategies, and Applications; Greenwood Publishing Group: Westport, CT, USA, 1995.
92. Hwang, C.L.; Yoon, K. Methods for multiple attribute decision making. Multiple Attribute Decision Making; Springer: Berlin, Germany, 1981; pp. 58-191.
93. Opricovic, S. Fuzzy VIKOR with an application to water resources planning. Expert Syst. Appl.; 2011; 38, pp. 12983-12990. [DOI: https://dx.doi.org/10.1016/j.eswa.2011.04.097]
94. Samanlioglu, F.; Ayağ, Z. An intelligent approach for the evaluation of innovation projects. J. Intell. Fuzzy Syst.; 2020; 38, pp. 905-915. [DOI: https://dx.doi.org/10.3233/JIFS-179458]
95. Mohanty, R.; Agarwal, R.; Choudhury, A.; Tiwari, M. A fuzzy ANP-based approach to R&D project selection: A case study. Int. J. Prod. Res.; 2005; 43, pp. 5199-5216.
96. Hassanzadeh, F.; Collan, M.; Modarres, M. A Practical Approach to R&D Portfolio Selection Using the Fuzzy Pay-Off Method. IEEE Trans. Fuzzy Syst.; 2012; 20, pp. 615-622. [DOI: https://dx.doi.org/10.1109/TFUZZ.2011.2180380]
97. Hassanzadeh, F.; Collan, M.; Modarres, M. A practical R&D selection model using fuzzy pay-off method. Int. J. Av. Manuf. Technol.; 2012; 58, pp. 227-236. [DOI: https://dx.doi.org/10.1007/s00170-011-3364-9]
98. Hajkowicz, S.; Collins, K. A review of multiple criteria analysis for water resource planning and management. Water Resour. Manag.; 2007; 21, pp. 1553-1566. [DOI: https://dx.doi.org/10.1007/s11269-006-9112-5]
99. Velasquez, M.; Hester, P.T. An analysis of multi-criteria decision making methods. Int. J. Oper. Res.; 2013; 10, pp. 56-66.
100. Liang, D.; Xu, Z.; Liu, D.; Wu, Y. Method for three-way decisions using ideal TOPSIS solutions at Pythagorean fuzzy information. Inf. Sci.; 2018; 435, pp. 282-295. [DOI: https://dx.doi.org/10.1016/j.ins.2018.01.015]
101. Carlsson, C.; Fullér, R.; Heikkilä, M.; Majlender, P. A fuzzy approach to R&D project portfolio selection. Int. J. Approx. Reason.; 2007; 44, pp. 93-105. [DOI: https://dx.doi.org/10.1016/j.ijar.2006.07.003]
102. Oral, M.; Kettani, O.; Lang, P. A Methodology for Collective Evaluation and Selection of Industrial Research and Development projects. Manag. Sci.; 1991; 37, pp. 871-885. [DOI: https://dx.doi.org/10.1287/mnsc.37.7.871]
103. Karasakal, E.; Aker, P. A multicriteria sorting approach based on data envelopment analysis for R&D project selection problem. Omega; 2017; 73, pp. 79-92. [DOI: https://dx.doi.org/10.1016/j.omega.2016.12.006]
104. Wei, H.; Xia, B.; Yang, Z.; Zhou, Z. Model and Data-Driven System Portfolio Selection Based on Value and Risk. Appl. Sci.; 2019; 9, 1657. [DOI: https://dx.doi.org/10.3390/app9081657]
105. Schiffels, S.; Fliedner, T.; Kolisch, R. Human behavior in project portfolio selection: Insights from an experimental study. Decis. Sci.; 2018; 49, pp. 1061-1087. [DOI: https://dx.doi.org/10.1111/deci.12310]
106. Bell, D.; Read, A. The application of a research project selection method. R&D Manag.; 1970; 1, pp. 35-42. [DOI: https://dx.doi.org/10.1111/j.1467-9310.1970.tb01196.x]
107. Ringuest, J.; Graves, S. The Linear Multi-Objective R&D Project Selection Problem. IEEE Trans. Eng. Manag.; 1989; 36, pp. 54-57. [DOI: https://dx.doi.org/10.1109/17.19984]
108. Ringuest, J.; Graves, S. The Linear R&D Project Selection Problem: An Alternative to Net Present Value. IEEE Trans. Eng. Manag.; 1990; 37, pp. 143-146. [DOI: https://dx.doi.org/10.1109/17.53718]
109. Henig, M.; Katz, H. R&D project selection: A decision process approach. J.-Multi-Criteria Decis. Anal.; 1996; 5, pp. 169-177. [DOI: https://dx.doi.org/10.1002/(SICI)1099-1360(199609)5:3<169::AID-MCDA94>3.0.CO;2-V]
110. Beaujon, G.; Marin, S.; McDonald, G. Balancing and optimizing a portfolio of R&D projects. Nav. Res. Logist.; 2001; 48, pp. 18-40. [DOI: https://dx.doi.org/10.1002/1520-6750(200102)48:1<18::AID-NAV2>3.0.CO;2-7]
111. Ringuest, J.; Graves, S. Formulating optimal R&D portfolios. Res. Technol. Manag.; 2005; 48, pp. 42-47.
112. Fang, Y.; Chen, L.; Fukushima, M. A mixed R&D projects and securities portfolio selection model. Eur. J. Oper. Res.; 2008; 185, pp. 700-715. [DOI: https://dx.doi.org/10.1016/j.ejor.2007.01.002]
113. Eckhause, J.; Gabriel, S.; Hughes, D. An integer programming approach for evaluating R&D funding decisions with optimal budget allocations. IEEE Trans. Eng. Manag.; 2012; 59, pp. 679-691. [DOI: https://dx.doi.org/10.1109/TEM.2012.2183132]
114. Bhattacharyya, R. A Grey Theory Based Multiple Attribute Approach for R&D Project Portfolio Selection. Fuzzy Inf. Eng.; 2015; 7, pp. 211-225. [DOI: https://dx.doi.org/10.1016/j.fiae.2015.05.006]
115. Storch de Gracia, M.D.; Moya Perrino, D.; Llamas, B. Multicriteria methodology and hierarchical innovation in the energy sector: The Project Management Institute approach. Manag. Decis.; 2019; 57, pp. 1286-1303. [DOI: https://dx.doi.org/10.1108/MD-07-2017-0676]
116. Aghdaie, M.H.; Zolfani, S.H.; Cavallaro, F.; Morshed Varzandeh, M.H.; Derakhti, A.; Pamucar, D. Decision making on exigent issues in organisations: A case study on R&D projects. Transform. Bus. Econ.; 2020; 19, pp. 369-392.
117. Lin, C.J.; Wu, W.W. A causal analytical method for group decision-making under fuzzy environment. Expert Syst. Appl.; 2008; 34, pp. 205-213. [DOI: https://dx.doi.org/10.1016/j.eswa.2006.08.012]
118. Kawakita, J. The Original KJ Method; Kawakita Research Institute: Tokyo, Japan, 1991.
119. Awasthi, A.; Chauhan, S.S. A hybrid approach integrating Affinity Diagram, AHP and fuzzy TOPSIS for sustainable city logistics planning. Appl. Math. Model.; 2012; 36, pp. 573-584. [DOI: https://dx.doi.org/10.1016/j.apm.2011.07.033]
120. Awasthi, A.; Omrani, H. A goal-oriented approach based on fuzzy axiomatic design for sustainable mobility project selection. Int. J. Syst. Sci. Oper. Logist.; 2019; 6, pp. 86-98. [DOI: https://dx.doi.org/10.1080/23302674.2018.1435834]
121. Garza-Reyes, J.A. Lean and green—A systematic review of the state of the art literature. J. Clean. Prod.; 2015; 102, pp. 18-29. [DOI: https://dx.doi.org/10.1016/j.jclepro.2015.04.064]
You have requested "on-the-fly" machine translation of selected content from our databases. This functionality is provided solely for your convenience and is in no way intended to replace human translation. Show full disclaimer
Neither ProQuest nor its licensors make any representations or warranties with respect to the translations. The translations are automatically generated "AS IS" and "AS AVAILABLE" and are not retained in our systems. PROQUEST AND ITS LICENSORS SPECIFICALLY DISCLAIM ANY AND ALL EXPRESS OR IMPLIED WARRANTIES, INCLUDING WITHOUT LIMITATION, ANY WARRANTIES FOR AVAILABILITY, ACCURACY, TIMELINESS, COMPLETENESS, NON-INFRINGMENT, MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE. Your use of the translations is subject to all use restrictions contained in your Electronic Products License Agreement and by using the translation functionality you agree to forgo any and all claims against ProQuest or its licensors for your use of the translation functionality and any output derived there from. Hide full disclaimer
© 2021 by the authors. Licensee MDPI, Basel, Switzerland. This article is an open access article distributed under the terms and conditions of the Creative Commons Attribution (CC BY) license (https://creativecommons.org/licenses/by/4.0/). Notwithstanding the ProQuest Terms and Conditions, you may use this content in accordance with the terms of the License.
Abstract
From small spin-offs deploying innovative software to big pharmaceutical complexes making vaccines, Research and Development (R&D) Project Portfolio Selection (PPS) is an essential strategic process for various companies. It was never easy to select a set of projects among many feasible possibilities, even for yesterday’s paces. However, the world is rapidly changing, and so is R&D PPS. The portfolio objectives excel profit in the same manner that model constraints go beyond budget limitations. In parallel, project selection approaches and solving algorithms followed the increase of computational power. Despite all those changes, the importance of Multi-Criteria Decision Making (MCDM) methods and the decision criteria used for R&D PPS, there is still room for a systematic literature review (SLR) for the topic. Thus, this paper offers an SLR of the existing literature from the half-century, 1970, and onward MCDM-based R&D PPS performed in Scopus and Web of Science Core Collection. We provide a comprehensive picture of this field, show how it is changing, and highlight standard practices and research opportunities in the area. We perform a broad classification of the MCDM methods, categorized by the nature of alternatives, types of integration approach, the MCDM method itself, and types of uncertainty, by the 66 studies in the SLR database. The portfolios’ classification obeys the application domain and the number of projects. We have also explored all the 263 criteria found in the literature by grouping them according to experts from five Brazilian R&D organizations that together manage portfolios valued around US$ 5 billion a year, accounting for 38% of all Brazilian annual expenditure in R&D projects. We also include a bibliometric analysis of the considered papers and research opportunities highlighted or not explored by researchers. Given the increasing number of decision-making approaches and new technologies available, we hope to provide guidance on the topic and promote knowledge production and growth concerning the usage of MCDM methods and decision criteria in R&D PPS.
You have requested "on-the-fly" machine translation of selected content from our databases. This functionality is provided solely for your convenience and is in no way intended to replace human translation. Show full disclaimer
Neither ProQuest nor its licensors make any representations or warranties with respect to the translations. The translations are automatically generated "AS IS" and "AS AVAILABLE" and are not retained in our systems. PROQUEST AND ITS LICENSORS SPECIFICALLY DISCLAIM ANY AND ALL EXPRESS OR IMPLIED WARRANTIES, INCLUDING WITHOUT LIMITATION, ANY WARRANTIES FOR AVAILABILITY, ACCURACY, TIMELINESS, COMPLETENESS, NON-INFRINGMENT, MERCHANTABILITY OR FITNESS FOR A PARTICULAR PURPOSE. Your use of the translations is subject to all use restrictions contained in your Electronic Products License Agreement and by using the translation functionality you agree to forgo any and all claims against ProQuest or its licensors for your use of the translation functionality and any output derived there from. Hide full disclaimer
Details
; dos Santos, Erivelton Antonio 2
; Nei Yoshihiro Soma 3 ; Carlos Eduardo Sanches da Silva 4 1 Division of Computer Science, Aeronautics Institute of Technology, São José dos Campos 12.228-900, Brazil;
2 Institute of Industrial Engineering and Management, Federal University of Itajubá, Itajubá 37.500-903, Brazil;
3 Division of Computer Science, Aeronautics Institute of Technology, São José dos Campos 12.228-900, Brazil;
4 Institute of Industrial Engineering and Management, Federal University of Itajubá, Itajubá 37.500-903, Brazil;