VHB-JOURQUAL2: Method, Results, and Implications of the German Academic Association for Business Research's Journal Ranking

Abstract

VHB-JOURQUAL represents the official journal ranking of the German Academic Association for Business Research. Since its introduction in 2003, the ranking has become the most influential journal evaluation approach in German-speaking countries, impacting several key managerial decisions of German, Austrian, and Swiss business schools. This article reports the methodological approach of the ranking's second edition. It also presents the main results and additional analyses on the validity of the rating and the underlying decision processes of the respondents. Selected implications for researchers and higher-education institutions are discussed. [PUBLICATION ABSTRACT]

Headnote

Abstract

VHB-JOURQUAL represents the official journal ranking of the German Academic Association for Business Research. Since its introduction in 2003, the ranking has become the most influential journal evaluation approach in German-speaking countries, impacting several key managerial decisions of German, Austrian, and Swiss business schools. This article reports the methodological approach of the ranking's second edition. It also presents the main results and additional analyses on the validity of the rating and the underlying decision processes of the respondents. Selected implications for researchers and higher-education institutions are discussed.

Keywords: Journal ranking, rating, impact factors, VHB-JOURQUAL

This article was commissioned by the Editor-in-Chief and went through the review process. Manuscript received September 10, 2009, accepted by Sönke Albers (Editor-in-chief) November 9, 2009.

(ProQuest: ... denotes formulae omitted.)

1 Introduction

VHB-JOURQUAL represents the official journal ranking of the German Academic Association for Business Research (Verband der Hochschullehrer für Betriebswirtschaftslehre - VHB). It rates and ranks international and German-language academic journals which are considered relevant for Germanspeaking business researchers based on the quality assessments of VHB members who comprise more than 90% of all professors and researchers with a Ph.D. in business administration at German, Austrian, and Swiss universities. Since its initial publication in 2003 (Hennig-Thurau, Walsh, and Schrader 2004), the ranking has become the most prominent business research-journal ranking in German-speaking countries.

The results of an online survey of VHB members in 2007 (n = 489) give evidence that VHB-JOURQUAL is widely accepted and broadly used for evaluating the scientific performance of business scholars in Austria, Germany and the Germanspeaking part of Switzerland (Schrader and Hennig- Thurau 2007). 59 % of the respondents assessed VHB-JOURQUAL as "good" or "very good", while only 9 % held a negative attitude toward the ranking. VHB-JOURQUAL is of special importance for the formal post-doctoral assessment in Germanspeaking countries - the so-called "Habilitation," a traditional requirement for obtaining a full professorship - and the appointment of full professors; in each case, 54 % of the respondents judged the ranking being of "high" or "very high" relevance for their institutions. Other areas for which VHBJOURQUAL is considered to have substantial relevance include the hiring and evaluation of assistant professors ("Junior-Professoren"), research performance evaluations (with a possible impact on budget allocation), and the evaluation of the rapidly growing number of cumulative doctoral dissertations.

VHB-JOURQUAL is considered a major driver of the radical transformation of the German businessadministration community into a much more research- focused and internationally active academic discipline (Homburg 2008). In an environment which Simon (1993) described as a "black hole" - with German-speaking business scholars absorbing their international colleagues' findings, but giving nothing back to the international community - , VHB-JOURQUAL offered for the first time a comparison of the scientific quality of articles published in German-language journals with those published in international journals. As the ranking provided strong evidence that the quality of even the best German-language journals is perceived to be substantially lower than the quality of leading international journals, VHB-JOURQUAL has since then become a strong motivating force for German scholars to publish their best work internationally and enter "global competition".

This article describes the methodology of VHBJOURQUAL and reports specific features and key results of the second edition of the ranking (VHBJOURQUAL2), which is based on a survey conducted among VHB members in 2008. We also investigate the ranking's validity and provide concluding remarks on the benefits and limitations of VHB-JOURQUAL.

2 Background: Survey-based vs. citation based journal-rankings

There are two basic approaches for conducting academic- journal rankings: citation-based rankings (e.g., Azar and Brock 2008; Ritzberger 2008; Vieira 2008) and survey-based rankings (e.g., Bräuninger and Haucap 2002; Hennig-Thurau, Walsh, and Schrader 2004). Hybrid rankings, as a third ranking type, combine both approaches (e.g., Franke and Schreier 2008; Schulze, Warning, and Wiermann 2008). We will compare the different approaches in terms of strengths and weaknesses and explain how VHB-JOURQUAL addresses them.

Citation-based journal rankings such as the ISI Journal Citation Reports are often considered "objective" (e.g., Ritzberger 2008). Here, the ranking position of a journal depends on the number of citations the papers in the journal receive. The idea behind it is that citations are "the scientific community's version of dollar voting by consumers for goods and services" (Laband and Piette 1994a: 641). Similar to the way economic theory considers payment at the cash desk as the result of a quality assessment by consumers, a citation-based ranking treats a citation as a proof of perceived quality. However, there are two major caveats to this approach, particularly when audiences are interested in the scientific quality of a journal (e.g., Schulze, Warning, and Wiermann 2008):

* Quality vs. impact. Citations are not objective indicators for an article's scientific quality. The decision to accept a submitted article, making it available for citations, is subjective and not only determined by scientific quality (Blank 1991; Frey and Rost 2008; Laband and Piette 1994b; Starbuck 2004), and authors' decision to cite an article is, in addition to quality, also influenced by the article's type and topic (e.g., state-of-theart review), its shortcomings as well as citation cartel memberships (e.g., Fabel and Heße 1999). Thus, citation indexes do not measure the scientific quality but rather the impact of a journal. The ISI Journal Citation Reports ranks a journal according to its "impact factor", not to a quality index. Impact and scientific quality should not be treated as synonyms since the empirical correlations between them can be weak or even negative in some cases (e.g., Maier 2006; Schlinghoff and Backes-Gellner 2002).

* Data availability. Citation data can only be obtained from a limited number of sources and is not available for a substantial number of journals. The market-dominating provider of interdisciplinary citation indexes is Thomson Reuters with the Social Science Citation Index (SSCI), the Science Citation Index-Expanded (SCI-X), the Arts & Humanities Citation Index (A&HCI), the Conference Proceedings Citation Index - Science (CPCI-S), and the Conference Proceedings Citation Index Social - Science & Humanities (CPCI-SSH), constituting the "ISI Web of Knowledge." Elsevier introduced the competing database Scopus in 2004, with limited market penetration so far. For business researchers, the most important index is the SSCI with its business, business/finance, and management categories, which - as reported later in the section on ranking validation - cover only about 20 % of the journals considered relevant for business scholars and included in VHB-JOURQUAL2, with only one German-language journal being represented there (BFuP - Betriebswirtschaftliche Forschung und Praxis). Considering all SSCI categories and other indexes mentioned above, these numbers will be only slightly higher (Clermont and Schmitz 2008; Dyckhoff and Schmitz 2007). In 2009, the ISI covered in total nine and Scopus 12 German-language journals ranked in VHB-JOURQUAL2, which have predominantly low ratings (Clermont 2009). Consequently, in German-speaking countries citation-based rankings are hardly able to measure publication productivity of business scholars (Albers 2009; Dilger 2000).

Survey-based rankings also face limitations. However, these obstacles appear less systematic and strongly depend on the mode of data collection and analysis when compared to those of citation-based rankings. Major threats to the validity of surveybased rankings include the following issues:1

* Sample. Respondents from whose assessments survey-based rankings are constructed are not always suited for assessing the scientific quality of academic journals. VHB-JOURQUAL aims to overcome this potential threat by including only VHB members who, at a minimum, hold a postdoctoral position, with the majority being full professors. VHB-JOURQUAL also requires active readership of a journal to rate its quality (it first asks the respondents to indicate which journals they have recently read and only in a second step asks respondents to evaluate those journals only) and uses an expertise factor to account for the respondents' differing levels of expertise (see also Heischmidt and Gordon 1993; Extejt and Smith 1990).

* Too few or too many journals. Survey-based rankings are often forced to limit the number of included journals to avoid overstraining respondents. VHB-JOURQUAL assures completeness of relevant journals by using a multistage process starting with the Journal Quality List by Anne-Wil Harzing (for VHBJOURQUAL2: Harzing 2007), limiting the danger of overstraining by a highly customized online survey design. Specifically, all journals are assigned to business sub-disciplines (e.g., accounting, finance), and respondents are asked to evaluate only journal titles relevant for their specific sub-discipline.

* Strategic answers. Opponents of the surveybased approach argue that researchers not always evaluate journals according to their actual quality perception, but rather in a way that is best for them (e.g., Schulze, Warning, and Wiermann 2008). Specifically, given that journal rankings can influence careers, scholars will have an interest that the journals they publish in or serve for as editors or reviewers are highly ranked; something which they can influence through their own ratings. As the incentive for such strategic answers strongly depends on the individual researcher's impact on a journal's rating, VHB-JOURQUAL only considers journals with at least ten individual ratings. Moreover, ratings in VHB-JOURQUAL2 are not anonymous - each respondent had to use a unique ID and agree that his data could be linked to his personal information by the authors of this study. This should reduce the motivation for strategic answers, since evident over- or underrating may become overt to the VHB-JOURQUAL authors. In addition, outlier judgments were systematically removed as will be explained later in more detail.

Finally, hybrid rankings combine data from expert surveys and citation indices. This extends the number of journals and bases the evaluation on a broader foundation. However, the hybrid approach does not heal limitations inherent in the sources which serve as necessary inputs such as missing citation data for German-language journals.

3 Measuring journal quality in VHB-JOURQUAL

Survey-based rankings usually solely focus on the overall quality of the articles that are published by a certain journal. In VHB-JOURQUAL, we conceptualize overall scientific quality of a journal as being defined by two quality dimensions which are measured separately, namely the quality of the articles published in a journal (article quality) and the quality of the review process of the journal (review quality). Both quality dimensions are treated as forma- tive indicators of overall journal quality (for a similar approach, see Rossiter 2002). Our distinction between article and review quality draws from quality literature in related fields such as service management, which has often been used as a template for higher-education research (e.g., Hennig-Thurau, Langer, and Hansen 2001).

Service research distinguishes between outcomerelated quality aspects and those aspects which are related to the process of the service production as dimensions of overall (perceived) service quality (e.g., Brady and Cronin 2001). While article quality serves as the equivalent of outcome quality in the context of academic-knowledge generation, review quality refers to the process of how articles (and their authors) are treated by the respective journal. In that sense, article and review quality measure different dimensions of academic-knowledge generation. Not all academics (as "customers") experience both quality dimensions; while article quality can be assessed by all readers of a journal's articles (including those who had only limited contact with a journal), the assessment of review quality requires deeper experiences and insight into a journal's internal processes. The inclusion of review quality not only helps to capture scientific quality in a more comprehensive way, but also adds dynamics and flexibility, as changes in the review process will be experienced much faster than their manifestation in printed articles.

The VHB-JOURQUAL index, on which the rating and ranking of journals is based, combines the two quality dimensions of article quality and review quality in a weighted additive composition (see equation 1). Both quality dimensions are measured with single items which is an adequate procedure for expert surveys with formative measures, as is the case with VHB-JOURQUAL (Rossiter 2002).

(1) ...

with JQIJ : VHB-JOURQUAL index value of journal J (10-point scale from 1 = 'very low' to 10 = 'very high'),

AQJ,i : Scientific quality of articles in journal J as perceived by respondent i (on a 10-point scale from 1 = 'very low' to 10 = 'very high'),

RQJ,i : Review quality of journal J as perceived by respondent i (on a 10-point scale from 1 = 'very low' to 10 = 'very high'),

nJ,AQ : Number of respondents who have evaluated the article quality of journal J,

nJ,AQ : Number of respondents who have evaluated the article quality of journal J,

Ei : Weighting factor for the expertise of respondent i,

aJ, bJ : Weighting factors for journal J, with aJ + bJ = 1.

Article quality is measured with the item "I consider the scientific standard of articles published in this journal to be ... [Please indicate a number between 1 = 'extremely low' und 10 = 'extremely high']". The original German wording of this item is: "Das wissenschaftliche Niveau der in dieser Zeitschrift veröffentlichten Artikel halte ich für...[Bitte Zahl zwischen 1 = 'extrem gering' und 10 = 'extrem hoch' angeben]." The item is evaluated by respondents who have read at least one new article in the respective journal in the last five years (VHB-JOUR QUAL2: 2003-2007).

Review quality is measured with the item "The scientific quality of the review process is ... [Please indicate a number between 1 = 'extremely low' und 10 = 'extremely high']". The original German wording of the item is: "Die wissenschaftliche Qualität des Review-Prozesses ist...[Bitte Zahl zwischen 1 = 'extrem gering' und 10 = 'extrem hoch' angeben]." The scientific quality of the review process is defined as "standards for submissions requested by reviewers and/or editors" (in German: "die Ansprüche, die Gutachter und/oder Schriftleiter bzw. Editoren an eingereichte Beiträge stellen"). Review quality is rated only by those who have either submitted at least one paper within the last five years or know the review process as reviewers or editors of the journal. In VHB-JOURQUAL1 only authors who had submitted papers were allowed to evaluate the review process. Also letting reviewers and editors evaluate review quality leads to a higher number of reviewprocess evaluations and reduces the likelihood that ratings and rankings are mainly or only based on article-quality assessments.

We weight the formative indicators of article quality and review quality equal (both aJ and bJ = 0.5), assuming both play the same role for determining overall scientific quality of a journal. However, to account for the limited number of review evalua tions for several journals and the loss in reliability associated with a small sample size for the reviewquality indicator, we adjusted the weights for article and review quality if the number of review quality assessments fell short of a certain threshold. Equation 2 lists the adjustments in weighting the two quality criteria for different numbers of ratings of a journal's review quality. If no respondent has rated a journal's review process, the weight for b becomes 0 and (since aJ = 1 - bJ) the overall JOURQUAL rating is solely based on the article-quality indicator (with aJ = 1).

(2) ...

Expertise is included in the calculation of the VHBJOURQUAL2 overall quality score as a weighting factor since we expect a positive correlation between the expertise of a respondent and the validity of her or his assessment. We argue that scholars who have published in a variety of journals, and particularly in leading international journals, will be better able to judge the quality of articles and review processes than colleagues who lack that experience. Specifically, we operationalize expertise by three indicators: (a) the number of different journals a respondent has published in during the last five years; (b) the number of publications in high-quality journals (defined as journals with an unweighted VHBJOURQUAL index value . 7) during the last five years; and (c) the international experience gathered by publications in English-language high-quality journals during the last five years. For the first two indicators the raw values are transformed to scales ranging from 1 to 2; for indicator (c) respondents with at least one international high-quality journal publication receive a 2, all others a 1. The three indicators are multiplied and then rescaled to an expertise scale ranging from 1 to 5. As a consequence, the evaluations of the respondents scoring with the highest expertise are weighted five times higher than the assessments of the respondents with the lowest expertise. We prefer a multiplicative function of expertise over an additive one, as we argue that academic expertise builds up in a non-linear way, with the three indicators' impact on overall expertise not being independent.

While the journal ranking results directly from the different VHB-JOURQUAL index values, the rating assigned to an index value requires a categorization. VHB-JOURQUAL sorts journals into rating categories based on their index values, using absolute numbers as thresholds for defining the categories for the lack of a more objective categorization. Table 1 lists the six rating categories ranging for A+ to E and the respective VHB-JOURQUAL scores.

4 Survey and sample of VHBJOURQUAL2

The VHB-JOURQUAL2 survey was conducted from February to April 2008. Every VHB member with a registered email address (N = 1,555) received an invitation email from the VHB chairmen and the authors of the study which contained a unique link to an online questionnaire. Every participant went through the highly individualized survey procedure depending on his or her specific research fields, reading habits, submission activities and reviewer or editor positions. 1,011 respondents started the evaluation process (response rate of 65 %), about 600 fully completed the questionnaire. Many respondents who "dropped out" had actually proceeded through major parts of the survey. We included all responses in our calculations regardless of technical completion.

The questionnaire included a total of 1,633 journals. In addition to academic journals in a narrow sense, the list also comprises yearbooks and proceedings with a homogeneous review process. This is due to the observation that in some business subdisciplines (such as information systems) refereed conference proceedings fulfill similar tasks than actual journals. The list of journals for VHB- JOURQUAL2 was selected in a multi-step process. First, we synchronized the VHB-JOURQUAL1 journal list with the established Harzing-List of business- administration journals (Harzing 2007). Second, we assigned the journals to different business sub-disciplines which strongly overlapped with the 16 sections of the VHB ("Wissenschaftliche Kommissionen"; e.g., finance and banking, taxation, international management). Third, in a joint effort with all chairpersons of the VHB sections, we added new or previously overlooked journals, eliminated those which have ceased publication and adjusted the categorization. Forth, we invited all VHB members to complete the journal list and asked them to indicate which journals they actually read in an online pre-survey (n = 489). The final list then included all journals which were either indicated by at least two respondents in the pre-survey or evaluated by at least five respondents in VHB-JOURQUAL1. Journals which did not meet one of these requirements were excluded as we expected them not to receive the minimum number of ten evaluations in the main survey required for the inclusion in the final ranking.

To reduce the number of strategic answers, we limited the anonymity of the study by informing the participants in the invitation mail that the authors of the study would be able to assign every rating to the individual respondent. Also, indicating the number of submissions to and publications in a journal was not sufficient; respondents also had to name respective papers' short titles. This information was used to assure a valid calculation of expert factors and that review processes are only evaluated by respondents who have actually experienced them within the period of observation. While we are aware that these measures cannot completely remove all kinds of strategic behavior, they certainly increased the psychological barriers to behave in such a way.

To further improve the quality of our data, we excluded outlier ratings when calculating the VHBJOURQUAL2 ratings, as we assumed outliers to be based on misunderstandings or strategic misevaluations. Specifically, for each journal, we kept all responses within the 99 % confidence interval (twosided) for both dimensions of quality (i.e. article and review quality) and deleted those responses outside the confidence interval. This procedure resulted in the removal of a total of 315 individual ratings. About 90 % of these deleted ratings were below the confidence interval; thus, the risk of overrating seems to be lower than the risk of underrating.

5 VHB-JOURQUAL2 results

5.1 General Results

742 journals received more than 10 ratings in VHBJOURQUAL2; journals which received less than 10 ratings were excluded to assure a sufficient level of reliability (Web-Appendix 1 contains the alphabetical list of all 742 journals).2 As VHB-JOURQUAL2 aims at business scholars, we only included journals in the ranking which - in addition to being read by at least 10 respondents - could be assigned to at least one sub-discipline of business administration (e.g., accounting, marketing) or whose review process was evaluated by a minimum of 5 respondents. 666 journals met at least one of these criteria and were subsequently included in the VHBJOURQUAL2 journal ranking.

Figure 1 shows the distribution of VHB-JOURQUAL2 (JQ2) ratings for all 666 journals for the overall VHB-JOURQUAL2 index value as well as the two quality dimensions and lists descriptive information. The mean of the overall JQ2 score is 6.22, equal to a C-rating.

Figure 2 displays the distributions for the quality weighting parameters and the respondent expertise parameter. The weighting factor for the two quality criteria varied between 0 and 0.5, with an average weighting for the review process of .24 (standard deviation = .18).

As can be seen in Figure 3, the expertise factor for the respondents has a mean value of 1.36 (standard deviation = .58) and is positively skewed (the forth quartile ranges from 1.74 to 5), that is, relatively few researchers have very high expertise scores. This is consistent with the finding of Dyckhoff and Schmitz (2007) that about two out of three German professors had no international refereed publication between 1990 and 2004; something which has changed only recently (Homburg 2008). Figure 3 also shows the distribution of the three expertise indicators.

Table 2 lists the A+ and A ranked journals according to their overall quality index value and provides additional information on the assessments of the VHB-JOURQUAL quality dimensions and changes between JQ1 and JQ2. The full list of all ranked journals is reported in Web-Appendix 2. In addition to the expertise-weighted index values, we also report the unweighted quality assessments for each journal.

Within the top ten journals are four marketing and three finance journals, with the Journal of Finance being listed as the number one journal. Administrative Science Quarterly, ranked fourth, has the highest JQ2 index value of all general management journals. 14 Journals (= 2 %) are rated A+, 50 (= 8 %) are rated A, 152 (= 23 %) are rated B, 186 (= 28 %) are rated C, 143 (= 22 %) are rated D, and 121 (= 18 %) are rated E. The best German-language journal is the B-rated Wirtschaftsinformatik (since 2009 also available in English as Business & Information Systems Engineering) ranked 169, followed by Schmalenbachs Zeitschrift für betriebswirtschaftliche Forschung (zfbf) ranked 177.

Considering the changes from JQ1 to JQ2, the ratings on average have decreased. Taking into account only those 326 Journals which were included in both rankings, the mean score for the overall quality has fallen from 6.86 to 6.22; 275 journals (or 84 %) received a lower and only 51 a higher quality assessment than five years ago. This indicates that respondents have not systematically overrated their preferred journals, but displayed a critical attitude in general. This attitude might be the result of a growing sensitivity among VHB members with regard to academic journal quality; a trend which has certainly been strengthened by the increasing number of VHB members which have experienced international journal review processes.

The results also reveal that the double blind reviewing process has become a condition sine qua non for a high-quality journal among VHB members, following international standards. The journals with the highest decrease (by percentage) in quality rating did not employ a rigorous review process during the last five years (Appendix 3 lists the journals and their respective changes since JQ1). Most of the journals with a high decrease in quality perception had already a below average rating in JQ1.

Table 3 lists the best-rated journals for 16 businessadministration sub-disciplines; the disciplines were selected based on the VHB sections structure. The table also reveals to which extent the journal-quality ratings are affected by judgments of researchers who do not belong to a specific sub-discipline. In other words: Does the "core audience" of a journal judge its quality differently than other scholars?

As can be seen, differences between quality ratings are usually very limited. Specifically, for the journals listed in Table 3, total ratings and those based on section members correlate with r = .91 (p < .01, n = 133) in the case of article quality and with r = .92 (p < .01, n = 96) for review quality. The average ratings are slightly higher for the section members (7.85 vs. 7.57 for article quality; 8.30 vs. 8.08 for reviewprocess quality), but the difference is not significant for any of the two quality indicators.

In our model, we assume that article quality and review quality define the overall scientific quality of an academic journal. Both constitute different, but related dimensions of quality, as a high-quality process will usually go hand in hand with high outcome quality; something which is also reflected by a correlation of r = .75 between the two quality dimensions. The merit of measuring quality via the two dimensions becomes apparent when studying those journals for which both quality criteria differ substantially. Table 4 lists those journals whose articles and review standards are perceived most differently by the respondents.

Among the journals whose articles receive better quality ratings than the review process are some which are explicitly positioned as "transfer" journals, linking scientific insights with managerial audiences (e.g., Harvard Business Manager, Sloan Management Review). Those journals which receive higher review ratings than article ratings, however, seem to have difficulties to fully transfer the quality of their review process into their final product, the published articles. An alternative explanation might be that these journals have increased the quality of their review through procedural changes only recently, but the change has not reached the majority of the journals' readership yet, since article-related ratings can be expected to be more resistant to change than review ratings.

5.2 Respondent-level Determinants of Quality Assessments

To learn which variables explain the interpersonal differences in journal-quality ratings and to shed more light on the unobserved heterogeneity which underlies the aggregated results, we conducted an additional post-hoc analysis. We focused on the three most prominent German-language businessresearch journals, namely Die Betriebswirtschaft - DBW, Zeitschrift für Betriebswirtschaft - ZfB, Schmalenbachs Zeitschrift für betriebswirtschaftliche Forschung - zfbf (Macharzina, Wolf, and Rohn 2004; Schlinghoff and Backes-Gellner 2002). This selection offers two main advantages: these journals (a) are read by a large number of respondents, and they (b) contain articles from different subdisci- plines of business administration, attracting a highly diverse readership.

We conducted OLS regressions for each of the three journals, with the perceived article quality serving as dependent variable. As independent variables, we included the individual respondent's expertise factor, his or her affiliation with the 16 VHB sections which represent business-administration subdisciplines (scholars can be affiliated with multiple sections), the respondent's status as a board member of the respective journal, his or her academic status (i.e. full professor or not), as well as age and gender as demographic characteristics. The regression results are reported in Table 5.

The results show a substantial amount of overlap between the three journals; a correlation analysis with the standardized regression coefficients as cases exhibits correlations of r = .66 (DBW/zfbf), r = .68 (DBW/ZfB), and r = .84 (zfbf/ZfB). Consistently, corporate taxation researchers tend to rate the German business-administration journals higher than scholars from other sub-disciplines; the same is true for accounting scholars. These findings might be attributed to the fact that in both sub-disciplines the domestic legal framework plays a crucial role, so that research in these fields will have a stronger focus on national issues. Consequently, the relative importance of the domestic scientific community and of its German-language journals might be higher than in other disciplines. As the top domestic journals are the best possible publication outlets for these researchers, competition for publication space and article quality might also be relatively higher.

Furthermore, we see that scholars with higher levels of research expertise tend to rate the Germanlanguage general business-administration journals lower; the effect is significant for two of the three journals. Obviously, experience with review processes of international journals - a major facet of research expertise - tends to heighten the researcher's comparison standard, resulting in lower evaluations for German business-administration journals. Also, established scholars tend to hold a more critical attitude - ratings of full professors tend to be lower for the German business-administration journals, with the effect being significant again for two of the three journals.

6 Validation of VHB-JOURQUAL2

To test the reliability and validity of VHBJOURQUAL2, we compare its results with a number of other international ratings: the first edition of VHB-JOURQUAL from 2003 (i.e. VHB-JOURQUAL1), the 2008 ISI Journal Citation Impact Factors (ISI Impact Factors), the British Association of Business Schools Academic Journal Quality Guide from 2009 (ABS09), the French Centre National de la Recherche Scientifique ranking from 2008 (CNR08), and the Dutch Erasmus Research Institute of Management Journals Listing from 2006 (EJL06). With the exception of the ISI Impact Factors (due to copyright issues), all these rankings are included in the Harzing list (Harzing 2009) and - for the journals ranked in VHB-JOURQUAL2 - also in Web-Appendix 2 of this paper.

The comparison with VHB-JOURQUAL1 allows us to assess the reliability of the results, as the method and population for both rankings is very similar. Although differences will result from changes in journal quality over time and perception changes, they should be of a somewhat limited size. VHBJOURQUAL1 itself has been successfully validated in comparison with leading international surveybased journal rankings (Hennig-Thurau, Walsh, and Schrader 2004). For the 666 businessadministration journals in VHB-JOURQUAL2, we collected the VHB-JOURQUAL1 index values of journals in the final ranking (nJ . 10). Data was available for 326 journals from VHB-JOURQUAL1. We find that the correlation is significant and substantial, with r = .94 (p < .01).

Regarding the comparison between VHB-JOURQUAL2 and the ISI Impact Factors, it is important to see that both ratings measure related, but distinct constructs. While VHB-JOURQUAL2 explicitly focuses on the scientific quality of a journal, ISI Impact Factors are an established measure which reflects the degree to which a journal's articles are read and actively cited by researchers. Since highquality articles are on average more likely to be cited by scholars than low-quality ones (e.g., Hult, Reimann, and Schilke 2009), a significant correlation between the two rankings can be expected. However, due to the conceptual differences between the two constructs, the correlation between VHBJOURQUAL2 and the ISI Impact Factors should be weaker than the correlation between the two versions of VHB-JOURQUAL. We collected the impact factors from the ISI Journal Citation Report Edition 2008 (Thomson Reuters 2009) for the disciplines business, business/finance, and management. Data was available for 137 journals listed in VHB-JOURQUAL2. We find that the correlation between VHBJOURQUAL2 and the ISI Impact Factors is r = .57 (p < .01); it is slightly higher (r = .59) when quadratic scores are used to account for the skewed distribution of the ISI Impact Factors. In addition to being significant and substantial, these correlations are also substantially lower than the one between VHB-JOURQUAL2 and VHB-JOURQUAL1, which is in line with our theoretical arguments.

Finally, when comparing the VHB-JOURQUAL2 ratings with the international journal rankings listed above, we ran pairwise comparisons and determined the correlation between VHB-JOUQUAL2 and each ranking. When doing this, we included all journals which are considered in VHB-JOURQUAL2 and the respective comparison ranking (n = 329 for ABS09; n = 233 for CNR08; n = 207 for EJL06). In each case the correlation is strong and significant (p < .01) with r = .64 for ABS09, r = .70 for CNR08, and r = .56 for EJL06. It is again consistent with our arguments that the correlation between VHB-JOURQUAL2 and EJL06 is relatively weaker than between VHB-JOURQUAL2 and the two other rankings, as EJL06 is partly based on citations.

In summary, we interpret these results as strong support for VHB-JOURQUAL2's reliability and validity.

7 Discussion, Implications, and Future Research Perspectives

7.1 Discussion and Implications

This article reports the results of VHB-JOURQUAL2, a survey-based ranking of 666 businessadministration journals, and details the underlying methodology. In addition to presenting the results for the different journals, we provide empirical evidence for the ranking's reliability and validity. Thus, we have confidence that VHB-JOURQUAL2 is a sound instrument to evaluate the journal-publishing achievements of business researchers. As such, we believe that the major contribution of this ranking is its ability to reduce the level of arbitrariness and the importance of non-performance-related network characteristics (such as "academic provenance") from key decisions made at universities - something which will benefit both universities and good scholars.

As its predecessor, VHB-JOURQUAL2 carries the potential to stimulate business researchers in Ger- many, Austria, and Switzerland to compete with colleagues from around the world for publication space in leading international journals, which are highlighted in this ranking. By doing this, VHBJOURQUAL2 might further raise the level of global competitiveness of the German-speaking businessadministration community, a trend which we already see as a result of the existence of VHBJOURQUAL1 (Homburg 2008). Furthermore, we hope that the ranking will also help to make scholars' intellectual achievements much easier to communicate to colleagues, department heads, deans, and rectors, a precondition for getting adequate rewards for such achievements.

Nevertheless, we believe that the results reported here have to be treated with great care. We have ambiguous feelings when we read job postings for full professorships which say that the "scientific performance is mainly evaluated by number and quality of scientific publications in international journals according to the VHB-JOURQUAL Ranking" (like the University of Siegen in 2008, own translation). Although this indicates that VHBJOURQUAL indeed influences the community (and that we have reached an objective we had when we once initiated it), we see the danger that the importance of the VHB-JOURQUAL ranking might be carried to excess. The scientific performance - not to say the overall performance - of an academic must not be solely judged on the basis of a single criterion, that is, a scholar's top journal publications according to VHB-JOURQUAL. As Albers (2009: 361) states, "we should be aware that any ranking can only provide a small piece of the overall performance picture". Business researchers have to be careful not to over-emphasize the part of the picture which is measurable by ratings and rankings today, since the result would be counterproductive for the whole profession (Adler and Harzing 2009). As a consequence, we see a strong need for additional rankings of scholarly performances. Alternative ratings that measure business researchers' contributions in journal articles might use VHB-JOURQUAL as a comprehensive and powerful source for integrating survey-based and citation-based approaches into hybrid rankings; other rankings might want to emphasize a journal's reputation or its importance for knowledge transfer. The discrepancy between scientific quality and relevance for business managers has been shown by Oesterle (2006), who reported a significant negative correlation between VHB-JOURQUAL results and academic-journal use by German managers with a PhD. In addition to different journal evaluations, powerful measurement tools for books, teaching, or university management achievements would be valuable to avoid the threat of one-dimensional university professors (Frey 2007).

However, VHB-JOURQUAL should not be blamed for its occasional misuse. It undisputedly covers a key facet of scholars' professional performance by providing a reliable and valid estimate of the scientific quality of a business researcher's journal articles. It should be treated as such, no more, but also no less.

7.2 Future Research Perspectives

While the current state of VHB-JOURQUAL provides an established tool for research-performance evaluation, it also raises questions which should be considered as opportunities for future research. Regarding the VHB-JOURQUAL methodology, it is obvious that some elements are based on pragmatic considerations and might be considered arbitrary, at least to a certain extent. So we encourage future research to identify more theoretically and/or empirically justifiable approaches for the following aspects of VHB-JOURQUAL and compare their results with the current ranking:

Definition and labeling of rating categories: Instead of measuring quality on a non-labeled 10- point scale and assigning the categories ex-post, the respondent might be asked to use the category labels themselves. While this was not possible in the first edition of the ranking (and perhaps might have raised substantial problems even when collecting data for the second edition), the category labels are now widely established among German-speaking business scholars. Alternatively, an empirical approach using empirical distributions might be chosen to assign rating categories, which reduces the danger that journals fall directly below the threshold between two categories.

Selection, weighting, and composition of indicators for the expertise factor: Regarding respondents' expertise, two relevant questions refer to the dimensions of expertise and their composition: Should other factors than those currently represented by the expertise construct be considered when measuring expertise? And is the current multiplicative composition optimal - and how would a different combination of expert dimensions affect the results? However, the current results show that the impact of the expertise factor should not be overrated, as expertise-weighted results and unweighted results do not differ substantially for most journals.

Weighting of article and review quality: The current version of VHB-JOURQUAL posits that article and review quality are of equal importance for constructing the overall quality score of a journal (with adjustments if only a small number of judgments exists for review quality). An alternative approach would be to empirically determine the relevance (or factor weights) of the two quality dimensions. For example, conjoint measurement approaches can support (or reject) our decision for weighting of both determinants equal. Also, the weight correction for the review quality dimension in the case of limited review-related judgments might be questioned and potentially improved. Especially the potentially strong effect of low numbers of review quality ratings which differs substantially from the usually much higher number of article ratings would deserve additional thought; maybe it would be advantageous to treat review assessments as outliers in such a case. At the same time, theoretical or empirical arguments would be valuable to demonstrate the superiority of alternative approaches.

Minimum number of article-quality evaluations: While the current version of VHB-JOURQUAL considers a minimum number of 10 ratings as the threshold for the inclusion of a journal, future research might address whether this number is adequate to guarantee sufficient reliability or if lower numbers are possible - or a higher number required.

In addition, future studies would be welcome which provide insight why most of the highest-ranked journals come from marketing and finance. Both are large and global disciplines with a long tradition in journal ratings, which had also a large number of respondents in VHB-JOURQUAL2. We tested for an impact of the number of raters of a journal on the journal's quality assessment, but found none within our sample (neither linear nor non-linear). We suspect that global competition for publication space is particularly strong in marketing and finance, which would contribute to the establishment of highly selective journals which are then perceived as of outstanding academic quality. The analysis of determinants for three German-language journals has shown that marketing and finance scholars do not have a general tendency to evaluate academic journals more positively than researchers from other business areas.

It is inevitable that the need for justification and adequate foundation of a journal ranking increases with its importance. Thus, we see the success of VHB-JOURQUAL as an obligation for its improvement. However, the potential trade-off between optimization and comparability of different JOURQUAL editions needs careful considerations.

Acknowledgments

Both authors contributed equally to this article. We thank the BuR reviewers; the board of the VHB, particularly its former chairman Sönke Albers and its current chairman Alfred Wagenhofer, for extensive valuable input to our study; Paul Marx for programming the online questionnaire and his help with the data handling; and all participating scholars for their investments in terms of time and effort, which has been essential for the success of this project. We also thank numerous colleagues for providing informational and motivational support. Last but not least, Uschi Hansen mentored our initial idea for this ranking several years ago - thank you!

Footnote

1 Please note that additional information on the VHB-JOURQUAL2 methodology is provided later in this article.

Footnote

2 Please note that this number differs from the initial results published on the VHB website as we deleted journals which were duplications of other journals with slightly different names or had ceased publication before the time frame considered in this ranking; we thank Robert Hofmeister from the Thurgau Institute of Economics at University of Konstanz for his valuable input. For duplicated journals, the evaluation scores were merged on an individual respondent level; if more than one journal version was rated by a respondent, the mean of his or her ratings for the respective journal was considered. As a result of this merging process, the scores for these journals can differ from those originally reported on the VHB website.

References

References

Adler, Nancy J. and Anne-Wil Harzing (2009): When Knowledge Wins: Transcending the Sense and Nonsense of Academic Rankings, Academy of Management Learning and Education, 8 (1): 72-95.

Albers, Sönke (2009): Misleading Rankings of Research in Business, German Economic Review, 10 (3): 352-363.

Azar, Ofer H. and David M. Brock (2008): A Citation-Based Ranking of Strategic Management Journals, Journal of Economics & Management Strategy, 17 (3): 781-802.

Blank, Rebecca M. (1991): The Effects of Double-Blind versus Single-Blind Reviewing: Experimental Evidence from the American Economic Review, American Economic Review, 81 (5): 1041- 1067.

Brady, Michael K. and J. Joseph Cronin Jr. (2001): Some New Thoughts on Conceptualizing Perceived Service Quality: A Hierarchical Approach, Journal of Marketing, 65 (3): 34-49.

Bräuninger, Michael and Justus Haucap (2002): What Economists Think of Their Journals and How They Use Them: Reputation and Relevance of Economics Journals, Discussion Paper, University of the Federal Armed Forces: Hamburg.

Clermont, Marcel (2009): Erfassung betriebswirtschaftlich relevanter Zeitschriften in Literaturdatenbanken: Analyse hinsichtlich Vollständigkeit und Schlussfolgerungen für die Literatursuche und die Messung der Forschungsleistung, Working Paper, RWTH: Aachen.

Clermont, Marcel and Christian Schmitz (2008): Erfassung betriebswirtschaftlich relevanter Zeitschriften in den ISI-Datenbanken sowie der Scopus-Datenbank, Zeitschrift für Betriebswirtschaft, 78 (10): 987-1010.

Dilger, Alexander (2000): Plädoyer für einen sozialwissenschaftlichen Zitationsindex, Die Betriebswirtschaft, 60 (4): 473-484.

Dyckhoff, Harald and Christian Schmitz (2007): Forschungsleistungsmessung mittels SSCI oder SCI-X? Internationale Sichtbarkeit und Wahrnehmung der deutschen Betriebswirtschaftslehre von 1990 bis 2004, Die Betriebswirtschaft, 67 (6): 640-664.

Extejt, Marian M. and Jonathan E. Smith (1990): The Behavioral Sciences and Management: An Evaluation of Relevant Journals, Journal of Management, 16 (3): 539-551.

Fabel, Oliver and Frank Heße (1999): Befragungsstudie vs. Publikationsanalyse: Zur Interpretation von Ranglisten der Forschungsaktivitäten deutscher betriebswirtschaftlicher Fachbereiche, Die Betriebswirtschaft, 59 (2): 196-204.

Franke, Nikolaus and Martin Schreier (2008): A Meta-Ranking of Technology and Innovation Management/Entrepreneurship Journals, Die Betriebswirtschaft, 68 (2): 185-216.

Frey, Bruno S. (2007): Evaluierungen, Evaluierungen ... Evaluitis, Perspektiven der Wirtschaftspolitik, 8 (3), 207-220.

Frey, Bruno S. and Katja Rost (2008): Do Rankings Reflect Research Quality?, Working Paper, SSRN.

Harzing, Anne-Wil (ed.) (2007): Journal Quality List, 26th ed., 5 March 2007, www.harzing.com/jql.htm (Access Date: 2007-03- 10).

Harzing, Anne-Wil (ed.) (2009): Journal Quality List, 34th ed., 3 July 2009, www.harzing.com/jql.htm (Access Date: 2009-08-28).

Heischmidt, Kenneth A. and Peter Gordon (1993): Rating of Marketing Publications: Impact of Accreditation and Publication History, Journal of Education for Business, 68 (3): 152-158.

Hennig-Thurau, Thorsten, Markus F. Langer, and Ursula Hansen (2001): Modeling and Managing Student Loyalty: An Approach Based on Relationship Quality, Journal of Service Research, 3 (4): 331-344.

Hennig-Thurau, Thorsten, Gianfranco Walsh, and Ulf Schrader (2004): VHB-JOURQUAL: Ein Ranking von betriebswirtschaftlich- relevanten Zeitschriften auf der Grundlage von Expertenurteilen, Schmalenbachs Zeitschrift für betriebswirtschaftliche Forschung, 56 (6): 520-545.

Homburg, Christian (2008): Internationalität, Praxisnähe, Spitzenforschung: Deutsche Universitäten im Zielkonflikt?, in: Technische Universität Bergakademie Freiberg (ed.): Zukunft deutscher Universitäten: Standpunkte und Perspektiven, Gabler: Wiesbaden, 31-60.

Hult, G. Tomas M., Martin Reimann, and Oliver Schilke (2009): Worldwide Faculty Perceptions of Marketing Journals: Rankings, Trends, Comparisons, and Segmentations, global EDGE Business Review, 3 (3): 1-10.

Laband, David N. and Michael J. Piette (1994a): The Relative Impacts of Economics Journals: 1970 - 1990, Journal of Economic Literature, 32 (2): 640-666.

Laband, David N. and Michael J Piette (1994b): Does the "Blindness" of Peer Review Influence Manuscript Selection Efficiency?, Southern Economic Journal, 60 (4): 896-906.

Macharzina, Klaus, Joachim Wolf, and Anne Rohn (2004): Quantitative Evaluation of German Research Output in Business Administration - 1992-2001, Management International Review, 44 (3): 335-359.

Maier, Gunther (2006): Impact factors and peer judgment: The case of regional science journals, Scientometrics, 69 (3): 651-667

Oesterle, Michael-Jörg (2006): Wahrnehmung betriebswirtschaftlicher Fachzeitschriften durch Praktiker, Die Betriebswirtschaft, 66 (3): 307-325.

Ritzberger, Klaus (2008): A Ranking of Journals in Economics and Related Fields, German Economic Review, 9 (4): 402-430.

Rossiter, John R. (2002): The C-OAR-SE Procedure for Scale Development in Marketing, International Journal of Research in Marketing, 19 (4): 305-335.

Schlinghoff, Axel and Uschi Backes-Gellner (2002): Publikationsindikatoren und die Stabilität wirtschaftswissenschaftlicher Zeitschriftenrankings, Schmalenbachs Zeitschrift für betriebswirtschaftliche Forschung, 54 (6): 343-362.

Schrader, Ulf and Thorsten Hennig-Thurau (2007): VHBJOURQUAL2: Planungsstand und Ergebnisse der Vorbefragung. Presentation at the 69th annual scientific conference of the VHB, June 2nd, Paderborn.

Schulze, Günther G., Susanne Warning, and Christian Wiermann (2008): Zeitschriftenrankings für die Wirtschaftswissenschaften: Konstruktion eines umfassenden Metaindexes, Perspektiven der Wirtschaftspolitik, 9 (3): 286-305.

Simon, Hermann (1993): Die deutsche Betriebswirtschaftslehre im internationalen Wettebewerb - ein schwarzes Loch?, in: Horst Albach and Klaus Brockhoff (eds.): Die Zukunft der Betriebswirtschaftslehre in Deutschland, Zeitschrift für Betriebswirtschaft, Special Issue 9/1993: 73-84.

Starbuck, William H. (2004): Vita Contemplativa: Why I stopped trying to understand the real world, Organization Studies, 25 (7): 1233-1254.

Thomson Reuters (2009): Journal Citation Reports, Edition 2008, http://admin-apps.isiknowledge.com/JCR/JCR (with licence, Access Date: 2009-08-28).

Vieira, Pedro Cosme da Costa (2008): An Economics Journals' Ranking that Takes into Account the Number of Pages and Coauthors, Applied Economics, 40 (7): 853-861.

Appendix

Web-Appendix 1:

List of all Journals Rated in VHB-JOURQUAL2 in Alphabetical Order

Web-Appendix 2:

VHB-JOURQUAL2 Ranking of Business Research Journals

Web-Appendix 3:

Changes of JQ Index Values from VHB-JOURQUAL1 to VHB-JOURQUAL2