I was interested to read ManMohan Sodhi's article in the April edition of OR/MS Today concerning journal rankings ["Journal Rankings: U.K. Perspective," p. 14]. It is certainly the case that the Research Assessment Exercise (RAE) is a major driver of university behavior in the United Kingdom. It inevitably results in league tables of university performance, which affect both funding and prestige.

I do have some concerns, however, about the use of citations and impact factors alone as a measure of journal quality. While they obviously have some advantages being readily available and quantitative — no doubt a plus point for management scientists — they also have important limitations:

• Not all journals are covered by the ISI database and so will not have citation rates or impact factors. Although this in itself could be seen as a sign of low quality, in fact some high-quality journals are not included, for example the Journal of the American Statistical Association, which is generally considered a top journal.
  
  
• The citation index favors those journals with a wide field and many readers at the expense of specialist journals that may be top in a narrow area. Hence the big difference between Management Science and Operations Research, perhaps.
  
  
• The impact factor only counts citations in the last two years. This tends to favor journals that publish in very current topical areas against those with long gestation periods or long publication lead times. These could be empirically based surveys and reviews rather than those concentrating on basic research. An example of this is the ACM Computing Surveys, which has a very high impact factor (7.5 for 2003) but is generally rated as relatively low quality.
  
  
• The factor may favor American journals, which tend to have large readerships and tend also to cite other American journals, at the expense of non-U.S. ones.
  
  
• Generally, we cannot assume that impact is the same as research quality, and there are many journals with low or non-existent impact factors that publish high-quality research as evidenced by peer-assessed rankings.

One of the best sources of information about journal rankings in business and management is the Harzing Web site (www.harzing.com), which contains 13 rankings by different universities since 1994 as well as five years of impact factors. This gives a good general overview of the similarities and differences between different rankings. I am currently analysing this dataset statistically. One analysis is to use principal components to see the dimensions underlying the variability of the data. The first principal component clearly related to the general quality level of journals and accounted for 61 percent of the variation. The second component was more interesting as it precisely highlighted the difference between impact factors and peer-assessed journal rankings. Journals scoring highly on this dimension were those with high impact factors but low rankings such as the ACM Computing Surveys mentioned above. Journals scoring poorly were those with high rankings but low impact factors such as the Journal of the American Statistical Association mentioned above.

An interesting study was carried out by Geary, Marriott et al. ["Journal rankings in business and management and the 2001 research assessment exercise in the U.K.," British Journal of Management, Vol. 15, pgs. 95-141] where they analyzed all the papers that were submitted to the 2001 RAE in business and management (nearly 10,000 publications by 3,000 faculty). One of their aims was to impute the quality of journals by considering the quality of the departments they were submitted by. Departmental quality was measured by the results of the RAE in terms of a seven-point scale. The reasoning was that high-quality departments will have high-quality faculty who will publish in high-quality journals. For each journal the average departmental score of its authors was calculated. Of interest to OR/MS Today readers is that one of the highest-rated journals was the Journal of Operational Research Society with a mean of 5.5 and a median of 6. This can be explained simply by the fact that two of the largest O.R. groups in the United Kingdom are at Lancaster and Warwick Universities, which were two of only three to gain the highest ranking in the RAE — the other being London Business School. On the other hand, top journals such as Management Science and Operations Research did poorly as relatively few U.K. academics publish in them.

In conclusion, impact factors are a useful way of judging journal quality but give only one perspective and should be used cautiously along with other peer-assessed rankings.

John Mingers
Kent Business School,
University of Kent, U.K.

Table of Contents

OR/MS Today Home Page


OR/MS Today copyright © 2005 by the Institute for Operations Research and the Management Sciences. All rights reserved.


Lionheart Publishing, Inc.
506 Roswell Rd., Suite 220, Marietta, GA 30060 USA
Phone: 770-431-0867 | Fax: 770-432-6969
E-mail: lpi@lionhrtpub.com
URL: http://www.lionhrtpub.com


Web Site © Copyright 2005 by Lionheart Publishing, Inc. All rights reserved.