The article dealt with the U.K.'s "Research Assessment Exercise" (RAE). Each U.K. faculty member in OR/MS (and in all other disciplines) is required to submit up to four publications for the RAE in 2008, which is driven by the government-led university funding authority. Submitted publications will then be ranked by a process of peer review by a panel of subject experts (including members from outside the United Kingdom) involving both academics and users of research (including industrial practitioners, government officials and other members of public bodies). Publications will be rated on a five-point scale that may be summarized as follows:

4* world-leading quality
3* internationally excellent
2* internationally recognized
1* nationally recognized
Unclassified

The RAE takes place every six years or so. Submitted publications are assessed by panels convened for each discipline, but this is the first time a publication ranking scale has been made quite so explicit. Obviously academics will try to "second guess" the ranking that the panel will place on each submitted publication. The first thing to emphasize is that it is individual papers (or other forms of publication) that will be ranked — the panels are supposed to judge each individual paper on it own merits. Even in terms of the perceived quality of journals, our view is that citation counts may not be a useful way on which to base estimates of rankings, and that there is clear evidence that academics did not use this approach in previous rounds of the RAE.

Our first piece of evidence comes from the paper by Geary et al. [2] that provides statistics on journal counts for publications submitted in the last RAE in 2001 that were included in the set of subjects falling under the Business and Management Unit of Assessment. (OR/MS falls under this unit of assessment, but also falls under Statistics and O.R., and partly into Information Science.) Where OR/MS is part of a business school, it is usually submitted for RAE purposes to the Business and Management Assessment Panel. Where it is part of a statistics or mathematics department, it is usually submitted to the Statistics and Operational Research Assessment Panel. Where it is part of an information science department, it would usually be submitted to the Information Science Assessment Panel.

Geary et al. state that of the 20 most frequently cited journals (of any discipline) in business and management submissions, Journal of the Operational Research Society (JORS) ranks fourth with 113 citations, International Journal of Operations and Production Management (IJOPM) ranks sixth with 85 citations, and European Journal of Operational Research (EJOR) is 14th with 61 submissions. No other O.R. journal or journal closely related to O.R. figures in the top 10. In addition, when only submissions from top-ranked departments (as ranked by the RAE 2001 exercise) are considered, JORS comes out first, EJOR seventh and IJOPM eighth. It is noteworthy that these three journals are not among those with the highest journal impact factors (a measure of citation count).

The second piece of evidence is from the year-on-year volatility of impact factors, which is well documented (including by Sodhi himself). The RAE is a relatively long-term exercise, but impact factors can change considerably over five years, so using impact factors as a ranking of journal quality has limitations.

The third piece of evidence is the artificial massaging of impact factors. Some editors (particularly U.S.) require authors to cite their own journals (one of us recently had an article returned from a prestigious U.S.-based O.R.-related journal with express instructions to include five references to that journal) — something JORS editors feel we should not necessarily insist on.

The fourth piece of evidence is that not all journals are ranked by impact factor (e.g. project management journals do not appear on the citation lists).

The fifth piece of evidence is that there is selectivity in the list of journals in Sodhi's article. Two interesting omissions from the list in [1] are Decision Sciences, which had a 0.891 impact factor on the same listing, and Omega, which had a 0.558 ranking, suggesting a certain inconsistency in arguments.

We would also disagree with Sodhi's comment about Journal of Operations Management (JOM) — this has a high ranking because it is good despite being anchored in real practice. We feel that journals such as EJOR, which are theoretical, are going to be cited more often due to the nature of academic publishing; JORS and Interfaces are cited less often because they seek to bring real case-oriented work to the OR/MS audience. We feel such work is more "real O.R."

The final piece of evidence is that the panel which will consider business and management submissions in the 2008 exercise has stated that they do not have a list of "approved" journals, nor do they plan to use citation-based journal rankings. Thus it is unwise to second-guess that an impact factor of >1.2 for a journal will guarantee that a paper is rated 4*.

Terry Williams and John Wilson



Terry Williams of the University of Strathclyde, Scotland, and John Wilson of Loughborough University, England, are joint editors of Journal of Operational Research Society.

References

1. M.S. Sodhi, 2005, "Journal Rankings: U.K. Perspective," OR/MS Today, Vol. 32, No. 2, p. 14.
   
   
2. J. Geary, L. Marriott and M. Rowlinson, 2004, "Journal rankings in business and management and the 2001 Research Assessment Exercise in the U.K.," British Journal of Management, Vol. 15, pp. 95-11.

Editor's note:
In keeping with the publication's policy, OR/MS Today offered ManMohan Sodhi an opportunity to reply. Here is his response:

Many thanks to Williams and Wilson for highlighting U.K. and European journals. Their "evidence," however, is based on the mistaken belief that I "outline the case for U.K. academics to use citation counts as provided by impact factors in order to rank OR/MS journals." I only described the process as it is conducted in the United Kingdom and at my school. I even called the whole thing "strange." My point was that journal rankings exist and, for better or worse, are used for evaluating research performance. Still, it is heartening to see that U.K. academics, especially journal editors, are thinking actively about these issues that pertain to research-performance evaluation including promotion and tenure.



My Favorite George Dantzig Story

Everyone has a favorite George Dantzig story. Here's mine.

It was the summer of 1968, and I was doing the final oral defense of my dissertation. George was one of the members of the committee. It was his turn to ask questions. At that time, the final defense included fielding questions on any subject in O.R. Sort of a Prelims Redux.

George asked me the usual questions and then came, I believe, to the Decomposition Principle. My mind when blank. I uttered a few generalities and then said, "George, that's all I know."

He wasn't fazed. Instead he said, "That's OK. Let me ask you something else. Are you married?"

"Yes."

"Is your wife in the sciences?"

"No."

"How do you explain operations research to her?"

I marveled at the brilliance of the question for a final oral and started replying. After a few sentences he was satisfied and interrupted me to say, "Thank you."

I passed.

Paul Gray
Irvine, Calif.



Paul Gray is professor emeritus of Information Science at Claremont Graduate University and a past president of The Institute for the Management Sciences.

Editor's note:
For more remembrances of George B. Dantzig, the "Father of Linear Programming" who passed away on May 13 at the age of 90, turn to page 40.

Table of Contents

OR/MS Today Home Page


OR/MS Today copyright © 2005 by the Institute for Operations Research and the Management Sciences. All rights reserved.


Lionheart Publishing, Inc.
506 Roswell Rd., Suite 220, Marietta, GA 30060 USA
Phone: 770-431-0867 | Fax: 770-432-6969
E-mail: lpi@lionhrtpub.com
URL: http://www.lionhrtpub.com


Web Site © Copyright 2005 by Lionheart Publishing, Inc. All rights reserved.