Open research evaluation system an integral part of WEA philosophy
Download the WEA commentaries issue ›
By Grazia Ietto-Gillies
The guidelines for our two new journals – World Economics Journal (WEJ) and Economic Thought (ET) – put forward a system of research evaluation based on: (a) pre- and post- publication commentaries; (b) double openness; and (c) a Discussion Forum accessible to all WEA members. The latter two characteristics feature also in our guidelines for conferences.
In this contribution I would like to explain how the system developed for WEA is in line with the overall philosophy put forward in our Manifesto and specifically with the aims of: plurality of approaches to economics; inclusivity; excellence in research; and full utilization of digital technologies.
The Peer Review (PR) system of research evaluation – in which a paper is published following positive reports by 2-3 referees in a double-blind process – has been in use in most subjects for many decades. However it has come under severe criticisms in the last two decades or so1. Recently the PR system has, indeed, been the subject of an enquiry by the Science and Technology Committee of the British House of Commons.
The criticisms levelled at the PR system are many and range from costs/efficiency issues to effectiveness particularly in terms of its inability to give assurance about detection of mistakes or fraud and its inability to detect the really original, ground-breaking work and secure publication for it. This latter criticism is the most serious and damaging one and it has been documented in several works (Horrobin, 1990; Gans and Shepherd, 1994; Legendre, 1995). Gillies (2008) uses a Kuhnian philosophical framework to explain why in several instances from the history of sciences the now acclaimed works were not recognized as significant by the contemporary peers.
Peers tend to go for orthodoxy and particularly so in high rated journals (Lee, 2007). The PR system, therefore, tends to reinforce the dominant paradigm by the related system of journals’ rating, by the selection of referees and by possible bias of referees who see research through their own approach to the subject.
The system used in the WEJ and ET is based on the following principles.
- The digital technologies have made the function of space allocation by journals almost irrelevant. Any paper can be posted on a web site without additional costs and indeed many researchers now post their papers on their web sites prior to publication.
- The digital technologies are being used extensively by journals’ publishers in the publication process of research papers. For example in communications between editors, referees and authors and in copy editing. However, so far, little use has been made of them for the evaluation process itself. The WEA system does just that: it involves a large number of potential commentators from many countries and belonging to a variety of economics schools.
- The PR system is based on the principles of assessment/rating and of exclusion. Because journal space is limited and the ratio of paper submission to acceptances is very high, the editors necessarily look for support and justification for the rejection of many submitted papers. In order to do so often referees look for faults rather than areas which are positive and could be further developed. These critical points do not mean to devalue the work of referees – many of whom labour very hard and often come up with helpful suggestions – but only to point out a problem in the system they are caught in: in the end no matter how helpful some of them may want to be, their reports are used to exclude papers from publication in specific journals. But again no blame can be attached to the editors who have to allocate limited space in their journals.
- Research can achieve best results when it is developed as a social activity not necessarily in the sense of two or three people working together on a project, though this is increasingly the case in many fields. The social context is seen here as researchers developing their own ideas on the basis of previous research – which is always the case – and benefiting from discussions and interchanges with peers in a constructive environment. The involvement of peers in the further development/evaluation of research is very useful. However, it does not have to be on a confrontational and rating basis. It can take place on the basis of exchange of ideas for the advancement of the specific topic of the paper.
- The involvement of many researchers in the evaluation process is preferable to only 2-3 reviewers because: (a) more people are more likely to spot plagiarism, mistakes, data problems; (b) if many people read a paper it is more likely that one or two of them spot the originality and value of a paper which is out of the ordinary and may thus appear strange and wrong to the few. Thus one of the major pitfalls of the PR system is less likely to manifest. Moreover, the involvement of many commentators increases the likelihood of getting contributions from researchers belonging to different schools/paradigms. This helps to achieve our aim of plurality of approaches to economics.
- Double-sided openness: the names of the author(s) and those of reviewers are revealed2. The attribution of comments to a specific paper encourages commentators to come forward and make known their – sometimes provisional – views knowing that they are posted with their name. Attribution may, therefore, eliminate reticence in putting forward very original comments; it may also encourage commentators to consider carefully their critical arguments and make sure that they are not inspired mainly by adherence to a specific paradigm and ideology.
- A common worry about open posting (where the names of authors and commentators are disclosed) is that commentators feel embarrassed to be critical. However, it is worth pointing out that: (a) reviewers of books – where a doubly open system is used as a matter of normal academic activity – are often quite critical; (b) moreover, if the process is online commentators and authors may be in very distant parts of the world and do not know each other; and (c) if the system is less confrontational than the PR system this is no bad thing: a critically positive system is more likely to lead to the advancement of research and to be less disheartening for young researchers.
- Post-publication commentary is as important for the advancement of research as pre-publication one. The life of a paper does not end with publication; hopefully that is only its beginning. Other researchers will read the paper for years to come. The attraction of readers for years and years after publication is evidence of the relevance of a paper. Some readers may develop further research of their own after reading an article and their research may lead to new publications of their own. Others may have points to make about it which do not amount to the development of a full research project or paper but that can, nonetheless, be relevant and useful for the further advancement of the field. A post-publication commentary as a standard feature of journals allows these people to have their comments published – at the discretion of the editors – with attribution. As with pre-publication comments, the attribution is important because, knowing that their name will appear, people may be more willing to come forward and publish their comments, rather than hold them back.
There may problems with the system we have devised ranging from too many uninformed comments to too few comments; comments that are insulting (they will not be posted!) to comments that are too bland and unhelpful for the development of the paper. At WEA we are aware that many problems may develop and are prepared to take action to correct them with the help and support of our excellent team of editors. We are developing a system that involves a change in the culture of research evaluation within the economics profession. We and our membership will go through a learning process. The participation and collaboration of the membership – of you all – is essential for this learning process and for the success of our system.
I have recently come across a similar system applied to the Journal Atmospheric Chemistry and Physics and have been greatly encouraged by its success. An article in Nature (Koop and Posch, 2006) by two of its editors describes the system and assesses it after 5 years of operation. Their system is very similar to the one we developed for WEA. The two editors write about their experience: “…Our statistics confirm that collaborative peer review facilitates and enhances quality assurance.” and “We believe that collaborative peer review with a two-stage publication process and interactive public discussion effectively resolves the dilemma between rapid scientific exchange and thorough quality assurance. It fosters scientific discussion, deters submissions of substandard manuscripts, conserves reviewing capacity and enhances the density of information in final papers”.
So, on this hopeful note I end my piece and invite all WEA members to get involved in our – your – journals and conferences and to help us make them successful.
References
Gans, J. S. and Shepherd, G.B. (1994), ‘How the mighty have fallen: Rejected classic articles by leading economists’, Journal of Economic Perspectives, 8: 165-79.
Gillies, D. (2008), How Should Research be Organised? London: College Publications.
Horrobin, D.F. (1990), ‘The philosophical basis of peer review and the suppression of innovation’, Journal of the American Medical Association, 263: 1438-41
Ietto-Gillies, G. (2008), ‘A XXI-century alternative to XX-century peer review” real-world economics review, 45: 10-22, March; www.paecon.net/PAEReview/issue45/IettoGillies45.
Legendre, A. M. (1995), ‘Peer review of manuscripts for biomedical journals’. Journal of the American Veterinary Medical Association, 207: 36-8.
Lee, F. (2007), The Research Assessment Exercise, the state and the dominance of mainstream economics in British universities, Cambridge Journal of Economics, 31: 309-25.
Grazia Ietto-Gillies
London Oct 2011
From: Pp.8-9 of World Economics Association Newsletter 1(1), December 2011 https://www.worldeconomicsassociation.org/files/newsletters/Issue1-1.pdf