Seminar
Peerin' into Peer Review with Agent-Based Models

Flaminio Squazzoni (University of Brescia)

14 December 2015
Room A, h. 13.00-14.30
Graduate School in Social and Political Sciences
via Pace 10 - Milan

Peer review is a cornerstone of science. Without it, there would be no science. On the one hand, it guarantees that the scientific community generates innovative, rigorous and solid knowledge. On the other hand, it is fundamental to help science to self-regulate through a decentralized, distributed and collective trial and error process.
Whether directly or indirectly, this complex, self-regulated institution determines how all the resources of the science system—including funding, positions, and reputation—are allocated. Recent failures of peer review due to judgment biases and parochialism as well as cases of misconduct have contributed to calls for a reconsideration of the rigour and quality of the process. Furthermore, increasing competition between scientists at all levels, the explosion of scholarly publications and application of peer review to a variety of complex knowledge output (e.g., grant proposals, research assessment, careers) have raised serious concern about the long-term sustainability of a process that is largely based upon voluntary contribution and unprofessional.
In this talk I shall present some simulation models built to understand large-scale implications of scientist behaviour on the quality and efficiency of peer review. In our models, we have simulated a population of scientists competing for publication in a "publish-or perish" context and asked to submit and review articles facing different trade-offs. We looked at peer review as a cooperation dilemma in which scientists' self interest is at odds with collective achievements.
Simulation findings showed that peer review outcomes are dramatically sensitive to scientist motivation and behaviour and that a random scenario, i.e., something typically considered as a nightmare for each scientist, in which reviewers follow random judgement, is not the worst case scenario. If reviewers follow self-interest, strategic motivations, e.g., trying to outperform potential competitors by providing unreliable opinion or being reliable depending on previous acceptance/rejection when authors, peer review generates publication bias larger than a random situation. We also found a trade-off between quality, which might be increased by extending the sample of reviewers for each author submission, and efficiency, the systemic capacity to allocate more resources on research than on reviewing.
Finally, I shall discuss recent achievements from an EU project (COST Action PEERE-New Frontiers of Peer Review, www.peere.org), which includes cooperation between scientists and publishers for data sharing on peer review. Without data there is not only no science. Without data there is also no science on peer review.

Recent relevant papers

Squazzoni F. and Takács K. (2011) Social Simulation that 'Peers into Peer Review', Journal of Artificial Societies and Social Simulation, 14(4), 3: http://jasss.soc.surrey.ac.uk/14/4/3.html
Squazzoni F. and Gandelli C. (2012) Saint Matthew Strikes Again. An Agent-Based Model of Peer Review and the Scientific Community Structure, Journal of Informetrics, 2012, 6, 2, pp. 265-275.
Squazzoni F., Bravo G. and Takács K. (2013) Does Incentive Provision Increase the Quality of Peer Review? An Experimental Study, Research Policy, 2013, 42(1), pp. 287-294.
Squazzoni F. and Gandelli C. (2013) Opening the Black Box of Peer Review: An Agent-Based Model of Scientist Behaviour, Journal of Artificial Societies and Social Simulation, 16(2) 3: http://jasss.soc.surrey.ac.uk/16/2/3.html
Cabotà J., Grimaldo F. and Squazzoni F. (2014) Do Editors Have Silver Bullets? An Agent-Based Models of Peer Review. In Squazzoni F., Baronio F., Archetti C. and Castellani M. (eds.), Proceedings of the European Conference on Modelling and Simulation 2014, ECMS 2014.
Bianchi F. and Squazzoni F. (2015) Is three better than one? Simulating the effect of reviewer selection and behavior on the quality and efficiency of peer review, IEEE Proceedings of the Winter Simulation Conference 2015, WSC 2015.

This seminar is part of the ResFron ESLS Cycle of seminars - 2016 Edition