Research Seminar - “Scoring vs. Ranking: An Experimental Study of Idea Evaluation Processes” by Zhijian Cui, Associate Professor, Nazarbayev University Graduate School of Business.

2018-04-19 11:24:50

Seminar Announcement Zhijian scoring

Dear students, faculty and staff,

Nazarbayev University Graduate School of Business invites you to Research Seminar

Scoring vs. Ranking: An Experimental Study of Idea Evaluation Processes” by Zhijian Cui, Associate Professor, Nazarbayev University Graduate School of Business. 

WHEN: Friday, April 13, 12:30 – 13:30

WHERE: Block C3 (GSB/GSPP Building), 3rd floor, Room 3038

ABSTRACT: (please see paper attached)

Using an online experiment, this study examines the efficacy of two idea evaluation processes: scoring vs. ranking in the context of innovation management. In the scoring process, the evaluators are asked to rate the quality of each idea by assigning it a score (e.g., from 0 to 10) while in the ranking process the evaluator simply orders all ideas according to their perceived qualities. Our results suggest that the scoring process strictly outperforms the ranking process in terms of the likelihood of selecting the highest-quality ideas. However, when the number of ideas to be evaluated is reduced from eight to three, the efficacies of the two idea evaluation processes become similar. When we did not allow the possibility of making a tie in the ranking process, there is a significant interaction effect between the information presented and the evaluation process. Specifically, the efficacy of the ranking process is significantly improved when additional information is provided, yet the efficacy of the scoring process does not change with more information. When the possibility of making a tie is allowed in the ranking process, the interaction effect disappears, i.e., providing additional information does not improve the efficacy of either the ranking or scoring process. Based on the observations from the experimental data, we propose and test an explanatory model in which the information becomes a cue that directs the participants’ efforts to evaluate the ideas (i.e., the time taken for evaluation). 


Before joining the Graduate School of Business, Nazarbayev University in August 2017, Dr. Zhijian Cui has been an Associate Professor of Operations Management at IE (Instituto de Empresa) Business School (Spain) and a Visiting Assistant Professor of Operations Management at HKUST Business School, the Hong Kong University of Science and Technology (Hong Kong).  Zhijian’s research is focused on the management of collaborations and conflicts in new product development (NPD) process. His research work has been published in Production and Operations Management, Decision Sciences, IEEE Transactions on Engineering Management, Cornell Hospitality Quarterly, Research-Technology Management, etc. and received numerous awards from different academic entities. Zhijian received “Best Ten Lecturers” award from the Hong Kong University of Science and Technology in 2016 and was voted as one of ten “UST Great Minds” speakers in the same year. He also received “Best Professor of Master of Management Program” award from IE Business School in 2015 and “Best Professor of the Year” award from the MBA Program of Shanghai International Studies University in 2014.