"Robust" but Tiny: Methodological Influences and Inter-Individual (Un)Stability of the Serial Order Effect in Creativity

Abstract

The serial order effect (SOE), i.e., the tendency for the creative quality of ideas to increase with increasing response position, has been widely documented in divergent thinking (DT) research. However, its magnitude is hardly reported, and its sensitivity to methodological effects (e.g., DT task used, scoring method) has rarely been investigated. More critically, the stability of inter-individual differences in SOE has not been investigated to date. Is the SOE a sizable, dependable (i.e., immune to methodological variations), and importantly, a trait-like psychological phenomenon? This study examined SOE strength and inter-individual consistency across three Alternate Uses Task (AUT) prompts and three scoring approaches: subjective ratings, frequency-based scoring, and semantic similarity. While the SOE emerged across all scoring methods and functional forms examined (linear and quadratic), effect sizes were consistently small in magnitude (explaining at most 3.5% of variance) and varied significantly by stimulus and scoring method. Crucially, evidence for stable individual differences in SOE slopes was absent (ICCs < 0.31 across AUT prompts), raising a potential “reliability paradox”: the SOE appears robust at the group level (i.e., reproducible, though small), yet unreliable in terms of inter-individual stability. These findings underscore the need for systematic evaluation of SOE effect sizes, greater attention to task and scoring differences, and closer investigations clarifying whether the SOE reflects a stable trait interpretable at the individual level, or an aggregate phenomenon with limited practical significance.