Compared to the wealth of debate surrounding replicability and transparency, relatively little attention has been paid to the issue of generalisability – the extent to which research findings hold across different samples, cultures, and other parameters. Existing research suggests that researchers in psychology are prone to generalisation bias, relying on narrow samples (e.g. drawn predominantly from US or European undergraduate samples) to draw broad conclusions about the mind and behaviour. While recent attempts to address generalisability concerns have been made – such as journals requiring explicit statements acknowledging constraints on generality – addressing this bias at root, and developing truly generalisable methods and results, requires a deeper understanding of how researchers perceive generalisability in the first place.

 

In the current study, Schmidt et al. (2023) tackle the issue of cross-cultural generalisability using four large-scale international studies that are being conducted as part of the Psychological Science Accelerator (PSA) – a globally distributed network of researchers in psychology that coordinates crowdsourced research projects across six continents. Specifically, participants (who will be PSA research members) will estimate the probability that an expected focal effect will be observed both overall and within regional subsamples of the PSA studies. They will also predict the size of these focal effects overall and by region.

 

Using this methodology, the authors plan to ask two main questions: first whether researchers can accurately predict the generalisability of psychological phenomena in upcoming studies, and second whether certain researcher characteristics (including various measures of expertise, experience, and demographics) are associated with the accuracy of generalisability predictions. Based on previous evidence that scientists can successfully predict the outcomes of research studies, the authors expect to observe a positive association between predicted and actual outcomes and effect sizes. In secondary analyses, the authors will also test if researchers can predict when variables that capture relevant cultural differences will moderate the focal effects – if so, this would suggest that at least some researchers have a deeper understanding as to why the effects generalise (or not) across cultural contexts.

 

The Stage 1 manuscript was evaluated over two rounds of in-depth review. Based on detailed responses to the reviewers' comments, the recommender judged that the manuscript met the Stage 1 criteria and therefore awarded in-principle acceptance (IPA).

 

URL to the preregistered Stage 1 protocol: https://osf.io/vwqsa (under temporary private embargo)
 
Level of bias control achieved: Level 6. No part of the data or evidence that will be used to answer the research question yet exists and no part will be generated until after IPA.
 
List of eligible PCI RR-friendly journals:
 

 

 

1. Schmidt, K., Silverstein, P. & Chartier, C. R. (2023). Registered Report: Researcher Predictions of Effect Generalizability Across Global Samples. In principle acceptance of Version 3 by Peer Community in Registered Reports. https://osf.io/vwqsa

What determines whether groups of people can come up with ideas that are both original and useful? Since the 1960s, this question has been intensively studied with the help of more or less structured group creativity activities such as brainstorming or creative problem solving, with subsequent rating of the generated ideas.

In this line of research, personal factors—such as personality traits, and other interindividual differences in emotion and cognition—have received substantial attention as potential correlates of creative outcomes of group activities. This has spawned a sprawling literature that, to date, has not yet been synthesized. Thus, empirical findings in this literature, which are also sometimes contradictory, have not yet been well-integrated.

In the present study, Fillon et al. (2022) conducted the first meta-analysis of correlations between personal factors and group creativity outcomes. The authors searched and synthesized the existing published literature according to predetermined criteria to (1) assess the overall relationship between a broad list of personal factors and creativity outcomes in group settings and (2) explore potential moderators of these relationships. 

In total, 11 studies could be included in the meta-analysis. They provided weak support for a positive correlation between self-efficacy and the three investigated group creative outcomes, number of ideas, originality of ideas, and usefulness of ideas. With respect to moderators, many of the planned analyses could not be conducted due to the low number of studies. The only finding that arose was weak evidence for the idea that time constraint moderates associations: relationships between personal factors and group creativity outcomes were slightly stronger for tasks limited to 20 minutes rather than 10 minutes. Statistical power overall was low across studies.

The authors conclude their meta-analysis with the observation that the available data on the topic are very limited. They suggest that to improve our knowledge of the topic, future studies should adhere to standardized creativity methods and protocols and implement expert ratings of creativity. They also call for increasing the availability of raw data in this field of study to improve the accumulation of knowledge about links between personal factors and the creative performance of groups.

The Stage 2 manuscript was evaluated over one round of in-depth review provided by the recommender and Chris Chambers, as the original reviewers were no longer available. Based on additional changes to the manuscript, the recommender judged that the manuscript met the Stage 2 criteria and awarded a positive recommendation.

 

URL to the preregistered Stage 1 protocol: https://osf.io/nybg6
 
Level of bias control achieved: Level 3. At least some data/evidence that was used to answer the research question had been previously accessed by the authors (e.g. downloaded or otherwise received), but the authors certified that had not yet observed ANY part of the data/evidence prior to IPA.
 
List of eligible PCI RR-friendly journals:
 

The N2pc is a lateralised ERP component that is often interpreted as a marker of attentional allocation, so much so that it is frequently used a tool in the attention literature for inferring that a stimulus was attentionally processed. This interpretation of N2pc can be traced back to the seminal work of Eimer (1996), which has been conceptually replicated many times but has never been replicated directly.

 

This registered direct replication by Constant et al. (2023) forms part of a larger series of large-scale, multi-lab replications of highly influential EEG papers by the #EEGManyLabs project (Pavlov et al., 2021). Seven labs (with the potential for more to sign up later), will conduct high-powered replications of the critical Experiment 2 of Eimer (1996), where in the crucial conditions, participants discriminate a target letter (M vs W) or colour (blue vs green) in the presence of a distractor. Using four preprocessing pipelines, including the original, the authors will test whether the N2pc is observed over parieto-occipital electrodes contralateral to target presentation.  

 

The Stage 1 manuscript was evaluated over one round of in-depth review and one additional round of minor corrections. Based on detailed responses to the reviewers' comments, the recommender judged that the manuscript met the Stage 1 criteria and therefore awarded in-principle acceptance (IPA).

 

 

Level of bias control achieved: Level 6. No part of the data or evidence that will be used to answer the research question yet exists and no part will be generated until after IPA (so-called “primary RR”)

 

 

 

 

1. Constant, M., Mandal, A., Asanowicz, D., Yamaguchi, M., Gillmeister, H., Kerzel, D., Luque, D., Pesciarelli, F., Fehr, T., Mushtaq, F., Pavlov, Y. G. & Liesefeld, H. R. (2023). A multilab investigation into the N2pc as an indicator of attentional selectivity: Direct replication of Eimer (1996), in principle acceptance of Version 3 by Peer Community in Registered Reports. https://osf.io/dw68r

 

2. Eimer, M. (1996). The N2pc component as an indicator of attentional selectivity. Electroencephalography and Clinical Neurophysiology, 99, 225-234. https://doi.org/10.1016/0013-4694(96)95711-9

 

3. Pavlov, Y. G., Adamian, N., Appelhoff, S., Arvaneh, M., Benwell, C. S., Beste, C., ... & Mushtaq, F. (2021). #EEGManyLabs: Investigating the replicability of influential EEG experiments. Cortex, 144, 213-229. https://doi.org/10.1016/j.cortex.2021.03.013

For many of us, music is far more than an auditory experience. It can trigger emotional reactions, evoke memories, and wide-ranging associations with other sensory modalities and cognitive states. Music also varies between different cultures in several ways. It remains unclear in how far the broader associations music has differs between cultural contexts, both in terms of the music itself and the listener. This study by Hadavi et al. (2024) seeks to better understand these relationships. Using an online survey targeted at 72 participants from anglophone Canada, Farsi-speaking Iran, and Japan (24 from each location), the researchers aim to address two straightforward hypotheses.

 

First, does faster tempo of music increase ratings of emotional arousal? Second, do participants match faster tempo music with denser visual line patterns? This latter measure aims to quantify the visual imagery evoked by the musical pieces. Imagery is a loaded term that is not used consistently across the cognitive neuroscience literature; one could argue that what the researchers are actually here is in fact mainly an association between tempo and a visual representation of tempo (or frequency). It certainly seems doubtful that persons listening to a piece of music will form a mental image of a bundle of horizontal lines. Yet, irrespective of how to interpret this experimental variable, it quantifies something about the impression listeners have when experiencing music, and whether these associations differ cross-culturally. The experiment has a balanced design, incorporating excerpts of musical pieces from each of the three cultural contexts, including both solo and group music. While the sample size is comparably low, given the online nature of data collection, it is based on a power analysis and relatively large expected effect sizes.

 

The proposed study was evaluated by three expert reviewers and the recommender over three rounds of in-depth review, plus a final round ironing out smaller issues. Reviewer Juan David Leongómez was recruited as a co-author after the first round of revisions. Following this process, the recommender decided that the manuscript met Stage 1 criteria and awarded in-principle acceptance (IPA).

 

 
Level of bias control achieved: Level 6. No part of the data or evidence that will be used to answer the research question yet exists and no part will be generated until after IPA.
 
List of eligible PCI RR-friendly journals:

 

References
 
1.Hadavi, S., Kuroda, J., Shimozono, T., Leongómez, J. D. & Savage, P. E. (2024). Cross-cultural relationships between music, emotion, and visual imagery: A comparative study of Iran, Canada, and Japan. In principle acceptance of Version 6 by Peer Community in Registered Reports. https://osf.io/zdnkm

The cardinality principle (CP) is a key concept in numerical cognition, which children typically acquire by the age of five. This is the understanding that each number in a counted sequence refers to a specific set of items, and that the final number represents the total size of the set. Here, Ivanova and colleagues (2024) propose an experiment to study the changes in neural activity that accompany the acquisition of this concept, focusing on activity within the parietal lobes as measured by functional near-infrared spectroscopy (fNIRS).

 

Typically-developing children aged from 2 years 9 months to 4 years 9 months will be assessed for their ability to give a specific number of balls from a pile: those who can give five or more correctly will be classified as CP-knowers, and those who can only give lower set sizes will be classified as subset-knowers. All children will then perform an auditory number word adaptation task while undergoing fNIRS. The adaptation task involves hearing the number word ‘two’ repeated, interspersed with deviant number words (‘four’ or ‘eight’) or a non-number word (‘rin’). The experimental hypotheses are that left parietal activations and bilateral parietal functional connectivity will be differentially greater for number than non-number deviants amongst CP-knowers than amongst subset-knowers. Each hypothesis will be tested by sequential Bayes factor analysis, with a minimum of 25 and a maximum of 46 participants per group, providing high sensitivity to detect a smallest effect size of d = .35. This study aims to provide insights into the neural underpinnings of the CP, informing theoretical models of symbolic knowledge acquisition.

The study plan was refined over four rounds of review, with input from two external reviewers, after which the recommender judged that the Stage 1 manuscript met the criteria for in-principle acceptance (IPA).

 

Level of bias control achieved: Level 4. At least some of the data/evidence that will be used to answer the research question already exists and is accessible in principle to the authors (e.g. residing in a public database or with a colleague) but the authors certify that they have not yet accessed any part of that data/evidence.
 
List of eligible PCI RR-friendly Journals:
 

• Advances in Cognitive Psychology
• Brain and Neuroscience Advances
• Cortex
• Imaging Neuroscience
• In&Vertebrates
• NeuroImage: Reports
• Peer Community Journal
• PeerJ
• Royal Society Open Science
• Studia Psychologica

 

 

Ivanova, E., Joanisse, M., Ansari, D., & Soltanlou, M. (2024). The origin of symbolic numerical knowledge in early development – an fNIRS Registered Report. In principle acceptance of Version 7 by Peer Community in Registered Reports. https://osf.io/gzpk5