Comparative research is how evidence is generated to support or refute broad hypotheses (e.g., Pagel 1999). However, the foundations of such research must be solid if one is to arrive at the correct conclusions. Determining the external validity (the generalizability across situations/individuals/populations) of the building blocks of comparative data sets allows one to place appropriate caveats around the robustness of their conclusions (Steckler & McLeroy 2008).

In the current study, Alley and colleagues (2023) tackled the external validity of comparative research that relies on subjects who are either university students or participating in experiments via an online platform. They determined whether data from these two types of subjects have measurement equivalence - whether the same trait is measured in the same way across groups.

Although they use data from studies involved in the Many Labs replication project to evaluate this question, their results are of crucial importance to other comparative researchers whose data are generated from these two sources (students and online crowdsourcing). The authors show that these two types of subjects do not often have measurement equivalence, which is a warning to others to evaluate their experimental design to improve validity. They provide useful recommendations for researchers on how to to implement equivalence testing in their studies, and they facilitate the process by providing well annotated code that is openly available for others to use.

After one round of review and revision, the recommender judged that the manuscript met the Stage 2 criteria and awarded a positive recommendation.

 

Level of bias control achieved: Level 2. At least some data/evidence that was used to answer the research question had been accessed and partially observed by the authors prior to Stage 1 IPA, but the authors certify that they had not yet observed the key variables within the data that were used to answer the research question AND they took additional steps to maximise bias control and rigour.

 

 

• Advances in Methods and Practices in Psychological Science
• Collabra: Psychology
• F1000Research
• Peer Community Journal
• PeerJ
• Royal Society Open Science
• Studia Psychologica
• Swiss Psychology Open

 

 

1. Pagel, M. (1999). Inferring the historical patterns of biological evolution. Nature, 401, 877-884. https://doi.org/10.1038/44766

 

2. Steckler, A. & McLeroy, K. R. (2008). The importance of external validity. American Journal of Public Health 98, 9-10. https://doi.org/10.2105/AJPH.2007.126847

 

3. Alley L. J., Axt, J., & Flake J. K. (2023). Convenience Samples and Measurement Equivalence in Replication Research [Stage 2 Registered Report] Acceptance of Version 2 by Peer Community in Registered Reports​. https://osf.io/s5t3v

For centuries, the relationship between creativity and mental health has been a subject of fascination, propelled by the impression that many of the most famous artists in history likely suffered from mood disorders or other mental illnesses. However, with the advent of psychological science – including more precise and diagnostic clinical measures – the empirical evidence for an association between creativity and depressive symptoms has been mixed, with some studies suggesting a positive relationship and others showing either no effect or indicating that the link, if there is one, may be driven by other personality characteristics (Verhaeghen et al., 2005).

 

In the current study, Lam and Saunders used an online design in 201 participants to ask whether creativity is associated with higher depressive traits, and further, whether that relationship depends on two additional variables that could explain an observed positive correlation: self-reflective rumination (repetitive thoughts that maintain a negative mood state) and the frequency with which individuals engage in reappraisal (a regulation strategy that involves reinterpreting an event or situation to diminish its negative impact).

 

Results showed mixed support for the hypotheses. Contrary to expectations, the relationship between creativity and depression was significantly negative rather than positive. However, even though the directionality of this relationship was opposite to predicted, the hypothesis that the association between creativity and depression is mediated by self-reflective rumination was supported. Finally, the results disconfirmed the hypothesis that reappraisal frequency contributes to the relationship between creativity and depression. Exploratory analyses indicated no reliable moderating role of gender. Overall, these findings suggest that creativity and depression may be only weakly related, and that self-reflective rumination could account for that relationship.

 

The Stage 2 manuscript was evaluated over one round of review provided by the recommender and Managing Board, as the Stage 1 reviewers were no longer available. Based on additional changes to the manuscript, the recommender judged that the manuscript met the Stage 2 criteria and awarded a positive recommendation.

 

URL to the preregistered Stage 1 protocol: https://osf.io/yub7n
 
Level of bias control achieved: Level 6. No part of the data or evidence that was used to answer the research question was generated until after IPA. 
 
List of eligible PCI RR-friendly journals:
 

References
 

1. Verhaeghen, P., Joormann, J., & Khan, R. (2005). Why we sing the blues: The relation between self-reflective rumination, mood, and creativity. Emotion, 5, 226-232. https://doi.org/10.1037/1528-3542.5.2.226

 

2. Lam, C. Y. & Saunders, J. A. (2023). Relationship between creativity and depression: the role of reappraisal and rumination [Stage 2 Registered Report]. Acceptance of Version 2 by Peer Community in Registered Reports. https://osf.io/z3ap5

‘Pseudoneglect’ is a small, lateralised bias of visuospatial attention towards the left side of space, and is typically observed in healthy adults. Recently, Strauch et al. (2022) reported that bright stimuli presented in the left visual field induced a greater constriction of the pupil (the pupillary light reflex) compared to the same bright stimuli presented in the right visual field. Further, the pupillary restriction bias was positively correlated with a behavioural measure of pseudoneglect (the greyscales task). This is potentially an important development for attention research, because the passive nature of the task, in addition to the ability to track the time course of the bias measures, could provide a new, and highly sensitive, method of studying spatial attention.

 

In this report, Burns and McIntosh (2023) aim to replicate and extend the study of Strauch et al. (2022). The extension centres around investigating whether the pupillary biases are influenced by recording pupillary responses from the right or left eye. In their pilot replication data, Burns & McIntosh identified a larger constriction in response to stimuli on the right side when recording from the right eye. They hypothesise that pupillary biases may be stronger to stimuli presented in the ipsilateral, rather than contralateral, side of space.

 

The Stage 1 manuscript was reviewed over 2 rounds by 2 reviewers, including the authors of the study being replicated. Based on detailed responses to the reviewers’ comments and edits to the Stage 1 report, the recommender judged that the manuscript met the Stage 1 criteria and therefore awarded in-principle acceptance.

 

Level of bias control achieved: Level 6. No part of the data or evidence that will be used to answer the research question yet exists and no part will be generated until after IPA.
 
List of eligible PCI-RR-friendly journals:
 

 

1. Strauch, C., Romein, C., Naber, M., Van der Stigchel, S. & Ten Brink, A. F. (2022). The orienting response drives pseudoneglect—Evidence from an objective pupillometric method. Cortex, 151, 259-271.

 
2. Burns, N. E. & McIntosh, R. D. (2023). Does pupillometry provide a valid measure of spatial attentional bias (pseudoneglect)? In principle acceptance of Version 2 by Peer Community in Registered Reports. https://osf.io/ua9jn

Measurement is a vital activity for all research areas, but we so often fail to provide sufficient clarity, rigor and transparency about it, undermining the validity of our studies' conclusions (Flake & Fried, 2020). This concern is of wide societal interest when applied to the domains of ideology and attitudes where measurements of both implicit and explicit attitudes are assumed to reflect the same underlying concept. The extent to which this can be accepted is undermined by mixed evidence demonstrating a lack of consensus on the extent to which relevant psychological factors similarly predict both implicit and explicit attitudes.

 

In the proposed study, Reid & Inbar (2023) question these assumptions through use of the Project Implicit dataset, exploring the extent to which social dominance orientation (SDO) and right wing authoritarianism (RWA) similarly predict implicit and explicit attitudes. This work is ideally suited for publication through the Registered Reports format because whilst we may expect that relationships between SDO/RWA are similar in effect size across measures of both implicit and explicit attitude (because they tap into the same underlying attitude), there is great scope to acknowledge a more complex set of findings which may not be immediately interpretable or coherent. The proposed work will help us unpack further the assumptions surrounding measurement of attitudes and can help us better understand the extent to which SDO and RWA predict atittudes. 

 

The Stage 1 manuscript was evaluated over three rounds of in-depth review. Based on detailed responses to the reviewers' comments, the recommender judged that the manuscript met the Stage 1 criteria and therefore awarded in-principle acceptance (IPA).

 

URL to the preregistered Stage 1 protocol: https://osf.io/zv4jw
 
Level of bias control achieved: Level 5. All of the data or evidence that will be used to answer the research question already exist, but are currently inaccessible to the authors and thus unobservable prior to IPA (e.g. held by a gatekeeper) 
 
List of eligible PCI-RR-friendly journals:

 

 

 

1. Flake, J. K. & Fried, E. I. (2020). Measurement schmeasurement: Questionable measurement practices and how to avoid them. Advances in Methods and Practices in Psychological Science, 3, 456-465. https://doi.org/10.1177/2515245920952393

 

2. Reid, J. & Inbar, Y. (2023). Implicit Ideologies: Do Right-Wing Authoritarianism and Social Dominance Orientation Predict Implicit Attitudes? In principle acceptance of Version 3 by Peer Community in Registered Reports. https://osf.io/zv4jw

Unconscious priming effects have fascinated not just psychologists but also ad-makers and consumers alike. A related phenomenon in perception is illustrated by presenting participants with two-tone images, which are degraded versions of images of objects and scenes. These two-tone images look like and are indeed judged as meaningless dark and light patches. Upon presenting the actual template image, however, the two-tone image is accurately recognized. This perceptual learning is abrupt, robust, and long-lasting (Daoudi et al., 2017). Surprisingly, Chang et al. (2016) showed that such perceptual disambiguation of two-tone images can happen even in the absence of conscious awareness of having seen the template image. 

 

Halchin et al. (2023) in the current study propose to conduct a conceptual replication of Chang et al. (2016) with important modifications to the procedures to address limitations with the earlier work. Specifically, there was no explicit manipulation of levels of conscious awareness of the template images in the original study. Therefore, miscategorization of low-confidence awareness as unaware could have led to an erroneous conclusion about unconscious priors guiding perceptual learning. Such miscategorization errors and how to tackle them are of interest to the broader field of consciousness studies. Furthermore, a conceptual replication of Chang et al. (2016) is also timely given that prior related work suggests that masking impairs not only conscious awareness of visual features but also blocks processing of higher-level information about the images (e.g. object category). 

 

To address the issues identified above, Halchin et al. (2023) propose to experimentally manipulate conscious awareness by masking the template image very quickly (i.e., a short stimulus onset asynchrony; SOA) or by allowing some more time to induce weak and strong conscious awareness, respectively. The SOAs were validated through pilot studies. Furthermore, they include a four-point perceptual awareness scale instead of the original yes/no options to gauge participants’ subjective awareness of the template images. The authors also propose multiple experiments to include different ways of testing participants’ objective ability to identify the masked template images. Last but not least, the proposed design includes a stronger control condition than the original study by using masked images created from related images (e.g. belonging to the same semantic category). Depending on the results obtained in the main experiments, the inclusion of this control allows the authors to conduct a third experiment to investigate whether the results in the first two can be explained by semantic priming. The proposed study is sufficiently powered (as demonstrated through simulations), and Bayesian statistical procedures will be used to test the main hypotheses. In summary, the proposed work offers a significant improvement in terms of experimental procedures over the original study. If the Chang et al. (2016) results are replicated, the stronger design in the current study is likely to lead to a better understanding of the mechanisms underlying unconscious priors guiding perceptual learning. On the other hand, a failure to replicate not just Chang et al. (2016)’s results but also effects across the three experiments in the current study would raise legitimate questions about the reality of unconscious information guiding perceptual learning. 

 

The study plan was refined across two rounds of review, with input from two external reviewers who both agreed that the proposed study is well designed, timely, and scientifically valid. The recommender then reviewed the revised manuscript and judged that the study met the Stage 1 criteria for in-principle acceptance (IPA).

 

URL to the preregistered Stage 1 protocol: https://osf.io/juckg
 
Level of bias control achieved: Level 3. At least some of the data/evidence that will be used to answer the research question already exists AND is accessible in principle to the authors BUT the authors certify that they have not yet accessed any part of that data/evidence.
 
List of eligible PCI-RR-friendly journals:

 

• Advances in Cognitive Psychology
• Collabra: Psychology
• Cortex
• Journal of Cognition
• Peer Community Journal
• PeerJ
• Psychology of Consciousness: Theory, Research and Practice
• Royal Society Open Science
• Studia Psychologica
• Swiss Psychology Open

 

 

1. Daoudi, L. D., Doerig, A., Parkosadze, K., Kunchulia, M. & Herzog, M. H. (2017). The role of one-shot learning in #TheDress. Journal of Vision, 17, 15-15. https://doi.org/10.1167/17.3.15 

 

2. Chang, R., Baria, A. T., Flounders, M. W., & He, B. J. (2016). Unconsciously elicited perceptual prior. Neuroscience of Consciousness, 2016. https://doi.org/10.1093/nc/niw008 

 

3. Halchin, A.-M., Teuful, C. & Bompas, A. (2023). Can one-shot learning be elicited from unconscious information? In principle acceptance of Version 3 by Peer Community in Registered Reports. https://osf.io/juckg