We are recruiting recommenders (editors) from all research fields!
Your feedback matters! If you have authored or reviewed a Registered Report at Peer Community in Registered Reports, then please take 5 minutes to leave anonymous feedback about your experience, and view community ratings.
Latest recommendations
| Id | Title * | Authors * | Abstract * ▲ | Picture | Thematic fields * | Recommender | Reviewers | Submission date | |
|---|---|---|---|---|---|---|---|---|---|
17 Nov 2022
STAGE 1
Removing barriers to plant-based diets: assisting doctors with vegan patientsStage 1 acceptance (IPA)Recommended by Zoltan Dienes based on reviews by Joshua Tasoff, Bence Palfi and Alaa Aldoh
Thank you for your careful response to the points of myself and the reviewers. I am now happy to award in principle acceptance (IPA). As requested, your submission is being awarded a private Stage 1 acceptance, which will not appear yet on the PCI RR website. Your Stage 1 manuscript has also been registered under the requested 4-year private embargo on the OSF (link below). URL to the preregistered Stage 1 protocol: https://osf.io/fc9gp Level of bias control achieved: Level 6. No part of the data or evidence that will be used to answer the research question yet exists and no part will be generated until after IPA. List of eligible PCI RR-friendly journals: | Removing barriers to plant-based diets: assisting doctors with vegan patients | Romain Espinosa, Thibaut Arpinon, Paco Maginot, Sébastien Demange, and Florimond Peureux | <div style="text-align: justify;">Shifting to plant-based diets can alleviate many of the externalities associated with the current food system. Spontaneous shifts in diet are often hindered by consumers’ imperfect knowledge about the health risks... | Medical Sciences, Social sciences | Zoltan Dienes | 2021-08-16 16:50:43 | |||
09 Feb 2023
STAGE 1
The Medusa effect: A registered replication report of Will, Merritt, Jenkins, and Kingstone (2021)Looking (again) at Medusa: does pictorial abstraction influence mind perception?Recommended by Chris Chambers based on reviews by Alan Kingstone, Brittany Cassidy and 3 anonymous reviewers
The Medusa effect is a recently described phenomenon in which people judge a person to be more mindful when they appear as a picture than as a picture within a picture. Across a series of experiments, Will et al. (2021) reported that at higher levels of abstraction, images of people were judged lower in realness (how real the person seemed), experience (the ability to feel) and agency (the ability to plan and act), and also benefited less from prosocial behaviour. The findings provide an intriguing window into mind perception – the extent to which we attribute minds and mental capacities to others.
In the current study, Han et al. (2023) propose a close replication of two experiments from the original report by Will et al. (2021), asking first, whether the level of pictorial abstraction influences ratings of realness, agency and experience, and second, whether it also influences prosocial behaviour as measured in the dictator game (with participants predicted to allocate more money to recipients presented as pictures than as pictures within pictures). In the event of a non-replication using the original materials, the authors will further repeat the experiments using newly generated stimuli that are better matched for cultural context and more tightly controlled along various dimensions.
The Stage 1 manuscript was evaluated over two rounds of in-depth review. Based on detailed responses to the reviewers' comments, the recommender judged that the manuscript met the Stage 1 criteria and therefore awarded in-principle acceptance (IPA).
URL to the preregistered Stage 1 protocol: https://osf.io/xj46z Level of bias control achieved: Level 6. No part of the data or evidence that will be used to answer the research question yet exists and no part will be generated until after IPA.
List of eligible PCI RR-friendly journals: References
1. Will, P., Merritt, E., Jenkins, R., & Kingstone, A. (2021). The Medusa effect reveals levels of mind perception in pictures. Proceedings of the National Academy of Sciences, 118(32), e2106640118. https://doi.org/10.1073/pnas.2106640118
2. Han, J., Zhang, M., Liu, J., Song, Y. & Yamada, Y. (2023).The Medusa effect: A registered replication report of Will, Merritt, Jenkins, and Kingstone (2021), in principle acceptance of Version 2 by Peer Community in Registered Reports. https://osf.io/xj46z
| The Medusa effect: A registered replication report of Will, Merritt, Jenkins, and Kingstone (2021) | Jing Han, Minjun Zhang, Jiaxin Liu, Yu Song, Yuki Yamada | <div>The medusa effect refers to the tendency of people to evaluate a "picture of a person" as more mindful than a "picture of a picture of a person". This phenomenon is strikingly intriguing because it suggests that when people evaluate the human... | Social sciences | Chris Chambers | Alan Kingstone, Anonymous, Anonymous, Anonymous | 2022-08-18 09:50:35 | ||
21 Jun 2022
STAGE 1
Pathway between Negative Interpretation Biases and Psychological Symptoms: Rumination as a Transdiagnostic Mediator in a Longitudinal StudyProbing the interaction between interpretation bias and repetitive negative thinking in subclinical psychopathologyRecommended by Chris Chambers based on reviews by Ariana Castro and Rita Pasion
Research in clinical psychology has found that interpretation bias (perceiving ambiguous information in a systematically negative or hostile way) and repetitive negative thinking (recurrent, prolonged worry or rumination) are associated with a range of psychopathologies – including depression, anxiety and paranoia – but the complex interplay between them in driving symptomatology is unclear. Here, Chung and Cheung (2022) propose a longitudinal examination of the directional relationship between interpretation bias and psychological symptoms in subclinical depression and paranoia, as well as the potential transdiagnostic mediating role of repetitive negative thinking. Using an online three-wave design, they ask whether the association between negative interpretation biases and psychological symptoms is bidirectional, whether negative interpretation biases are associated with repetitive negative thinking over time, and whether repetitive negative thinking is associated with psychological symptoms over time. They will also test whether negative interpretation biases and psychological symptoms exert reciprocal influences across dimensions through repetitive negative thinking, and whether repetitive negative thinking acts as a transdiagnostic mediator for depression and paranoid thoughts. Overall, the study aims to generate a clearer understanding of the relationship between interpretation biases and subclinical symptomatology, as well as clarifying the role of rumination as a transdiagnostic mechanism that mediates psychopathology.
The Stage 1 manuscript was evaluated over two rounds of in-depth review. Based on detailed responses to the reviewers' comments, the recommender judged that the manuscript met the Stage 1 criteria and therefore awarded in-principle acceptance (IPA).
URL to the preregistered Stage 1 protocol: https://osf.io/89n7u (currently under private embargo) Level of bias control achieved: Level 6. No part of the data or evidence that will be used to answer the research question yet exists and no part will be generated until after IPA.
List of eligible PCI RR-friendly journals:
References
1. Chung, H.-F. & Cheung, S.-H. (2022). Repetitive negative thinking as a transdiagnostic mediator in the interplay of interpretation biases and psychological symptoms in depression and paranoia: A three-wave longitudinal study, in principle acceptance of Version 4 by Peer Community in Registered Reports. https://osf.io/89n7u
| Pathway between Negative Interpretation Biases and Psychological Symptoms: Rumination as a Transdiagnostic Mediator in a Longitudinal Study | Chung, Ho-Fung, Cheung, Sing-Hang, | <h4>Background</h4> <p>The association between interpretation biases and content-relevant symptoms have been well-established but no studies have investigated their cause-and-effect relationship in a prospective longitudinal design. To date, the... | Social sciences | Chris Chambers | Rita Pasion, Ariana Castro | 2022-02-17 05:36:23 | ||
Putting things into perspective: Which visual cues facilitate automatic extraretinal symmetry representation?Understanding how visual cues influence extraretinal representation of planar symmetryRecommended by Grace Edwards and Zoltan Dienes based on reviews by Guillaume Rousselet and 2 anonymous reviewers
Visual symmetry is critical to our interaction with our environment so that when detected, symmetry automatically produces a neural marker in the form of an Event Related Potential (ERP) called Sustained Posterior Negativity (SPN). However, when symmetry is presented to the visual system slanted away from the viewer, there is a reduction in SPN, termed a perspective cost.
Considering objects are rarely presented front-on (or frontoparallel) in our natural environment, Karakashevska et al., (2023) examined the extent of the perspective cost with the addition of visual cues to facilitate extraretinal representation of the visual symmetry. The authors recorded electroencephalography (EEG) from 120 participants while they performed a luminance task on symmetrical and asymmetrical stimuli. The authors hypothesized that perspective cost would be reduced by three perspective cues: 1) monocular viewing, eliminating binocular cue conflict, 2) a static frame surrounding the symmetrical stimulus, adding a depth cue, and 3) a moving frame, providing a structure-from-motion 3D cue, prior to the symmetry onset. If the SPN was equivalent during frontoparallel and slanted presentation in a cue condition, the authors would have concluded extraretinal representation can be automatic when sufficient visual cues are available. The experiment was powered to detect a relatively small difference between perspective cue conditions.
The authors found that there was no impact of different visual cues on the perspective cost, as measured using the SPN. Perspective cost was consistent across all conditions, contrary to the pre-registered hypotheses. Karakashevska and colleagues conclude that the three perceptual cues tested in their design do not reduce perspective cost. The study prompts future research into the nature of the extraretinal representations of planar symmetry.
The Stage 2 manuscript was evaluated over four rounds by three expert reviewers. Two of our reviewers reviewed the Stage 1 manuscript, and one new reviewer. Following in-depth review and responses from the authors, the recommenders determined that the Stage 2 criteria were met and awarded a positive recommendation.
URL to the preregistered Stage 1 protocol: https://osf.io/yzsq5
Level of bias control achieved: Level 6. No part of the data or evidence that was used to answer the research question was generated until after IPA. List of eligible PCI-RR-friendly journals:
References 1. Karakashevska, E., Bertamini, M. & Makin, A. D. J. (2024). Putting things into perspective: Which visual cues facilitate automatic extraretinal symmetry representation? [Stage 2]. Acceptance of Version 5 by Peer Community in Registered Reports. https://doi.org/10.31234/osf.io/z9c28 | Putting things into perspective: Which visual cues facilitate automatic extraretinal symmetry representation? | Elena Karakashevska, Marco Bertamini and Alexis D.J. Makin | <p>Abstract</p> <p>Introduction: Objects often project different images when viewed from different locations. Our visual system can correct for perspective distortion and identify objects from different viewpoints that change the retinal image. T... | Life Sciences | Grace Edwards | 2024-06-03 21:00:08 | |||
18 Oct 2023
STAGE 1
Putting things into perspective: Which visual cues facilitate automatic extraretinal symmetry representation?Understanding how visual cues influence extraretinal representation of planar symmetryRecommended by Grace Edwards and Zoltan Dienes based on reviews by Tadamasa Sawada, Guillaume Rousselet, Benoit Cottereau and Deborah Apthorp
Visual symmetry is critical to our interaction with our environment so that when detected, symmetry automatically produces a neural marker in the form of an Event Related Potential (ERP) called Sustained Posterior Negativity (SPN). However, when symmetry is presented to the visual system slanted away from the viewer, there is a reduction in SPN, termed a perspective cost.
Considering objects are rarely presented front-on (or frontoparallel) in our natural environment, Karakashevska et al., (2023) plan to examine the extent of the perspective cost with the addition of visual cues to facilitate extraretinal representation of the visual symmetry. The authors will record electroencephalography (EEG) from 120 participants while they perform a luminance task on symmetrical and asymmetrical stimuli. The authors hypothesize perspective cost will be reduced by three perspective cues: 1) monocular viewing, when cue conflict caused by binocular viewing is eliminated, 2) a static frame surrounding the symmetrical stimulus, adding a depth cue, and 3) a moving frame, assisting 3D perception prior to the symmetry onset. If the SPN is equivalent during frontoparallel and slanted presentation in a cue condition, the authors will conclude extraretinal representation can be automatic when sufficient visual cues are available. The proposed experiment is powered to detect a relatively small difference between perspective cue conditions. This will solidify fundamental research on visual symmetry processing and will further our understanding of object perception and recognition.
The Stage 1 manuscript was evaluated over three rounds by four expert reviewers. Following in-depth review and responses from the authors, the recommenders have determined that Stage 1 criteria was met and have awarded in-principle acceptance (IPA).
URL to the preregistered Stage 1 protocol: https://osf.io/yzsq5
Level of bias control achieved: Level 6. No part of the data or evidence that will be used to answer the research question yet exists and no part will be generated until after IPA. List of eligible PCI-RR-friendly journals: References 1. Karakashevska, E., Bertamini, M. & Makin, A. D. J. (2023). Putting things into perspective: Which visual cues facilitate automatic extraretinal symmetry representation? [Stage 1 Registered Report]. In principle acceptance of Version 4 by Peer Community in Registered Reports. https://osf.io/yzsq5 | Putting things into perspective: Which visual cues facilitate automatic extraretinal symmetry representation? | Elena Karakashevska, Marco Bertamini and Alexis D.J. Makin | <p>A challenge for the visual brain is to identify objects from a range of different viewpoints. This study will determine the conditions under which the brain spends computational resources to achieve view-invariance. We focus on view-invariant r... | Life Sciences | Grace Edwards | 2023-04-17 21:52:26 | |||
31 May 2023
STAGE 1
Defacing biases in manual and automated quality assessments of structural MRI with MRIQCThe impact of removing facial features on quality measures of structural MRI scansRecommended by D. Samuel Schwarzkopf based on reviews by Catherine Morgan and Cassandra Gould van Praag
Data sharing is perhaps the most fundamental step for increasing the transparency and reproducibility of scientific research. However, the goals of open science must be tempered by ethical considerations, protecting the privacy and safety of research participants. Bridging this gap causes challenges for many fields, such as human neuroimaging. Brain images, as measured with magnetic resonance imaging (MRI), are unique to the participant and therefore contain identifying information by definition. One way to mitigate the risk to participants arising from public data sharing has been "defacing" the MRI scans, i.e., literally removing the part of the image that contains the face and surrounding tissue, while preserving the brain structure. This procedure however also removes information that is not (or at least minimally) identifiable. It also remains unclear whether defacing the images affects image quality and thus the information necessary for addressing many research questions.
The current study by Provins et al. (2023) seeks to address this question. Leveraging a publicly available "IXI dataset" comprising hundreds of T1-weighted structural MRI scans, they will assess the effect of defacing on manual and automatic estimates of image quality. Specifically, the researchers will compare image quality ratings by experts for a subset of 185 images. They hypothesise that images in which facial features have been removed are typically assigned higher quality ratings. Moreover, using a full data set of 580 images, which have been obtained across three scanning sites, they will also test the impact defacing MRI scans has on automated quality measures obtained with MRIQC software. The results of this study should have important implications for open science policy and for designing the optimal procedures for sharing structural MRI data in an ethical way. For example, if the authors' hypothesis is confirmed, studies relying on MRI quality measures might be better served by a custodianship model where identifiable data is shared under strict conditions, rather than relying on publishing defaced data. More generally, the outcome of this study may have significant legal implications in many jurisdictions.
The Stage 1 manuscript was evaluated at the inital triage stage by the Recommender and PCI:RR team, and another round of in-depth review by two experts. After a detailed response and substantial revisions, the recommender judged the manuscript met the Stage 1 criteria and awarded in-principle acceptance (IPA).
URL to the preregistered Stage 1 protocol: https://osf.io/qcket (under temporary private embargo)
Level of bias control achieved: Level 2. At least some data/evidence that will be used to answer the research question has been accessed and partially observed by the authors, but the authors certify that they have not yet observed the key variables within the data that will be used to answer the research question AND they have taken additional steps to maximise bias control and rigour. List of eligible PCI RR-friendly journals:
References
1. Provins, C., Savary, E., Alemán-Gómez, Y., Richiardi, J., Poldrack, R. A., Hagmann, P. & Esteban, O. (2023). Defacing biases in manual and automated quality assessments of structural MRI with MRIQC, in principle acceptance of Version 3 by Peer Community in Registered Reports. https://osf.io/qcket
| Defacing biases in manual and automated quality assessments of structural MRI with MRIQC | Céline Provins, Yasser Alemán-Gómez, Jonas Richiardi, Russell A. Poldrack, Patric Hagmann, Oscar Esteban | <p>A critical requirement before data-sharing of human neuroimaging is removing facial features to protect individuals’ privacy. However, not only does this process redact identifiable information about individuals, but it also removes non-identif... | Medical Sciences | D. Samuel Schwarzkopf | Catherine Morgan, Cassandra Gould van Praag, Abiola Akinnubi | 2022-11-28 10:59:32 | ||
16 Sep 2022
STAGE 1
Action-Inaction Asymmetries in Emotions and Counterfactual Thoughts: Meta-Analysis of the Action EffectCharting meta-analytic evidence for the action-effectRecommended by Chris Chambers based on reviews by Dan Quintana, Emiel Cracco and priyali rajagopal
Winston Churchill once famously quipped, “I never worry about action, but only inaction.” Churchill, however, may have been an exception to the rule, with psychological research suggesting that people are more concerned about the consequences of actions than inactions. During the so-called “action-effect”, first reported by Kahneman and Tversky (1982), people regret an action leading to a bad outcome more than they do an inaction leading to the same bad outcome In the current study, Yeung and Feldman (2022) propose a wide-ranging meta-analysis to characterise evidence for the action-effect, focusing in particular on emotions and counterfactual thoughts – that is, mental representations of alternative decisions (or “what if” thoughts). Consistent with the expected consequences of the action-effect on emotion, they predict that action will be associated with stronger negative emotions than inaction (when outcomes are negative), and with stronger positive emotions than inaction (when outcomes are positive). The authors also expect action to be associated with a greater abundance of counterfactual thought compared to inaction. In addition to examining the overall reliability of the action-effect (plus a range of exploratory questions), the study will also examine the extent to which the action-effect is moderated by temporal distance (with more recent events or behaviours predicted to associated with a stronger action effect), the type of study design, prior outcomes and social norms, the specificity (vs. generality) of the prior event, and whether the study employed a hypothetical scenario or a real-life event. The Stage 1 manuscript was evaluated over two rounds of in-depth review. Based on detailed responses to the reviewers' comments, the recommender judged that the manuscript met the Stage 1 criteria and awarded in-principle acceptance (IPA). URL to the preregistered Stage 1 protocol: https://osf.io/4pvs6 Level of bias control achieved: Level 2. At least some data/evidence that will be used to answer the research question has been accessed and partially observed by the authors, but the authors certify that they have not yet observed the key variables within the data that will be used to answer the research question References | Action-Inaction Asymmetries in Emotions and Counterfactual Thoughts: Meta-Analysis of the Action Effect | Siu Kit Yeung, Gilad Feldman | <p>Action-effect refers to the phenomenon in which people experience, associate, or attribute stronger emotions for action compared to inaction. In this registered report, we conducted a meta-analysis of the action effect literature (k = [enter nu... | Social sciences | Chris Chambers | 2021-07-16 14:02:56 | |||
Applying a Synergistic Mindsets Intervention to an Esports ContextSynergistic Mindset Intervention in Competitive SituationsRecommended by Veli-Matti Karhulahti based on reviews by Lee Moore, Ivan Ropovik and Jacob Keech
Mindset theories suggest that the mere belief in the malleability of human abilities can help one to develop related performance. On the other hand, one and the same performance situation can also be experienced in various affective ways, which differently contribute to performance outcomes. One theoretically justifiable premise is that appraising a performance situation as a “threat” instead of “challenge” is associated with maladaptive responses, such as impaired cardiovascular mobilization. If people could experience performance situations as positive challenges, this might also improve performance outcomes. Drawing from these connected premises, the synergistic mindset intervention was developed and tentatively found to help adolescents in stressful situations (Yeager et al., 2022).
In the present registered report, Behnke et al. (2024) built on the above to test whether the synergistic mindset intervention can help individuals in competitive gaming situations. The authors utilized one of the leading esport games, Counter-Strike: Global Offensive, and recruited its active players (N=300) into randomized control and intervention groups. The participants competed in a cash-prize tournament involving measures of affective experience and cardiovascular responses. Behnke et al. (2024) hypothesized the synergistic mindset group (SMI) to show greater challenge affective responses and superior performance outcomes. Although the SMI produced a number of positive outcomes such as more beneficial stress mindsets, the hypotheses were not corroborated but the results supported a null. This may be related to the observation that participants generally experienced the intervention positively, which, in turn, limits the potential for improving affective and physiological responses. These rigorous null results are informative by directing the SMI research program toward test designs where more participants experience strong negative stress responses. Moreover, the results encourage researchers to reassess the underlying auxiliary hypotheses regarding affective responses and performance outcomes, the relationships of which may be complicated by situational factors that are not yet fully understood. The Stage 2 manuscript was evaluated over one round of in-depth review. Three out of the four Stage 1 experts returned to review and, due to the study’s exceptionally high level of transparency, the reviewers had only minor requests for revision. As all the requested revisions were implemented carefully, the recommender judged that the manuscript met the Stage 2 criteria and awarded a positive recommendation. URL to the preregistered Stage 1 protocol: https://osf.io/z3adb Level of bias control achieved: Level 6. No part of the data or evidence that was used to answer the research question was generated until after IPA. List of eligible PCI RR-friendly journals:
References
1. Behnke M., Lakens D., Petrova K., Chwiłkowska P., Kaczmarek L. D., Jamieson J. P., & Gross J. J. (2024) Applying a Synergistic Mindsets Intervention to an Esports Context. Acceptance of Version 2 by Peer Community in Registered Reports https://osf.io/53z8e 2. Yeager D.S., Bryan C.J., Gross J.J., Murray J., Krettek D., Santos P., ... & Jamieson J.P. (2022) A synergistic mindsets intervention protects adolescents from stress. Nature 607, 512–520. https://doi.org/10.1038/s41586-022-04907-7
| Applying a Synergistic Mindsets Intervention to an Esports Context | Maciej Behnke, Daniël Lakens, Kate Petrova, Patrycja Chwiłkowska, Szymon Jęśko Białek, Maciej Kłoskowski, Wadim Krzyżaniak, Patryk Maciejewski, Lukasz D. Kaczmarek, Kacper Szymański, Jeremy P. Jamieson, James J. Gross | <p>Affective responses during stressful, high-stakes situations can play an important role in shaping performance. For example, feeling shaky and nervous at a job interview can undermine performance, whereas feeling excited during that same interv... | Social sciences | Veli-Matti Karhulahti | 2024-02-02 17:57:13 | |||
27 Mar 2023
STAGE 1
Optimizing Esports Performance Using a Synergistic Mindsets InterventionSynergistic Mindset Intervention for Competitive SituationsRecommended by Veli-Matti Karhulahti based on reviews by Lee Moore, Ivan Ropovik , Ivana Piterová and Jacob Keech
Mindset theories suggest that the mere belief in the malleability of human abilities can already help one to develop related performance. On the other hand, one and the same performance situation can also be experienced in various affective ways, which differently contribute to performance outcomes. Arguably, appraising a performance situation as a “threat” instead of “challenge” is associated with maladaptive responses, such as impaired cardiovascular mobilization. If people could experience performance situations as positive challenges, this might also improve performance outcomes. Drawing from these connected theoretical premises, the synergistic mindset intervention was developed and tentatively found to help adolescents in stressful situations (Yeager et al., 2022).
In the present registered report, Behnke et al. (2023) build on the above and test whether the synergistic mindset intervention can help individuals in competitive gaming situations. The authors utilize one of the leading esport games, Counter-Strike: Global Offensive, and recruit its active players into randomized control and intervention groups for two weeks. Ultimately, the participants compete in a cash-prize tournament involving measures of affective experience and cardiovascular responses. Behnke et al. (2023) hypothesize that the synergistic mindset group will show greater challenge affective responses and superior performance outcomes. As such, the study design has significant potential to generate valuable evidence for various theoretical models and the synergistic mindset model in particular. The Stage 1 manuscript was evaluated over two rounds by four experts with experimental psychology specializations in mindsets, stress, and statistics. Based on the comprehensive responses to the reviewers' feedback, the recommender judged that the manuscript met the Stage 1 criteria and therefore awarded in-principle acceptance (IPA). URL to the preregistered Stage 1 protocol: https://osf.io/z3adb Level of bias control achieved: Level 6. No part of the data or evidence that will be used to answer the research question yet exists and no part will be generated until after IPA. List of eligible PCI RR-friendly journals:
References
Behnke M., Lakens D., Petrova K., Chwiłkowska P., Kaczmarek L. D., Jamieson J. P., & Gross J. J. (2023) Optimizing Esports Performance Using a Synergistic Mindsets Intervention. In principle acceptance of Version 3 by Peer Community in Registered Reports. https://osf.io/z3adb Yeager D.S., Bryan C.J., Gross J.J., Murray J., Krettek D., Santos P., ... & Jamieson J.P. (2022) A synergistic mindsets intervention protects adolescents from stress. Nature 607, 512–520. https://doi.org/10.1038/s41586-022-04907-7 | Optimizing Esports Performance Using a Synergistic Mindsets Intervention | Maciej Behnke, Daniël Lakens, Kate Petrova, Patrycja Chwiłkowska, Lukasz D. Kaczmarek, Jeremy P. Jamieson, James J. Gross | <p>Affective responses during stressful high-stakes situations can play an important role in shaping performance outcomes. For example, feeling shaky and nervous at a job interview can undermine performance, whereas feeling pumped and excited duri... | | Social sciences | Veli-Matti Karhulahti | 2023-01-04 10:12:55 | ||
21 Mar 2023
STAGE 1
Convenience Samples and Measurement Equivalence in Replication ResearchDoes data from students and crowdsourced online platforms measure the same thing? Determining the external validity of combining data from these two types of subjectsRecommended by Corina Logan based on reviews by Benjamin Farrar and Shinichi Nakagawa
Comparative research is how evidence is generated to support or refute broad hypotheses (e.g., Pagel 1999). However, the foundations of such research must be solid if one is to arrive at the correct conclusions. Determining the external validity (the generalizability across situations/individuals/populations) of the building blocks of comparative data sets allows one to place appropriate caveats around the robustness of their conclusions (Steckler & McLeroy 2008).
In this registered report, Alley and colleagues plan to tackle the external validity of comparative research that relies on subjects who are either university students or participating in experiments via an online platform (Alley et al. 2023). They will determine whether data from these two types of subjects have measurement equivalence - whether the same trait is measured in the same way across groups. Although they use data from studies involved in the Many Labs replication project to evaluate this question, their results will be of crucial importance to other comparative researchers whose data are generated from these two sources (students and online crowdsourcing). If Alley and colleagues show that these two types of subjects have measurement equivalence, then this indicates that it is more likely that equivalence could hold for other studies relying on these type of subjects as well. If measurement equivalence is not found, then it is a warning to others to evaluate their experimental design to improve validity. In either case, it gives researchers a way to test measurement equivalence for themselves because the code is well annotated and openly available for others to use.
The Stage 1 manuscript was evaluated over two rounds of in-depth review. Based on detailed responses to the reviewers' comments, the recommender judged that the manuscript met the Stage 1 criteria and therefore awarded in-principle acceptance (IPA). Level of bias control achieved: Level 2. At least some data/evidence that will be used to answer the research question has been accessed and partially observed by the authors, but the authors certify that they have not yet observed the key variables within the data that will be used to answer the research question AND they have taken additional steps to maximise bias control and rigour (e.g. conservative statistical threshold; recruitment of a blinded analyst; robustness testing, multiverse/specification analysis, or other approach)
List of eligible PCI RR-friendly journals:
References
Alley L. J., Axt, J., & Flake J. K. (2023). Convenience Samples and Measurement Equivalence in Replication Research, in principle acceptance of Version 4 by Peer Community in Registered Reports. https://osf.io/7gtvf
Steckler, A. & McLeroy, K. R. (2008). The importance of external validity. American Journal of Public Health 98, 9-10. https://doi.org/10.2105/AJPH.2007.126847
Pagel, M. (1999). Inferring the historical patterns of biological evolution. Nature, 401, 877-884. https://doi.org/10.1038/44766
| Convenience Samples and Measurement Equivalence in Replication Research | Lindsay J. Alley, Jordan Axt, Jessica Kay Flake | <p>A great deal of research in psychology employs either university student or online crowdsourced convenience samples (Chandler & Shapiro, 2016; Strickland & Stoops, 2019) and there is evidence that these groups differ in meaningful ways ... | Social sciences | Corina Logan | 2022-11-29 18:37:54 |
