Announcements
We are recruiting recommenders (editors) from all research fields!
Your feedback matters! If you have authored or reviewed a Registered Report at Peer Community in Registered Reports, then please take 5 minutes to leave anonymous feedback about your experience, and view community ratings.
279 records found
Latest recommendations
Id | Title * | Authors * ▲ | Abstract * | Picture | Thematic fields * | Recommender | Reviewers | Submission date | |
---|---|---|---|---|---|---|---|---|---|
Beneath the label: Unsatisfactory compliance with ESRB, PEGI, and IARC industry self-regulation requiring loot box presence warning labels by video game companiesLeon Y. Xiao https://doi.org/10.31219/osf.io/asbcgFailure of industry self-regulation in loot box labellingRecommended by Chris ChambersPaid loot boxes – items bought for real-world money that offer randomised rewards – are a prevalent feature of contemporary video games (Zendle et al., 2020). Because they employ random chance to provide rewards after spending real money, loot boxes have been considered a form of gambling, raising concerns about risk of harm to children and other vulnerable users. In response, some countries have taken legal steps to regulate and even ban the use of loot boxes, with only limited success so far (Xiao, 2022). At the same time, the Entertainment Software Rating Board (ESRB) and PEGI (Pan-European Game Information) now expect games that contain loot boxes to be marked with warning labels that, in theory, will enable users (including parents) to make more informed decisions. These requirements by ESRB/PEGI are not legally binding and may be considered a form of industry self-regulation.
In the current study, Xiao (2023) investigated the effectiveness of self-regulation in the use of loot box labels. Study 1 examined the consistency of warning labels by the ESRB and PEGI, with the expectation that if self-regulation works as it should then these labels should always (or nearly always) co-occur. Study 2 established the compliance rate for labelling among popular games that are known to contain loot boxes, with a rate of ≥95% considered to be successful.
The results of both studies reveal deficiences in industry self-regulation. The consistency rate of warning labels by the ESRB and PEGI was just 39.4% in preregistered analyses, rising to 83.9% in an unregistered exploratory analysis that took into account industry responses to the findings. Even at this upper bound, this rate is lower than expected by complete (or near-complete) consistency. The results of Study 2 indicate that only 29% of games on the Google Play Store known to contain loot boxes were accurately labelled, indicating that 71% were non-compliant with industry requirements.
Following careful evaluation, the recommender judged that the manuscript met the Stage 2 criteria and awarded a positive recommendation.
URL to the preregistered Stage 1 protocol: https://osf.io/e6qbm Level of bias control achieved: Level 3. At least some data/evidence that was used to the answer the research question had been previously accessed by the authors (e.g. downloaded or otherwise received), but the authors certifed that they had not yet observed ANY part of the data/evidence prior to in-principle-acceptance.
List of eligible PCI RR-friendly journals:
References
1. Zendle, D., Meyer, R., Cairns, P., Waters, S., & Ballou, N. (2020). The prevalence of loot boxes in mobile and desktop games. Addiction, 115(9), 1768-1772. https://doi.org/10.1111/add.14973
2. Xiao, L. Y. (2022). Breaking Ban: Belgium’s ineffective gambling law regulation of video game loot boxes. Stage 2 Registered Report, acceptance of Version 2 by Peer Community in Registered Reports. https://doi.org/10.31219/osf.io/hnd7w 3. Xiao, L. Y. (2023). Beneath the label: Unsatisfactory compliance with ESRB, PEGI, and IARC industry self-regulation requiring loot box presence warning labels by video game companies, acceptance of Version 2 by Peer Community in Registered Reports. https://doi.org/10.31219/osf.io/asbcg
| Beneath the label: Unsatisfactory compliance with ESRB, PEGI, and IARC industry self-regulation requiring loot box presence warning labels by video game companies | Leon Y. Xiao | <p>Loot boxes in video games are a form of in-game transactions with randomised elements. Concerns have been raised about loot boxes’ similarities with gambling and their potential harms (e.g., overspending). Recognising players’ and parents’ conc... | Humanities, Social sciences | Chris Chambers | Pete Etchells , Jim Sauer | 2023-02-12 16:17:34 | View | |
25 Mar 2024
STAGE 1
![]() Assessing compliance with UK loot box industry self-regulation on the Apple App Store: a 6-month longitudinal study on the implementation processLeon Y. Xiao https://osf.io/7xft9Does self regulation by gaming companies for the use of loot boxes work?Recommended by Zoltan DienesVideo games may provide the option of spending real money in exchange for probabilistically receiving game-relevant rewards; in effect, encouraging potentially young teenagers to gamble. The industry has subscribed to a set of regulatory principles to cover the use of such "loot boxes", including 1) that they will prevent loot box purchasing by under 18s unless parental consent is given; 2) that they will make it initially clear that the game contains loot boxes; and 3) that they will clearly disclose the probabilities of receiving different rewards.
Can the industry effectively self regulate? Xiao (2024) will evaluate this important question by investigating the 100 top selling games on the Apple App Store and estimating the percentage compliance to these three regulatory principles at two time points 6 months apart.
The Stage 1 manuscript was evaluated over one round of in-depth review. Based on detailed responses to the reviewers' comments, the recommender judged that the manuscript met the Stage 1 criteria and therefore awarded in-principle acceptance (IPA).
URL to the preregistered Stage 1 protocol: https://osf.io/3knyb
Level of bias control achieved: Level 2. At least some data/evidence that will be used to answer the research question has been accessed and partially observed by the authors, but the authors certify that they have not yet observed the key variables within the data that will be used to answer the research question. List of eligible PCI RR-friendly journals:
References 1. Xiao, L. (2024). Assessing compliance with UK loot box industry self-regulation on the Apple App Store: a 6-month longitudinal study on the implementation process. In principle acceptance of Version 3 by Peer Community in Registered Reports. https://osf.io/3knyb | Assessing compliance with UK loot box industry self-regulation on the Apple App Store: a 6-month longitudinal study on the implementation process | Leon Y. Xiao | <p>Loot boxes in video games can be purchased with real-world money in exchange for random rewards. Stakeholders are concerned about loot boxes’ similarities with gambling and their potential harms (e.g., overspending). The UK Government has decid... | ![]() | Humanities, Social sciences | Zoltan Dienes | 2023-08-27 22:47:03 | View | |
Non-compliance with and non-enforcement of UK loot box industry self-regulation on the Apple App Store: A longitudinal study on poor implementationLeon Y. Xiao, Mie Lund https://osf.io/3re4nSelf regulation by gaming companies for the use of loot boxes is not workingRecommended by Zoltan DienesVideo games may provide the option of spending real money in exchange for probabilistically receiving game-relevant rewards; in effect, encouraging potentially young teenagers to gamble. The industry has subscribed to a set of regulatory principles to cover the use of such "loot boxes", including 1) that they will prevent loot box purchasing by under 18s unless parental consent is given; 2) that they will make it initially clear that the game contains loot boxes; and 3) that they will clearly disclose the probabilities of receiving different rewards.
Can the industry effectively self regulate? Xiao and Lund (2025) evaluated this important question by investigating the 100 top selling games on the Apple App Store and estimating the percentage compliance to these three regulatory principles at two time points 6 months apart. In all cases compliance was minimal, or even non-existent. The authors recommend stricter legal regulations of loot boxes.
The Stage 2 manuscript was evaluated over one round of review. The recommender judged that the manuscript met the Stage 2 criteria for recommendation.
URL to the preregistered Stage 1 protocol: https://osf.io/3knyb
Level of bias control achieved: Level 2. At least some data/evidence that was used to answer the research question had been accessed and partially observed by the authors prior to IPA, but the authors certify that they had not yet observed the key variables within the data that were used to answer the research question. List of eligible PCI RR-friendly journals:
References Xiao, L. & Lund, M. (2025). Non-compliance with and non-enforcement of UK loot box industry self-regulation on the Apple App Store: A longitudinal study on poor implementation [Stage 2]. Acceptance of Version 2 by Peer Community in Registered Reports. https://osf.io/3re4n | Non-compliance with and non-enforcement of UK loot box industry self-regulation on the Apple App Store: A longitudinal study on poor implementation | Leon Y. Xiao, Mie Lund | <p>Loot boxes in video games can be purchased with real-world money in exchange for random rewards. Stakeholders are concerned about loot boxes’ similarities with gambling and their potential harms (e.g., overspending money and developing gambling... | Humanities, Social sciences | Zoltan Dienes | 2025-01-13 12:43:11 | View | ||
04 Dec 2023
STAGE 1
![]() Self-Control beyond inhibition. German Translation and Quality Assessment of the Self-Control Strategy Scale (SCSS)Leopold H. O. Roth1, Julia Jankowski1, Georgia Clay1, Dominik Meindl1, Lisa-Marie Vogt1, Victoria Wagner1, Artemis Nordmann1, Loana Stenzel1, Olga Freiman1, Christopher Mlynski1, Veronika Job1; 1University of Vienna, Vienna, Austria https://osf.io/n2xsdStrategies for self control: German translation and evaluation of the Self Control Strategy ScaleRecommended by Zoltan DienesSelf-control has shown to be a trait related to beneficial outcomes, including health, academic achievement and relationship quality. It is mostly understood as the ability to surpress immediate urges in order to achieve long-term goals, such as not watching another episode and therefore reaching a healthy amount of sleep. An emerging perspective on self-control shows that there is broader variety in applied strategies, such as removing oneself from a tempting situation, or reminding oneself of one's long-term goal, or reinterpreting the temptation.
Katzir et al. (2021) developed a novel instrument, the Self-Control Strategy Scale, that measured the tendency to engage in eight such strategies. In the current study, Roth et al. (2023) propose to translate the scale into German and assess its psychometric properties. Further, they will determine which strategies are related to particular outcomes that may be beneficial; for example, amount of physical activity engaged in, how healthy the diet is, exam performance and life satisfaction.
The Stage 1 manuscript was evaluated over two rounds of in-depth review by the recommender and at least two expert reviewers, before issuing in-principle acceptance.
URL to the preregistered Stage 1 protocol: https://osf.io/s7qwk
Level of bias control achieved: Level 6. No part of the data or evidence that will be used to answer the research question yet exists and no part will be generated until after IPA. List of eligible PCI RR-friendly journals:
References 1. Katzir, M., Baldwin, M., Werner, K. M., & Hofmann, W. (2021). Moving beyond inhibition: Capturing a broader scope of the self-control construct with the Self-Control Strategy Scale (SCSS). Journal of Personality Assessment, 103, 762-776. https://doi.org/10.1080/00223891.2021.1883627
2. Roth, L. H. O., Jankowski, J., Clay, G., Meindl, D., Vogt, L.-M., Wagner, V., Nordmann, A., Stenzel, L., Freiman, O., Mlynski, C., & Job, V. (2023). Self-Control beyond inhibition. German Translation and Quality Assessment of the Self-Control Strategy Scale (SCSS). In principle acceptance of Version 3 by Peer Community in Registered Reports. https://osf.io/s7qwk
| Self-Control beyond inhibition. German Translation and Quality Assessment of the Self-Control Strategy Scale (SCSS) | Leopold H. O. Roth1, Julia Jankowski1, Georgia Clay1, Dominik Meindl1, Lisa-Marie Vogt1, Victoria Wagner1, Artemis Nordmann1, Loana Stenzel1, Olga Freiman1, Christopher Mlynski1, Veronika Job1; 1University of Vienna, Vienna, Austria | <p>Self-control is crucial for goal attainment and related to several beneficial outcomes, such as health and education. For a long time, it was predominantly understood in terms of inhibition, namely the ability to suppress immediate urges for th... | Social sciences | Zoltan Dienes | 2023-07-13 13:46:30 | View | ||
Self-Control Beyond Inhibition. German Translation and Quality Assessment of the Self-Control Strategy Scale (SCSS)Leopold H. O. Roth, Julia M. Jankowski, Dominik Meindl, Georgia Clay, Christopher Mlynski, Olga Freiman, Artemis L. Nordmann, Loana-Corine Stenzel, Victoria Wagner https://doi.org/10.31234/osf.io/gpmnvStrategies for self control: German translation and evaluation of the Self Control Strategy ScaleRecommended by Zoltan DienesSelf-control has shown to be a trait related to beneficial outcomes, including health, academic achievement and relationship quality. It is mostly understood as the ability to suppress immediate urges in order to achieve long-term goals, such as not watching another episode and therefore reaching a healthy amount of sleep. An emerging perspective on self-control shows that there is broader variety in applied strategies, such as removing oneself from a tempting situation, or reminding oneself of one's long-term goal, or reinterpreting the temptation.
Katzir et al. (2021) developed a novel instrument, the Self-Control Strategy Scale, that measured the tendency to engage in eight such strategies. In the current study, Roth et al. (2024) translated the scale into German and assessed its psychometric properties: internal consistency and retest reliability were sufficient for six or seven of the eight subscales. Further, different strategies (subscales) were related to particular outcomes; at least one strategy was related to each outcome for 20 out of 23 outcomes in health behavior, school/work achievement, life satisfaction, interpersonal functioning and pro-environmental behavior (though the particular pattern of similarities and differences would need confirming). Thus, the SCSS is a valid and reliable measure that can now be used in German.
The Stage 2 manuscript was evaluated over two rounds of in-depth review by the recommender and at least two expert reviewers. Following revision, the recommender judged that the manuscript met the Stage 2 criteria and awarded a positive recommendation.
URL to the preregistered Stage 1 protocol: https://osf.io/s7qwk
Level of bias control achieved: Level 6. No part of the data or evidence that was used to answer the research question was generated until after IPA. List of eligible PCI RR-friendly journals:
References 1. Katzir, M., Baldwin, M., Werner, K. M., and Hofmann, W. (2021). Moving beyond inhibition: Capturing a broader scope of the self-control construct with the Self-Control Strategy Scale (SCSS). Journal of Personality Assessment, 103, 762-776. https://doi.org/10.1080/00223891.2021.1883627
2. Roth, L. H. O., Jankowski, J., Meindl, D., Clay, G., Mlynski, C., Freiman, O., Nordmann, A., Stenzel, L., and Wagner, V. (2024). Self-Control beyond inhibition. German Translation and Quality Assessment of the Self-Control Strategy Scale (SCSS) [Stage 2]. Acceptance of Version 3 by Peer Community in Registered Reports. https://doi.org/10.31234/osf.io/gpmnv
| Self-Control Beyond Inhibition. German Translation and Quality Assessment of the Self-Control Strategy Scale (SCSS) | Leopold H. O. Roth, Julia M. Jankowski, Dominik Meindl, Georgia Clay, Christopher Mlynski, Olga Freiman, Artemis L. Nordmann, Loana-Corine Stenzel, Victoria Wagner | <p>Self-control is crucial for goal attainment and related to several beneficial outcomes, such as health and education. For a long time, it was predominantly understood in terms of inhibition, namely the ability to suppress immediate urges for th... | Social sciences | Zoltan Dienes | Eleanor Miles, Kaitlyn Werner, Sebastian Bürgler | 2024-06-28 11:50:25 | View | |
28 Jan 2025
STAGE 1
![]() See me, judge me, pay me: Gendered effort moralization in work and careLeopold H. O. Roth, Tassilo T. Tissot, Thea Fischer, S. Charlotte Masak https://osf.io/rz6yuA gender difference in effort moralization?Recommended by Adrien FillonEffort moralization is the well known idea that, unrelated to actual performance, people doing more effort are judged better, attributed more morality and seen as better collaborators than people doing less effort. However, the series of studies on this topic mostly used vignettes with a man or a neutral protagonist. The current study by Roth et al. (2025) proposes to tackle the gender problem by testing the difference in attribution morality between a man and a woman protagonist, and two contexts: a “care” and a “work” context, mirroring the stereotypes associated with men and women.
The authors included two different and adequate power analyses, various interpretation of the possible effects, and filtering to ensure a high quality of data collection. They also provide a supplementary repository including the qualtrics survey, R script, and simulated data.
The Stage 1 manuscript was evaluated over two rounds of in-depth review. Based on detailed responses to the reviewers’ and the recommender’s comments, the recommender judged that the manuscript met the Stage 1 criteria and therefore awarded in-principle acceptance. URL to the preregistered Stage 1 protocol: https://osf.io/xd87r Level of bias control achieved: Level 6. No part of the data or evidence that will be used to answer the research question yet exists, and no part will be generated until after IPA. List of eligible PCI RR-friendly journals:
References
Roth, L. H. O., Tissot, T. T., Fischer, T. & Masak, S. C. (2025). See me, judge me, pay me: Gendered effort moralization in work and care. In principle acceptance of Version 3 by Peer Community in Registered Reports. https://osf.io/xd87r
| See me, judge me, pay me: Gendered effort moralization in work and care | Leopold H. O. Roth, Tassilo T. Tissot, Thea Fischer, S. Charlotte Masak | <p>The display of high effort at work is commonly rewarded with more positive moral judgements and increased cooperation partner attractiveness. This effect was shown to hold, even if higher effort is unrelated to better performance. Yet, current ... | Social sciences | Adrien Fillon | 2024-09-09 15:12:30 | View | ||
One and only SNARC? Spatial-Numerical Associations are not fully flexible and depend on both relative and absolute magnitudeLilly Roth, John Caffier, Ulf-Dietrich Reips, Hans-Christoph Nuerk, Annika Tave Overlander, Krzysztof Cipora https://osf.io/ajqpkA Registered Report demonstration that the SNARC effect depends on absolute as well as relative number magnitudeRecommended by Robert McIntoshThe Spatial-Numerical Association of Response Codes (SNARC) effect refers to the fact that smaller numbers receive faster responses with the left hand, and larger numbers with the right hand (Dehaene et al., 1993). This robust finding implies that numbers are associated with space, being represented on a mental number line that progresses from left to right. The SNARC effect is held to depend on relative number magnitude, with the mental number line dynamically adjusting to the numerical range used in a given context. This characterisation is based on significant effects of relative number magnitude, with no significant influence of absolute number magnitude. However, a failure to reject the null hypothesis is not firm evidence for the absence of an effect. In this Registered Report, Roth and colleagues (2024) report two large-sample online experiments, with a Bayesian statistical approach to confirm—or refute—a role for absolute number magnitude in modulating the classic SNARC effect (smallest effect size of interest, d = 0.15).
Experiment 1 closely followed Dehaene’s (1993) original methods, and found strong evidence for an influence of relative magnitude, and moderate-to-strong evidence against an influence of absolute magnitude. Experiment 2 was designed to exclude some potential confounds in the original method, and this second experiment found strong evidence for both relative and absolute magnitude effects, of comparable effect sizes (in the range of d = .24 to .42). This registered study demonstrates that the SNARC effect is not ‘fully flexible’, in the sense of depending only on relative number magnitude; it is also shaped by absolute magnitude.
This Stage 2 manuscript was evaluated by the recommender and one external reviewer. Following appropriate minor revisions, the recommender judged that the manuscript met the Stage 2 criteria for recommendation.
URL to the preregistered Stage 1 protocol: https://osf.io/ae2c8
Level of bias control achieved: Level 6. No part of the data or evidence that was used to answer the research question was generated until after IPA. List of eligible PCI RR-friendly journals: References 1. Dehaene, S., Bossini, S., & Giraux, P. (1993). The mental representation of parity and number magnitude. Journal of Experimental Psychology: General, 122, 371–396. https://doi.org/10.1037/0096-3445.122.3.371
2. Roth, L., Caffier, J., Reips, U.-D., Nuerk, H.-C., Overlander, A. T. & Cipora, K. (2023). One and only SNARC? Spatial-Numerical Associations are not fully flexible and depend on both relative and absolute magnitude [Stage 2]. Acceptance of Version 3 by Peer Community in Registered Reports. https://osf.io/epnd4
| One and only SNARC? Spatial-Numerical Associations are not fully flexible and depend on both relative and absolute magnitude | Lilly Roth, John Caffier, Ulf-Dietrich Reips, Hans-Christoph Nuerk, Annika Tave Overlander, Krzysztof Cipora | <p>Numbers are associated with space, but it is unclear how flexible these associations are. We investigated whether the SNARC effect (Spatial-Numerical Association of Response Codes; Dehaene et al., 1993; i.e., faster responses to small/large num... | Life Sciences | Robert McIntosh | 2024-06-10 15:00:30 | View | ||
28 Nov 2023
STAGE 1
![]() One and only SNARC? A Registered Report on the SNARC Effect’s Range DependencyLilly Roth, John Caffier, Ulf-Dietrich Reips, Hans-Christoph Nuerk, Krzysztof Cipora https://osf.io/gr94fIs the SNARC effect modulated by absolute number magnitude?Recommended by Robert McIntoshThe Spatial-Numerical Association of Response Codes (SNARC) effect refers to the fact that smaller numbers receive faster responses with the left hand, and larger numbers with the right hand (Dehaene et al., 1993). This robust finding implies that numbers are associated with space, being represented on a mental number line that progresses from left to right. The SNARC effect is held to depend on relative number magnitude, with the mental number line dynamically adjusting to the numerical range used in a given context. This characterisation is based on significant effects of relative number magnitude, with no significant influence of absolute number magnitude. However, a failure to reject the null hypothesis, within the standard frequentist statistical framework, is not firm evidence for the absence of an effect. In this Stage 1 Registered Report, Roth and colleagues (2023) propose two experiments adapted from Dahaene’s (1993) original methods, with a Bayesian statistical approach to confirm—or rule out—a small effect (d = 0.15) of absolute number magnitude in modulating the classic SNARC effect.
The study plan was refined across two rounds of review, with input from two external reviewers and the recommender, after which it was judged to satisfy the Stage 1 criteria for in-principle acceptance (IPA).
URL to the preregistered Stage 1 protocol: https://osf.io/ae2c8
Level of bias control achieved: Level 6. No part of the data or evidence that will be used to answer the research question yet exists and no part will be generated until after IPA. List of eligible PCI RR-friendly journals:
References Dehaene, S., Bossini, S., & Giraux, P. (1993). The mental representation of parity and number magnitude. Journal of Experimental Psychology: General, 122(3), 371–396. https://doi.org/10.1037/0096-3445.122.3.371
Roth, L., Caffier, J., Reips, U.-D., Nuerk, H.-C., & Cipora, K. (2023). One and only SNARC? A Registered Report on the SNARC Effect’s Range Dependency. In principle acceptance of Version 3 by Peer Community in Registered Reports. https://osf.io/ae2c8
| One and only SNARC? A Registered Report on the SNARC Effect’s Range Dependency | Lilly Roth, John Caffier, Ulf-Dietrich Reips, Hans-Christoph Nuerk, Krzysztof Cipora | <p>Numbers are associated with space, but it is unclear how flexible these associations are. In this study, we will investigate whether the SNARC effect (Spatial-Numerical Association of Response Codes; Dehaene et al., 1993), which describes faste... | Social sciences | Robert McIntosh | 2022-11-30 12:36:08 | View | ||
03 Mar 2025
STAGE 1
![]() Shape of SNARC: How task-dependent are Spatial-Numerical Associations? A highly powered online experimentLilly Roth, Krzysztof Cipora, Annika T. Overlander, Hans-Christoph Nuerk, Ulf-Dietrich Reips https://osf.io/gsajbShedding light on task influence in the SNARC effectRecommended by Mario DalmasoThe Spatial-Numerical Association of Response Codes (SNARC) effect (Dehaene et al., 1993) is a key phenomenon in numerical cognition. It describes the tendency for individuals to respond faster to smaller numbers with a left-side key and to larger numbers with a right-side key, suggesting a mental mapping of numerical magnitudes onto space. While this effect has been widely replicated, its precise nature remains debated, particularly regarding its task dependency.
In this Stage 1 Registered Report, Roth et al. (2025) present a highly powered study that systematically investigates whether the SNARC effect differs in two widely used numerical cognition tasks: magnitude classification (MC) and parity judgment (PJ). In the MC task, participants determine whether a presented number is smaller or larger than a reference value (typically 5). This task explicitly requires magnitude processing, making numerical magnitude directly relevant to the response. In contrast, the PJ task requires participants to judge whether a number is odd or even, a decision that does not explicitly involve numerical magnitude.
The authors address a fundamental theoretical question in numerical cognition: while the SNARC effect in PJ is often considered continuous, does it follow a categorical pattern in MC? To investigate this, the study directly compares continuous and categorical representations of the SNARC effect across these two tasks, using Bayesian statistical approaches to determine the best-fitting model. By systematically analysing the SNARC effect in these widely used paradigms, this work aims to refine our understanding of how numerical magnitudes are mapped onto space and whether this mapping depends on task demands. The findings of this study will provide crucial insights into the cognitive mechanisms underlying numerical-spatial associations, highlighting the extent to which task structure shapes the emergence of the SNARC effect.
Three expert reviewers provided valuable feedback across multiple rounds of review. Based on detailed responses to the reviewers’ and recommender’s comments, the recommender judged that the manuscript met the Stage 1 criteria and therefore awarded in-principle acceptance.
Level of bias control achieved: Level 6. No part of the data or evidence that will be used to answer the research question yet exists and no part will be generated until after IPA.
List of eligible PCI RR-friendly journals:
References
1. Dehaene, S., Bossini, S., & Giraux, P. (1993). The mental representation of parity and number magnitude. Journal of Experimental Psychology: General, 122(3), 371–396. https://doi.org/10.1037/0096-3445.122.3.371
2. Roth, L., Cipora, K., Overlander, A. T., Nuerk, H.-C., & Reips, U.-D. (2025). Shape of SNARC: How task-dependent are Spatial-Numerical Associations? A highly powered online experiment. In principle acceptance of Version 3 by Peer Community in Registered Reports. https://osf.io/968ad
| Shape of SNARC: How task-dependent are Spatial-Numerical Associations? A highly powered online experiment | Lilly Roth, Krzysztof Cipora, Annika T. Overlander, Hans-Christoph Nuerk, Ulf-Dietrich Reips | <p>Spatial-Numerical Associations (SNAs) are fundamental to numerical cognition. They are essential for number representation and mathematics learning. However, SNAs are highly dependent on the experimental situation and task. Understanding this d... | Life Sciences | Mario Dalmaso | Peter Wühr | 2024-05-27 13:14:25 | View | |
21 Mar 2023
STAGE 1
![]() Convenience Samples and Measurement Equivalence in Replication ResearchLindsay J. Alley, Jordan Axt, Jessica Kay Flake https://osf.io/32unbDoes data from students and crowdsourced online platforms measure the same thing? Determining the external validity of combining data from these two types of subjectsRecommended by Corina LoganComparative research is how evidence is generated to support or refute broad hypotheses (e.g., Pagel 1999). However, the foundations of such research must be solid if one is to arrive at the correct conclusions. Determining the external validity (the generalizability across situations/individuals/populations) of the building blocks of comparative data sets allows one to place appropriate caveats around the robustness of their conclusions (Steckler & McLeroy 2008).
In this registered report, Alley and colleagues plan to tackle the external validity of comparative research that relies on subjects who are either university students or participating in experiments via an online platform (Alley et al. 2023). They will determine whether data from these two types of subjects have measurement equivalence - whether the same trait is measured in the same way across groups. Although they use data from studies involved in the Many Labs replication project to evaluate this question, their results will be of crucial importance to other comparative researchers whose data are generated from these two sources (students and online crowdsourcing). If Alley and colleagues show that these two types of subjects have measurement equivalence, then this indicates that it is more likely that equivalence could hold for other studies relying on these type of subjects as well. If measurement equivalence is not found, then it is a warning to others to evaluate their experimental design to improve validity. In either case, it gives researchers a way to test measurement equivalence for themselves because the code is well annotated and openly available for others to use.
The Stage 1 manuscript was evaluated over two rounds of in-depth review. Based on detailed responses to the reviewers' comments, the recommender judged that the manuscript met the Stage 1 criteria and therefore awarded in-principle acceptance (IPA). Level of bias control achieved: Level 2. At least some data/evidence that will be used to answer the research question has been accessed and partially observed by the authors, but the authors certify that they have not yet observed the key variables within the data that will be used to answer the research question AND they have taken additional steps to maximise bias control and rigour (e.g. conservative statistical threshold; recruitment of a blinded analyst; robustness testing, multiverse/specification analysis, or other approach)
List of eligible PCI RR-friendly journals:
References
Alley L. J., Axt, J., & Flake J. K. (2023). Convenience Samples and Measurement Equivalence in Replication Research, in principle acceptance of Version 4 by Peer Community in Registered Reports. https://osf.io/7gtvf
Steckler, A. & McLeroy, K. R. (2008). The importance of external validity. American Journal of Public Health 98, 9-10. https://doi.org/10.2105/AJPH.2007.126847
Pagel, M. (1999). Inferring the historical patterns of biological evolution. Nature, 401, 877-884. https://doi.org/10.1038/44766
| Convenience Samples and Measurement Equivalence in Replication Research | Lindsay J. Alley, Jordan Axt, Jessica Kay Flake | <p>A great deal of research in psychology employs either university student or online crowdsourced convenience samples (Chandler & Shapiro, 2016; Strickland & Stoops, 2019) and there is evidence that these groups differ in meaningful ways ... | Social sciences | Corina Logan | 2022-11-29 18:37:54 | View |
FOLLOW US
MANAGING BOARD
Chris Chambers
Zoltan Dienes
Corina Logan
Benoit Pujol
Maanasa Raghavan
Emily S Sena
Yuki Yamada