Does incorporating open research practices into the undergraduate curriculum decrease questionable research practices?

This preregistration aims to test the effect on statistics confidence and awareness of questionable research practices when undergraduate dissertations are preregistered.  I think this is a necessary empirical investigation that will facilitate understanding and methods of the open science movement.  The introduction section is thorough, with sufficient background to support the proposed study.  I suggest adding an additional paragraph commenting on the impact to the open science movement of the findings of the proposed study.  The hypotheses, predictions, and interpretation of alternative results could be more detailed and I make specific comments on these in the attached track-changes pdf document.  The planned analyses seem sufficient to address the 3 research questions, however I do not have experience in the best analysis practices for survey-based studies.

“Evaluating the pedagogical effectiveness of study preregistration in the undergraduate dissertation: A Registered Report” is an interesting manuscript that proposes an innovative design to study the effects of implementing preregistration into research methods pedagogy on student learning and attitudes. Overall, I think the manuscript is well-written and informative, asks an important question for both research and practice, and has a sound research design for testing the proposed hypotheses. The authors addressed most of the concerns that came to mind as I read the manuscript, so I only have a few minor questions and suggestions for the authors to consider prior to carrying out their study.

My first question is about why there is such a heavy focus on statistics as an outcome rather than other elements of research design. One of the things that I appreciate about preregistration is that it forces researchers (and students) to think about the relationship between many elements in the research process (strength of manipulation, effect sizes, sample characteristics that might moderate processes, etc.). The statistical elements are of course important, but the preregistration process reveals more than that—it forces us to wrestle with how all elements of research design are related. I was surprised to not see other research design related competencies (other than statistics and QRPs) being measured.

My second question is about the “forced entry” strategy for minimizing missing data…is that allowed? This could be a country-level difference so feel free to ignore this question if it is, but in the US we are not allowed to use forced responses—it is considered an ethical violation; we can use “request response” to nudge them toward answering, but participants have to have the option to not answer questions that they do not want to.

Overall, this is a great manuscript, and I commend the authors for conducting this important research.

I want to gratulate the authors to this interesting and thoroughly planned Registered Report, which I enjoyed reading and reviewing very much. 

Summary: The authors want to conduct a study among undergraduate psychology students in the UK, to assess if preregistration of the final-year dissertation influences attitudes towards statistics, QRPs, and open science. For this, a targeted sample size of 200 students will be recruited, that plan or plan not to preregister their dissertation. The design follows a 2 (pregistration: yes vs. no) between x 2 (timepoint: before and after dissertation) within subjects design.

I have added comments to the PDF of the Registered Report (see attached). Most of these concern minor points, such as:
1) Spelling.
2) Structure of the text.
3) Consistency of notation and formatting.
4) Depth and clarity of descriptions. Most often I added some questions I had during reading, for which I would find it beneficial if the authors would answer them in the text to enhance clarity and reproducibility of the study.
5) Enumeration of hypotheses. I would find it beneficial if the authors would differentiate between different sub-hypotheses and enumerate them, to reference them later during the analysis section.
6) Bot protection. Since the authors plan to advertise their study via social media, I recommend to implement some kind of bot protection.
7) Definition of preregistration in the study. I recommend that the authors update their definition of preregistration, since the current definition describes that preregistration is possible "before you collected your data". However, to address secondary data analysis preregistration, I would add that preregistration is also possible before data is analyzed.
8) Attention check. I recommend to implement a different attention check, since participants that simply agree to anything in the study, will pass the current attention check. 
9) Addition of some questions in the study. This is only based on personal interest.
10) Combination of frequentist and Bayes statistics. Unfortunately, I am no expert to Bayes statistics, but I understand that one method (frequentist vs. Bayes) would be sufficient, however, here both are combined. Maybe the other reviewers can contribute additional input to this question.
11) One hypothesis concerns the "understanding/awareness" of open science. Since understanding is not directly tested, but rather perceived understanding is assessed, I would recommend to use the term "perceived understanding".
12) Sometimes, an ANOVA is implemented for a comparison of two groups (preregistration: yes vs. no). From my perspective, a t-test would be the more appropriate analysis.
(more detailed descriptions of these comments are provided in the attached PDF)

Additionally, I would like to discuss some major points which I think are important to consider before conducting the study:
1) I would like to discuss more the data collection at Time 1. First, I suggest that quota sampling could be used to control for different group sizes (preregistration: yes vs. no). Alternatively, if no quota sampling is used, I think the authors should discuss more what would be the consequences of the case that there may be very unequal group sizes.
2) Also, in general, I am unsure if enough participants can be collected at Time 1, since only a short time period is indicated for this data collection (September - October 2021). I would appreciate it if the authors could elaborate on how they will proceed if less than the targeted 240 students can be collected by the end of October, or if only a small proportion of participants indicate planning to preregister.
(more detailed descriptions of these comments are provided in the attached PDF)​

Overall, I think that the study addresses a valid research question with coherent and plausible hypotheses. I perceived the described methods as sound, feasible, and for the most parts, clear. Whenever open questions remain, I have marked them in the attached PDF. Additionally, I thought it was excellent that the authors had already addressed possible limitations themselves.  I feel like the study will be an important contribution and I hope that my comments will help the authors improve their study and manuscript.

All the best,
Lisa Spitzer

This proposed study aims to test the influence of study pre-registration on undergraduate attitudes towards statistics and questionable research practices. Given the enthusiasm for open science practices among early career researchers, and the certainty with which many of those practices are endorsed by some members of the field, it is important to empirically test the influence of those practices on meaningful research outcomes. Open science practices themselves should indeed be subject to the scrutiny of the Registered Report format, so I commend the team for taking on this sort of research. In order to meet the demands of this format, however, I believe the current report could use more detail in several key areas related to establishing the scientific premise, justifying the choice of measures and associated hypotheses, and ensuring that any results can be interpreted with confidence.

Overall, I think the study motivation and scientific premise should be strengthened. While undergraduate teaching and learning are inherently important, it’s a bit unclear exactly what issue this RR is tackling, or what guides the choice of measures and hypotheses. For instance, the intro lays out a number of dissertation struggles that undergraduates may face related to anxiety, disengagement, writing ability, and supervisory relationships (p. 6). Is pre-registration meant to mitigate these concerns? If so, how? It’s unclear to me whether the focus is on student well-being, research quality, or pedagogical value. If it is all of these things, I think the intro should more clearly establish why each is important to study in this context, and make a more explicit case for why/how pre-registration should be expected to improve these problem areas. Is the thinking that the mere act of preregistration will impart these benefits? Or will these students undergo some additional learning, which might have the same benefits without the actual preregistration? The intro also implies that open science practices are universally endorsed, but there is in fact a vocal pushback against many of these practices (e.g., Szollosi et al., 2020, TICS). I mention that, not to undermine the current study, but because I believe this perspective should be recognized and may offer further fuel to justify the proposed work.

After establishing the problem, I also think the report could clarify why these particular proposed measures were chosen, and also offer stronger justification that the measures will test what we think they’ll test. It’s not totally clear to me that these self-report questionnaires actually tap into the pedagogical effectiveness of pre-registration. For instance, is there any evidence that student attitudes toward statistics relate to their knowledge or competence with statistics? Instead of asking for confidence in open science terms, why not actually measure ability to define the terms accurately? I think the report could be strengthened overall if it included more objective measures of knowledge (stats and open science), and engagement in QRPs (i.e., did you do any of these things?), rather than just attitudes.

I would also like to see more details about recruitment and study criteria. What do we know right now about the uptake of pre-registration in this cohort? As in, how feasible will it be to recruit equal numbers in both groups, and for those groups to be matched? The report acknowledges this might be an issue, but I think it would be a rather serious one if all the pre-registration participants end up coming from a certain set of targeted schools, and all the non-pre-registerers are from different schools.

Is the *only* inclusion criteria that participants be a final-year undergraduate student, studying Psychology at a U.K institution. For instance, what happens at Time 1 if they respond that they’ve already pre-registered their study? Is the only possible reason that data would be excluded if a participant fails the attention check? I’m just asking to ensure that all the methods are specified in enough detail to be truly reproducible and transparent. 

Regarding the sample size, I understand that the proposal is based on time and resource considerations, but we would still need to be reasonably assured that the outcomes would be valuable. What was the input to the power analyses, and do we have reason to believe that the cited effect size (80% statistical power to detect an effect size of np2 = .04) would constitute a meaningful effect in this context?

I very much appreciate that hypotheses and tests are specified in a table, but I think the interpretation, alternative outcomes, and theory portions of the table should be more explicit.  For instance, the final column applies broadly to all hypotheses and tests, stating “This may call into question various situational, contextual, and personal factors that impact how preregistration may be useful to students in this context “ Ideally, this section would include straightforward, concrete predictions that are guided by theory, for each hypothesis, and specific implications if findings do or don’t match expectations.

The “Limitations” section acknowledges that pre-registrations may differ in their rigor, but states that this issue is beyond the scope of the current study. This strikes me as an incredibly consequential issue, if we truly want to understand the impact of pre-registration on student outcomes. I understand why a common, plain-language definition of preregistration is provided, but I think there needs to be some further confirmation that the participants actually conducted a valid preregistration. For instance, is it acceptable if students just saved a personal pdf with their “plan,” or do they need to have submitted it to an accepted repository? There is then WIDE variability in the detail with which a pre-registration is specified. OSF alone provides 8 possible options for preregistration. There is then also WIDE variability in adherence to the registered methods/analyses. What if a large number of students submit a bare minimum, superficial pre-registration, just because it’s required of their program? Many scientists have speculated that this may become the norm, and I worry that it could set up this study up for failure to find any differences between groups. Why not ask whether the participants actually followed through with the registration, or to what extent they deviated?

In the end, if you find no differences between groups with these methods, would you confidently declare that preregistration has no benefits? If not, why not? If there is a difference, would we know it is due to the act of pre-registering, or could it be explained by some other factor? As in, do students in a pre-registration context just get more exposure to open science issues as part of their process, and could the same benefits therefore be achieved by briefly teaching students about QRPs and Open Science? The above concerns all boil down to ensuring that any result is meaningful (null or otherwise), which should be a primary goal of the design and RR proposal. I know some of these concerns may seem persnickety, but they are aimed at the overarching goals of ensuring that any results can be confidently interpreted, and that the methods are precisely reproducible.