Ensuring universal access to sexual and reproductive health and reproductive rights is a global concern, exemplified by goal 5.6 of the Sustainable Development Goals (UN General Assembly, 2015). Whilst the range of contraceptive options have increased, our understanding of the impacts of use for women are inadequate and represent a key barrier to positive change in policies and practices. In particular, we have few consensuses on the expected impacts of hormonal contraceptive use on women's sexuality and wellbeing.

 

In the current programmatic submission, Botzet et al. (2023) argue that this inconclusive evidence base could be due to the wide heterogeneity in responses, the impacts of this heterogeneity upon attrition, differences in contraceptive methods and dosage effects, confounders, and the potential for reverse causality. Tackling some of these potential factors, Botzet (2023) explore whether hormonal contraceptive use influences sexuality and well-being outcomes, and whether (and to what extent) the effects vary between women. To achieve this they have proposed analysis of longitudinal data from the German Family Panel (PAIRFAM) which includes annual waves of data collection from >6500 women, with separate Stage 2 submissions planned to report findings based on sexuality and well-being. The proposed work will progress our understanding of the impact of hormonal contraceptives by overcoming limitations of more common research approaches in this field, and has the potential to contribute to a more contextualised view of the impact of their impacts in real-world practice.

 

The Stage 1 manuscript was evaluated over three rounds of in-depth review. Based on detailed responses to the reviewers' comments, the recommender judged that the manuscript met the Stage 1 criteria and therefore awarded in-principle acceptance (IPA).

 

 

Level of bias control achieved: Level 3: At least some of the data/evidence that will be used to answer the research question already exists AND is accessible in principle to the authors BUT the authors certify that they have not yet accessed any part of that data/evidence.

 

 

• Collabra: Psychology
• Peer Community Journal
• PeerJ
• Royal Society Open Science
• Studia Psychologica
• Swiss Psychology Open
  
  

 

Botzet, L. J., Rohrer, J. M., Penke, L. & Arslan, R. C. (2023). Hormonal Contraceptive Use and Women's Sexuality and Well-Being: Estimating Treatment Effects and Their Heterogeneity Based on Longitudinal Data. In principle acceptance of Version 3 by Peer Community in Registered Reports. https://osf.io/kj3h2

 

UN General Assembly (2015). Transforming our world : the 2030 Agenda for Sustainable Development, 21 October 2015, A/RES/70/1. Available at: https://www.refworld.org/docid/57b6e3e44.html [accessed 27 September 2023]

Measurement is a vital activity for all research areas, but we so often fail to provide sufficient clarity, rigor and transparency about it, undermining the validity of our studies' conclusions (Flake & Fried, 2020). This concern is of wide societal interest when applied to the domains of ideology and attitudes where measurements of both implicit and explicit attitudes are assumed to reflect the same underlying concept. The extent to which this can be accepted is undermined by mixed evidence demonstrating a lack of consensus on the extent to which relevant psychological factors similarly predict both implicit and explicit attitudes.

 

In the proposed study, Reid & Inbar (2023) question these assumptions through use of the Project Implicit dataset, exploring the extent to which social dominance orientation (SDO) and right wing authoritarianism (RWA) similarly predict implicit and explicit attitudes. This work is ideally suited for publication through the Registered Reports format because whilst we may expect that relationships between SDO/RWA are similar in effect size across measures of both implicit and explicit attitude (because they tap into the same underlying attitude), there is great scope to acknowledge a more complex set of findings which may not be immediately interpretable or coherent. The proposed work will help us unpack further the assumptions surrounding measurement of attitudes and can help us better understand the extent to which SDO and RWA predict atittudes. 

 

The Stage 1 manuscript was evaluated over three rounds of in-depth review. Based on detailed responses to the reviewers' comments, the recommender judged that the manuscript met the Stage 1 criteria and therefore awarded in-principle acceptance (IPA).

 

URL to the preregistered Stage 1 protocol: https://osf.io/zv4jw
 
Level of bias control achieved: Level 5. All of the data or evidence that will be used to answer the research question already exist, but are currently inaccessible to the authors and thus unobservable prior to IPA (e.g. held by a gatekeeper) 
 
List of eligible PCI-RR-friendly journals:

 

 

 

1. Flake, J. K. & Fried, E. I. (2020). Measurement schmeasurement: Questionable measurement practices and how to avoid them. Advances in Methods and Practices in Psychological Science, 3, 456-465. https://doi.org/10.1177/2515245920952393

 

2. Reid, J. & Inbar, Y. (2023). Implicit Ideologies: Do Right-Wing Authoritarianism and Social Dominance Orientation Predict Implicit Attitudes? In principle acceptance of Version 3 by Peer Community in Registered Reports. https://osf.io/zv4jw

Measurement is a vital activity for all research areas, but we so often fail to provide sufficient clarity, rigor and transparency about it, undermining the validity of our studies' conclusions (Flake & Fried, 2020). This concern is of wide societal interest when applied to the domains of ideology and attitudes where measurements of both implicit and explicit attitudes are assumed to reflect the same underlying concept. The extent to which this can be accepted is undermined by mixed evidence demonstrating a lack of consensus on the extent to which relevant psychological factors similarly predict both implicit and explicit attitudes.

 

In the current study, Reid & Inbar (2023) question these assumptions through use of the Project Implicit dataset, exploring the extent to which social dominance orientation (SDO) and right wing authoritarianism (RWA) similarly predict implicit and explicit attitudes. This work was ideally suited for publication through the Registered Reports format because whilst it may have been expected that relationships between SDO/RWA are similar in effect size across measures of both implicit and explicit attitude (because they tap into the same underlying attitude), there was great scope to acknowledge a more complex set of findings which may not be immediately interpretable or coherent.

 

As expected, the results were not completely unambiguous, but the mostly consistent relationships between implicit attitudes and RWA/SDO provided evidence towards both implicit and explict measures capturing the same underlying construct. These results also provide a useful step forward in our discussion of measurement in this domain, acknowledging the complexity of the tradeoff between reliability and specificity.

 

The Stage 2 manuscript was evaluated over two rounds of in-depth review. Based on detailed responses to the reviewers' comments, the recommender judged that the manuscript met the Stage 2 criteria and awarded a positive recommendation. 

 

URL to the preregistered Stage 1 protocol: https://osf.io/zv4jw
 
Level of bias control achieved: Level 5. All of the data or evidence that was used to answer the research question existed before the research commenced but was inaccessible to the authors and thus unobservable prior to IPA.
 
List of eligible PCI-RR-friendly journals:
 

References
 
1. Flake, J. K. & Fried, E. I. (2020). Measurement schmeasurement: Questionable measurement practices and how to avoid them. Advances in Methods and Practices in Psychological Science, 3, 456-465. https://doi.org/10.1177/2515245920952393
 
2. Reid, J. & Inbar, Y. (2024). Implicit Ideologies: Do Right-Wing Authoritarianism and Social Dominance Orientation Predict Implicit Attitudes? [Stage 2]. Acceptance of Version 6 by Peer Community in Registered Reports. https://doi.org/10.31234/osf.io/8udps

The replication crisis/credibility revolution has driven a vast number of changes to our research environment (Korbmacher et al., 2023) including a much needed spotlight on issues surrounding measurement (Flake & Fried, 2020). As general understanding and awareness has increased surrounding the 'garden of forking paths' or 'researcher degrees of freedom' (Simmons et al., 2011), and the various decisions made during the scientific process that could impact the conclusions drawn by the process, so too should our interest in meta-research that tells us more about the methodological processes we follow, and how discretionary decisions may influence the design, analysis and reporting of a project.

 

Peeters et al. (2025) have proposed a systematic literature review of this nature, mapping the methodological variation in experience sampling methods (ESM) from the design stage all the way to dissemination. It starts this journey by mapping how ESM studies vary e.g., in design, considering a variety of factors like sample size, number of measurements, and sampling scheme. It also evaluates reporting quality, rationales provided, and captures the extent of open science practices adopted. Covering many parts of the research process that get assumed, unreported or otherwise unjustified, the proposed work looks set to springboard an important body of work that can tell us more effectively how to design, implement and report ESM studies.

 

The Stage 1 submission was reviewed over one round of in-depth review with two reviewers. Based on detailed responses to reviewers’ feedback, the recommender judged that the manuscript met the Stage 1 criteria and therefore awarded in-principle acceptance.

 

 
Level of bias control achieved: Level 1. At least some of the data/evidence that will be used to the answer the research question has been accessed and observed by the authors, including key variables, but the authors certify that they have not yet performed any of their preregistered analyses, and in addition they have taken stringent steps to reduce the risk of bias.
 
List of eligible PCI-RR-friendly journals:

 

 

1. Flake, J. K., & Fried, E. I. (2020). Measurement schmeasurement: Questionable measurement practices and how to avoid them. Advances in Methods and Practices in Psychological Science, 3, 456-465. https://doi.org/10.1177/2515245920952393

 

2. Korbmacher, M., Azevedo, F., Pennington, C. R., Hartmann, H., Pownall, M., Schmidt, K., ... & Evans, T. (2023). The replication crisis has led to positive structural, procedural, and community changes. Communications Psychology, 1, 3. https://doi.org/10.1038/s44271-023-00003-2

 

3. Peeters, L., Van Den Noortgate, W., Blanchard, M. A., Eisele, G., Kirtley, O., Artner, R., & Lafit, G. (2025). Mapping Methodological Variation in ESM Research from Design to Data Analysis: A Systematic Review. In principle acceptance of Version 3 by Peer Community in Registered Reports. https://osf.io/ztvn3

 

4. Simmons, J. P., Nelson, L. D., & Simonsohn, U. (2011). False-positive psychology: Undisclosed flexibility in data collection and analysis allows presenting anything as significant. Psychological Science, 22, 1359-1366. https://doi.org/10.1177/0956797611417632

This recommendation concerns the plan of two studies that are intended to be conducted simultaneously, using the same data collection approach, and to result in two manuscripts that will be submitted for assessment at Stage 2. The Stage 1 manuscript containing these protocols was submitted via the programmatic track.

 

Protocol 1 concerns “Open Scholarship and Feedback in Applied Research: A Qualitative Registered Report”. With this study, the authors aim to explore how applied researchers integrate feedback processes into their work, in relation to transparency and rigor in particular. They will investigate whether their sample are aware of and use feedback mechanisms from the open science movement, such as registered reports, which makes this study nicely metascientific. Through interviews with 50 applied researchers from various fields, the study will examine current feedback practices. The authors intend to use the findings of this first study to inform recommendations on how open science practices can be incorporated into research workflows.

 

Protocol 2 concerns “Understanding the Role of Climate Change in Applied Research: A Qualitative Registered Report”. This study aims to explore how applied researchers address climate change in their work, including the ways their practices are influenced by and respond to climate challenges. It addresses how their approaches may evolve, and they plan to look into the barriers and opportunities climate change presents in practice. Interviews with 50 applied researchers will be analysed to help understand these dynamics. The authors aim to provide recommendations to help applied researchers and their employers adjust their priorities to align with the urgency of climate action. One reviewer did not comment on this second protocol, as the content was outside of their own research area. Although I would have found a reviewer who specializes in this area directly ideally, I found I could still rely on the other two reviewers and my own knowledge to assess this protocol.
 
General comments: As I mentioned in my initial assessment text, these studies were well planned from the get-go and the protocol nicely articulated those plans. The use of different colour highlighting clearly helped the reviewers target different elements of the protocol and give direct feedback on specific parts. It also helped prevent me from getting lost in all the details! Well done, once again, to the authors for making the distinction between the two studies so clear. I was also pleased at how well they balanced the information between the two protocols – this made it easier to see if there were deficiencies in either one somehow. Finally, I loved that reflexivity was considered by the authors. One suggestion by me is that the authors might consider providing a collective positionality statement to go with the trainees’ reflexivity statements (if this is already in the plan and I missed it, please forgive the oversight!) in the final studies, even if as part of an appendix. This is because the open science movement and climate change can both be controversial, and because of the nature of the qualitative approach I would like to understand a little of the stance the group takes towards these issues collectively if the authors think it’s appropriate. I understand that with a big group that might be difficult or impossible, but if it is possible I would like to see it. I would also like to see initials used in the manuscripts to indicate who was responsible for what analysis elements where possible. This allows for accountability and to attribute interpretation to specific individuals involved in the data analysis. Alternatively, individuals can be attributed in a statement at the end of each manuscript to serve the same purpose and be less awkward in the text. If this won't work for some reason, please motivate this decision. 

 

The three reviewers that took the time to go through the reports nevertheless had useful comments, most of which would have contributed to strengthening the plan and minimizing problematic bias later on. The authors took these comments seriously, and thoughtfully (cheerfully even) responded to each. In my estimation, each of the suggestions of the reviewers were satisfied by the authors’ response to reviews letter. Other than my earlier comment about the positionality statement, I have no further comments for the Stage 1 protocol, and I wish the authors all the best with running the studies and writing up Stage 2 for each.

 

URL to the preregistered Stage 1 protocol: https://osf.io/jdh32
 
Level of bias control achieved: Level 6. No part of the data or evidence that will be used to answer the research question yet exists and no part will be generated until after IPA.
 
List of eligible PCI RR-friendly journals:
 

 

 

Evans, T. R. et al. (2024). Open Scholarship and Feedback in Applied Research/Understanding the Role of Climate Change in Applied Research: A Qualitative Registered Report. In principle acceptance of Version 2 by Peer Community in Registered Reports. https://osf.io/jdh32