Understanding how applied researchers address open scholarship, feedback and climate change in their work

Thank you for your submission to PCI RR. I am reviewing "Understanding the Role of Climate Change in AppliedResearch: A Qualitative Registered Report​," which as I understand it is the green section of this double manuscript. I hope my comments are helpful as the authors hone their registered report.

INTRODUCTION

The introduction is generally well-written; if there's one improvement I would suggest, it would be to elaborate specifically on how applied researchers have actually influenced policy or practice. One or two examples would probably suffice, especially if there are any relevant to climate change. There's some general discussion of this early on, but I struggled to make the connection between what applied researchers can do and what the authors are suggesting, that there is evidence that researchers have actually impacted climate change practice. I would argue that increased attention is not actually change. This may be a nitpicky point, but I think it would help people have a clearer picture of what kinds of changes are being discussed. If there aren't specific examples, it may be beneficial to walk back the sentences that suggest applied researchers have influenced practice and stick to the responsibility of applied researchers moving forward.

METHOD

I just wanted to note I appreciate the international data collection efforts! Given the burden of climate change and relative power different countries have to impact pollution levels of their own and other countries, I think it would be good to pay attention to this aspect when analyzing the results. That is to say, I wonder if geographic region will affect people's reporting of their experiences, actions, and impacts. I don't think it's necessary to explicitly compare, but would be helpful to consider how this could be attended to.

I also appreciate the reflexivity that is being built into the process, and the balancing of privacy needs and reflection.

PARTICIPANTS

Point of clarification: If the PhD work is applied in nature, is it still considered? In other words, are the authors excluding people who are just doing research because their degree requires it, without consideration of applied benefits, or are they excluding anyone doing PhD-level research, even if it's applied in nature? I think the authors mean the former, but want to be clear.

ANALYSIS

Are there any reflexive practices that are built in for the analysts?

I have no other comments. I am happy to look at the manuscript again if needed, but am also fine if the recommender feels the authors have addressed my comments fully (or that they can ignore certain suggestions). I look forward to seeing how the registered report turns out!

This is my first PCI review and first time reviewing programmatic Registered Reports, though I've reviewed many other documents over the past decade across disciplines.

I found the study ideas intriguing and am most curious about the Open Scholarship Registered Reports given my interest in applied research. I'm looking forward to reading the eventual paper! My critical comments are below.

---------------------------------------------------------------------------------

Programmatic Registered Report 1
Open Scholarship

Introduction

Citation/explication needed for "However, such open scholarship practices have been nearly exclusively applied to basic and quantitative academic research, and there are many reasons why these practices may not be common or considered so favorably in applied settings..." You should find several specific citations, but here is one to get started with citation tracing:

Survey of open science practices and attitudes in the social sciences
By Joel Ferguson, Rebecca Littman, Garret Christensen, Elizabeth Levy Paluck, Nicholas Swanson, Zenan Wang, Edward Miguel, David Birke, John-Henry PezzutoContainer: Nature CommunicationsYear: 2023Volume: 14Issue: 1DOI: 10.1038/s41467-023-41111-1URL: https://www.nature.com/articles/s41467-023-41111-1
 

Citation needed for each practice mentioned in "wide range of practice." Great list, by the way! This is my first encounter with pre-mortems, and they remind me of methodological review panels (Lakens: https://pubmed.ncbi.nlm.nih.gov/36596953/). You can also remove the parentheses in that paragraph mentioning the "wide range of practices" and simply italicize each practice like "premortems to try and anticipate...." This is for ease of reading.

RQ1: Remove the dual commas: ", and using," because they are not needed.

I would revise: "contemporary practices and feedback mechanisms" to read "contemporary research reform practices and study feedback mechanisms for authors..." as that is more specific and clear if readers are skimming. In general, your use of "feedback mechanisms" seems broad throughout the paper and a more specific term would help. I like "research plan feedback mechanisms", "study feedback strategies", or something similar.

pg 4, paragraph 3

"Applied research" here needs to be grounded in concrete examples across 3+ domains with vignettes. Although you defined it earlier, we need to know the details of the term to make sense of this. I interpret this to mean action research (which changes as it is being done). Please clarify this part with specific methods and examples.

RQ2

This is a question that I am mulling over myself in my institution (informally). I am curious to read your findings later. I expect that these feedback processes will be informal and operate for lab meetings, colloquia, conferences, and listservs. I see great potential for systematically investigating this topic and using the results to support the diffusion of author feedback mechanisms. I hope that you make a distinction between formal and informal feedback mechanisms in your papers.

Can you write a brief section on why RR1 is so much shorter than RR2? According to the PCI website, additional information is recommended that you included in RR2: "Authors of a programmatic Stage 1 RR should ensure that all the usual criteria for a RR are met, including detailed specification (where applicable) of theory, hypotheses, procedures, and analysis plans (see review criteria in Section 2.1)." The Method and analysis plan is most critical for RR1.

-----------------------------------------------

Programmatic Registered Report 2

Understanding the Role of Climate Change in Applied Research: A Qualitative Registered Report

Are "Occupational Psychologists" synonymous with "Industrial and Organizational Psychologists?" The latter term is more familiar to me (US resident). If so, it might be good to mention that. If not, please define the term.

RQs 1 and 2 are interesting, though should be introduced a bit earlier if possible.

Good justification for analysis plan: content analysis

pg. 15: I recommend avoiding violin plots for readability and using density plots instead.

Regarding sampling, the variety of settings (virtual) seems sensible. You could also supplement with visits to conferences and making special requests if needed. I don't have a sense of the difficulty of recruiting applied researchers, but the venues you listed seem very open, and I foresee enthusiasm to participate. Sampling success could be a nice topic to include in a Registered Report by brainstorming reasons for success and failure. What do studies with similar samples struggle with? Financial incentives?

Appendix A:

Looks good, though at some point a distinction can be made between formal (pre-registrations and Registered Reports) vs. informal feedback mechanisms (social media, colleagues nearby).

Appendix B:

Looks good and seems helpful to include.

Appendix C:

1 to 10 pt rating scales are too granular according to modern psychometrics research. 1 to 5 would be sufficient, and participants would find a simplified scale easier to process.

You may also want to visualize that scale here and add labels for each value in the scale for clarity.

All nine prompts in Appendix C are positively framed and may lead to biased results, so you should find a way to word them neutrally or negatively. This is often advised by survey methodologists. Please note for readers whether this will be administered verbally or via computer. Given the "Open Text Box" line at the end, I assume this will be on a computer. So then you should just create nine additional negatively worded prompts like "I did not enjoy contributing to this project" to ensure that you don't skew responses to the positive end of the rating scale. Then you can perform some simple arithmetic to create sum scores per theme (e.g., enjoyed contributing, learned a lot, developed my research skills, etc.). Additional information can be found in textbooks like Survey Methodology 2nd Edition by Robert M. Groves, Floyd J. Fowler Jr., Mick P. Couper, etc.