Recommendation
DOI or URL of the report: https://osf.io/3rxt5
Version of the report: https://osf.io/3rxt5
, posted 12 Mar 2024, validated 12 Mar 2024Dear Maciej Behnke and co-authors,
We were lucky to have 3 out of 4 reviewers from Stage 1 to return at Stage 2. All reviewers are widely satisfied and appreciate your diligent reporting. They have very minor comments, which I let you address respectively in the revision.
As noted by LM, the results section could use some clarifications especially in reporting the hypothesis tests. I personally prefer a structure that clearly reports “H1 = outcome, H2 = outcome etc”, as it helps readers to easily locate and assess key findings. That's an option to consider. I also highly support IR’s request to add descriptives.
Regarding the optimality of the created conditions and the limits they set to the intervention, I agree with LM and IR that some further engagement with previous related knowledge would make the discussion even more informative. E.g., to my memory, already the first-ever doctoral dissertation on videogames (Elizabeth Ellen Moulds, 1978) investigated arousal in the competitive gaming situation. It could be useful to review the experimental literature of the past decades in detail to further assess whether the stress response in the present study is unique or echo those in previous studies involving similar conditions.
JK and IR point out some wordings in the discussion that should be rephrased. I made similar notes and would also suggest removing the first three words of the phrase “For this reason, we observed no effects..” (p. 47). Many other reasons could be too. I would also rethink some wordings in the last sentences: "However, we found the limited impact of SMI on the performance outcomes and challenge/threat affective responses... In sum, we recommend the SMI as a prevention tool." Please reconsider the conclusion of ‘limited impact’ for inferred no effects. As to the framing “we recommend the SMI”, would it carry similar meaning if phrased “the findings lead us to recommend the SMI”? I think it's a small but important difference.
Please add the date July 5 2023 as the time of updating the randomisation plan.
Last, I recommend stating all theoretical implications in the discussion, even if briefly, as per the last column of the design table (Theory that could be shown wrong). They're already implicitly in the text, but being explicit would be preferred. In my experience, readers tend to draw their own theoretical inferences if they’re not spelled out. You may also add a new results column to the design table; some Stage 2 RRs have it and it's often informative. If you wish to discuss any of the revisions before the next (perhaps final) version, I can be contacted as usual.
Veli-Matti Karhulahti
I enjoyed reading this Stage 2 PCI registered report entitled 'Applying a synergistic mindsets intervention to an esports context'. When reflecting on the criteria and key issues to consider at Stage 2, I felt the authors did a great job with the manuscript. Indeed:
> A direct url to the approved protocol was included.
> The introduction was largely the same as the stage 1 manuscript with only minor changes made and these were appropriate and transparently flagged.
> The authors stayed true to their approved Stage 1 protocol and where they deviated this was appropriate, clearly articulated, and well-justified (e.g., additional exploratory analyses).
> Additional exploratory analyses were informative, justified, performed appropriately, methodological sound, and clearly distinguished from the pre-registered analyses.
> The conclusions aligned to the preregistered aims/hypotheses and were appropriate, evidence-based, and well-communicated.
Beyond these points, I just have a series of relatively minor suggestions that I believe could help the authors improve the manuscript further:
1) Remove (including stress) from the abstract, stress is a process, not an emotion linked to negative affect (note: if the authors are referring to feeling stressed, the wording should better reflect this).
2) In the methods, it would be good to see a breakdown of gender/sex (i.e., proportion of the sample that were male, female, etc.). I also felt more information could be added beyond stage 1 details relating to missing data and outliers (e.g., stating the precise number linked to physiological variables).
3) Regarding the results, some proof-reading of the inferential statistics reported is needed (e.g., page 44, paragraph 2), units should be included for relevant variables (e.g., bpm for HR), and the formatting of tables/figures should match the rest of the text (e.g., font style). I also found some of the wording tricky to interpret (e.g., a lack of increased challenge cardiovascular reactivity [TPT reactivity), and the results were generally relatively difficult to follow and link to the hypotheses noted on pages 12 and 13. I would therefore recommend the authors reflect and consider how best to restructure and rewrite the results section so it is as clear to the reader as possible.
4) The discussion would benefit from being more evidence-based and via the inclusion of more relevant citations/references (e.g., page 48, paragraph 3; page 52, paragraph 2). In particular, I would like to see the authors expand on their discussion regarding the post-intervention test not being high-pressured or stressful enough; what other factors could they have manipulated beyond performance-contingent rewards (see Baumeister and Showers [1986] for some suggestions like performance-contingent punishments, social comparison and evaluation, ego relevance, etc.)? Relatedly, other key discussion points could be elaborated upon (e.g., page 53, paragraph 2, lines 1206-1207). Finally, the writing could be checked to ensure it remains formal and scientific throughout (e.g., avoid colluqualisms like 'don't').
My review of the Stage II report is going to be way shorter than my review of the Stage I report. That is because have done a great job making their research workflow very open, accountably owning the minor mistakes, and being frank about the lack of clear signal from the data in several respects. Due to the fact that the data do not support a more straightforward stylized narrative, I think that the paper would have much harder time in many mainstream journals that are rather “novelty-oriented”. In my view, it is always very refreshing to see a study that reads like a report from a real study.
Now to the point of what should be the target of my comments:
1. The data are able to test the authors’ proposed hypotheses (or answer the proposed research question) by passing the approved outcome-neutral criteria.
In my view, the study is still informative, even though the situational context created to observe and test the target effect did not turn out to be optimal. This should probably be assessed by a substantive reviewer knowledgeable about gaming and affective responses, as it is potentially a limiting factor. Yes, the fact that esports competition may not represent a threat-type situation may be seen as logical, but only now, ex post. Although this aspect has a foul taste of failure, I think it is equally worthy of being published.
In that respect, I think the reader of the study would benefit from seeing the descriptives for the study variables to see how much limiting factor are these floor effects. Please, in the revision, include a table with basic descriptives. I understand that fitting it into the manuscript may be difficult, but there is no reason not to include it in the supplementary materials. I also cannot see the generated file with the analysis outputs – a knitted html, pdf or something of that sort.
2. The introduction, rationale and stated hypotheses (where applicable) are the same as the approved Stage 1 submission
The track changes document is not very clear but I believe it is the case. As a sidenote, I think it okay not to track some low-level edits (like changing to past tense) as it clutters the track changes document a lot.
3. The authors adhered precisely to the registered study procedures. Any unregistered exploratory analyses are justified, methodologically sound, and informative.
Yes, they are, IMO. Of course there were several acknowledged deviations. To me, several of the deviations are not really deviations _from the research plan_ at all. But okay, it is great to see that the authors were really very diligent about disclosing any change. I think that the deviations had little chance to jeopardize the integrity of the RR.
4. The authors’ conclusions are justified given the evidence
The authors offer a measured interpretation of the results, the narrative is not overblown. The discussion would, however, benefit from taking more into account the constrains on generality posed by the present design and tightening several too-much-generalizing claims accordingly.
A few minor points:
- I now see that the authors refer to supplementary materials, I just cannot see them on the osf repo.
- I tend to agree how the authors resolved unexpected situations with the pre-reg deviations, except for the randomization schedule. Unequal groups would have induced no causal bias whatsoever – only some drop in power if H0 is indeed false. But I agree that the fix is an okay solution, albeit a one to a non-problem.
- I do not see how the present data support the claim that SMI is a universal prevention tool.
- The model did not fit the data well. Even with very modest N and low magnitudes of interrelationships, the formal test of the model, the chi-square test, had enough teeth to reject the exact fit hypothesis. As you can see, the value of the chi^2 is almost 4 times as large, which is not terrible, but it points to model-data deviations inconsistent with purely random sampling variability. Also, the CFI is not reaching the (rather benevolent) threshold preached by Hu & Bentler (1999). I would say that even the approximate fit is mediocre at best.
Thanks for the opportunity to read your study and best wishes,
Ivan Ropovik
Well done to the authors for completing this study and submitting a well-written Stage 2 Registered Report. In my view, all necessary information has been adequately described, the analyses have been competently conducted, and the manuscript is well written. I have three minor comments which I recommend addressing:
1. Double check the sentence structure for accuracy where the SMI abbreviation has been used throughout. For example, a “the” is missing in this sentence: “we adapted and validated SMI in real-world performance”.
2. For the measure of Situational Affect Regulation, is there any indication of validity when using half of the items from the original scale?
3. Line 2068: This phrase seems to overstep the results “that the SMI is the universal prevention tool that is recommended”. The data suggest that the intervention may be promising, but there is no evidence yet that it is effective for universal prevention.
