Recommendation

Synergistic Mindset Intervention in Competitive Situations

ORCID_LOGO based on reviews by Lee Moore, Ivan Ropovik and Jacob Keech
A recommendation of:

Applying a Synergistic Mindsets Intervention to an Esports Context

Abstract

EN
AR
ES
FR
HI
JA
PT
RU
ZH-CN
Submission: posted 02 February 2024
Recommendation: posted 30 March 2024, validated 08 April 2024
Cite this recommendation as:
Karhulahti, V.-M. (2024) Synergistic Mindset Intervention in Competitive Situations. Peer Community in Registered Reports, 100712. 10.24072/pci.rr.100712

This is a stage 2 based on:

Optimizing Esports Performance Using a Synergistic Mindsets Intervention
Maciej Behnke, Daniël Lakens, Kate Petrova, Patrycja Chwiłkowska, Lukasz D. Kaczmarek, Jeremy P. Jamieson, James J. Gross
https://osf.io/sp6av

Recommendation

Mindset theories suggest that the mere belief in the malleability of human abilities can help one to develop related performance. On the other hand, one and the same performance situation can also be experienced in various affective ways, which differently contribute to performance outcomes. One theoretically justifiable premise is that appraising a performance situation as a “threat” instead of “challenge” is associated with maladaptive responses, such as impaired cardiovascular mobilization. If people could experience performance situations as positive challenges, this might also improve performance outcomes. Drawing from these connected premises, the synergistic mindset intervention was developed and tentatively found to help adolescents in stressful situations (Yeager et al., 2022).
 
In the present registered report, Behnke et al. (2024) built on the above to test whether the synergistic mindset intervention can help individuals in competitive gaming situations. The authors utilized one of the leading esport games, Counter-Strike: Global Offensive​, and recruited its active players (N=300) into randomized control and intervention groups. The participants competed in a cash-prize tournament involving measures of affective experience and cardiovascular responses. Behnke et al. (2024) hypothesized the synergistic mindset group (SMI) to show greater challenge affective responses and superior performance outcomes. 
 
Although the SMI produced a number of positive outcomes such as more beneficial stress mindsets, the hypotheses were not corroborated but the results supported a null. This may be related to the observation that participants generally experienced the intervention positively, which, in turn, limits the potential for improving affective and physiological responses. These rigorous null results are informative by directing the SMI research program toward test designs where more participants experience strong negative stress responses. Moreover, the results encourage researchers to reassess the underlying auxiliary hypotheses regarding affective responses and performance outcomes, the relationships of which may be complicated by situational factors that are not yet fully understood.
 
The Stage 2 manuscript was evaluated over one round of in-depth review. Three out of the four Stage 1 experts returned to review and, due to the study’s exceptionally high level of transparency, the reviewers had only minor requests for revision. As all the requested revisions were implemented carefully, the recommender judged that the manuscript met the Stage 2 criteria and awarded a positive recommendation.
 
URL to the preregistered Stage 1 protocol: https://osf.io/z3adb
 
Level of bias control achieved: Level 6. No part of the data or evidence that was used to answer the research question was generated until after IPA.
 
List of eligible PCI RR-friendly journals: 
 
 
References
 
1. Behnke M., Lakens D., Petrova K., Chwiłkowska P., Kaczmarek L. D., Jamieson J. P., & Gross J. J. (2024) Applying a Synergistic Mindsets Intervention to an Esports Context. Acceptance of Version 2 by Peer Community in Registered Reports https://osf.io/53z8e
 
2. Yeager D.S., Bryan C.J., Gross J.J., Murray J., Krettek D., Santos P., ... & Jamieson J.P. (2022) A synergistic mindsets intervention protects adolescents from stress. Nature 607, 512–520. https://doi.org/10.1038/s41586-022-04907-7
 
Conflict of interest:
The recommender in charge of the evaluation of the article and the reviewers declared that they have no conflict of interest (as defined in the code of conduct of PCI) with the authors or with the content of the article.

Evaluation round #1

DOI or URL of the report: https://osf.io/3rxt5

Version of the report: https://osf.io/3rxt5

Author's Reply, 27 Mar 2024

Decision by ORCID_LOGO, posted 12 Mar 2024, validated 12 Mar 2024

Dear Maciej Behnke and co-authors,
 
We were lucky to have 3 out of 4 reviewers from Stage 1 to return at Stage 2. All reviewers are widely satisfied and appreciate your diligent reporting. They have very minor comments, which I let you address respectively in the revision.
 
As noted by LM, the results section could use some clarifications especially in reporting the hypothesis tests. I personally prefer a structure that clearly reports “H1 = outcome, H2 = outcome etc”, as it helps readers to easily locate and assess key findings. That's an option to consider. I also highly support IR’s request to add descriptives.
 
Regarding the optimality of the created conditions and the limits they set to the intervention, I agree with LM and IR that some further engagement with previous related knowledge would make the discussion even more informative. E.g., to my memory, already the first-ever doctoral dissertation on videogames (Elizabeth Ellen Moulds, 1978) investigated arousal in the competitive gaming situation. It could be useful to review the experimental literature of the past decades in detail to further assess whether the stress response in the present study is unique or echo those in previous studies involving similar conditions.
 
JK and IR point out some wordings in the discussion that should be rephrased. I made similar notes and would also suggest removing the first three words of the phrase “For this reason, we observed no effects..” (p. 47). Many other reasons could be too. I would also rethink some wordings in the last sentences: "However, we found the limited impact of SMI on the performance outcomes and challenge/threat affective responses... In sum, we recommend the SMI as a prevention tool." Please reconsider the conclusion of ‘limited impact’ for inferred no effects. As to the framing “we recommend the SMI”, would it carry similar meaning if phrased “the findings lead us to recommend the SMI”? I think it's a small but important difference.

Please add the date July 5 2023 as the time of updating the randomisation plan.
 
Last, I recommend stating all theoretical implications in the discussion, even if briefly, as per the last column of the design table (Theory that could be shown wrong). They're already implicitly in the text, but being explicit would be preferred. In my experience, readers tend to draw their own theoretical inferences if they’re not spelled out. You may also add a new results column to the design table; some Stage 2 RRs have it and it's often informative. If you wish to discuss any of the revisions before the next (perhaps final) version, I can be contacted as usual. 
 
Veli-Matti Karhulahti

Reviewed by , 22 Feb 2024

I enjoyed reading this Stage 2 PCI registered report entitled 'Applying a synergistic mindsets intervention to an esports context'. When reflecting on the criteria and key issues to consider at Stage 2, I felt the authors did a great job with the manuscript. Indeed:

> A direct url to the approved protocol was included.

> The introduction was largely the same as the stage 1 manuscript with only minor changes made and these were appropriate and transparently flagged.

> The authors stayed true to their approved Stage 1 protocol and where they deviated this was appropriate, clearly articulated, and well-justified (e.g., additional exploratory analyses).

> Additional exploratory analyses were informative, justified, performed appropriately, methodological sound, and clearly distinguished from the pre-registered analyses.

> The conclusions aligned to the preregistered aims/hypotheses and were appropriate, evidence-based, and well-communicated.

Beyond these points, I just have a series of relatively minor suggestions that I believe could help the authors improve the manuscript further:

1) Remove (including stress) from the abstract, stress is a process, not an emotion linked to negative affect (note: if the authors are referring to feeling stressed, the wording should better reflect this).

2) In the methods, it would be good to see a breakdown of gender/sex (i.e., proportion of the sample that were male, female, etc.). I also felt more information could be added beyond stage 1 details relating to missing data and outliers (e.g., stating the precise number linked to physiological variables).

3) Regarding the results, some proof-reading of the inferential statistics reported is needed (e.g., page 44, paragraph 2), units should be included for relevant variables (e.g., bpm for HR), and the formatting of tables/figures should match the rest of the text (e.g., font style). I also found some of the wording tricky to interpret (e.g., a lack of increased challenge cardiovascular reactivity [TPT reactivity), and the results were generally relatively difficult to follow and link to the hypotheses noted on pages 12 and 13. I would therefore recommend the authors reflect and consider how best to restructure and rewrite the results section so it is as clear to the reader as possible.

4) The discussion would benefit from being more evidence-based and via the inclusion of more relevant citations/references (e.g., page 48, paragraph 3; page 52, paragraph 2). In particular, I would like to see the authors expand on their discussion regarding the post-intervention test not being high-pressured or stressful enough; what other factors could they have manipulated beyond performance-contingent rewards (see Baumeister and Showers [1986] for some suggestions like performance-contingent punishments, social comparison and evaluation, ego relevance, etc.)? Relatedly, other key discussion points could be elaborated upon (e.g., page 53, paragraph 2, lines 1206-1207). Finally, the writing could be checked to ensure it remains formal and scientific throughout (e.g., avoid colluqualisms like 'don't').

Reviewed by , 19 Feb 2024

My review of the Stage II report is going to be way shorter than my review of the Stage I report. That is because have done a great job making their research workflow very open, accountably owning the minor mistakes, and being frank about the lack of clear signal from the data in several respects. Due to the fact that the data do not support a more straightforward stylized narrative, I think that the paper would have much harder time in many mainstream journals that are rather “novelty-oriented”. In my view, it is always very refreshing to see a study that reads like a report from a real study.

Now to the point of what should be the target of my comments:

1. The data are able to test the authors’ proposed hypotheses (or answer the proposed research question) by passing the approved outcome-neutral criteria.

In my view, the study is still informative, even though the situational context created to observe and test the target effect did not turn out to be optimal. This should probably be assessed by a substantive reviewer knowledgeable about gaming and affective responses, as it is potentially a limiting factor. Yes, the fact that esports competition may not represent a threat-type situation may be seen as logical, but only now, ex post. Although this aspect has a foul taste of failure, I think it is equally worthy of being published. 
In that respect, I think the reader of the study would benefit from seeing the descriptives for the study variables to see how much limiting factor are these floor effects. Please, in the revision, include a table with basic descriptives. I understand that fitting it into the manuscript may be difficult, but there is no reason not to include it in the supplementary materials. I also cannot see the generated file with the analysis outputs – a knitted html, pdf or something of that sort.

2. The introduction, rationale and stated hypotheses (where applicable) are the same as the approved Stage 1 submission

The track changes document is not very clear but I believe it is the case. As a sidenote, I think it okay not to track some low-level edits (like changing to past tense) as it clutters the track changes document a lot.

3. The authors adhered precisely to the registered study procedures. Any unregistered exploratory analyses are justified, methodologically sound, and informative.

Yes, they are, IMO. Of course there were several acknowledged deviations. To me, several of the deviations are not really deviations _from the research plan_ at all. But okay, it is great to see that the authors were really very diligent about disclosing any change. I think that the deviations had little chance to jeopardize the integrity of the RR.

4. The authors’ conclusions are justified given the evidence

The authors offer a measured interpretation of the results, the narrative is not  overblown. The discussion would, however, benefit from taking more into account the constrains on generality posed by the present design and tightening several too-much-generalizing claims accordingly.

A few minor points:
- I now see that the authors refer to supplementary materials, I just cannot see them on the osf repo.
- I tend to agree how the authors resolved unexpected situations with the pre-reg deviations, except for the randomization schedule. Unequal groups would have induced no causal bias whatsoever – only some drop in power if H0 is indeed false. But I agree that the fix is an okay solution, albeit a one to a non-problem.
- I do not see how the present data support the claim that SMI is a universal prevention tool.
- The model did not fit the data well. Even with very modest N and low magnitudes of interrelationships, the formal test of the model, the chi-square test, had enough teeth to reject the exact fit hypothesis. As you can see, the value of the chi^2 is almost 4 times as large, which is not terrible, but it points to model-data deviations inconsistent with purely random sampling variability. Also, the CFI is not reaching the (rather benevolent) threshold preached by Hu & Bentler (1999). I would say that even the approximate fit is mediocre at best.

Thanks for the opportunity to read your study and best wishes,
Ivan Ropovik

Reviewed by , 12 Mar 2024

Well done to the authors for completing this study and submitting a well-written Stage 2 Registered Report. In my view, all necessary information has been adequately described, the analyses have been competently conducted, and the manuscript is well written. I have three minor comments which I recommend addressing:
 
1. Double check the sentence structure for accuracy where the SMI abbreviation has been used throughout. For example, a “the” is missing in this sentence: “we adapted and validated SMI in real-world performance”.
 
2. For the measure of Situational Affect Regulation, is there any indication of validity when using half of the items from the original scale?
 
3. Line 2068: This phrase seems to overstep the results “that the SMI is the universal prevention tool that is recommended”. The data suggest that the intervention may be promising, but there is no evidence yet that it is effective for universal prevention.

User comments

No user comments yet