Recommendation
In this Stage 1 Registered Report, Karhulahti and colleagues (2021) propose a qualitative, interview-based study of videogame play, with the central aim to understand key phenomological differences between gaming behaviour that is associated with vs. without health problems. This question is particularly important given the recent inclusion of "gaming disorder" in the WHO's International Statistical Classification of Diseases and Related Health Problems (ICD).
In recent years, the validity of "gaming disorder" as an identifiable mental illness has been controversial (e.g. Van Rooij et al, 2018), as has the debate concerning purported harms or benefits of gaming for mental health. This Stage 1 manuscript describes a rigorous qualitative investigation that should provide new insights on this question, and will also include a longitudinal component to examine changes in phenomonology over time, as well as an examination of the extent to which the phenomonology of gaming is reflected in the experiences of medical experts such as doctors, nurses, and therapists who have worked with gaming-related health problems.
More broadly, the manuscript breaks new ground for Registered Reports, being one of the first to focus on qualitative methods, while also making use of the Programmatic submission track in which the approved Stage 1 manuscript is intended to produce two Stage 2 manuscripts focusing on different elements of the project.
Three expert reviewers with a variety of field-specialist and qualitative methodological expertise assessed the Stage 1 manuscript over two rounds of in-depth review. Following revision, the reviewers and recommender agreed that the manuscript met the Stage 1 criteria and therefore awarded in-principle acceptance (IPA).
URL to the preregistered Stage 1 protocol: https://osf.io/a2rwg
Level of bias control achieved: Level 4. At least some of the data/evidence that will be used to answer the research question already exists AND is accessible in principle to the authors (e.g. residing in a public database or with a colleague), BUT the authors certify that they have not yet accessed any part of that data/evidence.
List of eligible PCI RR-friendly journals:
References
DOI or URL of the report: https://mfr.de-1.osf.io/render?url=https://osf.io/gr9bk/?direct%26mode=render%26action=download%26mode=render
Version of the report: v2
Dear Dr. Chambers and Reviewers,
Thank you for the constructive second round feedback. This has been an extermely helpful process.
We hope to have responded to the follow-up reviews well and look forward to carrying out the study.
On behalf of the team, sincerely,
Veli-Matti Karhulahti
* Image credit: Anniecho
, posted 14 Sep 2021The revised Stage 1 manuscript was returned to the three reviewers who evaluated the original submission. The good news is that the reviewers are broadly satisfied and we are now close to being able to award Stage 1 in-principle acceptance (IPA). Among the remaining issues to address are methodological clarifications and minor presentational issues, together with assurances regarding the anonymisation of participant data.
Other than these points, the one remaining significant issue is the suitability of the proposed design for the programmatic RR track in which there will be two separate Stage 2 outputs. After the first round of review, I was not fully convinced that the second proposed Stage 2 output was sufficiently robust and substantive to justify a separate RR (in accordance with Stage 1 criterion 1C).
This concern is revisited here by one of the three reviewers (Malte Elson), and I admit I do share the reviewer's concern. However, after re-reading the revised manuscript myself, including especially the authors' contingency plan for the designation of the second Stage 2 output as either a complete qualitative report or a case study, I have decided that the best course of the action in this case is to grant the authors' the benefit of the doubt, much as with many Stage 1 RRs that, in spite of careful planning, face a non-zero risk of recruitment failure. To comprehensively settle this issue, in this final revision, I would like to see (in both the manuscript and in the authors' response to reviewers) a clear description of the conditions under which the second RR would be impossible -- i.e. presumably if the sample size is below the minimum needed even for a case study. Although this may seem obvious, I believe it needs to be clearly articulated, primarily for the authors' own sake. As noted in the PCI RR policy, once a programmatic IPA is awarded, the prespecified boundaries defining different Stage 2 outputs are treated as design elements; therefore, like any other design element, Stage 1 IPA will be contingent on authors adhering to the prespecified and approved article boundaries at Stage 2. Where one of these elements becomes impossible to complete, authors must ensure that they are aware of the conditions under which a Withdrawn Registration would be required, and the resulting reporting requirements: "For programmatic RRs, entire components that are planned as separate Stage 2 outputs can be withdrawn without affecting the IPA of the remaining components. In such cases, each remaining Stage 2 output must include a URL to the Withdrawn Registration of the withdrawn component." Therefore, while I am satisfied for the authors to split the project as described, the authors should bear this in mind in reaching their own final decision about the risks and benefits of the programmatic track.
Overall, given the enthusiastic evaluations of the reviewers, provided the authors are able to respond comprehensively to these remaining points in a final revision, Stage 1 IPA should be forthcoming without requiring further in-depth review.
, 19 Aug 2021This is a revised version of a manuscript previously submitted proposing a study on the phenomenology of gaming disorder. I would like to thank the authors for their efforts and revisions, which clearly have improved this study proposal.
My remarks on this revision are relatively brief. Speaking just for myself, I can say that the authors have in principle addressed all my earlier comments. With regards to the research questions, the path for study 2 is now much clearer to me. That being said, I am still not fully convinced that the authors can sufficiently guarantee the value of study 2 independently of the results from study 1 given the many eventualities. I think the authors have probably done their best in addressing this concern (getting in touch with the participants regularly during the 12 month waiting period, distinguishing between longitudinal IPA and longitudinal case study) – I’m simply worried that if some things go wrong (imagine, e.g., they end up with 2 subjects at time 2), there will be regrets based on the decision we make now.
I will honestly admit that this is just a gut feeling that also stems from my own inexperience in dealing with programmatic qualitative interview-based RRs. Being merely a reviewer, and not an editor or recommender, I will take the easy way out and leave this puzzle to Dr. Chambers to solve. This is probably something that can be resolved better in a dialog with the authors (e.g., taking into account how important it is to them to have study 2 in this RR).
I have one minor comment on an issue that came up due to the revisions. I commend the authors for aiming to share the literal transcripts after allowing the participants to review them. However, I take issue with the method by which the authors claim to “anonymize” the transcripts (p. 12). Clearly, simply removing the name of someone from a transcript may not be enough to anonymize their utterances, particularly when they go into personal details that could make them easily identifiable to others. The risk of de-anonymization can be minimized by removing names and other references to places or people (which the authors should do before the transcripts are published), but I don’t think there is good way make in-depth interview data truly anonymous in the sense of the GDPR. I recommend that the authors explain to their participants that they will do everything in their power, but that an element of risk remains.
Signed
Malte Elson
, 03 Sep 2021Thanks for such a considered and detail response to the three reviews. I think the changes clarify your analytic and longitudinal focus. I also appreciate the practical challenges of submitting this registered review while having to meet your obligations to the funder. Consequently, I want to say thanks. I've learnt a lot from this process and wish you the best of luck with this research. You do mention about the challenge of word limit. I imagine that the final article will be much shorter but readers will benefit incredibly from access to your RR.
Thank you to the authors for their careful consideration of all of our remarks. I have only a few additional suggestions and questions to consider.
Questions:
" both Group 1 and 2 participants will fill out various paper-based screening tools (Appendix 2);" –Appendix 2 shows 3 separate scales and a single-item question to measure gaming problems, as well as 6 other self-reported scales. Will all scores be reported in a table as part of the description? I assume these will not only be used as part of participants descriptions, but in later work; this might be explained a bit further. It might also be good to list any demographic information that will be collected.
"we expect these meaningsthe above to produce significant life conflicts for the treatment-seeking"—do you expect that the meaningfulness they derive to be the thing that produces conflict? or can the conflict derive from aspects that aren't seen as meaningful and positive, like symptoms/impairment? E.g., play that feels like a compulsion—you're no longer getting anything out of it but you still feel like you have to keep doing it. Any counterbalancing positive aspects/meaning are outweighed by the loss of control/continuance despite problems. This, I think, would be more likely to cause life conflict rather than the meaning itself. I think " because that meaningful behavior clashes with their other key valuesabove." can be deleted, unless your hypothesis is predicated on the meaning itself causing the conflict.
"forums by recommendation of the Finnish Esports Federation (https://seul.fi), with whom we collaborate –" this potential conflict of interest should be explained in greater detail.
Suggestions:
Link the self-reported health problems to gaming in a consistent way, e.g.
will help modelling the differences-->will help model the differences
those who play videogames daily to have better psychosocial adjustment-->have better
has suggested similar meaningfulness to be part of those who meet the DSM5 criteria for ‘internet..."--> has suggested similar meaningfulness to be found by those who meet the DSM5 criteria for ‘internet
depth and breath—depth and breadth
longitudinal case study,-- perhaps a case series rather than a case study
DOI or URL of the report: https://mfr.de-1.osf.io/render?url=https://osf.io/8q4ym/?direct%26mode=render%26action=download%26mode=render
Cover Letter attached, manuscript link updated to v2 and the materials link: https://osf.io/7v5bj/
, posted 16 Jul 2021Three reviewers with a range of field-specialist and qualitative methodological expertise have now assessed the Stage 1 manuscript. As you will see, the comments are generally encouraging and constructive, while also critical, noting a range of areas that will need careful attention to achieve Stage 1 IPA. At a broad level, the reviews identify shortcomings that cut across several Stage 1 criteria, including justification of the hypotheses (with one reviewer especially identifying concerns with QH2), feasibility, and the degree of detail provided concerning the study procedures and analysis plans.
One reviewer also questions whether the follow-up study proposed for the second of the two proposed Stage 2 outputs is sufficiently substantive, robust, and resistant to recruitment failure to include now as part of a programmatic Stage 1 RR -- or whether it may be preferable to focus on the first part and then later submit a second Stage 1 RR that includes a more adequately defined, and risk-controlled longitudinal study. This evaluation falls appropriately under Stage 1 criterion 1C, which, for programmatic RRs, includes "whether the separate study components are sufficiently robust and substantive to justify separate Stage 2 outputs". At present, given the comments of this reviewer, I think that test is not yet met, but I am open to being convinced otherwise through a substantive and major revision that addresses this concern.
Overall, the manuscript falls within scope for a Major Revision. Please note that a revised submission is likely to be returned to the reviewers for further consideration.
, 13 Jul 2021Review for:
Phenomenological Strands for Gaming Disorder and Esports Play: A Qualitative Registered Report
This protocol proposes a qualitative interview-based study on the phenomenology of gaming disorder. The authors plan to conduct a series of interviews with people from three different groups: (1) video game players that have sought help with health problems related to their video game use, (2) esport gamers who have not sought any such help, (3) medical experts familiar with gaming disorder and/or gamers with health-related problems. The gamers (group 1 and 2) are interviewed again after 12 months to study how their experiences have evolved.
There is much to like about the proposed design. It is very much committed to the kind of groundwork that should have been conducted before turning a poorly understood phenomenon of a widespread activity into a mental disorder. In this regard, the research might fill an important gap. That being said, I think there may be some things the authors might consider in order to improve the impact of their proposed study.
Looking at the literature on internet gaming disorder, I think there are two major issues currently not at all explored by the manuscript:
1. The incoherence and lack of precision on the conceptual level of IGD is inherited by attempts to operationalize and measure it. The arbitrariness of measures, cutoffs, thresholds, computations etc results in a literature that is difficult to synthesize -- not only because such flexibility offers opportunity for p-hacking, but even more so because it is not sufficiently clear whether different studies on IGD are actually measuring the same thing. These larger points are discussed, e.g., in van Rooij et al (2018); and recently, Satchell et al. (2021) published a study supporting the problem of arbitrariness with the mock-development of a friendship addiction scale.
2. It is not sufficiently established whether "problematic game-playing" (however it is operationalized) should be its own diagnosis (which implies that the medium itself may be an immediate cause), or whether it is merely a symptom of another underlying condition (e.g., depression). I think this point deserves particular attention not because it is simply part of the larger discourse around IGD, but also because it has implications for the present study design:
The authors are aiming to recruit participants for group 1 by inviting gamers who report mental health problems related to their gaming habits. I don't think this by itself is problematic at all compared to, e.g., using a screening instrument, but I think the point that gaming itself may not necessarily be the proximate or distal cause of the disorder should be further discussed.
There are some further concerns with regards to the study design that I would like to at least mention, although I understand that not all of them can or should be addressed (but maybe clarified in the manuscript):
3. While I find the comparison "gamers with disorders" with "esport players without disorders" an interesting approach, it naturally has some limitations for the goal of the proposed research. For example, it might be conceivable that a disorder is a major hurdle in becoming a competitive, (semi-) professional esport player. As an experimental psychologist, my toolbox woud suggest sampling gamers from a group that is similar except for the variable of interest (whether or not they suffer from mental health problems); of course, I understand that the nature of this study does not demand such strict control of confounding factors, but maybe this point deserves some further attention beyond what is currently proposed in the paper.
4. I understand that this manuscript is submitted as a programmatic stage 1 RR for two separate outcomes. The value in what is currently conceptualized as "study 1" is clear to me and I can see this study filling an important gap; however, this is less so for "study 2", which is essentially another round of interviews 12 months later focusing on the change in gaming experiences. My feeling was that the route for study 1 was quite clear, but that study 2 was quite vague in contrast; part of the reason is of course that the outcome of the first round of interviews could dramatically change what is currenty planned for the second round. That, however, sort of defeats the purpose of Registered Reports. I also see some concerns regarding potential dropouts, which are currently not accounted for. Assuming, for example, the authors are able to recruit 7 interview candidates for round 1, of which 3 are no longer available for round 2, the outcome of study 2 could be quite limited.
My own feeling about this is that the authors should rewrite this as an RR for study 1 and use their findings to write a much clearer, separate RR for study 2 at a later point, but maybe this too strongly interferes with the authors' project roadmap.
5. I'm not 100% clear which parts of the data the authors will be able to share. The recordings or literal transcripts of the interviews seem out of question, so maybe this point could be clarified.
With kind regards
Malte Elson
REFERENCES
Satchell, L.P., Fido, D., Harper, C.A. et al. Development of an Offline-Friend Addiction Questionnaire (O-FAQ): Are most people really social addicts?. Behav Res 53, 1097–1106 (2021). https://doi.org/10.3758/s13428-020-01462-9
van Rooij, A. J., Ferguson, C. J., Colder Carras, M., Kardefelt-Winther, D., Shi, J., Aarseth, E., … Przybylski, A. K. (2018). A weak scientific basis for gaming disorder: Let us err on the side of caution. Journal of Behavioral Addictions, 1–9. https://doi.org/10.1556/2006.7.2018.19
, 09 Jul 2021This sounds like an interesting and unique longitudinal interpretative phenomenological study of ostensibly disordered Esports play. I wish the researchers the best of luck with the research. Below, I've added feedback that is structured by questions for reviewing a Stage 1 proposal.
1. The scientific validity of the research question(s).
This sounds like an interesting and unique longitudinal interpretative phenomenological study of ostensibly disordered Esports play.
The use of 'qualitatively' (penultimate paragraph before the methods section) is extremely broad and means the reader doesn't really have a sense of the approach you are taking. 'Qualitatively' could encompass everything from positivist to social constructionist approaches. You mention that you will conduct phenomenological interviews in the abstract and in the Sample Justification you mention IPA. Can you instead describe your specific phenomenological approach and how and why this is appropriate for your research questions? Plus, can you consistently refer to this phenomenological approach throughout the paper? Personally, it would be good to see a brief elaboration on how you can use a phenomenological approach longitudinally perhaps with reference to the longitudinal qualitative, such as the Timescapes projects: https://timescapes-archive.leeds.ac.uk/timescapes/ or others that are specifically phenomenological.
2. The logic, rationale, and plausibility of the proposed hypotheses.
Not applicable although (following the Mixed Methods Article Reporting Standards; https://apastyle.apa.org/jars/mixed-table-1.pdf) can you specify your aim or aims for this study?
The use of hypotheses is interesting. My first thought that hypotheses unnecessary. Next, I was persuaded by your argument as significance as meaning, so I was open to see how you used them. The hypotheses outline what you 'expect', so I can see how they might be comparable to a confirmatory hypothesis. Last, I'm sitting on fence. I can see the potential benefit of outlining your expectations; but I also think they could also be presented as expectations rather than hypotheses. As such, I'm not saying you should or shouldn't change it - just giving my perspective on reading it.
3. The soundness and feasibility of the methodology and analysis pipeline (including statistical power analysis where applicable).
Archiving qualitative data is a delicate topic; is it worth exploring this briefly in the ethics section and explaining how you plan to get consent and the approach to anonymity and confidentiality? In an interview study I was involved in, qualitative researchers could see the benefit of secondary use of their data but raised concerns about achieving valid consent and the risks to participants privacy (DOI: 10.1080/14780887.2019.1605477).
NB: I thought the red font was unnecessary; the paper makes sense without it (and the sentence about the red font in the abstract doesn't seem to fit with the format of an abstract).
Can you follow the Journal Article Reporting Standards for Qualitative Methods (https://apastyle.apa.org/jars/qual-table-1.pdf0 and include the researcher description and the researcher-participant relationship.
Useful and honest account (mentioning the role of the funding application and subsequent ethical approval) of the sample size.
4. Whether the clarity and degree of methodological detail would be sufficient to replicate the proposed experimental procedures and analysis pipeline.
In the abstract, can you clarify your how you will analyse the data. I note that you specify you will approach the questions with 'phenomenological interviews'; a phenomenological data analysis is therefore implicit. There are a wide range of phenomenological approaches to data analysis, so could be make it explicit by specifying? You mention IPA in the method section - if this is an IPA study, can you mention this consistently throughout? From my reading, this seems to be a longitudinal IPA study.
In the abstract, the group 'those who play esports more than 4 hours per day without problems' is 'without problems' too simplistic? While playing, they may still have problems, such as neglecting childcare or other responsibilities. Later in the sample size justification, you use the term 'self-reported health problems'. Is there are better way of describing this group (and can you use the same term consistently throughout the paper)?
As you have two time points, can you clarify how you will synthesise the data between the two time points? Will you, for example, keep the idiographic focus usually seen in IPA and, if so, how? In the analysis section, you mention that 'the process will be repeated in 12 months'. Wouldn't this give you two separate analyses (which would be incredibly useful) but doesn't explain how you will consider time and/or synthesis findings from the two time points.
For group 3, the interview type is 'non-phenomenological'. Can you specify what it is rather than what it is not?
Interesting use of the Phenomenology of Play interview frame; can you highlight this in the abstract?
As the interviews will be in Finnish, can you clarify which language the analysis will be conducted in and when translation will occur. E.g., will you conduct the IPA in Finnish and translate theme names and illustrative quotes into English for publication?
For the analysis plan, can you clarify that it will be idiographic (e.g. One person at a time); from your description, this could also be thematic analysis.
Can you provide the reference for the 'phenomenological manual'?
5. Whether the authors provide a sufficiently clear and detailed description of the methods to prevent undisclosed flexibility in the experimental procedures or analysis pipeline.
There is disclosed flexibility in Stage 2 that is consistent with qualitative analysis.
6. Whether the authors have considered sufficient outcome-neutral conditions (e.g. absence of floor or ceiling effects; positive controls; other quality checks) for ensuring that the results obtained are able to test the stated hypotheses.
Not applicable for this design.
Peter Branney
Please see attached review.
Download the review