How can the experiences of those who engage in video games in healthy and unhealthy ways be systematically organised?
Reconstructing Gaming Disorder: A Taxonomy by Registered Report
Abstract
Recommendation: posted 15 April 2023, validated 15 April 2023
Dienes, Z. (2023) How can the experiences of those who engage in video games in healthy and unhealthy ways be systematically organised?. Peer Community in Registered Reports, . https://rr.peercommunityin.org/articles/rec?id=322
Recommendation
URL to the preregistered Stage 1 protocol: https://osf.io/ekm8x
Level of bias control achieved: Level 4. At least some of the data/evidence that will be used to answer the research question already exists AND is accessible in principle to the authors (e.g. residing in a public database or with a colleague) BUT the authors certify that they have not yet accessed any part of that data/evidence
List of eligible PCI RR-friendly journals:
The recommender in charge of the evaluation of the article and the reviewers declared that they have no conflict of interest (as defined in the code of conduct of PCI) with the authors or with the content of the article.
Evaluation round #2
DOI or URL of the report: https://osf.io/9nm6x
Version of the report: v2
Author's Reply, 12 Apr 2023
Decision by Zoltan Dienes
, posted 03 Apr 2023, validated 03 Apr 2023
The reviewers are largely happy with the paper, with some minor changes for clarification. Concerning the smallest effect size of interest, I prefer to work in raw regression slopes, as they reflect units one should be interested in unaffected by criterion reliability (https://online.ucpress.edu/collabra/article/7/1/28202/118660/Obtaining-Evidence-for-No-Effect), but you may wish to argue,as you have done, that you will not be claiming an effect of interest exists vs does not, in which case, defining what is of interest is moot. Nonetheless, in addressing Ferguson's point, you may wish to alert readers about these issues in forming their own conclusions about what meaningfully exists, given that readers may well jump to conlusions you are are wise enough not to!
Reviewed by Lukas J. Gunschera, 30 Mar 2023
Review: Reconstructing Gaming Disorder: A Taxonomy by Registered Report (STAGE2)
I enjoyed reviewing this registered report and think the authors did well at implementing the suggested changes. I have added some minor comments below:
1) Participants from groups A-C will provide frequent diary-like entries over multiple years. The mere act of monitoring one’s gaming activity may affect participants and their appraisal of their gaming. This is something that the data will be able to address, and it would be interesting to analyse and/or mention in the manuscript, given ‘thoughtless’ gaming may have distinct effects.
2) The authors mention that “following our critical Bayesian epistemology, each discovered taxon will be established with a confidence level (0-1), and when evidence accumulates, confidence can decrease or decrease” (p. 14). It does not become clear whether these confidence intervals are based on data or on the subjective judgments of the authors. Arguably, the latter is insufficient to be termed “Bayesian”. In this case, the use of “Bayesian epistemology” is misleading, as it is traditionally used to refer to formal approaches of belief updating with high degrees of precision.
Moreover, the addition of a confidence level raises the question what level of confidence is acceptable. Please specify how these confidence levels will be used to guide later interpretation.
3) Typo: “while 41.1 % think the Finns think gaming is harmful” (p. 23)
Reviewed by Christopher Ferguson, 12 Mar 2023
I think these revisions mostly look good.
The only exception is that the authors MUST have a SESOI (smallest effect size of interest) of minimum of r (or beta) of .10. I think the authors misunderstood me here...this has nothing to do with hypothesis testing or sample size, only avoiding misinterpreting noise for "true" effects. There's a risk of noise being misinterpreted below effect sizes of .10, as there's a general "hum" of garbage results below that effect size. This is true whether a study is exploratory or confirmatory. The only issue about sample size is that if the sample size is small, it won't be powerful enough to detect effects that low anyway. Again, please read Ferguson and Heene (2021) for an explanation for why this is critical. But this is an unambiguous MUST for me to sign off on this registered report.
Reviewed by Michelle Carras, 16 Mar 2023
The revised addresses all of our concerns well--I appreciate the opportunity to remain involved and very much look forward to your progress.
The attached copy of your tracked changes version includes a few more comments for slight changes to wording that could be considered.
Thank you again.
Michelle
Download the reviewEvaluation round #1
DOI or URL of the report: https://osf.io/56ugm
Version of the report: v1
Author's Reply, 24 Feb 2023
Dear Recommender and Reviewers,
Thank you for your patience. It took a lot of time with the revisions. Three remarks:
1. We reformatted the file for clarity. Therefore, the track changes file and the "clean" file have minor differences. We didn't mark track changes in the supplements because the changes were so major that basically everything was rewritten. OSF was also reorganized for clarity. A pdf-version is also available here: https://osf.io/xwubz
2. One reviewer asked further details, which we have promised to share confidentially via private repository link. Here is the link, which remains active until end of March (password sent to recommender):
https://nextcloud.jyu.fi/index.php/s/c7DojEsYzLaNx5d
3. We have started first steps in data collection in Finland and Slovakia, fully following this v2 protocol. First adolescents have been interviewed, and one treatment-seeker. This keeps our bias level at 4.
One more thank you for the helpful feedback and time investments with this long manuscript.
Sincerely,
Veli-Matti Karhulahti
Decision by Zoltan Dienes, posted 27 Dec 2022, validated 27 Dec 2022
Dear Matti
I now have three thorough and thoughtful expert reviews. All are positive about your research and all ask for important clarifications. I was not sure if you intended quantitiive analyses as part of the Stage 2s, and if so there would of course be a benefit of specifying them in the Stage 1. You should also consider the extra clarity that would be provided by using a Desiggn Table, adapting it where possible for qualitative methods.
best
Zoltan
Reviewed by Michelle Carras, 27 Nov 2022
Please see the attached file.
Download the reviewReviewed by Christopher Ferguson, 18 Dec 2022
I appreciated the opportunity to review this RR. I have just a few comments below that I wonder if these may improve some of the ideas in here.
First, when mentioning the WHO "gaming disorder", it might be helpful to note the specific controversies around this. Many scholars directly opposed the WHO (Aarseth et al., 2017) as did the APA and Psychological Society of Ireland's media psychology divisions in an open statement (https://div46amplifier.com/2018/06/21/an-official-division-46-statement-on-the-who-proposal-to-include-gaming-related-disorders-in-icd-11/). The authors do a nice job discussing the wider controversies, but I think it's good to note specific criticisms of the WHO's decision.
Related to this conceptually, is the lack of interest (by the WHO and, frankly, most researchers) over the larger issues of behavioral overuse disorders...why is there so little interest in overeating, work, exercise, shopping or religion "addiction" (Griffiths, to his credit, has papers on dance and fishing "addiction".) There's an important question about why we're always hyperfocused on technology (and I'm not convinced the answer is that technology is inherently more addictive than these other things.
This I think is also a practical issue for the paper...particularly given lack of clear guidance on what constitutes "gaming disorder"...responses by both clinicians and gamers may simply invoke societal moral panics in open response formats. I'm thinking in particular of the clinician ratings here. Given how easy it is to portray anything addictive...see the offline friends addiction scale (Satchell et al), I'd like to see a bit more attention to this.
I think the authors include a lot of good and important variables about gaming contexts and health, but I think it will be possible to find spurious correlations here, without controlling for other variables. I assume that T1 health/mental health data will be controlled. However, particularly as we have some decent information to suggest that academic and family stress is causal for problematic gaming, I think some of these larger contexts would need to be measured and controlled.
The authors need to set a smallest effect size of interest (SESOI) for their quantitative analyses, otherwise they may inadvertently misinterpret "statistically significant" noise effects as hypothesis supportive when they should not be interpreted as such. See Ferguson & Heene (2021) for discussion and guidance.
In conclusion, I support the authors' efforts here and hope my suggestions are constructive.
Signed,
Chris Ferguson
