Recommendation

Self regulation by gaming companies for the use of loot boxes is not working

ORCID_LOGO based on reviews by Chris Chambers
A recommendation of:

Non-compliance with and non-enforcement of UK loot box industry self-regulation on the Apple App Store: A longitudinal study on poor implementation

Abstract
Submission: posted 13 January 2025
Recommendation: posted 08 April 2025, validated 08 April 2025
Cite this recommendation as:
Dienes, Z. (2025) Self regulation by gaming companies for the use of loot boxes is not working. Peer Community in Registered Reports, 100971. 10.24072/pci.rr.100971

Recommendation

Video games may provide the option of spending real money in exchange for probabilistically receiving game-relevant rewards; in effect, encouraging potentially young teenagers to gamble. The industry has subscribed to a set of regulatory principles to cover the use of such "loot boxes", including 1) that they will prevent loot box purchasing by under 18s unless parental consent is given; 2) that they will make it initially clear that the game contains loot boxes; and 3) that they will clearly disclose the probabilities of receiving different rewards.
 
Can the industry effectively self regulate? Xiao and Lund (2025) evaluated this important question by investigating the 100 top selling games on the Apple App Store and estimating the percentage compliance to these three regulatory principles at two time points 6 months apart. In all cases compliance was minimal, or even non-existent. The authors recommend stricter legal regulations of loot boxes.
 
The Stage 2 manuscript was evaluated over one round of review. The recommender judged that the manuscript met the Stage  2 criteria for recommendation.
 
URL to the preregistered Stage 1 protocol: https://osf.io/3knyb
 
Level of bias control achieved: Level 2. At least some data/evidence that was used to answer the research question had been accessed and partially observed by the authors prior to IPA, but the authors certify that they had not yet observed the key variables within the data that were used to answer the research question.
 
List of eligible PCI RR-friendly journals:
 
 
References
 
Xiao, L. & Lund, M. (2025). Non-compliance with and non-enforcement of UK loot box industry self-regulation on the Apple App Store: A longitudinal study on poor implementation [Stage 2]. Acceptance of Version 2 by Peer Community in Registered Reports. https://osf.io/3re4n
Conflict of interest:
The recommender in charge of the evaluation of the article and the reviewers declared that they have no conflict of interest (as defined in the code of conduct of PCI) with the authors or with the content of the article.

Evaluation round #1

DOI or URL of the report: https://doi.org/10.31219/osf.io/xmwgy

Version of the report: 1

Author's Reply, 08 Apr 2025

Download author's reply Download tracked changes file

Please find (i) my response to the recommender and reviewer comments and (ii) the manuscript file with all changes tracked separately attached below. Thank you!

All files (including a clean version of the manuscript with all changes confirmed) are available as one document via https://osf.io/3re4n. The final file has also been uploaded and will be available via https://doi.org/10.31219/osf.io/xmwgy (Version 2) upon approval by OSF moderators.

Decision by ORCID_LOGO, posted 07 Apr 2025, validated 07 Apr 2025

Sorry for the delay in getting back. We do have one review; and based on my own reading this is sufficient as you have followed the guidelines for a Stage 2 very well. The reviewer asks you optionally to consider wording in a few places.

 

best

Zoltan

Reviewed by ORCID_LOGO, 12 Feb 2025

I enjoyed reading this completed Stage 2 submission, having served as a (non-specialist) reviewer at Stage 1. In my view, the report already comes very close to meeting the Stage 2 criteria -- the authors adhered faithfully to their preregistered protocol, and as best as I can tell have documented all deviations very thoroughly and transparently. The reporting of the results is clear, with appropriate robustness tests where required, and the conclusions are justified by the evidence.

The judgment of findings and their importance does not form part of Stage 2 evaluation, so the following is pure commentary, but I do want to note for the record that I found the lack of compliance - and the apparent toothlessness of the relevant regulators - quite dispiriting, even if unsurprising. It is hard to see these findings and draw any conclusion other than that the regulation of loot boxes in the UK has failed. I hope this work has an impact in stimulating the necessary reforms.

I have one stylistic suggestion that I believe would further increase the impact of this work - although the authors may disagree and it will be the recommender's decision as to whether my comment falls within the purview of Stage 2 evaluation. At various places in the Stage 2 manuscript, I recommend replacing emotive/superlative language with more dispassionate phrasing, and keeping interpretations strictly in line with the evidence. Some examples (and there may be others) include "incredibly low", "compliance is abysmal”, "feel betrayed”, "destroyed their own reputation”, etc. These opinions are understandable, but the eventual Stage 2 RR will be harder for regulators and politicians to ignore if it meets the highest standards of discipline and sticks to the facts and their evidence-based interpretration - whereas emotive language may let them off the hook by pointing to the work as advocacy or a personal attack rather than what it is: rigorous preregistered scientific research. The more emotive language would be better suited, in my view, to an op-ed or other news-style article that the authors may want to write at a later date once the RR is recommended.

Minor points

Lines 1098-1105 – this is a very long sentence that I found difficult to parse; I suggest rephrasing or breaking into two sentences for clarity.

User comments

No user comments yet