How effective is self-regulation in loot box labelling?

based on reviews by Pete Etchells and Jim Sauer
A recommendation of:

Beneath the label: Assessing video games’ compliance with ESRB and PEGI loot box warning label industry self-regulation

Submission: posted 17 September 2022
Recommendation: posted 17 January 2023, validated 17 January 2023


Paid loot boxes – items bought for real-world money that offer randomised rewards – are a prevalent feature of contemporary video games (Zendle et al., 2020). Because they employ random chance to provide rewards after spending real money, loot boxes have been considered a form of gambling, raising concerns about risk of harm to children and other vulnerable users. In response, some countries have taken legal steps to regulate and even ban the use of loot boxes, with only limited success so far (Xiao, 2022). At the same time, the Entertainment Software Rating Board (ESRB) and PEGI (Pan-European Game Information) now expect games that contain loot boxes to be marked with warning labels that, in theory, will enable users (including parents) to make more informed decisions. These requirements by ESRB/PEGI are not legally binding and may be considered a form of industry self-regulation.
In the current study, Xiao (2023) will investigate the effectiveness of self-regulation in the use of loot box labels. Study 1 examines the consistency of warning labels by the ESRB and PEGI, with the expectation that if self-regulation works as it should then these labels should always (or nearly always) co-occur. Study 2 establishes the compliance rate for labelling among popular games that are known to contain loot boxes, with a rate of ≥95% considered to be successful. The findings should prove useful in identifying the success or failure of self-regulation as a means of ensuring industry compliance with loot box labelling.
The Stage 1 manuscript was evaluated over two rounds of in-depth review. Based on detailed responses to the reviewers' comments, the recommender judged that the manuscript met the Stage 1 criteria and therefore awarded in-principle acceptance (IPA).
URL to the preregistered Stage 1 protocol:
Level of bias control achieved: Level 3. At least some data/evidence that will be used to the answer the research question has been previously accessed by the authors (e.g. downloaded or otherwise received), but the authors certify that they have not yet observed ANY part of the data/evidence.
List of eligible PCI RR-friendly journals:
1. Zendle, D., Meyer, R., Cairns, P., Waters, S., & Ballou, N. (2020). The prevalence of loot boxes in mobile and desktop games. Addiction, 115(9), 1768-1772.

2. Xiao, L. Y. (2022). Breaking Ban: Belgium’s ineffective gambling law regulation of video game loot boxes. Stage 2 Registered Report, acceptance of Version 2 by Peer Community in Registered Reports. 
3. Xiao, L. Y. (2023). Beneath the label: Assessing video games’ compliance with ESRB and PEGI loot box warning label industry self-regulation, in principle acceptance of Version 3 by Peer Community in Registered Reports.
Cite this recommendation as:
Chris Chambers (2023) How effective is self-regulation in loot box labelling?. Peer Community in Registered Reports, .
Conflict of interest:
The recommender in charge of the evaluation of the article and the reviewers declared that they have no conflict of interest (as defined in the code of conduct of PCI) with the authors or with the content of the article.

Evaluation round #2

DOI or URL of the report:

Version of the report: v2

Author's Reply, 03 Jan 2023

Download author's reply Download tracked changes file

Please find (i) my response to the recommender and reviewer comments and (ii) the manuscript file with all changes tracked separately attached below. Thank you!

All files (including a clean version of the manuscript with all changes confirmed) are available as one document via (version 3).

Decision by , posted 16 Dec 2022, validated 16 Dec 2022

Thanks for the careful attention to the previous comments. Both reviewers returned to evaluate your revised manuscript, and I'm happy to say that both are broadly satisfied. You will see some remaining clarifications to address in the review from Sauer, after which I expect we can issue Stage 1 IPA.

I look forward to receiving your revised manuscript. Please note that due to the December shutdown at PCI RR, you won't be able to submit your revised manuscript until we reopen on January 3rd. I will action your manuscript as quickly as possible at that time.

Reviewed by , 14 Dec 2022

The responses to the previous reviews are thoughtful, open, and concientious and have largely addressed my concerns. I have only a couple of relatively minor comments on the revised submission.

1. RQ2 and H2

H2 currently reads "Video games previously known to be high-grossing and contain loot boxes and presently containing loot boxes on the Google Play Store will accurately display the IARC ‘In-Game Purchases (Includes Random Items)’ label".

I think there are a couple of issues here. First, the clause "previoulsy known to be high grossing" feels vague (e.g., feels like you'd need to operationalise "high grossing" and the manner through which this was "previously known"?). Second, the hypothesis is a bit wordy, and a little difficult to follow. Perhaps the hypothesis could be simplified? Maybe somthing like: "All titles in this sample of video games previously known to contain loot boxes, and available on the Google Play Store, will accurately display the IARC ‘In-Game Purchases (Includes Random Items)’ label"? 

2. Representativeness

Given the revisions to the RQ2 and H2, the sample can now provide data capable of testing hte hypothesis, so that issue has been resolved. And the revised ms also explicitly considers the issues relating to the representativeness of the sample used, and I appreciate the openess here. The question, however, is will compliance data from this sample of very popular and highly scutinised games tell us something useful about compliance behaviour more generally? On the one hand, compliance may well be higher for these games than for other, less popular and less scrutinised games, and therefore the current investigation may produce a overestimate of compliance rates. On the other hand, these are the popular games; the games interacted with most by gamers. I've waivered back and forth on this issue while writing this review. Ultimately, I think, the proposed methodology/sample is suitable. Although it may overestimate compliance for the market as a whole, it addresses the titles most gamers play (and therefore provides data of broad enough relevance to be pratically useful) and I'm not unsympathetic to the arguments relating to expediency and efficiency (and the impracticality of assessing "all games").

3. 95% cut-off

I understand that any cut-off of this sort is going to be largely arbitrary and, as long as this is acknowledged, all is well.

4. Minecraft and Roblox

Interesting issue, eh? I think the proposed treatment of these titles is a sensible approach.


Reviewed by , 22 Nov 2022

Thanks to the researcher for addressing my comments in the previous round of reviewer. These have been done to a satisfactory standard, and I have no further suggestions to add. I note that there was some disagreement between myself and the second reviewer regarding how Minecraft and Roblox should be treated in Study 2; I think that the proposed solution has merit, and does a good job of balancing the need to treat these games as unusual entities with regards to loot box systems, while also considering them in the context of future avenues for research. 

Evaluation round #1

DOI or URL of the report:

Version of the report: v1

Author's Reply, 20 Nov 2022

Download author's reply Download tracked changes file

Please find (i) my response to the recommender and reviewer comments and (ii) the manuscript file with all changes tracked separately attached below. Thank you!

All files (including a clean version of the manuscript with all changes confirmed) are available as one document via (version 2).

Decision by , posted 18 Nov 2022, validated 18 Nov 2022

I have now obtained two very helpful reviews of your submission. As you will see, both evaluations are cautiously positive while also noting various aspects of the design and rationale that would benefit from clarification or modification. Key issues to address include the testability of the hypotheses, the timeframe of the data extraction (with the useful suggestion by Etchells to use a broader and more principled range), and the justification of specific assumptions and elements of the rationale.

One particular issue that requires careful attention is whether the answer to RQ2 is a foregone conclusion given known information (noted by Sauer). In order to be eligible for consideration as a Registered Report at PCI RR, the conclusions of the research must not be known (or inferrable with certainty) before the study is conducted. Please consider this issue carefully and, in turn, whether the bias control level for your submission is set appropriately.

Based on these reviews, I am happy to invite a revision, which I will return to the reviewers for another look.

Reviewed by , 15 Nov 2022

This is a well-considered and straightforward study, and I have no major concerns in general.

Study 1 carries with it a certain level of risk, given that the relevant games lists from both PEGI and and ESRB have already been collated, but I appreciate the necessity of this, and the author is honest in their reporting. However, I do wonder whether the time frame limitation is somewhat arbitrary. While I appreciate that using the entire database (c. 31000 games) is beyond the scope of the study, limiting to the year leading up to September 2022 could be better justified. As such then, I would suggest expanding the time frame scope to capture a broader range of games and be grounded in a sounder rationale. Given that the announcement to include the relevant warning tags came in April 2020, I wonder whether this would serve as a useful time point to start with. Expanding the time frame would also allay concerns that readers may have about the potential for 'data peeking'. 

Study 2 seems well thought through, with a clear processing pipeline for gathering game data. I appreciated the level of detail regarding sampling method, but felt that this became a bit cumbersome with regards to Roblox and Minecraft. The author does an admirable job of providing a clear justification for scenarios where these two games (or a combination thereof) will or will not be included in the final sample, but I wonder whether it just makes sense to include them both anyway. Given the unique nature of these two games in terms of compliance issues, they serve well as test cases for future sandbox games that may run up against third-party content, and therefore would be well-placed to be included in such considerations. 


Reviewed by , 17 Nov 2022

User comments

No user comments yet