Recommendation

Loot boxes remain prevalent in Belgium despite their “ban”

based on reviews by Andrew Moshirnia, Joseph Macey and Jason Chin
A recommendation of:
picture

Breaking Ban: Belgium’s ineffective gambling law regulation of video game loot boxes

Abstract
Submission: posted 28 July 2022
Recommendation: posted 14 November 2022, validated 14 November 2022

Recommendation

Several countries currently struggle to legally interpret and deal with loot boxes, i.e. gambling-like mechanisms that have become common especially in contemporary videogame design. One of the few countries to take a clear regulation stance is Belgium, which officially announced that they interpret paid loot boxes as games of chance that violate their Gaming and Betting Act and can thus be criminally prosecuted (Naessens 2018). This announcement four years ago laid the basis for a unique social experiment where companies offering loot boxes in Belgium had to decide whether to modify their games, exit the market, or continue to monetize with loot boxes. In the present study, Xiao (2022) investigated the outcomes of this experiment both via hypotheses testing and exploratory analysis.
 
Using the 100 highest-grossing iPhone games in Belgium as data and applying comprehensive qualitative mechanical analysis to each title, Xiao (2022) tested three preregistered hypotheses regarding loot box prevalence. None of the hypotheses were confirmed: the prevalence rate of loot boxes in the Belgian App Store was not null but extremely high (82.0%), also among mobile games designed for minors (54.2–80.2%), and significantly more compared to global standards when assessed by a binomial test (p<.001). Corroborating a fourth hypothesis, Xiao was also able to access various UK loot boxes in Belgium. In exploratory research, Xiao received a confirmation from the Belgian Gaming Commission that even “simulated gambling games” (that do not yield monetary wins) also legally constitute gambling in Belgium.
 
The results are the first to describe outcomes of ban-driven loot box regulation globally. Despite Belgium’s clear statement that “paid loot boxes must be removed from the video games in order to comply with the Belgian Gaming and Betting Act” (Naessens 2018, p. 16), almost all highest-grossing iPhone games in the Belgian App Store keep offering them. The finding is important especially for the legal authorities around the world who are currently assessing their own positions in potential regulation: simple statements are unlikely to have immediate effects on international companies in local markets. On the other hand, it remains unknown how these companies have interpreted the statement and what are their distinct reasons for continuing to offer loot boxes in the Belgian App Store. It would be important in future research to investigate the companies’ perspectives.
 
This study (Xiao 2022) is important also from a meta-scientific perspective. It paves the way for Registered Reports in new disciplines and methods, and involves an exceptional field experiment that was carefully documented with open data and materials. Three external experts reviewed the Stage 2 manuscript twice, based on which the recommender awarded a positive recommendation.
 

URL to the preregistered Stage 1 protocol: https://osf.io/5mxp6

Level of bias control achieved: Level 6. No part of the data or evidence that was used to answer the research question existed prior to Stage 1 in-principle acceptance.

List of eligible PCI RR-friendly journals:

 

References
 
1. Naessens, P. (2018) Research Report on Loot Boxes [English Translation]. Belgian Gaming Commission. URL: https://www.gamingcommission.be/sites/default/files/2021-08/onderzoeksrapport-loot-boxen-Engels-publicatie.pdf
 
2. Xiao, L.Y. (2022) Breaking Ban: Belgium’s ineffective gambling law regulation of video game loot boxes. Stage 2 Registered Report, acceptance of Version 2 by Peer Community in Registered Reports. https://doi.org/10.31219/osf.io/hnd7w 
Cite this recommendation as:
Veli-Matti Karhulahti (2022) Loot boxes remain prevalent in Belgium despite their “ban”. Peer Community in Registered Reports, . https://rr.peercommunityin.org/articles/rec?id=264
Conflict of interest:
The recommender in charge of the evaluation of the article and the reviewers declared that they have no conflict of interest (as defined in the code of conduct of PCI) with the authors or with the content of the article.

Evaluation round #2

DOI or URL of the report: https://doi.org/10.31219/osf.io/hnd7w

Version of the report: v5

Author's Reply, 07 Nov 2022

Download author's reply Download tracked changes file

Please find my response to recommender and reviewer comments and the manuscript file with all changes tracked separately attached below. Thank you! All files (including a clean version of the manuscript with all changes confirmed) are available as one document via https://doi.org/10.31219/osf.io/hnd7w (version 6).

Decision by , posted 06 Nov 2022, validated 06 Nov 2022

Dear Leon Y. Xiao,
 
Thank you for the carefully revised manuscript. All reviewers are widely satisfied with the revisions. You will have an opportunity to address their last minor suggestions, after which I will write the recommendation. My own minor points:
 
P. 19: the Stage 1 text has been correctly moved to the first page and is no longer needed here.
P. 20: I still believe that presenting confidence intervals would be appropriate, at least with the main prevalence rates in Table 1 (but this remains a suggestion and is not required for recommendation).
P. 23: as the reviewers didn’t express concern for social casino games, we can leave it as it is. Nonetheless, it’s important to keep in mind that the exploratory outcome of 3.2.2 will not be cross-nationally comparable, e.g., with the previous studies that motivated this extra analysis (Zendle et al., 2020; cf. Zendle et al., 2022), as they negatively coded such products. This could be highlighted (again, only a suggestion).
#732: “The Belgian Gaming Commission does recognise the randomised monetisation methods in ‘simulated casino games’ games as constituting ‘gambling’..” This is lacking a citation. Some other parts of the discussion are also relying strongly on “general knowledge” without citation (e.g., “obvious loot box contraventions being … highly popular” #1187). Please consider adding further references to back up such statements. 
#751: if I understand correct, here “p = 0.05” refers to the alpha level, right? If yes, please correct and/or state earlier that an alpha level 0.05 was used for all analyses (also #466 and #702).
P. 39: Section numbers 4.8 onwareds seem to be incorrect.
 
One final note -- this doesn’t need a reply if deemed irrelevant, but please consider. Because it remains unclear how the Belgian ban should work or what are its goals/premises (as it was never properly implemented to begin with), some statements in the discussion like “Belgian ban on loot boxes is not working at present” (#1459) might still be better phrased. For example, the gambling monopoly of Finland is based on a specific system of ethics: money generated in gambling by Finnish companies should be directed to charity (thus only government companies are allowed provide local gambling services to ensure the rule is followed). As it remains unknown what the gambling regulation principles in Belgium are, it seems also possible that -- if they’d be based on similar local corporate control (rather than people control) -- it may also have, at least partially, succeeded if no Belgian game companies offer loot boxes? In this case, notions such as “circumventing” the ban wouldn’t fully apply either (that said, the quote of Article 4(2) from 1999 seems to imply that some people control is involved). Just a thought. 
 
As there will be no further review after the next final version, I can be contacted directly before submission if there is unclarity about how to respond to any of the points.
 
Sincerely,
Veli-Matti Karhulahti 

Reviewed by , 04 Nov 2022

Reviewed by , 31 Oct 2022

Thank you for the opportunity to review the revisions of this paper. I will take my previous comments as to the suitability of 2a-2d as read. I continue to recommend this paper. 

In 2e, I appreciate the softening of some of the bolder language addressed in my previous comments. 

I think the point at p28, ln 910 is an important one but there are some logical issues with the the current conclusion that higher ranked non-compliant games are gaining users from the removal of compliant games. If, for example, Nintendo provides games 1, 2, 3 on a ranked list, and are then removed, the rank of the remaining non-compliant games would go up, but not necessarily their user numbers (Nintendo fans could simply not play). This conclusion presupposes that at least some users of compliant games would migrate to non-compliant games (I am fine with this, as it seems intuitive, but it still should be addressed and suitably qualified). I mention this because off the added language regarding crime essentially paying under the current regime -- this would need to have more support. The author does address this assumption "assuming that the amount of money spent by players on video games did not 1047 change following the ban" but that qualification should be moved up and emphasised if the point retained. This push is contradicted somewhat by the addition at 4.3, which notes that some users were likely prevented from accessing loot box content, and addressing this assumption earlier would strengthen both points. 

The inclusion of "provided that 1515 reasonably strong technical measures have been implemented to prevent such 1516 circumvention. Belgium should therefore consider requiring companies and 1517 platform providers to implement sufficiently difficult-to-circumvent technical 1518 measures" forces the author to take a position as to (1) the existance of such measures and (2) their feasibilty (implementation, cost, subsequent countermeasures -- the eventual failure of essentially every DRM is instructive here). It might be wise to qualify the language with those concerns in mind. 

 

 

 

 

Reviewed by , 01 Nov 2022

I wish to thank the author for addressing or otherwise engaging with all the comments during what has been a very thorough review process.

I am satisifed that all oustanding issues have been rectified and have no further comments regarding potential amendments.

I believe the positionality statement is an excellent suggesiton and has been well implemented.

I would also like to acknowledge that the author can be in something of an impossible position when attempting to compile a reference table for loot box design (see Dr. Karhulahti's comment h). While such a table would, undoubtedly be of interest and use to readers, I believe that such an undertaking would be unduly onerous given the huge range of implementations that exist (as per the example of the game "wild rift" provided by Dr. Karhulahti).


Evaluation round #1

DOI or URL of the report: https://doi.org/10.31219/osf.io/hnd7w

Author's Reply, 23 Oct 2022

Download author's reply Download tracked changes file

Please find my response to recommender and reviewer comments and a compressed version (due to the file size limit on the web upload portal) of the manuscript file with all changes tracked attached below. Thank you. All files (including a clean version of the manuscript with all changes confirmed) are available as one document via https://doi.org/10.31219/osf.io/hnd7w (version 5).

Decision by , posted 29 Aug 2022

Dear Leon Xiao,
 
Thank you for submitting the Stage 2 manuscript. We were lucky to have all three original reviewers to re-review and they have provided valuable comments. I briefly comment on these reviews to help you navigate them, after which I give my own feedback, as before. 
 
1. It’s rare to see three reviewers agree on something, as they do here by explicitly asking a shorter and more focused discussion. While R3 suggests half of the discussion out, R1 & R2 ask individual sections to be removed. I tend to agree: the very benefit of using RRs is that you don’t have to sell the results, but the results speak for themselves. I leave it for you to assess how much to cut, but indeed, please consider moving significant parts of the discussion either to supplements or -- perhaps better -- to a separate article systematically dealing with different sides of loot box regulation.
2. Again, all reviewers agree that the anecdotal Figure 4 is used as a basis for too many not-so-strong discussion points (especially 4.6.2.). Please carefully assess the strength of each section and erase or rewrite. We have no word limits and exploratory analyses are allowed -- but considering the already-large scale, introducing new data (such as user discussions) may not improve the quality of the article but rather leaves it open to criticism. Sometimes less is more.
3. All reviewers also point at the use of unnecessarily strong language. Although I personally believe author voice can/should be present, some expressions (and directions taken in the discussion) are certainly stronger than needed. As the main problem in such voice is that readers don’t know how to interpret that, I suggest something that is a standard in qualitative studies: positionality statement, which describes your own beliefs and context from which the text is coming. This would make your position transparent for readers (while your statement would relate explicitly to the theme of this paper, see some great statements from a different field by another PCI report here). This is a suggestion, not a demand.
4. I agree with R3 that speculating about the possible removal of Game 36 doesn’t belong to results but appendix/footnote/supplement/out. Please also follow R3’s advise and move all exploratory analysis to the appropriate section and clearly explain how simulated casino games were coded e.g., in a readme file, as per TOP guidelines (I can see they’re included in the coding file, but replicators need to know *how* such decisions were made) 
 
My additional comments:
 
a. Please move the Stage 1 link near the abstract, as per PCI RR author guidelines. Also, please exclude from the abstract all points (i)-(viii), which are not based on the collected data and their confirmatory analysis; they are only hypothetical avenues for discussion. We *don’t* want to give readers (esp. stakeholders & media) the false impression that these are supported research findings. Consider using general language, such as "Implications of inefficient regulation are discussed."
b. Please return the Study Design Template, which has been removed. Later this year, PCI RR will also start asking Stage 2 reports to include a new outcome column, which briefly states the result. I suggest adding this column. See example.
c. I know this is not always easy, but the Stage 1 part of the MS is occasionally hard to read (especially for a non-native speaker like myself) due to the chosen past tenses, e.g., pp. 17–18. I suggest carefully rereading the Stage 1 part and proposing updated language edits that are more coherent, e.g., “It was planned to…” or “Author planned to…”, “H4 was decided to be accepted if…” etc. 
d. I didn’t notice this earlier, but on page 4 it states: “this paper omits further Manx law discussion…” However, Manx law *is* discussed later. I give permission to remove this factually incorrect statement or propose alternative action for correction.
e. As one reviewer also notices, there was a promise (p. 9) to approach the Commission about possible loot box licenses, however, there is no such document in the supplements. In fact, on p. 44 there is a sentence (without reference) that appears to conflict with the above: “Belgian gambling law does not allow loot boxes to be licensed at all.” The source for this is likely the meeting noted on p. 23, which also references a license list provided by the Commission. Please clarify and preferably address the communication with the Commission in detail in a separate subsection or supplement (as preregistered). This is highly important information. 
f. Please report the software (Stata?) and version that were used to carry out statistical analysis, as per TOP guidelines. I also recommend reporting confidence intervals and exact p-values (instead of p<).
g. On page 24, exploratory analysis is reported for the debated “social casino games.” The exploratory analysis is based on the alternative view, which does not consider such games as gambling. However, for the analysis, these games have been removed from the data (N=85) instead of classified as non-gambling games. In my view, appropriate analysis would use the original sample, including non-gambling social casino games (N=100). 
h. I don’t have the resources to assess each of the analyzed games myself, but I did review the data/materials and took a closer look at some titles. To pick an example, #100 (Wild Rift) states: “It did not appear possible to directly buy the Poros Energy that is required to open the loot boxes. However, by buying the season pass with premium currency (which can be bought with real-world money), it is possible to obtain those Poros Energy, so the player can indirectly buy the Poros Energy.” Although I have no access to the Belgium version, to my knowledge the season pass provides only indirect rewards, i.e., players earn extra content when playing games. So, if one needs to a) buy a pass to b) play more and c) change one’s rewards to d) earn poro energy to e) finally access a loot box, can we say loot boxes were sold? I’m not claiming “no”, but this brings us back to issues of the method, as discussed in the ongoing debate (“If everything is a loot box, nothing is” etc.). I would’ve hoped the results/discussion to have dedicated more space on making transparent the interpretive challenges, which otherwise end up being criticized in future commentaries and responses. E.g., a table with a breakdown of loot box types/structures and their classification challenges would make the article stronger. 
i. Following the above, the limitations only mention the possibility that the obtained prevalence was lower than true value, while different interpretations could evidently result in the true value being significantly lower as well.
j. I try helping to balance the discussion. First, I am not fully convinced that a ban like this necessarily aims to be immediately effective (page 36: “intended perfect or near-perfect elimination of the risks”), and comparisons of loot boxes to drug and weapon control (p. 33) don’t seem appropriate. That said, the present results should not be deemed conclusive evidence of “ineffectiveness” either. For example: in Finland where I live, our traffic law has demanded cyclists to wear a helmet since 2003, yet with it, it was also explicitly stated by the officials that cyclists won’t be fined for not wearing a helmet. Despite there being no enforcement, helmet use has slowly climbed from 35% to 60%, having potentially saved thousands of lives (1% annual improvement). Was the law ineffective? Considering the near-zero-cost, many would say it is highly effective (if it caused the improvement). The point being: effectiveness is difficult to measure and goes way beyond short-term prevalence. For this reason, it could be a strength for the article to take a more neutral stance in the current early stage, until we have more data. It’s ok to speak of ineffectiveness, as preregistered, but taking limitations seriously can also add to scientific credibility.
k. I don’t know if you decide to save 4.6.3., but some years ago we drafted a small framework explicitly for competitive design ethics. The dynamics of fair design are complexly divided between money, time, skill, luck, and occasion factors (combined), and it feels the discussion section in the MS might not be sufficient for addressing such complexity (e.g., removing loot boxes could also advantage Belgian players because item purchases would no longer depend on luck). 
 
I hope the feedback by the reviewers and myself help making this important study as good as possible. Because these are major revisions, I invite the reviewers again. As usual, I’ll be happy to respond to any questions meanwhile. 
 
Veli-Matti Karhulahti

Reviewed by , 21 Aug 2022

​​​I recommend this paper for publication. The data collected is sufficient to test the stated hypotheses concerning the incidence of loot boxes in Belgian games. The author has employed a thorough approach and offered a well-written account of results.

Though I have some concerns regarding conclusions drawn from the given evidence, especially concerning technical measures employed by Game 50 and the effectiveness/desireability of bans. These might be better addressed by being framed as issues for future study/examination.

2a. The data gathered is sufficient to test the hyptheses and answer the proposed research questions. 

2b - 2d - Sound method and adherence, though I have some concerns as relates to exploratory analyses regarding games 50 and 78. 

2e. As noted above I have two concerns:

Conclusions drawn as to the technical measures in G.o.P 3 (game 50). The text concludes that the likely IP address based block was easily circumventable (by users) through the use of a VPN (or physical travel) and that this explains why the game was ranked in the top 50. However, there are several other possibilities which may not require user efforts to circumvent: the most likely to my mind is that the IP block may be poorly or incompletely implemented. Indeed, the temporary failure of the IP block at the Brussels International Airport (without the use of a VPN) indicates that the block is inconsistent. While the text's conclusion may ultimately be true, it would be worthwhile to look at digital communities to see if VPN and/or other circumvention strategies are advanced/discussed (this would mirror the text's use of consumer reviews to prove frustration). The distinction does matter: the conclusion that VPN/travel would defeat the ban (as shown in testing) is not the same as the conclusion that dedicated users are in fact using those methods.  

The text's conclusion that "If the target [of the ban] is . . . highly dedicated players, then the measure is evidently not working (and realistically cannot ever work)" strikes me as greatly improved by deletion of the parenthetical, because one could imagine systems (IP spoof detection, etc) that could be implemented. A similiar concern arises with "is in fact impossible to achieve)".  

Section 4.6.2 seems a bit too broad on the current evidence. While I appreciate the bold conclusions in this section, the author would to well to further note the need for far greater study to these particular effects. It is hard to conclude as to the balance of harm reduction and diminished game enjoyment (and the negative reviews for Game 50 don't provide a rich enough sample to conclude one way or the other). I am slightly confused by the incidence rate comparison here -- if under an imagined effective ban 99 users have a slightly diminished experience, but 1 user is saved from crippling debt and attendant depression, it would be strange to argue incidence over severity. 

Again, the criticisms above should not be taken as disqualifying. The paper is an impressive read and is a valuable contribution to the literature. It strikes me that the concerns in 2e provide fertile ground for further study, and I would recommend they be framed as such in the current text. 

 

Reviewed by , 09 Aug 2022

Q. 2a. Data collected is suitable for answering all stated hypotheses.

Q. 2b. All consistent with registered report, as per tracked changes.

Q. 2c. Author adhered to registered procedure.

Q. 2d. Additional analyses are justified, sound, and add to the informational content of the paper, with the exception of the user comments highlighted in figure 4 (see below).

Q. 2e. In general, the author’s conclusions are justifiable given the data, however, some of the discussion/conclusion content may require some revision. As I indicated in the first review round, on occassion the author allows his personal opinion to shine through to an extent that it detracts from the objectivity of the work. While I understand the author may feel strongly about certain issues, and is fully entitled to do so, I would reccommend that he adopts a more neutral tone in order that the results speak for themselves. For example, in line 1124 I do not believe there is a need to describe the Belgian regulatory approach as ”extreme”, and the argument loses nothing if this word were removed. 

Additionally, I would query the position that Belgium’s ban leads to ”worse gameplay experience” (section 4.6.2) as this is largely speculative and was supported by just two or three comments from an online forum, the collection of which was not described in any of the preceding sections. Neither does it appear that these comments were subject to any methodological form of analysis, there is no information regarding frequency counts of positive or negative comments, why the source of comments was selected in place of others, etc. Finally, the comments chosen to support this position are taken from a SCG game, as far as I am able to tell, which replicates poker. These are precisely the types of games which prior research has found to have the most problematic relations with traditional gambling, and it necessarilly follows that those who play would like more opportunities to play by purchasing currencies/spins/whatever. The point would be better illustrated by analysing user comments of an explicitly non-gambling game with loot boxes, subject to an appropriate means of data sampling and analysis.

Reviewed by , 24 Aug 2022

User comments

No user comments yet