Loot boxes remain prevalent in Belgium despite their “ban”
Breaking Ban: Belgium’s ineffective gambling law regulation of video game loot boxes
Recommendation: posted 14 November 2022, validated 14 November 2022
Karhulahti, V. (2022) Loot boxes remain prevalent in Belgium despite their “ban”. Peer Community in Registered Reports, 100264. https://doi.org/10.24072/pci.rr.100264
This is a stage 2 based on:
URL to the preregistered Stage 1 protocol: https://osf.io/5mxp6
Level of bias control achieved: Level 6. No part of the data or evidence that was used to answer the research question existed prior to Stage 1 in-principle acceptance.
List of eligible PCI RR-friendly journals:
- Addiction Research & Theory
- Peer Community Journal
- Royal Society Open Science
- Swiss Psychology Open
The recommender in charge of the evaluation of the article and the reviewers declared that they have no conflict of interest (as defined in the code of conduct of PCI) with the authors or with the content of the article.
Evaluation round #2
DOI or URL of the report: https://doi.org/10.31219/osf.io/hnd7w
Version of the report: v5
Author's Reply, 07 Nov 2022
Decision by Veli-Matti Karhulahti, posted 06 Nov 2022, validated 06 Nov 2022
Dear Leon Y. Xiao,
Thank you for the carefully revised manuscript. All reviewers are widely satisfied with the revisions. You will have an opportunity to address their last minor suggestions, after which I will write the recommendation. My own minor points:
P. 19: the Stage 1 text has been correctly moved to the first page and is no longer needed here.
P. 20: I still believe that presenting confidence intervals would be appropriate, at least with the main prevalence rates in Table 1 (but this remains a suggestion and is not required for recommendation).
P. 23: as the reviewers didn’t express concern for social casino games, we can leave it as it is. Nonetheless, it’s important to keep in mind that the exploratory outcome of 3.2.2 will not be cross-nationally comparable, e.g., with the previous studies that motivated this extra analysis (Zendle et al., 2020; cf. Zendle et al., 2022), as they negatively coded such products. This could be highlighted (again, only a suggestion).
#732: “The Belgian Gaming Commission does recognise the randomised monetisation methods in ‘simulated casino games’ games as constituting ‘gambling’..” This is lacking a citation. Some other parts of the discussion are also relying strongly on “general knowledge” without citation (e.g., “obvious loot box contraventions being … highly popular” #1187). Please consider adding further references to back up such statements.
#751: if I understand correct, here “p = 0.05” refers to the alpha level, right? If yes, please correct and/or state earlier that an alpha level 0.05 was used for all analyses (also #466 and #702).
P. 39: Section numbers 4.8 onwareds seem to be incorrect.
One final note -- this doesn’t need a reply if deemed irrelevant, but please consider. Because it remains unclear how the Belgian ban should work or what are its goals/premises (as it was never properly implemented to begin with), some statements in the discussion like “Belgian ban on loot boxes is not working at present” (#1459) might still be better phrased. For example, the gambling monopoly of Finland is based on a specific system of ethics: money generated in gambling by Finnish companies should be directed to charity (thus only government companies are allowed provide local gambling services to ensure the rule is followed). As it remains unknown what the gambling regulation principles in Belgium are, it seems also possible that -- if they’d be based on similar local corporate control (rather than people control) -- it may also have, at least partially, succeeded if no Belgian game companies offer loot boxes? In this case, notions such as “circumventing” the ban wouldn’t fully apply either (that said, the quote of Article 4(2) from 1999 seems to imply that some people control is involved). Just a thought.
As there will be no further review after the next final version, I can be contacted directly before submission if there is unclarity about how to respond to any of the points.
Reviewed by Jason Chin, 04 Nov 2022
Reviewed by Andrew Moshirnia, 31 Oct 2022
Reviewed by Joseph Macey, 01 Nov 2022
Evaluation round #1
DOI or URL of the report: https://doi.org/10.31219/osf.io/hnd7w
Author's Reply, 23 Oct 2022
Decision by Veli-Matti Karhulahti, posted 29 Aug 2022
Dear Leon Xiao,
Thank you for submitting the Stage 2 manuscript. We were lucky to have all three original reviewers to re-review and they have provided valuable comments. I briefly comment on these reviews to help you navigate them, after which I give my own feedback, as before.
1. It’s rare to see three reviewers agree on something, as they do here by explicitly asking a shorter and more focused discussion. While R3 suggests half of the discussion out, R1 & R2 ask individual sections to be removed. I tend to agree: the very benefit of using RRs is that you don’t have to sell the results, but the results speak for themselves. I leave it for you to assess how much to cut, but indeed, please consider moving significant parts of the discussion either to supplements or -- perhaps better -- to a separate article systematically dealing with different sides of loot box regulation.
2. Again, all reviewers agree that the anecdotal Figure 4 is used as a basis for too many not-so-strong discussion points (especially 4.6.2.). Please carefully assess the strength of each section and erase or rewrite. We have no word limits and exploratory analyses are allowed -- but considering the already-large scale, introducing new data (such as user discussions) may not improve the quality of the article but rather leaves it open to criticism. Sometimes less is more.
3. All reviewers also point at the use of unnecessarily strong language. Although I personally believe author voice can/should be present, some expressions (and directions taken in the discussion) are certainly stronger than needed. As the main problem in such voice is that readers don’t know how to interpret that, I suggest something that is a standard in qualitative studies: positionality statement, which describes your own beliefs and context from which the text is coming. This would make your position transparent for readers (while your statement would relate explicitly to the theme of this paper, see some great statements from a different field by another PCI report here). This is a suggestion, not a demand.
4. I agree with R3 that speculating about the possible removal of Game 36 doesn’t belong to results but appendix/footnote/supplement/out. Please also follow R3’s advise and move all exploratory analysis to the appropriate section and clearly explain how simulated casino games were coded e.g., in a readme file, as per TOP guidelines (I can see they’re included in the coding file, but replicators need to know *how* such decisions were made)
My additional comments:
a. Please move the Stage 1 link near the abstract, as per PCI RR author guidelines. Also, please exclude from the abstract all points (i)-(viii), which are not based on the collected data and their confirmatory analysis; they are only hypothetical avenues for discussion. We *don’t* want to give readers (esp. stakeholders & media) the false impression that these are supported research findings. Consider using general language, such as "Implications of inefficient regulation are discussed."
b. Please return the Study Design Template, which has been removed. Later this year, PCI RR will also start asking Stage 2 reports to include a new outcome column, which briefly states the result. I suggest adding this column. See example.
c. I know this is not always easy, but the Stage 1 part of the MS is occasionally hard to read (especially for a non-native speaker like myself) due to the chosen past tenses, e.g., pp. 17–18. I suggest carefully rereading the Stage 1 part and proposing updated language edits that are more coherent, e.g., “It was planned to…” or “Author planned to…”, “H4 was decided to be accepted if…” etc.
d. I didn’t notice this earlier, but on page 4 it states: “this paper omits further Manx law discussion…” However, Manx law *is* discussed later. I give permission to remove this factually incorrect statement or propose alternative action for correction.
e. As one reviewer also notices, there was a promise (p. 9) to approach the Commission about possible loot box licenses, however, there is no such document in the supplements. In fact, on p. 44 there is a sentence (without reference) that appears to conflict with the above: “Belgian gambling law does not allow loot boxes to be licensed at all.” The source for this is likely the meeting noted on p. 23, which also references a license list provided by the Commission. Please clarify and preferably address the communication with the Commission in detail in a separate subsection or supplement (as preregistered). This is highly important information.
f. Please report the software (Stata?) and version that were used to carry out statistical analysis, as per TOP guidelines. I also recommend reporting confidence intervals and exact p-values (instead of p<).
g. On page 24, exploratory analysis is reported for the debated “social casino games.” The exploratory analysis is based on the alternative view, which does not consider such games as gambling. However, for the analysis, these games have been removed from the data (N=85) instead of classified as non-gambling games. In my view, appropriate analysis would use the original sample, including non-gambling social casino games (N=100).
h. I don’t have the resources to assess each of the analyzed games myself, but I did review the data/materials and took a closer look at some titles. To pick an example, #100 (Wild Rift) states: “It did not appear possible to directly buy the Poros Energy that is required to open the loot boxes. However, by buying the season pass with premium currency (which can be bought with real-world money), it is possible to obtain those Poros Energy, so the player can indirectly buy the Poros Energy.” Although I have no access to the Belgium version, to my knowledge the season pass provides only indirect rewards, i.e., players earn extra content when playing games. So, if one needs to a) buy a pass to b) play more and c) change one’s rewards to d) earn poro energy to e) finally access a loot box, can we say loot boxes were sold? I’m not claiming “no”, but this brings us back to issues of the method, as discussed in the ongoing debate (“If everything is a loot box, nothing is” etc.). I would’ve hoped the results/discussion to have dedicated more space on making transparent the interpretive challenges, which otherwise end up being criticized in future commentaries and responses. E.g., a table with a breakdown of loot box types/structures and their classification challenges would make the article stronger.
i. Following the above, the limitations only mention the possibility that the obtained prevalence was lower than true value, while different interpretations could evidently result in the true value being significantly lower as well.
j. I try helping to balance the discussion. First, I am not fully convinced that a ban like this necessarily aims to be immediately effective (page 36: “intended perfect or near-perfect elimination of the risks”), and comparisons of loot boxes to drug and weapon control (p. 33) don’t seem appropriate. That said, the present results should not be deemed conclusive evidence of “ineffectiveness” either. For example: in Finland where I live, our traffic law has demanded cyclists to wear a helmet since 2003, yet with it, it was also explicitly stated by the officials that cyclists won’t be fined for not wearing a helmet. Despite there being no enforcement, helmet use has slowly climbed from 35% to 60%, having potentially saved thousands of lives (1% annual improvement). Was the law ineffective? Considering the near-zero-cost, many would say it is highly effective (if it caused the improvement). The point being: effectiveness is difficult to measure and goes way beyond short-term prevalence. For this reason, it could be a strength for the article to take a more neutral stance in the current early stage, until we have more data. It’s ok to speak of ineffectiveness, as preregistered, but taking limitations seriously can also add to scientific credibility.
k. I don’t know if you decide to save 4.6.3., but some years ago we drafted a small framework explicitly for competitive design ethics. The dynamics of fair design are complexly divided between money, time, skill, luck, and occasion factors (combined), and it feels the discussion section in the MS might not be sufficient for addressing such complexity (e.g., removing loot boxes could also advantage Belgian players because item purchases would no longer depend on luck).
I hope the feedback by the reviewers and myself help making this important study as good as possible. Because these are major revisions, I invite the reviewers again. As usual, I’ll be happy to respond to any questions meanwhile.