Recommendation

Has the “ban” of loot boxes eliminated them from Belgian mobile games?

based on reviews by Jason Chin, Joseph Macey and Andrew Moshirnia
A recommendation of:
toto

Breaking Ban: Assessing the effectiveness of Belgium’s gambling law regulation of video game loot boxes

Abstract
Submitted: 07 February 2022, Recommended: 07 April 2022

Recommendation

Paid loot boxes, i.e. randomised monetization methods that are similar to lottery-type gambling, have become prominent features of contemporary gaming (e.g., Macey & Bujić, 2022). Because the design structures of loot boxes vary and the value of their virtual rewards is not always clear-cut, many countries now struggle how to deal with them legally and in practice (see Drummond et al., 2020). Belgium is one of the few countries that have officially interpreted loot box monetization to widely belong under gambling regulation. Mobile games that monetize with paid loot boxes in Belgium should thus apply for a gambling license, and companies should generally not offer paid loot boxes to local underage players at all.
 
In this Stage 1 Registered Report, Xiao (2022) has constructed a careful plan for testing whether the “ban” in Belgium has made the local mobile game market distinct in terms of paid loot boxes. The work builds on a rapidly accumulating literature and evolving methods (e.g., Xiao et al., 2021). The author will carry out a systematic qualitative investigation of the country’s top 100 (iPhone) mobile games to investigate whether paid loot box design components have indeed been removed from the products -- and if not, whether related game companies operate with a required gambling license. Additionally, Xiao (2022) will assess Belgium’s overall paid loot box prevalence in comparison to other countries and carry out a field experiment to test whether players can easily circumvent the local regulation by transporting or downloading different versions of software.
 
The study will produce valuable evidence regarding the effectiveness of loot box regulation in general, and more specifically, the results should be of utmost interest to Belgian legal authorities. To ensure the transparency and validity of the chosen methods as well as upcoming interpretations, the registered report format allowed the research design to be reviewed in three rounds before data collection. Three experts, representing the fields of law and gaming, reviewed the Stage 1 manuscript twice and agreed upon the acceptance of all details. Finally, the recommender carried out a third iteration with further requested revisions, which was followed by in-principle acceptance. 
 

URL to the preregistered Stage 1 protocol: https://osf.io/5mxp6

Level of bias control achieved: Level 6. No part of the data or evidence that will be used to answer the research question yet exists and no part will be generated until after IPA.

List of eligible PCI RR-friendly journals:

References

  • Drummond, A., Sauer, J. D., Hall, L. C., Zendle, D., & Loudon, M. R. (2020). Why loot boxes could be regulated as gambling. Nature Human Behaviour, 4(10), 986-988.
  • Macey, J., & Bujić, M. (2022). "The Talk of the Town: Community Perspectiveson Loot Boxes." In Ruotsalainen et al. (eds), Modes of Esports Engagement in Overwatch (pp. 199-223). Palgrave Macmillan.
  • Xiao, L. (2022) “Breaking Ban: Assessing the effectiveness of Belgium’s gambling law regulation of loot boxes.” Stage 1 Registered Report, in principle acceptance of Version 5 by Peer Community in Registered Reports.
  • Xiao, L. Y., Henderson, L. L., Yang, Y., & Newall, P. W. (2021). Gaming the system: suboptimal compliance with loot box probability disclosure regulations in China. Behavioural Public Policy, 1-27.
Cite this recommendation as:
Veli-Matti Karhulahti (2022) Has the “ban” of loot boxes eliminated them from Belgian mobile games?. Peer Community in Registered Reports, . https://rr.peercommunityin.org/articles/rec?id=168

Evaluation round #2

DOI or URL of the report: https://osf.io/f3yab/?view_only=33e5875516d144ed98509c7871242b31

Version of the report: v4

Author's Reply, 29 Mar 2022

Download author's reply Download tracked changes file

Please find my response to recommender and reviewer comments and the manuscript file with all changes tracked attached below. Thank you. All files are available as one document via https://osf.io/f3yab/?view_only=33e5875516d144ed98509c7871242b31 (version 5).

Decision by , 29 Mar 2022

 Dear Leon Xiao,
 
Thank you for submitting the revised Stage 1 of the manuscript. You’ve done comprehensive work and carefully responded to all requests. All the reviewers are satisfied with the revisions, and I agree that the manuscript is now closer to the IPA. However, I must request a few more revisions before we meet the PCI RR standards. 
 
1. Regarding the philosophy of hypothesis testing in general, there should always be a good reason for testing a hypothesis (PCI RR criterion 1B). Although a lot of relevant information is included in the manuscript, we are still lacking explicit justifications, i.e., *why* does the study expect that each hypothesis will be true. With H1 and H2, for instance, I would agree that because loot boxes are ‘banned’ in the country, there is a good reason to expect virtually no loot boxes at all. But you should explicitly provide such rationale, e.g. “Based on loot boxes being banned in Belgium, there will be no loot boxes in Belgium”. The justification can be inside or before/after the hypothesis, but there must always be a clear justification. 
2. As an example of the above issue: on page 8, after stating the hypotheses, a conflicting interpretation of H1 and H2 seems to occur: “Hypotheses 1 and 2 mean that a Belgian loot box prevalence rate of *less*than or equal to 2% will be found”, while the actual H1 and H2 state a *higher* prevalence. Lower prevalence, again, is suggested on page 13, but higher is suggested in the table at the end of the MS. Please clarify and justify which result is expected. Also, please note that, as currently H6 expects that at least 3 games will include loot boxes, this would conflict with expecting less than 3% loot box prevalence (H1 and H2). Clearly stating the rationale in each case will solve this.
3. Justification is also needed in Type 1 error control. It currently reads: “The rate of 2% was chosen instead of 0% to provide type 1 error control.” But one needs to justify why 0.02 was chosen as a control. I provided one such example in my previous letter (a previous study found 1 false positive, which was doubled to be safe). As I have now read your new commentary paper, which demonstrates no less than 22.9% (!) disagreement rate between two studies, it might be justifiable to use even more control than 2% (one of the examples in the commentary reminds me of the problematics in assessing the level of skill, when evaluating a mechanic to be considered gambling/loot box, see:
 
Lipton, M. D., Lazarus, M. C., & Weber, K. J. (2005). Games of skill and chance in Canada. Gaming Law Review, 9(1), 10-18.)
 
(I mention this paper just it in case it might be useful later)
 
To be clear, it is up to you to decide what error control to use, and you can definitely keep 2%, but whatever number you use, please justify it (at least briefly explain why it was chosen). For these hypotheses, perhaps also address the possibility that some loot boxes operate with a license and, if such evidence is found, they will not count in H1/H2. In general, it could be useful to refer to “unlicensed loot boxes” maybe?  
 
4. Regarding H4 and H5, there seems to be no justification and, as you say, the numbers are based on mere intuition (theoretically, you could craft 101 unique hypotheses for the prevalence and one of them would be true!). Therefore, I must ask you remove these hypotheses (unless a good justification is found). When a research field is not yet at a stage where hypotheses can be crafted based on good existing knowledge, more work is needed before hypothesis testing can be started. See e.g., 
 
Scheel, A. M., Tiokhin, L., Isager, P. M., & Lakens, D. (2021). Why hypothesis testers should spend less time testing hypotheses. Perspectives on Psychological Science, 16(4), 744-755.
 
That said, you can surely provide an estimation of how many loot boxes there might be, but that wouldn’t be hypothesis testing (there would be no hypothesis to test, but RRs can also be used for transparent estimation in order to provide unbiased estimates). 
 
Although this goes beyond the present study, I could imagine that reviewing the global literature and data on gambling regulation in general could yield a reasonable prior concerning the effect of successful regulation strategies. For instance, we do know that illegal gambling exists around the world, but perhaps less so in successfully regulated countries. Whether loot box regulation is useful or successful could be, in the future, based on this previous knowledge of regional (online) gambling regulation. But again, I highlight that using this kind of (or similar) approach in the present study could lead to new issues and the need for new reviews, for which removing H4 and H5 is likely the better option (again: you may still discuss the level of prevalence when the results are known, but you cannot make related *confirmatory* claims).
 
5. Binomial testing in H3 should be ok now and the justification for 0.65 is reasonable. However, I am still concerned about the power analysis. 0.15 has been chosen as the effect size but there is still no justification for why is that effect suitable? So again, we need an explicit justification (PCI RR criterion 1C). What would be the smallest effect that is meaningful for this study and why? The paper I referred to previously also provides a good overview regarding different ways for defining a meaningful effect.  
 
Dienes, Z. (2021). Obtaining evidence for no effect. Collabra: Psychology, 7(1), 28202.
https://doi.org/10.1525/collabra.28202
 
In other words, how few (= how much less than 0.65) loot boxes should there be in Belgium for that number to be relevant at all? This question needs to be answered before statistical power can be calculated. 
 
6. Still related to H3, there seem to be 2-sided and 1-sided tests, both, carried out for no reason. Please delete the 1-sided tests, which are duplicates.
7. Regarding RQ1 and RQ2, I would suggest combining them along the following lines: “Has the Belgian ban succeeded in eliminating paid loot boxes from mobile games?” You can answer this RQ via both H1 and H2. 
8. Regarding RQ3, I would suggest simplifying it along the lines “Has the Belgian ban on paid loot boxes been effective?” This is what you test with H3. 
9. If you remove H4 and H5, you can also remove RQ4. Of course, the results will still include the prevalence rate, so scholars can speculate about the exact effectiveness of the ban post hoc. RQ5 is good.
10.  … going briefly back to the hypotheses, with error control now in H1 and H2, you have rephrased them as “More than two…” Please note that hypotheses are not statistical statements, but they apply to the world in general. Essentially, we’re expecting the absence of loot boxes, and error control only reflects our awareness that testing and methods aren’t prefect. So, the hypotheses can well be “The highest-grossing iPhone games in Belgium do not contain paid loot boxes” -- and this will be accepted even if 1 or 2 do contain loot boxes, but only because we acknowledge the possibility of error in analysis/methods (alternatively you can include the justification directly inside the hypothesis, as exemplified in #1)
11.  The above applies to H3 as well. I would suggest following Macey’s suggested wording with the necessary modification: “Of the highest-grossing iPhone games, fewer will contain paid loot boxes in Belgium than in countries that have not banned loot boxes.”
12.  If you wish, H6 could also be clarified more (but this is optional, as it’s not making explicit statistical statements): “Games known to contain paid loot boxes will continue to offer them for sale even when the phone is within geographical and jurisdictional Belgium.”
13.  Following the above, it is also not clear what outcome will corroborate H6 (or null). E.g., if one game continues to offer loot boxes but 2 games do not, what would be the conclusion? The criteria for interpreting the results for a hypothesis must always be clear.
 
A few smaller notes/suggestions, which may be considered.
 
-       On page 2 it reads: “there are two types of loot boxes” --> perhaps rephrase into “loot boxes can be divided in two types” (because there are dozens of different types of loot boxes)
-       On page 3 it reads: “and therefore does not possess real-world monetary value” --> consider specifying e.g., “direct real-world monetary value” (because accounts can still be sold onward, right?)
-       It reads that “The following hypotheses will be preregistered at <[OSF registry link]>” but since this is an RR, you don’t need to separately register hypotheses.
-       On page 11 it reads: “A ‘paid loot box’ will be defined as being either an Embedded-Isolated random reward mechanism or an Embedded-Embedded random reward mechanism” --> please explain to the reader what these concepts mean
-       On page 11 it reads: “95.4% of games were coded through gameplay and only 4.6% of games had to be coded through internet browsing.” Does this refer to games or games with loot boxes? As the % of how many games in general must be coded via internet depends on the prevalence rate (e.g., with 0.5 prevalence one would need to code 50% of data with internet), the % of loot boxes found would be more informative.
-       On page 15, I would still suggest removing the following sentence: “… and conclude that the Belgian measure was likely ineffective.” Already the anecdotal evidence cited in the manuscript shows that the ban has had some effect (= some companies adjusted their design), so it feels wrong to conclude that the measure was likely ineffective, unless direct evidence is found (and justified what effectiveness would be). The section reads very well otherwise, so I suggest just dropping this sentence. 
 
I hope this feedback is helpful and I believe we are very close to IPA after the above revisions have been implemented. Please let me know if any of the comments are unclear -- I will be happy to clarify.
 
-       Veli-Matti Karhulahti

Reviewed by , 21 Mar 2022

I would like to thank Mr. Xiao for his revisions. My concerns have been addressed and I reiterate my recommendation of the piece. 

The Dutch reversal is an interesting addition and might we worth investigation if firms perceive uncertainty in the law and the likelihood of leniety based on that uncertainty. 

The Minecraft and Roblox discussion is necessary and well addressed. Of course it opens a whole new line of inquiry (merging modding/playbour with loot-box/consumer protection), but one outside the scope of the current experiment. 

Reviewed by , 18 Mar 2022

I enjoyed reading the response of the author and the other reviewers' reviews. The author responded satisfactorially to all my comments and questions. 

There were methological issues I missed in my initial review due to my lack of knowledge about the subject matter (e.g., the third party involment issue). I am happy to leave it to the editor, reviewers, and author to work out those details.

I always sig my reviews,

​Jason Chin (ORCID: 0000-0002-6573-2670)​​

Reviewed by , 24 Mar 2022

I thank the author for their comprehensive response to the comments of myself and the other reviewers, I believe all points have been addressed and am happy to accept the revised submission.

 

A small note, in their response to my 2nd point, the author queried as to whether further action is required:

"For transparency, I would be happy to add that “I have argued elsewhere…” before the section on overregulation (i.e., immediately following the section quoted above), if Dr Macey may think that would improve the fair presentation of the arguments."

I do not believe any further changes are neccessary, as I feel my original concern was appropriately addressed and is no longer valid, as such the author is free to make any further changes based on their own judgement.


Evaluation round #1

DOI or URL of the report: https://osf.io/8fvt2/

Author's Reply, 28 Feb 2022

Download author's reply Download tracked changes file

Please find my response to reviewer comments and the manuscript file with all changes tracked attached below. Thank you. All files are available as one document via https://osf.io/f3yab/?view_only=33e5875516d144ed98509c7871242b31 (version 4).

Leon Y. Xiao

Decision by , 28 Feb 2022

Dear Leon Xiao,
 
Thank you for submitting your Stage 1 manuscript to PCI RR. To my knowledge, this is the first RR in the domain of law, and as such a highly interesting manuscript to handle. I have now received all three reviews, collectively representing expertise of gaming and law as well as the related methods. The reviews are very positive, but also highlight issues that need revision. In general, the reviewers are consistent and do not express conflicting views, for which I will merely follow-up on some of their points and add a few comments of my own. I start by moving chronologically through the MS with minor issues, and in the end, I discuss some bigger methodological issues.
 
1.     On page 2, I would remove the part “rather than, e.g., wealthy players” because wealthy players can also be at-risk players. Later the same page, a parenthesis is not closed (“and therefore be…”).
2.     On page 5, you introduce Netherlands for the first time, while previously addressing only Manx and UK law. It would improve readability to briefly note earlier that Netherlands is also a candidate (yet its role differs from the other two). 
3.     On page 6, you introduce (i) and (ii), but as one reviewer points out, they are not stated as RQs. Please reformulate them into explicit RQs. When doing that, carefully ensure that your hypotheses/methods match with and able to answer the RQs. If you have a reason to do otherwise, please explain in the response letter. 
4.     On page 7, you note how the Belgian Gaming Commission will be contacted and their response discussed. I expect the response will take time and it is possible that you will not have it by the time of Stage 2 review; thus, I suggest obtaining a permission to share their response publicly and storing it in the OSF when it comes (if after Stage 2). Alternatively, if no permission for sharing is gained, you could add a summary to the OSF so that future readers will find the information via DOI. 
5.     On the same page, a small typo (just before “because”)
6.     Methods: as the reviewers point out, the time of data collection is very critical. If possible, I would suggest finding out the Top 100 list in Belgium for the compared time (June 2021). While post hoc analysis is not possible, it would at least allow the reader to assess the fluctuation of the titles on the list (and perhaps you to address that briefly in the discussion, if relevant).
7.     On page 10, you say that “game will be assumed by the coder to contain paid loot boxes without the need for the coder to identify and screenshot such a mechanic.” I might have missed something, but I don’t see how third-party involvement would automatically ensure that loot boxes are present. E.g., if there is a known avenue for generating paid loot boxes in sandbox games that cannot be interfered by companies, please cite that.
8.     On page 13, you have the ethics statement “No ethics approval will be required because the present study examines and records publicly available information.” Please elaborate, according to what university/country. Different universities and countries have difference ethics policies (e.g., according to the ethics policy of IT University of Copenhagen and the Danish Code of Conduct for Research Integrity, the study did not require ethics assessment).   
 
Methods
 
a)    I notice that the prevalence of 0.77 comes from a preprint that has not been reviewed yet. I am flagging it because if the paper remains un-reviewed at Stage 2, or if its peer review ends up affecting the result 0.77, there is a chance that this RR would have to be rejected based on criterion 2B (changes in hypothesis). As it is your own co-authored paper, we can proceed without changes; however, you should be aware that at Stage 2, if the results of the cited paper are still pending, we possibly cannot provide IPA.
b)    Related to the above, I can see that in a previous study Zendle et al. (2020) found 0.59 in the UK, and this number is from 2019. Although I understand that you prefer to use a more recent prevalence rate, we do not have evidence that the change is due to time alone, i.e., there can be variation in samples for other reasons, too. Considering that your to-be RQs are interested in whether Belgium has a lower rate vs other countries, and previous studies have found UK 0.59/0.77, Australia 0.62, and China 0.91, a bit more justification is needed why 0.77 has been selected, and as one reviewer points out, why would 0.4 be a remarkably low prevalence.
c)     There in issue with the method for testing H3. You are planning to use the binomial test, which assumes that the Belgian Top 100 list provides a random variable, but the compared UK Top 100 list is fixed. Yet since both lists produce outcomes as similar random variables, what you seem to need for testing is a 2x2 contingency table. 
d)    I also encourage thinking about the comment from one reviewer regarding the effect, i.e. what effect would be societally beneficial for a regulation like this to be useful in practice. Depending on how you proceed, you will then also need to recalculate power, depending on what your final RQ + hypotheses + method is. Please note that PCI RR does not demand any particular power as long as it is justified; however, some journals do, so you should double check that if you have a specific journal in mind (see the next point). 
e)    I would also like to highlight some parts of the conclusions. On page 12, it says: “if no significant difference is found, then the present study will conclude that the Belgian ban did not appear to affect paid loot box prevalence in Belgium, thus disconfirming Hypothesis 3. The present study will then conclude that the measure is likely ineffective and should not be adopted by other countries.” However, not being able to find effect is not the same thing as finding evidence for no effect. You cannot conclude ineffectiveness based on non-significance alone. If you want to obtain evidence for no effect, see e.g., Dienes (2021)
 
Dienes, Z. (2021). Obtaining evidence for no effect. Collabra: Psychology, 7(1), 28202.
https://doi.org/10.1525/collabra.28202
 
f)     This is a small issue, but in H1 and H2, one reviewer notes how absolute null might not be optimal, and I agree some type 1 error control would be appropriate here. I would suggest keeping it simple, e.g., considering that Zendle et al. (2020) found 1 false positive, you could just double that to be safe and corroborate H1/H2 if more than 2 cases occur (more than 2% prevalence). Although confirming the positives should be rather easy due to the sample size, some control seems reasonable because many things can affect the obtained the sample. If you wish, you may add that in case H1 or H2 is not corroborated but 1 or 2 instances are found, these games will be investigated in-depth as an exploratory analysis. I also encourage you to consider setting alternative/competing hypotheses, as suggested by one reviewer (but you can choose not to, if you so prefer). 
 
Finally, I must ask you to revise the table at the end of the MS. Please include all tested hypotheses, and carefully think in each case what can and cannot be deduced from their outcomes. In addition to all the above, please see and respond to the reviewers’ respective feedback. Needless to say, if you disagree with some the requested revisions, you are free to justify alternative choices. Do not hesitate to contact me if something is unclear. I look forward to reading the next version, based on which I will see if another external review round is needed.
 
Sincerely, 
Veli-Matti Karhulahti

Reviewed by , 10 Feb 2022

Thank you for the opportunity to review this paper at stage 1. I am familiar with Dr. Xiao's work in this area. Overall, I would heartily recommend this experiment, with some slight revisions. I note these below:

In reviewing the document, I flagged the assertion that a value out mechanism would constitute gambling under most laws/national codes. This statement should be softened, as the value out mechanism would not become gambling provided that there is guarenteed value of the purchase in any event (this is the legal manuever that renders collectibe card cards legal and non-gambling, even if there is a secondary market in which cards may be exchanged for value). 

The research question makes sense: an interpretation of law has been announced and in theory compliance should be absolute or near absolute. The hypotheses are perhaps too strict to test this, however, as near perfect compliance (presence of 2% of top 100 containing loot boxes) would return the same hypothesis rejection as complete refusal to comply (presence of 98% of top 100 containing loot boxes). In light of this it may be useful to insert an alternate hypothesis (with a cut off of 2% or 5%) rather than an abolute (as currently stated), because the rejection of the null may lead to less meaningful post-hoc if these alternates are not established before-hand. 

Hypothesis #3 is a simple comparative and I agree with the approach (one-tailed) and the method. 

Hypothesis #4 is an interesting question based primarily on what terms of service will control (locality of play or locality of installation). The method may be improved by also setting the locality of the phone to Belgium through the OS, but this may not be needed. 

The sample size is sufficient to provide meaningful results. 

The use of 1 hour of game play is a reasonable means of arriving at a result, but the author is surely aware that some games' loot boxes function primarily as forgiveness or pity counters (that is, the player is losing frequently so loot box item is offered to increase victory chances). The 1 hour of play should then include deliberate losses by player to solicit these offers (if present). This may account for prior interrater disagreements. 

 

 

Reviewed by , 17 Feb 2022

1A. The scientific validity of the research question(s). 

-        No research question is explicitly presented by the author(s), instead the aims of the research are presented in the body of the text, for example:

“Given that there is significant interest in emulating this regulatory approach, it is important to assess whether this Belgian ‘ban’ on loot boxes has been effective.”

And

“… a survey replicating the methodology of previous loot box prevalence studies [3–5] will be conducted in Belgium to assess: (i) the effectiveness of the Belgian Gaming Commission’s threat to criminally prosecute video game companies for

implementing paid loot boxes without a gambling licence (i.e., the Belgian ‘ban’) [44] and (ii) whether the loot box prevalence rate in Belgium is consequently lower than in other Western countries where no loot box regulation has been enforced, e.g., the UK. Doing so sheds light on whether the Belgian ban has effectively changed video gaming companies’ behaviour.”

Whilst the research is both well-designed and justified, it would benefit from the research question(s) being clearly presented, in lines with the requirements of PCI RR.

 

Although not directly connected to the validity of research questions, I would urge the author(s) to revise the following content:

“The restrictive course of action taken by … who would never have been harmed [58].”

In its current form, the language used clearly reflects the author(s) opinion rather than a neutral assessment of arguments supporting or opposing the Belgian approach (further emphasised by the use of 2 prior papers by the same author to support the statements).

 

1B. The logic, rationale, and plausibility of the proposed hypotheses, as applicable.

-        The 4 hypotheses are appropriate and the rationale for their development is logical and easy to follow. However, H3 would benefit from some minor revision as the way it is presented may cause confusion to some readers. I would suggest something along the lines of:

“Of the highest-grossing iPhone games, fewer will contain paid loot boxes in Belgium than in the UK.”

Of course, the author(s) are free to make any changes, or not, as they see fit.

 

1C. The soundness and feasibility of the methodology and analysis pipeline (including statistical power analysis or alternative sampling plans where applicable). 

-        The described methodology, including sampling procedures, variables recorded, and analytical approach appears feasible and well-planned. The sample sizes used to address the aims of the research appear to be more than sufficient. However, the justification for selecting the 3 games referenced in H4 could be further expanded. The fact that they represent offerings from game companies in 3 different regions (US, Europe, and China) is appreciated, but the reader would benefit from a more detailed explanation of why these particular games were chosen; are they the highest-ranked examples from each chosen reason (either in terms of number of players, or of revenue raised) or did other considerations guide the author(s)?

-        The authors describe the analytical approach and the conditions under which the different hypotheses will be considered to be a) met, or b) rejected. In reference to H3 the authors are frank in their presentation when they discuss how the presented methods cannot offer a clear assessment of a), they state that any conclusion will be discussed in terms of “possibility” that Belgian legislation affected paid loot box offerings. While it is likely impossible that author(s) will be able to access earlier versions, it may be worth supplementing the game analysis with additional analysis of company statements (if any) regarding their reaction to the Belgian legislation.

 

1D. Whether the clarity and degree of methodological detail is sufficient to closely replicate the proposed study procedures and analysis pipeline and to prevent undisclosed flexibility in the procedures and analyses. 

-        The method of data collection and analysis is clearly presented and comprehensible, allowing replication.

 

1E. Whether the authors have considered sufficient outcome-neutral conditions (e.g. absence of floor or ceiling effects; positive controls; other quality checks) for ensuring that the obtained results are able to test the stated hypotheses or answer the stated research question(s).

-        The author(s) state that only a single researcher will be coding the data sample, using prior studies to justify the approach. Given the nature of the analysis required, and the conditions under which coding will be conducted, this is likely to be acceptable.

Reviewed by , 27 Feb 2022

User comments

No user comments yet