Dear Philip Newall and co-authors,

I have now received the review of your revised PCI-RR submission from one reviewer who also reviewed your initial submission. Based on their and my own evaluations (below), I am recommending a revision of this paper. Satisfactory responses to the reviewer (and editor) comments is likely to result in a Recommendation.

Reviewer 1 makes two particularly good points which I also endorse. Regarding the reproducibility of the power analyses for H1 and H2, I suggest that you upload supporting information to the OSF showing how you arrived at these numbers -- having ran into stumbling blocks myself with this, I now upload these to remind myself exactly what I did. Regarding the equivalence tests, Reviewer 1 is correct that these are informative (and should be conducted) for both significant and non-significant effects (to discern a significant result from a practically meaningful one).

Here is my own review of the manuscript; please address these comments along with those of Reviewer 1s.

Editor Comments/Review

1.       The Abstract is rather confusing because it first reads as though you will test two conditions of the house edge format with 2000 participants, but then later (after explaining the DVs), states that you will also compare this to an existing suboptimal format with 1000 gamblers. It would be clearer to state the full design within the Abstract: specifically, the three conditions, the total sample size recruited (3000?) and then the DVs.

2.       Introduction, Page 4, Line 5: I suggest removing the term “Similarly” to improve this sentence (“Gambling is another public health issue…”)

3.       Page 4, Line 9-11: You state that “However, by comparison, gambling information can be criticized on the grounds of a lack of prominence, and suboptimalities with which it is communicated (Newall, Walasek, et al., 2022). This Registered Report contributes to this second issue, by experimentally testing two different phrasings of relevant information about gambling products”, but it’s not clear that you are doing this in order to get over this suboptimality. Could you end this sentence by clarifying why you are doing this/why it’s important?

4.       The Introduction provides some examples of the return-to-player and the house edge communication, but these odds seem very high in favour of the gambler (“This game has an average percentage payout of 90%”). Are these factual examples? Are the odds sometimes this high in gambling?

5.       Page 5, Line 6 states “However,  seeing as how replication is an important aspect of gambling psychology research (Heirene, 2021), a secondary aim of the present research is to attempt to replicate findings on rates of understanding and perceived chances of winning from the original studies on this topic (Newall, Walasek, & Ludvig, 2020a, 2020b)”, but this is quite confusing because you haven’t yet outlined the main aim specifically.

6.       As per my comment about the Abstract, Page 6 outlines the present research but does not make it clear that there are actually three conditions, with the two phrasings of ‘house edge’ also being compared to a ‘return-to-player’ condition. Can you make this clearer? The reader needs to be able to fully understand the design at this point.

7.       Please also give the manuscript a thorough proofread – there are some instances of missing words (e.g., Page 7, ‘the’ in the sentence ‘as this is the closest round number which exceeds the required sample size\e in each of the below power analyses”.

8.       Please note that at Stage 2, the approved Stage 1 text cannot be revised (unless there is a significant error or you are changing tense). For this reason, ensure that there is no information that you’d wish to remove or change in some way at Stage 2: e.g., the sentence that states “For the reviewers, a link to the experiment is here: https://bristolexppsych.eu.qualtrics.com/jfe/preview/previewId/b380c47d-7c9a-4128-a469-65082fabdabe/SV_b1TvgkAn3B7Er7U?Q_CHL=preview&Q_SurveyVersionID=current”

9.       What is the rationale for a change in H1’s outcome from 4.1 to 3.8/4.4 (representing an effect size of 0.188) and change in accuracy of 6% (representing an effect size of d = 0.133. Put simply, why an effect of 0.188/0.133 and not any other (small) effect? You should provide a rationale for your smallest effect size of interest (SESOI).

With best wishes,

Charlotte

Dear Philip Newall and Olivia Maynard,

Thank you for submitting your Stage 1 Registered Report “How does the phrasing of house edge information affect gamblers’ perceptions and level of understanding? A registered report” for consideration by PCI Registered Reports.

I have now received comments from three expert reviewers in this field. As you will see, these reviews are overall positive and based on these reviews and my own assessment, I would like you to revise your manuscript accordingly. You will see that the majority of the reviewer comments are minor, but I would like you to pay attention to the following, which will be a particular focus on re-review. These two criteria are essential for Recommendation at PCI Registered Reports. 

Review criteria 1C. The soundness and feasibility of the methodology and analysis pipeline (including statistical power analysis or alternative sampling plans where applicable).

1.    Please ensure that the power analysis is consistent with the analysis being used: as one Reviewer points out, it seems as though the power analysis is based on M/SD and an effect size of Cohen’s d, but the main planned analysis is a regression which is inconsistent with this. [but also, see Point 3 below which needs to be considered in addressing this]. Importantly, your plan is to compare two groups based on the phrasing of the message, but the justification of your power analysis is not currently based on this and is instead based on an average point on a Likert scale. Perhaps I am misunderstanding something here, which could simply be clarified, but again the power analysis should be based on the specific analysis you plan to conduct. 

2.    You state that the sample size would be able to “detect even relatively small effects”, which is ambiguous – what specific small effect could be detected? Have you considered a Smallest Effect Size of Interest (SESOI) and powered your study according to this?

3.    As per the Reviewer comment, the study is not sufficiently powered for the ‘understanding’ variable. An explicit reason should be given for this, which can also include resource limitations (i.e., funding/time), but must be apparent.

4.    You need to ensure that your data will still be informative if you arrive at a null result (e.g., p > .05) rather than stating that it is not “statistically significant/different”. This requires either Bayesian analyses or equivalence tests. Please see our Stage 1 criteria for guidance on this, which includes supporting references that will be of use. Note that this will have implications for your power analysis too: a power analysis should be based on the analysis tests you will conduct.

Review criteria 1E. Whether the authors have considered sufficient outcome-neutral conditions (e.g. absence of floor or ceiling effects; positive controls; other quality checks) for ensuring that the obtained results are able to test the stated hypotheses or answer the stated research question(s).

1. You will see from the Reviewers that there are different perceptions of including an attention/manipulation check in this study. On one hand, the study is short which allows the Authors to speculate that participants will be attentive. On the other hand, we know that data quality from online crowdsourcing platforms can be impacted by careless responders (see Jones et al., 2022). This should be carefully considered. One reviewer suggests adding an attention check at the end of the experiment which asks participants to select the statement they have just seen and to analyse the data with those that fail this included and excluded. I agree with this, but also highlight that you will need to consider such exclusions in your sample size planning: in Jones et al. (2022), we estimated that around ~12% of participants were careless responders through failed attention checks or implausible response times, which can give you a base estimate for your own study.

Reference:

Jones et al. (2022). Careless responding in crowdsourced alcohol research: A systematic review and meta-analysis of practices and prevalence. Experimental & Clinical Psychopharmacology, 30, 381-399. Link: https://pubmed.ncbi.nlm.nih.gov/35130007/ OA: https://psyarxiv.com/rs9xh  

2. The third Reviewer also makes an important point regarding the replication of the original framing difference. Whilst acknowledging that this will increase the sample size considerably, a justification of the exclusion of this control condition is required. The Authors may want to list this as a limitation in their Discussion section at Stage 1, showing that they considered this now and not later in the review process.

3. I agree with the Reviewer that you should run some simulated data to check your analysis pipeline and to document the analysis syntax. This avoids problems later down the line at Stage 2 if a planned analysis doesn’t seem appropriate and/or reduces any errors at the planning stage.

Other minor points:

- I am unsure whether you’ve considered the journal you’d like to submit this too given a positive Stage 2 acceptance but wanted to flag that your manuscript is not consistent with APA 7th edition style. For example, anything over 2 authors can now be stated as ‘et al.’ if you are planning to use this formatting style.

- Under design, the instructions to participants state: ““Imagine that you are a member of an online casino. You have played many of this online casino’s games over the last year.” Shouldn’t this be ‘these’ (online casino games)?

-  Please refer to Table 1, the design table, within the manuscript itself and also provide a title for this. Previous Stage 1 accepted Registered Reports can help with guide you with this.

- This links back to what I state under Review criteria 1C, point 1, but in the design table I do not understand the following sentence: “In order to detect a reduction on this outcome from 4.1 (see main Methods) to 3.8 (SD = 1.6), with 95% power and an alpha of 0.05 requires 741 participants in each condition”. This reads as though this is a within participant design where the phrasing of the sentence reduces a Likert point average from 4.1 to 3.8, but this is not the case given your design is between-participants with no baseline measures. Can you clarify this throughout the manuscript, please? It may be that you change your power analysis given the points above, which would mitigate this.

- A minor note is that the term ‘Registered Report’ is usually written in capitals and there are numerous times that you use this term (in lowercase) in your manuscript.

Please note, you will need to download some of the reviewer’s comments from the PCI-RR portal; others will be shown without the need to download a file.

I look forward to receiving your revision in due course. 

Yours sincerely,

Dr Charlotte Pennington