Expanding the Intervention Potential of Tabletop Role-Playing Games

Dear Authors,

Three reviewers have generously provided detailed rapid feedback, considering your hard deadline. They are all positive, but some critical things need to be carefully considered. The MS sits between an exploratory pilot and a confirmatory intervention: a key goal is to explore feasibility, but there are also hypotheses to be tested. As reviewers point out, hypothesis testing would require solid corroboration/falsification rules and clarity when success would be left undecided. A complete data analytic plan regarding how efficacy will be measured would be needed for assessing hypothesis testing. It also remains possible to register this as an exploratory pilot, in which case evaluation is more flexible (but you cannot make confirmatory claims at Stage 2). Although I personally see the exploratory option most feasible -- especially considering your time limit -- below is a list to help you revise if you wish to pursue hypothesis testing (skip this if you choose the exploratory path). 

1. There are discrepancies between the hypotheses on p. 8- and the expected outcomes on p. 22-. E.g., PO1 concerns gaming frequency, but this is not among the previously named hypotheses. It’s important to consistently justify each hypothesis; you may also set expectations without testing them (= no confirmatory claims at Stage 2), but they need to be clearly distinguished from tested hypotheses.
2. Justify the smallest effect of interest. Currently only the term “reduction” is used, but we need to be more specific. E.g., reduction of gaming by 1min/day would hardly be meaningful. Each effect/hypothesis used for confirming effectiveness needs a justification, respectively. See e.g., Anvari et al. (2022; https://doi.org/10.1177/17456916221091).
3. All outcomes are currently expected both at the end of the TTRPG-based program (P1A) and at the 3-month follow-up (P1B). We need to agree beforehand which of these, or what combination thereof, corroborate/falsify hypothesis. E.g., what if we see no reduction at P1A but reduction at P1B, would this corroborate hypotheses? 
4. Considering that some effects will not be meaningful, please specify when the result will be considered null, i.e. what are the results that will conclude the intervention had no meaningful effect or a non-meaningful effect.
5. Carefully consider how dropouts are assessed. E.g., what if you have 50% (10/20) dropouts and find meaningful effects in the remaining participants, would this be considered corroborating hypotheses? 
6. What about missing data, e.g., if a participant fails to deliver P1B data, will this be considered a dropout? What is the overall rule structure, considering all scenarios, for corroboration and falsification of hypotheses? 
7. A complete data analytic plan would be required for each to-be tested hypotheses. 

Because constructing a robust hypothesis testing design within the present time limitations may be challenging, you may also choose a simplified confirmatory design where only feasibility is tested. Following the main goal of the study (“to test the feasibility e.g., number of dropouts -- ability of the participants to complete regularly the online assessment”), you could formalize this into feasibility hypotheses:

1. Define what counts as dropout and justify success/failure by the number of dropouts, e.g., in relation to common dropouts in similar interventions. Consider the degree of flexibility, e.g., with confidence intervals. 
2. Define and quantify online assessments to be completed by participants and justify a sufficient completion rate that will qualify successful and unsuccessful intervention. 

The above would allow you to make confirming claims about the practical feasibility of the intervention at Stage 2 with relatively little revision. Note that you can (and should!) also report the current primary/secondary outcomes, but only as non-confirmatory, tentative results that will inform future efficacy testing of the design. 

In case you choose either of the two confirmatory designs, please add each hypothesis separately in the design table with justifications. Note that currently some of the explanations are not fully sufficient. E.g., regarding sample justification, you have stated it to be non-relevant, but there should be a justification for having n=20 and not e.g., n=1 or n=200. I see this is already touched on p. 11. See e.g., Lakens (2022; https://doi.org/10.1525/collabra.33267). Also the rationale for confirming and disconfirming hypotheses still appears to be highly relevant for this design (if tested as confirmatory).

Note that if you choose not to test any hypotheses, a fully exploratory approach is totally ok and does not need the design table (or any of the other confirmation concerns either). In this case, make sure to remove the hypotheses and/or clearly state that they will not be tested.

Minor points

Title: Because “registered reports” include preregistration, it might be more informative to use the former term in the title.

Figure 2: We’re in mid-February, which is the time for filling consent forms. Please update how far the recruitment is when you return the revision. It’s totally ok if some data have already been collected (e.g., participant demographics are known), but then we just take this into consideration with bias control (author guidelines section 2.6).

P. 10: Will one of the team members serve as a game master or is this an external expert? Please clarify. 

P. 11: Because participants with as few as 1/9 IGD symptoms are included, it remains a bit unclear how this will affect the analytic strategy and the interpretation of results. E.g., there is some evidence that 2/9 symptoms are connected to lower wellbeing (Ballou & Zendle 2022: https://doi.org/10.1016/j.chb.2021.107140 ), but it’s not clear how the reduction from 1/9 to 0/9 symptoms should be interpreted. Would it imply the participant’s health/wellbeing improved?

P. 13: The participants will be randomly distributed into 4 groups, but is that optimal? Considering that the study addresses social anxiety, taking into consideration e.g. gender in group distribution seems relevant. Imagine you have 5 women and 15 men; having mixed groups would likely lead to different outcomes vs if all men and women would be in gender-based groups. Which would be better in the light of current knowledge?

P. 20: Qualitative feedback is collected. Please also explain how and what kind of, and how it will be analyzed in this study (if it is). 

P. 22: PO1 mentions frequency and hours, both. In my understanding, frequency refers the number of times of engagement (“three times per day”), not the total time of engagement (“three hours per day”). Please clarify. 

P. 23: It is noted that deviations will be justified at Stage 2, but I must note that PCI RR guidelines (section 2.10) advise authors to consult the recommender for deviations immediately and prior to the completion of data collection whenever possible. If you choose to have this as a fully exploratory RR, deviations are more flexible. Especially if any confirmatory elements remain, it remains important to notify of them as soon as possible. 

Scales: because at least some of the scales (like DSM-based IGDT-10) include both core and peripheral construct criteria, it feels reporting omega would be better than alpha.

Please also consider the reviewers’ separate comments. I hope you find the reviewers’ feedback and my additions helpful. You may contact me directly for any clarifications if needed. This is a highly interesting and promising study, and I’m happy do my best to support it.

Best wishes,
Veli-Matti Karhulahti