Dear Veli-Matti Karhulahti and co-authors,

I have now received two peer-reviews of your Stage 2 Registered Report submission: “Ontological Diversity in Gaming Disorder Measurement: A Nationally Representative Registered Report”. As you will see, both reviewers are very positive about your submission and request some minor revisions and some discussion/thought before your manuscript is accepted.

Reviewer 1 makes a good suggestion about the conclusion of your paper with regards to measurement problems: “How do we prevent it happening again with other phenomena or technologies?”. This is worth thinking about in your response. 

Reviewer 2 asks about the open data, but this was solved through a temporary, accessible link. The data will be made permanently available via FSD once verified/approved. An update on this would be helpful and an updated statement (and link to the data when possible) should be added to the manuscript. 

Minor points from myself:

Table 1: the formatting for the 95% CI on the top row should be centered also. 

Tables should follow APA style, particularly if you want to go to a journal that has this requirement. 

In Table 2 there is a right-square bracket but no left-square bracket around the exploratory probabilities – please see R2’s comment also about making this clearer in the table that these are exploratory probabilities; at first glance they can be mistaken for upper and lower confidence intervals. Looking at Table 2 alone, it is not clear what the ‘overlap’ values actually represent – could this be made any clearer? 

For result H3c (Page 9), there is a plus and minus sign for the t-test result; please revise: t(323.22)=+-2.72, p< .01.

Exploratory analyses, Page 9: “Although this might reflect  the poor attention skills of respondents who have GRHPs…”. Is there evidence to suggest that this is the case? If so, please provide a reference to support this assertion; if not, I would remove this sentence from this section. 

On Page 10, you mention that ‘post hoc’ power analyses are reported in the supplementary materials – do you mean ‘sensitivity’ power analyses here? Post-hoc power is essentially meaningless but sensitivity power analyses would provide the estimated effect size that could be found with N, power and alpha. See Lakens, D. (2022). Sample size justification. Collabra: Psychology, 8(1), 33267. Relatedly, for some of these analyses you are comparing very large samples with very small ones (e.g., 8186 vs. 31) – is this appropriate? What was the ES that could be found for these analyses; I know this is reported in the supplementary materials but you may want to include this within the table too, given your conclusions (“The exploratory analyses regarding the mental health of gaming and non-gaming populations did not yield any  meaningful differences. […] This implies a construct difference in terms of mental health, but confirmatory research is needed to corroborate it”). 

Yours sincerely,

Dr Charlotte R. Pennington

[Recommender]

Dear Veli-Matti Karhulahti and co-authors,

Thank you for submitting your Stage 2 Registered Report “Ontological Diversity in Gaming Disorder Measurement: A Nationally Representative Registered Report” for consideration by PCI Registered Reports.

Before sending this for in-depth peer-review, there are a few reassurances and edits required. I explain these in detail below and also provide in-text comments on your Stage 2 preprint to make it easier to see where I think changes/clarity is required.

Yours sincerely,

Dr Charlotte R. Pennington

 
Recommender Comments:

Major

1.     Open data. You provide a valid justification as to why the data cannot be made openly available at this point in time. To adhere to the PCI RR TOP guidelines, data should be made publicly available, or a legal/ethical justification should be included. You state that the verification of a dataset can take approximately 3-months. Can the data be made openly available by the time of (potential) Stage 2 acceptance, do you think? Could the data be uploaded to the OSF, or do Blendi not allow for this?

2.     You contacted me to explain the error regarding one item in the PROMIS Global Physical Health 2 scale (GPH-2) meaning that they couldn’t test half of H3. Along with the Managing Board, we agreed that this was OK and to continue with Stage 2 analyses. However, in the Stage 2 submission, a footnote explains further that this error happened because additional measures were added, which weren’t explained or signed off by us: namely, anxiety, depression, and a question about the war in Ukraine (see below). Are any of these questions analysed in the Stage 2 manuscript? Are any of the questions combined with other questionnaire indices? For clarity, I would appreciate if you could update the footnote to explain that the Managing Board signed off on the questionnaire-item measure and then explain that extra control measures were included and their reasoning. These two things are mutually exclusive from my point of view of ‘signing’ one off.

“as an extra control measure, our team agreed to enlarge the survey with two additional measures: anxiety (validated Finnish translation of GAD-2: Kujanpää et al. 2014) and depression (validated Finnish translation of BDI-6: Aalto et al. 2012). To add further means for assessing the effects of the drastic world events, we included a single item that asks the participants to self-report the negative mental health impact of the war in Ukraine. As a byproduct of these last-minute changes and several extra test iterations, a mistake occurred in our team and an erroneous GPH-2 item—PROMIS Global Health item #09 instead of #06, which is very similar in wording—ended up being included in the final survey. We noticed this soon after the data had been collected and immediately contacted the recommender who, after discussing with the managing board, advised us to proceed without confirmatory GPH-2 analysis in H3. We thus report physical health exploratively in this section with only one GPH-2 item (“GPH-1”)”.

3.     The confirmatory analyses look good and I have reproduced them. Thank you for providing the R-script, too. Further information is, however, required for the exploratory analyses to allow me to send this out for in-depth review. First, what are the Results when participants are removed for mischievous responding? It seems imperative to me that you need to check whether the Results hold when individuals are removed for failing the first control item. Second, what are the exploratory analyses outlined in Table 4? You need to explain to the reader what these are in the immediate text where they follow: I looked up and down the manuscript but couldn’t quite work out what these analyses were referring to: a reminder for the reader would be very helpful.

4.     You state the below in your cover letter. You need to make transparent ANY deviations from Stage 1 to Stage 2 in the main text, so this needs to be acknowledged within the manuscript itself:

"One further point is worth noting: in the Stage 1 we promised to test different endorsement criteria for instruments to see if the results vary. We soon realized that even testing *one* set of different criteria for a single instrument results in a massive set of analyses, with lots of excessively lengthy reporting (and R code). The 2-page long Table 4 illustrates this, with only one alternative endorsement option tested. Therefore, we decided not to analyze and report further endorsement options to keep the article length reasonable and workload manageable. Regardless, we do fulfill our Stage 1 promise by reporting alternative endorsement criteria for THL1."

I recap some of these comments via in-text comments your Stage 2 manuscript (attached). Additional, minor editing points are provided for you there also. 

I understand you are concerned about word count, so I’d be happy for additional analyses etc. to be reported in supplementary files and uploaded to the project’s OSF Page. Nevertheless, the manuscript itself needs to read clearly enough for the reader to understand exactly what you are referring to (i.e. clarifying the exploratory analyses in Table 4).