Thanks for the kind invitation to review “Gold in, gold out. Quality appraisal and risk of bias tools to assess non-intervention studies for systematic reviews in the behavioural sciences: A scoping review”.

I think this study would address a very important issue. In systematic reviews of RCTs, assessing primary studies with tools such as Cochrane's Risk of Bias is very common. However, outside this context, including such assessments is much less frequent. This may be due to a lack of tools that may be applicable or a lack of knowledge of the authors conducting the reviews. Therefore, I believe that conducting a review to highlight what tools currently exist, what aspects these tools assess, and for what contexts they could be used would be very valuable for authors. I also believe that it will serve to identify gaps that are not currently assessed by existing tools but are aspects that should be examined.

Overall, I think the protocol for the scoping review is very well designed. I commend the authors for their efforts to be transparent and to increase the reproducibility of their work. I also liked very much the comprehensive checklist of items to be evaluated for each tool. There are some points that I think the authors could reconsider, although I do not think it is mandatory to make these changes and I would also be happy if the authors argue why it would be better not to make them.

Regarding the introduction, I believe that the outline on the three types of biases is very illustrative. It explains the potential sources of bias that a meta-analysis may have and mentions the tools or guidelines that can be followed to address them. However, I notice that the 'Study bias' section is as extensive as 'Literature bias' and 'Researcher bias.' I was wondering if it might be appropriate to expand the 'Study bias' section by introducing some key concepts when evaluating the quality of primary studies. For example, I think it could be helpful to already introduce the concepts of construct, external, internal, and statistical validity, as well as provide some examples. It could also be mentioned that some tools cover certain aspects but not all—for example, the Cochrane Risk of Bias scale focuses primarily on internal validity (Hartling et al., 2009).

Other considerations regarding the 'study bias' section. When mentioning concepts like 'selection bias,' 'interviewer bias,' or 'citation bias,' it might be helpful to add some references to studies that discuss these biases in more detail. However, I don't quite understand why citation bias is placed in the analysis stage. I think it might correspond more with the discussion stage (where only studies that confirm certain results are cited). Also, I also don't fully understand placing "analytical flexibility" there. I think it could be more appropriate to describe it as "inadequate use of statistical tests", which might occur when there is analytical flexibility.

Regarding data extraction, the authors mentioned that have piloted this process. If I understand correctly from the shared files, this coding has been done from one tool (ROBINS-I). The procedure that is indicated to be followed is “items from each tool will be extracted by one reviewer and items marked with an asterisk will be independently validated by a second reviewer, who will focus solely on these tagged items”. I wonder whether it might not be more appropriate to assume double coding of all items for at least a percentage of the tools (e.g. 25%, although this might depend on the total number of tools encountered) to ensure that all items are being well understood (the pilot has only been done with one tool). Of course, I understand that this would be more costly, and that it may be a waste of time to perform a double coding if the inter-rater reliability is very high, but perhaps it could avoid errors. I leave it to the authors to decide.

Regarding the data to be extracted from each tool, I think this should be explained more in detail in the main text. In the current version of the manuscript, I found it difficult to see that this information could be found in the ‘Data Extraction Instructions’ file. I think the main manuscript should at least mention the domains for which information is going to be extracted (even if ‘ad-hoc’ items are added later). For example, you could explain that you will extract metadata such as title, format or whether they offer support, as well as content information related to aspects such as ‘Open & Reproducible Scholarship’ or ‘Validity’. I would also reference explicitly that all the items can be found in the ‘Data Extraction Instructions’ document (https://osf.io/ewm7x).

Regarding the data extraction items, I have some concerns. First, about the section ‘Open & Reproducible Scholarship Content’, I wonder whether items such as ‘Open access publication’, ‘Open source software used’ or ‘preprinting archiving’ are really necessary. While these are of course desirable properties of a study, I don't see how these aspects relate to the quality of a primary study. I would be surprised if any quality assessment tool for primary studies assessed this aspect. Therefore, I think not assessing them could save time. Besides, I don't understand how ‘Sample size estimator’ relates to ‘Open & Reproducible Scholarship Content’. Shouldn't this be an aspect of internal validity (underpowered studies have higher probability of Type I error and, because of the lower precision also Type II error)?

Similarly, regarding the section ‘Integrity Assessment’, I am not sure whether ‘Methodology assessment’ or ‘Analytical approach assessment’ are in the right place. Wouldn't ‘internal validity’ be more appropriate as well?

Also, although the current checklist of items is very comprehensive, it is likely that the tools you find will assess aspects that you have not yet considered. I see you mentioned that you will add them as ‘ad hoc’ items. Do I understand correctly that if you find this item in the middle of the data extraction process, it would be evaluated retrospectively for all other tools?

Lastly, another aspect that might be interesting to know is whether there are studies that assess the validity and/or inter-rater reliability of the tools. However, I understand that looking for such studies may take considerable work and may be outside the scope of this review, but I think it is very important to know that these tools are adequate. Related to this, it might also be interesting to know if any training is specified as necessary to apply these tools.