Recommendation
and D. Samuel Schwarzkopf based on reviews by 2 anonymous reviewers
DOI or URL of the report: https://osf.io/rg872?view_only=f28f60cb55574d54a04571a189eca5f7
Version of the report: 1
and D. Samuel Schwarzkopf, posted 15 Nov 2023, validated 15 Nov 2023Dear Dr. Gautier,
Thank you for your detailed response to the reviewers. Both reviewers are now satisfied with the content of your Stage 1 manuscript.
Before we move to an in-principle acceptance, we have a few of remaining comments we would like you to address:
You consider the inconsistent results in prior literature to be due to the discrepancies in higher functioning levels of the two ASD samples (e.g. IQ). In the introduction, the discrepancy between the ASD groups of each study is one of the main motivations for further examination into the vocal-sound processing in ASD. It is still not totally clear how you will control for higher functioning level in your ASD sample. It is plausible that with a larger sample of ASD participants, you will cover a broader variance in higher-level functioning, culminating in a result which is more applicable to the ASD population as a whole. If this is the case, you would need to consider your sample size to account for the wide range of IQs in the population. However, it is also plausible that without controlling for higher level function (in this case, IQ), you may recruit closer to one end or the other which could significantly impact interpretation of the result, falling into the same trap as the prior literature. If you indeed plan to control for variables such as IQ to overcome discrepancies in the prior literature, a more substantial reconsideration of planned analyses would be necessary.
You have included a memory task to determine if there is a difference in attention to the stimuli between the two groups. This addition seems important, but you should add a hypothesis and run a power analysis to ensure you have the power for your planned one-way ANOVA. On that note, it seems like a t-test would be more appropriate for this analysis as there are only two groups. Could you either clarify why an ANOVA is the correct route or change the manuscript accordingly. Furthermore, more information is necessary with regard to the task: how many sounds will be tested? Will the participants be told of the memory task prior to the fMRI data collection? If so, how might the knowledge of a memory task influence the processing of the stimuli during the scan? We believe it would be better to inform your participants of the memory task prior to the scan, as a post-scan behavior difference could explain any functional differences. However, informing the participants prior to the scan changes the interpretation of a potential functional difference between groups. The interpretation cannot be of an automatic processing of early voice processing, but a task-relevant processing of early voice processing.
In the Design Table you should add/clarify some information. For example, you should state the power level you used (it's in the text but should be there as well). Also, under Rationale for deciding sensitivity you only write p<0.02 - you should support the p-value with a rationale by moving the information about previous effect size from Sampling plan into here. You already explain in the text why this is the effect size you use, so you would not need to add any extra this information.
It is important that you also remove their references to exploratory analyses in the study table. The mention of the individual differences exploratory analysis in the introduction is an exception. For readability, you need to explain the collection of data that will only support exploratory analyses (i.e. the individual evaluations) in the introduction. Other exploratory analyses are acceptable at Stage 2, but then they are clearly labelled and will be scrutinised accordingly. With your current design (which does not include an equivalence test), if your chi-square test finds no significant difference, the main message of your Stage 2 would be that the null hypothesis cannot be rejected.
We look forward to hearing from you soon.
Best,
Grace & Sam
Although the authors did not take into account my suggestion regarding Bayesian hypothesis or frequentist equivalence testing, they responded to this as well as my other questions/comments in a satisfactory way and I have no further remarks regarding this proposed study.
The authors addressed all of my comments and I don't have any further questions or remarks.
DOI or URL of the report: https://osf.io/egam2?view_only=978ca2543e7b434f9beea92a52f202b5
Version of the report: 1
and D. Samuel Schwarzkopf, posted 23 Aug 2023, validated 23 Aug 2023Dear Dr. Gautier,
We have been fortunate to receive insightful comments from two expert reviewers. We agree with the reviewers that you have outlined a clear rationale for your proposed research question, which is supported by a strong design. There remain some areas where the reviewers have requested clarification, to highlight a few:
Reviewer 1 has highlighted a potential difference in groups in relation to attentiveness to the stimuli, which in turn may explain activation differences – could the authors comment on how they might account for potential attentiveness differences? Reviewer 1 also indicated that Bayesian hypothesis testing or frequentist equivalence testing may be appropriate if there is no difference between your two groups in the proportion of individuals showing STS/STG activation in response to non-vocal sounds. Given that the two prior studies examining TVAs in ASD found atypical (Gervais et al., 2004) and typical (Schelinski et al., 2016) activation, you might consider registering an equivalence test as a contingency if your hypothesis is not upheld. In which case, you would need to also run a power analysis for an equivalence test. Bayesian hypothesis testing is another route, but it would be ideal to not mix hypothesis testing frameworks. This route would require the authors to change to the Bayesian framework for all analyses. It is also plausible to explore a lack of difference in Stage 2, however powering your study for this potential outcome in Stage 1 is advised.
Reviewer 2 expressed interest in the clinical profile of the typically developing group. As the hypothesis hinges on there being a difference in the clinical profile of the two groups, please comment on how you might screen individuals included in the typically developing group. Reviewer 2 also commented on the potential exploratory analysis involving the clinical profiles of the individuals with ASD and the presence or absence of TVA response. They suggest also examining individual differences of the those in the TD group. We request the exploratory analysis be removed from the abstract in line with registered reports formatting, however, it may be noted in brief towards the end of the introduction.
With regard to registered reports formatting, we have a further few items for the authors to address:
1. Version control
After each revision, it is ideal if you replace the old pdf with a new clean (no track changes) pdf on OSF to enable PCI-RR to record the version number. The tracked changes document should be uploaded to PCI-RR separately in the same page where you submit your reply to the reviewers and recommenders.
2. Study design table
In the final column of the Study design table, we encourage you to think about the theoretical implications of either finding or not finding a difference in TVA activation between groups.
Minor:
Typo on page 3: “founding resources”
We believe your manuscript has potential for Stage 1 in-principle acceptance, therefore we request revision and resubmission. Please address each of the reviewers’ and the recommenders comments and revise your manuscript accordingly.
Best,
Grace & Sam
1A. The scientific validity of the research question(s).
The authors clearly describe the gap in the current evidence on brain processing of vocal sounds in ASD, since this question has been addressed only in two studies providing opposite findings.
1B. The logic, rationale, and plausibility of the proposed hypotheses, as applicable.
The main hypothesis about a lower proportion of individuals with ASD showing significant responses to vocal vs. non-vocal sounds compared to non-autistic individuals is plausible and specific enough.
I wonder why the authors would explore individual characteristics of only those ASD participants, who do not show any significant voice-specific responses. As no previous study specifically looked on links between brain processing of voice and individual participant’s characteristics, a dimensional approach of looking on the whole continuum of vocal brain responses in ASD could be more informative. A dimensional approach would also provide more power compared to splitting up the sample into responders and non-responders for categorical comparisons (particularly if the non-responders are expected to be in the minority).
1C. The soundness and feasibility of the methodology and analysis pipeline (including statistical power analysis or alternative sampling plans where applicable).
The authors provide a high level of methodological detail in their Stage 1.
How is the alpha level of .02 justified? Is it based on any previous findings? If yes, please state this.
1D. Whether the clarity and degree of methodological detail is sufficient to closely replicate the proposed study procedures and analysis pipeline and to prevent undisclosed flexibility in the procedures and analyses.
The level of detail in the analysis of brain responses is sufficient.
I miss however more details in the description of the sample which is one hand relevant for replication of the study and on the other hand for the exploratory analysis in ASD individuals.
- Do the authors plan to assess hearing abilities and auditory processing abilities to ensure that different brain responses are not due to other factors than ASD.
- Do the authors plan to assess clinical profiles of non-autistic controls? Will they screen for other forms of neurodiversity? This is particularly important since the authors coin their control group as “typically developing”. If no such screening is planned, I recommend using the terms “non-ASD” or “non-autistic” as preferred by the community. Please pay more attention to more current vocabulary for describing ASD.
Please provide the description and examples of non-vocal sounds.
1E. Whether the authors have considered sufficient outcome-neutral conditions (e.g. absence of floor or ceiling effects; positive controls; other quality checks) for ensuring that the obtained results are able to test the stated hypotheses or answer the stated research question(s).
The authors have considered outcome-neutral conditions and controls. I recommend additionally to assess participants’ hearing abilities to control for their confounding effects on auditory brain responses.
1A. The scientific validity of the research question(s)
The authors have provided a clear rationale for the validity of research question, and they have cited relevant research that supports the need for the study. However, in my opinion, the research question may be stated more explicitly in the introduction. Although the goal of the study is mentioned in the abstract, the introduction of the paper does not mention the research question very clearly.
1B. The logic, rationale, and plausibility of the proposed hypotheses (where a submission proposes hypotheses)
The authors hypothesize that a lower proportion of individuals with ASD will show a specific response to vocal sounds in the STS/STG (i.e., a TVA activation) than in the typically developing sample. This hypothesis is plausible and supported by previous research that has shown that individuals with ASD have difficulty processing social cues, including voices. This suggests that they may also have difficulty showing a specific response to vocal sounds in the STS/STG. It is not clear to me whether the authors intend to interpret a negative result as evidence that the effect is absent. If so, using an inferential method that is capable of drawing such a conclusion (i.e. Bayesian hypothesis or frequentist equivalence testing) would be advised.
1C. The soundness and feasibility of the methodology and analysis pipeline (including statistical power analysis or alternative sampling plans where applicable)
The authors propose to use a well-established fMRI protocol for collecting and preprocessing the data including a denoising step to improve the quality of the data as well as appropriate statistical methods to test their hypotheses. The power analysis is described clearly and seems reasonable.
I do have some questions about the experimental task and the participant characteristics: 1) what will the duration of the presented sounds and the sound intensity (dB) levels be? 2) At what time intervals will the sounds be presented and is this a fixed rate or jittered? 3) Will the sounds be randomly presented to participants or in a fixed order? 4) I think it would be good to have an idea as to how well participants were engaged with the “task”. Some participants may be more attentive than others to the sounds and this may impact the results; especially if this where to be a between group difference. Would it be possible to add some sort of response task interspersed between the passive task stimuli to check if participants are (still) paying attention? Or, if that is not possible/desirable, ask a few questions after the experiment on how engaged participants were? This data then can be used to find out if differences in terms of task engagement/attention occurred between both groups for instance. 5) It is stated that participant’s IQ will be evaluated. Does this mean participants will be matched on IQ as well? It may not be feasible, but if participants would not only be matched on gender and age but also on IQ that would be preferable.
1D. Whether the clarity and degree of methodological detail is sufficient to closely replicate the proposed study procedures and analysis pipeline and to prevent undisclosed flexibility in the procedures and analyses
In principle, the methodology is described in sufficient detail to be replicated (but see comments above). The authors have provided a clear and concise overview of the methods, and they have included all the important data preprocessing and analysis steps. The only thing that was not clear to me in the fMRI scanning procedure is the phrase how “structural anomalies or irreparable artifacts will be replaced”. Does this mean that these will not be taken into account at all or will they be replaced by other data of some sort?
1E. Whether the authors have considered sufficient outcome-neutral conditions (e.g. absence of floor or ceiling effects; positive controls; other quality checks) for ensuring that the obtained results are able to test the stated hypotheses or answer the stated research question(s).
It seems to me the stated hypothesis can be tested with the experiment proposed when taking into account the abovementioned suggestions.
DOI or URL of the report: https://osf.io/d7rh8?view_only=f28f60cb55574d54a04571a189eca5f7
Version of the report: 1
Dear recommenders,
The authors thank you for the insightful commentaries on the registered report initially entitled ‘Cortical voice processing in Autism and Food Addiction’. Following the revision request, extensive changes has been made to the manuscript now entitled ‘Cortical voice processing in Autism Spectrum Disorder’. We hope that the revisions we made to the initial submission enabled us to further improve the quality of our manuscript. Here, we address each of the revisions required by the recommenders. Please note that all modifications has been highlighted in the revised manuscript.
1A. The scientific validity of the research questions.
The introduction and motivation are well considered. The authors may want to provide a clearer description of vocal sounds as an indicator of lower-level social information processing.
The authors emphasized the fact that a specific brain response to vocal sounds constituted an indicator of lower-level social information processing:
TVAs have been suggested to be involved in the first step of voice perception (i.e., structural analysis of voices, Belin et al., 2011). More elaborated steps of voice perception, such as perception of identity information and perception of vocal affective information are hypothesized to rely on the communication and integration of information between TVAs and regions from the extended voice processing network (Belin et al., 2004, 2011; Brück et al., 2011; Maguinness et al., 2018). These results emphasize the role of this functionally defined network in social functioning.
1B. The logic, rationale, and plausibility for the proposed hypotheses, as applicable.
Cortical voice processing in autism spectrum disorder
The authors build in outcome-neutral and replication hypotheses which provide a strong foundation for following analyses. Although clearer in the study plan, the hypotheses in the main text could be made more specific. For example, when referring to the TVAs in the ASD group, the authors write that there will be an absence of specific activation to vocal sounds. Does this mean equal activity in TVAs to other regions, or a total lack of response to vocal sounds?
The authors further described what would constitute a TVA activation and a lack of TVA activation:
Such a response would manifest as a higher activation level for vocal compared to non-vocal stimuli. As the opposite, a lack of response to vocal sounds corresponds to an equal activation level between the vocal and non-vocal conditions.
The authors also hypothesize TVA activation will be associated with social functioning level. This hypothesis is plausible with large variance in social functioning level in ASD, do the authors plan to control the distribution of social functioning level?
This analysis has been removed from the current version of the registered report
Cortical voice processing in food addiction
Similar to the ASD study, is it plausible to hypothesize a difference between groups for food addiction? This initial analysis may provide the foundation for all following analyses as is intended in the ASD study.
Could the authors consider the variance in addiction and how this may impact TVA activation, similar to the ASD study?
The food addiction part has been removed from the current version of the registered report.
1C. The soundness and feasibility of the methodology and analysis pipeline (including statistical power analysis or alternative sampling plans where applicable).
Power analyses for both ASD and FA studies:
· The reasoning for the proposed number of participants is not clear for the ASD and FA studies. The power analyses suggest a need for 23 participants, so why were 25 and 28 selected for ASD and TD, respectively, or 25 per group in the FA study? Is this selection due to attrition rates? Furthermore, for the ASD study, why select differing numbers of participants if the goal is to match sex and age?
· Running power analyses without prior studies which have examined the exact same effects which are of interest to the authors can be complicated. However, the registered report guidelines require a mention of the minimum effect sizes. The authors do not run power analyses for the between groups analysis for TVA activation, for the association between TVA activation and social functioning level, or for the functional connectivity analysis for the ASD or FA studies. Although there may not be prior studies which have examined the exact effects of interest, is it plausible to locate effect sizes for similar research questions in prior literature which would support power analyses? For example, prior studies which examine functional connectivity differences between ASD and TD controls?
· We appreciate the use of the simulation to determine the effect size that can be achieved for the functional connectivity analysis. Although an effect size of d=0.65 may be reasonable for your effect, it should be justified to minimize the risk of inconclusive results (i.e. by presenting prior studies which find moderate to large effect sizes for similar research questions in the literature). Without this justification, the authors should remove the functional connectivity analysis from the study plan. The analysis can always be tested as an explicitly labelled exploratory analysis in Stage 2.
· On the note of exploratory analyses, without a power analysis and justification of effect size for the association between TVA activation and social functioning level in the ASD study, this analysis should also be tested as an explicitly labelled exploratory analysis in Stage 2.
· Finally, we encourage authors to consider minimum effect sizes in terms of first principles and meaningful absolute effects rather than Cohen’s d.
· Ensure the significant threshold and power is stated for each test (see Behavioral data analysis for example of missing information). Please note, PCI-friendly journals have requirements based on alpha levels and power for frequentist statistics, check each journals’ requirements in order to be eligible for the journals of interested (https://rr.peercommunityin.org/about/pci_rr_friendly_journals).
The authors thank the recommenders for their insightful comments about the power analyses. However, the analysis plan has been completely reconsidered for this version of the registered report. Hence, the previous comments no longer apply to the current submission.
· The authors mention that fMRI data with an artifact or anomaly will not be replaced – a registered report should avoid flexibility in the sampling plan and should conform to the proposed number of participants dictated by the power analyses of the smallest effect size.
For the current version of the registered report, a power analysis has been conducted. Final sample size has been defined based on the funding resources for the study. Although the power analysis indicated a required sample size inferior to the sample size defined using monetary constraints, the authors prefer to maximize their sample size in order to maximize power for optional exploratory analyses.
1D. Whether the clarity and degree of methodological detail is sufficient to closely replicate the proposed study procedures and analysis pipeline and to prevent undisclosed flexibility in the procedures and analysis.
The authors provide a high level of methodological detail in their Stage 1. There seem to only be a few of items missing:
· The authors do not mention the task during the presentation of the sounds, could the authors explicitly outline the task or lack thereof.
The voice localizer consists for the participant of passively listening vocal and non-vocal sounds. The lack of behavioral output for this task has been emphasized in the main text:
The task consists of passively listening to the sounds, and no behavioral output is required.
· Could the authors outline the number of voxels they would consider a cluster when determining activation within the TVA mask?
As a cluster correction for multiple comparisons is planned, there is no need to pre-specify the number of voxels which would constitute a cluster. This has been emphasized in the main:
Once the model estimated, a contrast for vocal vs. non-vocal sounds will be computed and individual maps will be thresholded at the 0.02 alpha level, cluster corrected,with a cluster defining threshold of Z = 2.05 (corresponding to an alpha level of 0.02).
· Could the authors specify the cut-off scores for the ADOS/ ADI and Yale Food Addiction Scale 2.0 (YFAS) which results in acceptance into the ASD and FA groups, respectively? Note, the acronym for ADOS/ ADI are not currently unpacked in the text.
No cut-offs will be used for the ADOS and ADI questionnaires, although the scores will be reported for descriptive purposes. This has been emphasized in the main:
Diagnoses will be established by one of the psychiatrist coauthors of the study according to the DSM-5 criteria. In addition, we will report scores from the ADOS (Autism Diagnostic Observation Schedule) or ADI-R (Autism Diagnostic Interview-Revised) for descriptive purposes (these scores are not necessarily considered for the establishment of diagnoses).
Authors ensured that acronyms where unpacked in the main text.
1E. Whether the authors have considered sufficient outcome-neutral conditions (e.g. absence of floor or ceiling effects; positive controls; other quality checks) for ensuring that the obtained results are able to test the stated hypotheses or answer the stated research question(s).
The authors have considered outcome-neutral conditions and controls. One additional outcome-neutral test would be to determine a difference in food addiction between groups, as mentioned previously.
The food addiction part has been removed from the current version of the registered report.
Thank you very much for your time and consideration,
Sincerely,
Raphaël GAUTIER
and D. Samuel Schwarzkopf, posted 08 Jun 2023, validated 08 Jun 2023