Testing the facilitatory effect of high-frequency transcranial random noise stimulation through enhancement of global motion processing

(Apologies for the slight delay - I have been away for the past week.) Your revised RR is now very nearly ready to be accepted. However, because this step will make your Stage 1 plan formal, I think it is worth making two very minor changes, in relation to the statement of your sample size plan.

"Based on these power analyses we predict 42 participants to be the maximum number of
participants necessary for our within-subjects design."

This statement is potentially misleading, because it implies that this is the most participants that you would need for the study (actually it is the number required to achieve the desired power), as if the study could be adequately conducted with fewer. I think that you simply mean that it is the number of participants necessary to achieve .9 power to test your minimum effect size of interest. I suggest that you should re-state this to avoid confusion. Also, you should make it clear throughout when you discuss sample size that 42 is the number of participants required (after exclusions) (i.e. the number of valid datasets).

While you make these changes (or tell me why they are not needed), I will draft a recommendation text, so that we should be able to issue IPA immediately upon receipt, as I know that you are keen to get started with data collection.

Best wishes,

Rob

Thank you for submitting your revised Stage 1 plan. Both external reviewers are happy with the changes made. However, before we proceed to issue In Principle Acceptance, there are a few minor loose ends that may be worth tidying up. One is the typo that you have mentioned to me in an email, which you should amend as you suggest ("In our Stimulation Parameters and Positioning section of the manuscript, we say we will localize hMT+ at 3 cm dorsal of the inion and 4 cm leftward. It should read 5 cm"). The others, I list below.

1) Minor typo. On reflection, I think it is best to remove reference to 'placebo' from question 2 in the design table: "Question 2: Does the facilitation of contralateral motion coherence induced by hf- tRNS targeted at left hMT+ exceed that of the placebo effect of the application of the electrodes to the same area with no stimulation". Placebo would refer to a specific mechanism of change that you are not testing, so just refer instead to "the effect of".

2) Minor typo. In several places throughout the manuscript, e.g. in statement of effect sizes, you use four decimal places, which seems like spurious precision. In general, 2 decimal places should be used unless you have a specific reason to require more.

3) Minor tweak. You are drawing your effect size estimate from the study of Ghin et al (2018). Using a central estimate of effect size from a standard published (not Registered Report) study runs the risk of overestimating the effect size. You might acknowledge this fact, before you state (as you already do) that the effect size estimate is still conservative in the context of your design, because it is from a between-subject design where yours is within subject.

4) Minor clarifications. I still do not fully follow your description of how you determined the effect size from Ghin et al (2018). Was it based on the values reported in their paper, or on a re-analysis of their paper (because I can't easily find your quoted values in the paper itself). Also, the statements you make are of the general form:

"We used a true mean of 10.51% related to the difference in contralateral and ipsilateral motion coherence threshold for hMT+ targeted hf-tRNS, null hypothesis mean of 2.59% related to the difference in contralateral and ipsilateral motion coherence threshold for hMT+ targeted sham tRNS, and standard deviation of 14.8% from Experiment 1 of Ghin et al (2018; Cohen’s d: 0.5351)."

It's not really accurate to call these "true mean" and "null hypothesis mean"; they are just the estimated mean differences for the treatment and relevant control conditions. When you state the standard deviation, you do not make it clear what this is the standard deviation of. Is it the pooled standard deviation of those two differences, which you are using to calculate the effect size of the between-subjects difference between conditions?

Finally, is it inaccurate to imply that the effect size for hypothesis 1 (the ipsi-contra difference) is a between-subjects comparison in their design?

I hope that these points will be relatively straightforward to consider, and to clarify.

Best wishes,

Rob

Thank you for your revised Stage 1 manuscript. Both of the original reviewers have re-assessed the manuscript, and are happy with the revisions made.

However, I have a few small remaining issues that you may wish to give some consideration before we issue In Principle Acceptance for your protocol. These relate to the precise statement of hypotheses, and relation to the statistical tests proposed, which are critical features of the RR.

1) In Methods Section IX (and elsewhere) you refer to “Expected results”. It would generally be better to talk about the results that are predicted by the hypotheses that they test (as in your design table) than to talk about your personal “expectations”, which seem like they could be more subjective. (*Also, roman numerals is these days a very unconventional system for numbering sub-sections – you may want to reconsider this, as you will probably be forced to change it if eventually publishing in a journal. In order to enhance comparability between the Stage 1 and Stage 2 documents, I would suggest looking at your target journal’s sub-section conventions, and following these.)

2) In your design table, the statement of your research questions (column 1) seems like it could do with some tightening up.

For Question 1, you ask “Does stimulation targeted at left hMT+ facilitate motion processing in the contralateral visual field only?” However, this is not exactly what your statistical comparison will test, which is whether left hMT+ facilitates motion processing in the contralateral visual field more than in the ipsilateral visual field.

For Question 2, you ask “Is hf-tRNS stimulation targeted at left hMT+ necessary to elicit the contralateral motion coherence change, or is the effect caused by the placebo effect of the application of the electrodes alone”. Technically, you cannot ask whether left hMRT+ stimulation is “necessary” without testing every other conceivable stimulation site. It might be better to ask whether the facilitation of contralateral motion coherence induced by left hMRT+ exceeds that of the placebo effect of the application of electrodes to that same area.”

For Question 3, a similar comment applies about the use of the word “necessary”.

3) It seems to me that Questions 2 and 3 are not really separate questions but are conjoined sub-questions for one larger question, which is whether the facilitation of contralateral motion coherence induced by left hMRT+ exceeds that of appropriate control conditions, which will allow you to conclude in favour of a causal role of left hMRT+”. You would need both outcomes to be significant in order to conclude that the effect is caused specifically by the stimulation of left hMRT+. If this is correct, then it should be indicated in some appropriate way in the design table that your conclusion will be drawn across these two results. (If I have misinterpreted the design, and this is not the intention, then you still need to indicate how the overall conclusions will be informed by the combination of outcomes across questions 2 and 3.)

4)In the section on power calculation, you give the raw size of the targeted effect, and a measure of SD, but it might be useful to the reader if you were also to express the targeted effect size in terms of the standardised effect size measure (e.g. dz) that you have entered into the power calculation.

As ever, you are free to reject any of these suggestions with an appropriate rationale, but it would be good to consider them before the protocol is finalised.

Best wishes,

Rob McIntosh

PCI-RR Recommender

I have now received comments on this Stage 1 RR submission from two expert reviewers (R#1 has identified himself as Sam Westwood). Both reviewers are positive about the aims of the experiment, and have only relatively minor suggestions to make. In particular, I would draw your attention to SW's second point, which concerns the possible additional between-session variability introduced into your within-subjects design by practice effects. Before you embark on this experiment (with its present design), it would be worth carefully considering the possible impact of practice effects on your data and, crucially, on your ability to test your stated hypotheses at the desired level of power.

Your revision should be accompanied by a full responses-to-reviewers document, and a tracked version of the manuscript in which any changes made are highlighted.

Best wishes,

Rob McIntosh

PCI-RR recommender

Deart Dr. Edwards,

Thank you for this resubmission and for the modifications that you have made to take account of the initial round of triage comments. In general, these modifications are good, but there are some outstanding issues that it would be wise to address before seeking external review. Only the first of these is major, but I raise the other two for completeness.:

1) Your analysis plan for critical hypotheses is based upon a mix of frequentist power analysis (assuming an alpha of 0.02 and a power of .90; note, you state in the cover letter that this is one-tailed, but this does not seem to be specified in the paper) and inferential hypothesis test based on a Bayes factor (with ‘critical’ values of 6 and 1/6). This mixture of strategies is not really coherent and may not meet the criteria of relevant PCI-friendly journals. The frequentist power analysis would be fine if your hypothesis test is to be a one-tailed t-test with alpha .02. However, if your inferential tests are Bayesian, then the frequentist concept of ‘power’ does not apply, and you should instead demonstrate that your sample size is sufficient that it will lead to a high enough probability of returning a sensitive result by your Bayes criterion.

For instance the guidance from the Cortex RR guide for authors, states: "For inference by Bayes factors, authors must be able to guarantee data collection until the Bayes factor is at least 6 times in favour of the experimental hypothesis over the null hypothesis (or vice versa). Authors with resource limitations are permitted to specify a maximum feasible sample size at which data collection must cease regardless of the Bayes factor; however to be eligible for advance acceptance this number must be sufficiently large that inconclusive results at this sample size would nevertheless be an important message for the field.” I recommend that you read the whole relevant section of that guidelines document.

Often, in these cases, we would suggest a formal Bayes Factor Design Analysis (BFDA) (see this paper by Schönbrodt and Wagenmakers (2018): https://link.springer.com/article/10.3758/s13423-017-1230-y). In the BFDA method, you run simulations to determine the probability that your experiment will return evidence in favour of H1 or H0. This method allows you to estimate--given your chosen BF threshold, your smallest effect size of interest, and your maximum sample size--what proportion of studies will stop because the evidential threshold has been crossed, and what proportion because max n has been reached. With BFDAs for each hypothesis, you can plot the simulation results to show the sensitivity of your design given your assumptions (see Figs 2 and 3 of the following paper for an example of how the simulation results can be presented): http://dx.doi.org/10.1037/bne0000345).

It is entirely up to you whether you want to go down a Bayesian or frequentist route for hypothesis testing, but the sample size calculation should be appropriate to the chosen method.

2) In addition to your critical hypothesis tests, you propose some exploratory analyses to inform your interpretation. You can state an intention to add these exploratory analyses, as you do at the bottom of p7, but it is not necessary (or appropriate) to then describe in any detail how these will be conducted, or to include them in the core design table, because they are exploratory. The Stage 1 RR should specify the registered experiment, and any further exploratory analyses should be added at Stage 2. (On the other hand, if these exploratory analyses are essential to your purpose, they should be made into registered hypotheses, and specified as such.) Note that your Stage 2 conclusions must not be inappropriately guided by exploratory parts of your analyses,

3) I previously suggested that you should be more precise in the distinction between replication and reproduction. On p6 you now state, “In order to implement some additional controls, we propose to reproduce the contralateral impact of hf-tRNS over left hMT+, rather than perform an exact replication. In contrast to an exact replication, we include only a selection of the original conditions, and have introduced extra within-subject stimulation controls in our design.”

Actually, in most widespread usage, what you describe is a form of replication (just not a direct or exact replication). In my understanding, the term ‘reproduction’ is generally taken to imply computational reproducibility of the same outcomes from exactly the same dataset.

I hope that these further comments are helpful, and can be addressed (or rebutted) relatively easily, so that we can proceed with the review process.

Best wishes,

Rob McIntosh

Dear Dr. Edwards,

Thank you for submitting your Stage 1 manuscript to PCI-RR. The proposed experiment looks interesting, and the manuscript is generally readable and well-prepared. PCI recommenders routinely triage initial submissions for suitability for the RR format, before sending for external review. I think that there are a number of issues that you might want to consider in this regard. These are mostly related to the specific requirements of RR, rather than being topically-focused.

My comments follow at the end of this email. I think it would be worth giving consideration to the issues raised (especially 4-10), before the Stage 1 plan is sent for external review.

The comments are advisory and are based upon a single person’s reading (mine). If you choose to respond to these issues then please indicate how you have done in an accompanying letter, and provide a tracked-version of the manuscript, in addition to uploading a clean preprint.

I hope you find these comments useful, and I look forward to a revised submission in due course.

Yours sincerely,

Robert D McIntosh (PCI-RR recommender)
 

The Introduction is generally well-written but I see three main ways in which it could be strengthened: (1) when reviewing prior studies, especially the study that provides the target effect for replication (Ghin et al, 2018), it would be useful to give some quantitative details (e.g. sample size, effect size etc); (2) the review seems to be restricted to tRNS interventions, but it would be good to consider also the evidence regarding other NIBS interventions at MT+ on motion processing; (3) I would like to see a clearer rationale for why this particular effect has been targeted – if the aim is to select an effect just to validate the effect of tRNS, then is this particular effect the best candidate? Or if the interest is more specifically in motion processing, then why? (Also note, elsewhere in Introduction, keep the conceptual distinction between replication and reproduction clear.)

(4) Hypothesis 1 is that tRNS will cause a differential reduction of contralateral motion coherence threshold, greater than that in either control condition. Rather than propose a repeated-measures ANOVA with planned comparisons, it would be more targeted to specify the exact comparisons of interest and focus upon these (the full ANOVA can be added in subsequent exploration if you wish). The dependent variable of interest for each treatment condition would seem to be the subtraction of contralateral from ipsilateral thresholds, and your critical prediction that this difference will be significantly larger in tRNS than in both control conditions. This seems to suggest that you should perform two independent t-tests to make these comparisons. Because both outcomes would need to be significant to support your hypothesis, no correction for multiple comparisons would be required; see e.g. Rubin 2021). (It is even possible that your theory predicts the conjunction of three effects: significant difference between hemifields in tRNS condition, and significantly greater inter-hemifield difference in tRNS condition as compared with each of the control conditions.)

(5) If that is a correct formulation of your Hypothesis 1, then your targeted effect size of interest should correspond specifically to that effect, rather than (as at present) simply to the hemifield difference within the tRNS condition.

(6) Your effect size estimate is drawn directly from Ghin et al (2018) (although as noted above you may need to adjust precisely which effect size you are estimating). It is possible that this effect is an upwardly-biased estimate, considering that it is a small-n (n=16) study that has been published with a positive finding (see e.g. Button et al, 2013). If so, would it be advisable to be more conservative in setting your effect size of interest? One strategy would be to define the smallest expected effect size, for instance by estimating the lower bound on prior relevant effects in the literature. Ideally, you would draw your smallest expected effect size from consideration of more than a single prior study, although I understand that this may not be possible if there is only one directly relevant study. An alternative strategy is to define the smallest effect size that would be of theoretical interest, and to target that (see Dienes 2019; or for related discussion Lakens, 2022). By powering your study for a conservative or smallest interesting effect size, you would increase the potential informativeness of a null result. Of course, this is likely to mean that you may need to increase the sample size.

(7) The second hypothesis is based on a correlational relationship. If this is a critical hypothesis, then you similarly need to motivate your smallest effect size of interest, and show how much power your study has to detect it.

(8) Make sure that you state the significance threshold for each test, and whether it is one- or two-tailed. You can choose any level of power that you wish, but your combination of power and alpha will constrain the pool of eligible journals in which you might choose to place your Stage 2 manuscript, if it receives recommendation. For a list of PCI_friendly journals, and criteria, see https://rr.peercommunityin.org/about/pci_rr_friendly_journals. You are not obliged to publish your work in a journal at all; I am just making sure that you appreciate how your design choices may affect the pool of potential journals if you do.

(9) Appendix C reports threshold estimation method for your MLP. I think that this may be sufficiently central to the the experiment that it should be fully described within the main Methods.

(10) Make sure that you have considered and clearly stated all relevant exclusion criteria (at trial and participant level). Also consider whether your design has (or needs) any critical manipulation checks/outcome-neutral criteria. These are criteria that must be satisfied in order for your experiment to be deemed capable of testing the hypothesis of interest (which might include the absence of floor or ceiling effects, the presence of positive control effects, or other essential quality checks orthogonal to the main hypotheses.)

References

Button, K. S., Ioannidis, J., Mokrysz, C., Nosek, B. A., Flint, J., Robinson, E. S., & Munafò, M. R. (2013). Power failure: why small sample size undermines the reliability of neuroscience. Nature reviews neuroscience, 14(5), 365-376.

Dienes, Z. (2019). How do I know what my theory predicts?. Advances in Methods and Practices in Psychological Science, 2(4), 364-377.

Lakens, D. (2022). Sample size justification. Collabra: Psychology, 8(1), 33267.

Rubin, M. (2021). When to adjust alpha during multiple testing: a consideration of disjunction, conjunction, and individual testing. Synthese, 199(3), 10969-11000.