Comparing statistical word learning in bilinguals and monolinguals

Thank you for making these changes- its now clear. Just one small change: where you say you will get the values to inform the model of H1 from your pilot/ previous papers- can you state the actual values you will use? They should be in log odds, to match the data summary (though you could also give the odds/ odds ratios as these are a more familiar measure of effect size).

If you could add these into the text and table then I will be happy to approve this.

===

Note from Managing Board: To accommodate reviewer and recommender holiday schedules, PCI RR will be closed to submissions from 1st July — 1st September. During this time, reviewers will be able to submit reviews and recommenders will issue decisions, but no new or revised submissions can be made by authors. If you wish your revised manuscript to be evaluated before 1st September, please be sure to resubmit no later than 11.00am CEST on Monday 1st July.

Thank you for resubmitting your stage 1 report.

I am happy that you have addressed reviewer comments and I didn't see the need to send it back out for further peer review. However, there are some issues with the analysis plan and interpretation of statistics. I outline these below and they will need to be addressed in a revision. I plan to assess it myself, but note that if necessary I might seek additional statistical support.

Comments on analyses plan:

(1)    Your “interpretation” column” in the big table contains some errors. For example, for Hypothesis 1:

a.      you are right that a BF greater than 3 would mean that you have evidence that participants are more accurate in 1:1 than 1:2  - assuming (i) that you are testing a one tailed hypothesis- which is the case if you use the half normal, as stated in footnote 5, and (ii) that 1:1 is coded as 0.5 in the model and 2:1 as -0.5 (so that a positive effect of mapping type is in the direction of higher accuracy in the 1:1 condition)

b.      but a BF lower than 1/3 does NOT indicate that participants are more accurate with 1:2 than1:1. Assuming you conducted the test as I laid out in (a), a BF < 1/3 indicates you have evidence against the H1 that it is easier to learn 1;1 than 1:2 mappings. But note that this is consistent with either both conditions being the same OR with a reverse effect.

c.      Also, a BF between 1/3 and 3 for this test does NOT indicate that they are equally accurate with the two types. Instead, it indicates that the data collected do not provide much evidence to distinguish your H1 from the null (the data are insensitive). So, you can’t say you have evidence that 1:1 mapping is learned better than 1:2 mapping, but you also cant say that you have evidence that this isn’t the case.

(I suggest looking at Zoltan’s excellent webpage https://users.sussex.ac.uk/~dienes/inference/Bayes.htm and following links to paper and short explainer videos in that page. Note that if there are places where you want do test a hypothesis in two directions you can do that either by testing a two tailed hypothesis (in which case evidence for the null will mean evidence that the groups don’t differ in either direction- i.e. they are the same) or testing two one tailed hypotheses.)

(2)    Where you explain analyses in the main paper on p 23, what is in footnote (5) should be incorporated into the text and you need to be clearer that though you run a full model, your Bayes factor to test each hypothesis is computed by extracting a beta and SE for one specific coefficient. Then you will then use the calculator to test if these two statistics (which constitute the data summary) are more likely under the H1 than H0,  where H0 is a point null and H1 is represented as a half normal (i.e. a one-tailed test, because you are testing prediction in a particular direction) with a mean of 0 and sd set to a rough estimate of effect size x.  You need to explicitly lay out what x is for each hypothesis you are going to test (i.e. give the predicted beta value for the fixed effect) and explain where that value came from (though NB this information will also need to be in the table at the end – see next point- so here in the text you could refer the reader to that table.)

(3)    In the table, the analysis plan column should not only say what model you will run but also what fixed effect coefficient you will be extracting for this this specific analysis to test this specific hypothesis. So, for the Hypothesis 1, that is mapping type – the other fixed effects aren’t tested (though having them in the model is appropriate to account for the variance appropriately – and its right that you explain that that is the structure of the model). Again, it needs to be clear that you will be extracting the SE and estimate for that coefficient, and then using these two values as the model of the data that you feed it into the bayes factor calculator. As noted above, the other number that is critical to include in this column is the value of x you will be using as the value of the sd of the half normal. (You can also say  where this value came from).

(4)    Within the text, there is still at least one place where you use the term “significant” (bottom of page 23). Also there are places where you talk about “not finding” an effect (e.g. a difference between monolinguals and bilinguals) but if you are using Bayes factors you need to be more precise and differentiate between the case where you end up with insensitive data (where you don’t find evidence for H1, but also not for H0- i.e. BF between 1/3 and 3)  and data where there is evidence for the null (BF<1/3)

(5)    Power analysis: You say in the letter you are going to do optional stopping but are doing a power analysis to see what the minimum N you need is. I don’t fully follow this. If you are doing optional stopping, I would think what is key is to establish the maximum at which point you will stop even if you haven’t found BF>6 or BF<1/6? Also, if I have understood, the frequentist analysis you plan to do will tell you the sample size you need to find a z>2 in (90%?)  of runs for each of the fixed effects. This is likely (but not guaranteed) to correspond reasonably well to the sample necessary to find BF>3 for the same fixed effects if H1 is true.  However, for Bayes Factors we are also interested in establishing the sample necessary to find evidence for H0 if H0 is true (i.e. if the beta for the fixed effect is truely 0). There may be packages for this, but I only know how to do it with simulation. I am happy to share simulation scripts with if you if that is helpful – feel free to email. I also think what is in the supplementary materials could be a bit more fulsome – maybe include some of the output plots as well as just the script (unless I missed them on OSF- in which case maybe give clearer pointers).

Thank you for resubmitting this Stage 1 RR and for the considerable work you have put into it. It has been reviewed by the same two expert reviewers as before. One of the reviewers is happy with the document, barring some minor points. The other reviewer continues to have some concerns. One concerns the disconnect between the fact that questions about bilingual balance are now exploratory, and the framing in the introduction. I agree with this. For example, The Current Study section (p15) talks about looking at effects of extent of balancedness in the list of “critical components” of the experiment. Its ok to mention it, but it should be made clear that this is an exploratory element, mentioned after the factors related to the key hypotheses. I also got quite confused where you talk about the “Hypotheses 1-4” (p 7) as these don’t seem to match hypotheses later in the paper.

My other concern regards the Bayes Factors. First, can you clarify that you are moving to use Bayes Factors throughout as your statistic for inference, rather than significance? There are places where the word significance is used (e.g. p 28) and these will need to be changed. If you are planning to mix analyses that needs to be very explicit and well justified (it generally isn't recommended -see Dienes, Z. (2023, August 14). Use one system for all results to avoid contradiction: Advice for using significance tests, equivalence tests, and Bayes factors. https://doi.org/10.31234/osf.io/2dtwv). Second, while I applaud the move to these analyses, they are currently under specified. The author information for PCIRR says

“For studies involving analyses with Bayes factors, the predictions of the theory must be specified so that a Bayes factor can be calculated. Authors should indicate what distribution will be used to represent the predictions of the theory and how its parameters will be specified.  For example, will you use a uniform distribution up to some specified maximum, or a normal/half-normal distribution to represent a likely effect size, or a JZS/Cauchy distribution with a specified scaling constant?”

You say you will follow a similar approach to Silvey et al. In that paper, we represented H1 as a half normal distribution with a mean of zero and SD set to x, where x is a rough estimate of predicted effect size. That may be reasonable here too (for one sided predictions assuming small effects are more likely) but you need to explicit say what distribution you are using and why. Most critically, for each hypothesis you need to specify the parameter x – i.e. what size of effect you predict. For where to get these values- I think for this study you have values from pilot data / previous study. If not, Silvey et al suggest some possible ways to estimate main effects and interactions which may (or may not) be appropriate and there are more suggestions in these papers by Dienes:

Dienes, Z. (2021). How to use and report Bayesian hypothesis tests. Psychology of Consciousness: Theory, Research, and Practice, 8(1), 9. https://osf.io/preprints/psyarxiv/bua5n

Dienes, Z. (2021). Obtaining evidence for no effect. Collabra: Psychology, 7(1), 28202. https://osf.io/preprints/psyarxiv/yc7s5

You also need to explain more clearly how you will use the information from the mixed models to get the data summary which is fed to the calculator etc. I know that information is in Silvey et al, but you can’t assume the reader will look at that and it isn't a "standard approach".

A related point is that you talk about doing “power analysis”, but that implies that what you are going to be interpreting is the p value. If instead you are going to compute Bayes factors you need to do analyses that BOTH (1) show that if H1 is true, you have a reasonable chance of finding your threshold BF (e.g. BF>3) AND (2) show that if H0 is true you have reasonable chance of finding BF<0 (which btw is generally harder). I think that you used simulation to do your power analyses, in which case you should be able to adapt these to do the Bayesian version. (For looking at 1, you can run the same simulation but see how often you get BF>3 (or whatever threshold you are choosing) rather than how often you get a p <.05; For looking at (2) you will have to run versions of the model where the relevant coefficient in the mixed model is set to 0, and then you can see how often you find BF below your threshold fo 1/3). I would be happy to share some scripts with you where I did something similar if that is helpful (feel free to email me). I would also request that you share your own scripts for power analyses on OSF as it will make it easier for me – and future readers- to evaluate the approach you are taking.

In a revision, please make sure to address these and the reviewers points (or justify why you choose not to do so).  

First, I would like to apologize for the unusual length of time in getting this decision to you. I had difficulty finding reviewers, and then once reviewers were found there were some further unavoidable delays.  

The reviewers are generally positive about this paper, both in terms of the topic and the design of the study. I agree- this research has a lot of potential.   

One point made by both reviewers is that you need to more clearly motivate your hypotheses. I agree that you can do more to explicitly spell out the links to theory, particularly for the predicted differences between different type of bilinguals.   

The reviewers make several other good points, and I would like to invite you to address these in a revision. I will pick up on a few, mostly concerning analyses.  

**Sample plan 
  
In the text you tell us that you that you planed on effect size “based on the effect size reported by Poepsel and Weiss (2016) and Escudero et al., (2016)”. As the reviewers point out, this is underspecified. Is this an effect size for the main effect of language group, or language group by trial type interaction? What actually is this effects size? What analysis did you do with this value?  
 
In addition, you currently do not have separate justifications of the sample required for each of your hypotheses, but this is a requirement for being accepted as a Stage 1 RR. That is: every row in the big table at the end has to have its own sample size justification. (If there are places where you really aren’t able to do that, you can still do the analysis as an exploratory analysis, but they won’t be part of your pre-registered analysis plan and this means they can’t feature in the abstract and if you talk about them in the discussion they can’t be the main focus.) 
 
I think the only way you are going to be able to get all of these values is through simulation. If I have understood correctly, you only currently use simulation-based power analysis when looking at the effect of trial type. The reason was that you were using your pilot data, and you couldn’t use this method for the hypotheses where the relevant main effect/interaction was not significant. I see the difficulty here, however it should be possible for you to simulate mixed models using a combination of some parameters extracted from a models run over your pilot data (e.g. for the random effects and for the effects of trial type and block) but replacing the estimate for the main effect of language group, and for the language group by trial type interaction with relevant values from Poepsel and Weiss (2016) and Escudero et al., (2016) etc. (You could either use beta estimates from directly from equivalent glmer models, or if those aren’t reported you could compute odd ratios - and from these log odds- based on the relevant differences between means. If there are places where there really is no previous value in the literature you can use for a fixed effect, as a last resort you could consider estimating using rule-of-thumb estimates for odd ratios as indices of effect sizes  (see e.g. https://easystats.github.io/effectsize/articles/interpret.html#odds-ratio-or,). Similarly, for the analyses looking at trial by trial data, you can eithr use relevant estimates from the pilot or from values extracted from models from previous datasets (or some combination). 
 
(NB- I am not sure whether you did the power simulations using a package or if you wrote your own code. If the former and you would like help with code for simulating data for mixed effect models (i.e. to make it a bit more flexible) I would be happy to share some scripts with you.  (They might not do exactly what you need but might save you a bit of time in getting started). Feel free to reach out by email if that would be helpful.) 
 
** Assessing evidence for H0 
One of the reviewers asks you to consider including a measure to assess the strength of evidence for H0/H1. I agree with this and would strongly encourage you to consider using either Bayes factors or equivalence tests (Lakens, D., McLatchie, N., Isager, P. M., Scheel, A. M., & Dienes, Z. (2020). Improving inferences about null effects with Bayes factors and equivalence tests. The Journals of Gerontology: Series B, 75(1), 45-57.) 

This isn’t a requirement for stage 1 acceptance, however if you don’t have any method to assess evidence for the null, then you cannot make statements such as the following: 

“Thus, if we do not observe a performance difference between language groups (either as an   
interaction with mapping type or as a main effect), we will conclude that bilinguals are not   
better at learning words statistically than monolinguals.-- “ 

A p value greater than .05 doesn’t provide any evidence against an effect, and so you can’t draw this conclusion (Dienes has written many papers on this – see his website http://www.lifesci.sussex.ac.uk/home/Zoltan_Dienes/inference/Bayes.htm for references)  
 
If you did decide to use Bayes Factors and want to combine this with logistic mixed effect models, you might also find this preprint https://psyarxiv.com/m4hju (on which I am an author) useful for some practical demonstrations of how you can combine linear/logistic mixed effect models run using the lmer package with Bayes Factors. An alternative (and more commonly used) approach is to use the brms package and use model comparison (this is also briefly discussed at the end of our preprint). Both approaches require you to have an estimation of your predicted effect sizes to inform model of H1. However, you could use the same estimates you use for the fixed effects in power simulations. If you instead compute equivalence tests you will need to come up with ways of estimating the smallest effect sizes of interest for each hypothesis. 
 
 
** Other smaller points 
 
-random structures: is the mapping of labels and words randomized across participants?  If not, I think you might need a nested random effects structure for these?  (If it is randomized and I missed this, maybe remind the remind the reader of this when you talk about the random effects)  

-target count (H4) as a fixed effect: you say this will be log transformed, will it also be centered around 0 as you do for your other fixed effects? 
“we will exclude any participants with a linear slope of the accuracy of −1 SD deviation from average (slope will be calculated across bins of 36 trials; c.f., Roembke et al., 2018).” this needs justification beyond a reference to another paper  

-I agree with the reviewer that it is odd to use terminology of “balanced” and “unbalanced” bilinguals where your criteria seem to be more about onset of acquisition than about proficiency