Different ways of describing problematic substance use and its treatment influence public stigma

Dear Charlotte

We are getting there but a few points. In a few places you say things like:

"participants in the drug use condition allocated greater punishment compared to those in the health concern condition (d= .14, 99% CI = .02, .26), with the upper CI significantly outside of the equivalence range"

The upper limit of the CI is not "significantly outside" the equivalence range, it is just "outside" of it. That phrasing makes it sound like .26 is significantly different from .2 - but that has not been tested. What matters is what one can say about the population mean; and the sample mean is not significantly outside the equivalence region, so the set of possible population means, i.e. what is inside the CI and hence cannot be rejected, includes values deemed too small to be interesting i.e. practically equivalent to zero. You are allowed to say the drug use condition allocated greater punishment compared to those in the health concern condition; but the following clause should be deleted. (Also consider other similar sentences.) But given your claims about what effects are meaningful, and your Stage 1 decision procedure, even this phrasing is not the one you should use (see below).

You say "We refer to an effect as ‘significant’ if, given α = .01, the mean difference is
significantly different from zero and the 99% CI falls outside of the equivalence range" - but this is not the rule that has been applied, at least on a straightforward interpretation of what "falling outside" means, and given the logic of wanting to exclude all meaningless values in order to conclude there was a meaningful one; and also given how such reasoning has proceeded since Greenwald (1975) onwards; and given the need for consistency in a decision procedure for when a difference is regarded as good enough to be meaningful. If this rule were applied then you would declare a meaningful difference if the bottom limit of the CI were above the upper limit of the equivalence region (and vice versa, for a CI below the equivalence region). That rule, as I have just stated it, makes perfect sense. In the stage 1 you phrased it thus: "Equivalence will therefore be asserted if, given α = .01, the 99% confidence interval of the mean difference lies within this equivalence region, and rejected if the 99% CI lies outside of this region" and that states what I have just stated - equivalence is only rejected if the CI lies outside the equivalence region. Now I see you haven't distinguished "completely" from "partially" - but "partially" leads to self contradiction in what erffects are regarded as meaningful. Thus, the striaghtforward interptation of what you have stated that preserves self consistency is that outside means completely  outside - and I presume you wish to avoid self contradiction.  Conversely, equivalence only is declared when the CI lies within the equivalence region. THis means you suspend judgment in all other cases - namely if the CI overlaps both the equivalence region and the region of values deemed meaningful.  Notice this condition for suspending judgment is different from the one you have just stated; for example, if a sample mean is significantly different fropm zero, yet the CI spans equivalent and meaningful values, you suspend judgment. This rule needs to be consistently applied.

Concerning whether an effect size of .15 is meaningful for some tests but not others, in the absence of a principled argument for why, this seems arbitrary. So bear this in mind in how you interpet results.

Thank you for your revision!  I still see some inconsistencies in applying the logic of a CI being within or outside a suitably defined equivalence region:

1) "For the Financial Discrimination Task, participants in the drug use condition allocated greater punishment compared to those in the health concern condition, with the observed effect size (d= .14, 99% CI = .02, .26) significantly outside of the equivalence range"

But the sample mean is not significantly outside the equivalence region. If it were signfiicantly outside the equivalence region, the lower limit of the CI would be equal to or above the upper limit of the equivalence region. This problem occurs in various sentences. If by assumption values within the equivalence region are too small to be of interest - that is what an equivalence region is - then a CI that includes such values does not establish that an effect of interest was found.

2) "The difference for continued care (d= .01, CI = -.15, .17) was not statistically different to zero and equivalent."  Also: "and for punishment (d= .007, 99% CI = -.16, .17) was equivalent" Yet, as you say elsewhere, these CIs include effect sizes previously regarded as of interest.  This point needs to be explicitly made. Also you should explicitly state that your equivalence region may include values of actual interest, at a point in the discussion where you conclude equivalence.

best

Zoltan

One reviewer from the Stage 1 has responded and indicated their approval of the Stage 2. Incidentally their review read to me as possibly about the Stage 1 (a reviewer sees links to both Stage 1 and Stage 2), but I have double checked with them - they definitely read the Stage 2, they were positive and there were no problems.

Just one thing on my side. As per my comments for the Stage 1, equivalence cannot be concluded unless there is justification for the equivalence limit being so small that it is only just interesting/uninteresting from the point of view of the scientific theory tested. Resource limits obviously do not provide that justification, as they are arbitrarily related to theory. Indeed several of your CIs within the "equivalence region" include effect sizes that past studies have used as support for the theory. So remove references to having found equivalence, and just e.g. leave it as a case of having estimated the possible effect size.