Understanding the role of health condition, aetiological labels, and attributional judgements in public stigma toward problematic substance use
To help or hinder: Do the labels and models used to describe problematic substance use influence public stigma?
People suffering from substance misuse problems are often stigmatised. Such public stigma may impair such people obtaining help and the quality of help that they receive. For this reason, previous research has investigated the factors that may reduce stigma. Evidence has been found, but not consistently, for the claim that labelling the condition as "chronically relapsing brain disease" vs a "problem" reduces stigma; as does "a health concern" vs " drug use". Another potentially relevant difference that may explain different previous results is describing how effective treatment can be.
In this Stage 1 Registered Report, Pennington et al. (2022) describe how they will investigate if any of these factors affect two different measures of stigma used in previous work, with a study well powered for testing whether the 99% CI lies outside or inside an equivalence region. While the CI being outside the region will straightforwardly justify concluding an effect of interest, a CI within the region will need to be interpreted with due regard to the fact that some effects within the region may be interesting.
The Stage 1 manuscript was evaluated over two rounds of review (including one round of in-depth specialist review). Based on comprehensive responses to the reviewers' comments, the recommender judged that the manuscript met the Stage 1 criteria and therefore awarded in-principle acceptance (IPA).
URL to the preregistered Stage 1 protocol: https://osf.io/4vscg
Level of bias control achieved: Level 6. No part of the data or evidence that will be used to answer the research question yet exists and no part will be generated until after IPA.
List of eligible PCI RR-friendly journals:
- Addiction Research & Theory
- Advances in Cognitive Psychology
- Journal of Cognition
- Peer Community Journal
- Royal Society Open Science
- Swiss Psychology Open
Pennington, C. R., Monk, R. L., Heim, D., Rose, A. K., Gough, T., Clarke, R., Knibb, G., & Jones, A. (2022). To help or hinder: Do the labels and models used to describe problematic substance use influence public stigma? Stage 1 preregistration, in principle acceptance of version 2 by Peer Community in Registered Reports. https://osf.io/4vscg
Zoltan Dienes (2022) Understanding the role of health condition, aetiological labels, and attributional judgements in public stigma toward problematic substance use. Peer Community in Registered Reports, . https://rr.peercommunityin.org/articles/rec?id=144
Evaluation round #2
DOI or URL of the report: https://osf.io/dk694/
Version of the report: v2
Author's Reply, 21 Jan 2022
Decision by Zoltan Dienes, 17 Jan 2022
I now have two reviews which are largely positive about your submission. Sinclair-House asks you to comment on the difference between alcohol and opioids in previous studies and the bearing this may have had on different results; and Giner-Sorolla asks you to comment on the quality of the attributional manipulation if it weren't simply trying to resolve differences between previous studies.
Giner-Sorolla raises questions about Bonferroni (interestingly I made almost the exact same point about Bonferroni here https://psyarxiv.com/pxhd2 pp 19-20). On the other hand, given you give yourself some interpretational flexibility in choosing which measure is better simply by whether it yields the conclusion of a difference, I think some statistical conservatism is fine.
You still have a two-step procedure which is inferentially incoherent, i.e. test whether the mean is within or outside the equivalence region only after a signfiicant result against the H0 of no effect. Why not simply test whether the x% CI is completely outside or within (or only partially within) the equivalence region? That is, drop the initial significance test again the H0 of zero effect, an H0 you have implied is of no relevance by claiming there is minimally interesting effect size.
For RQ3 in the design table, do not label it as exploratory ( to keep things clean don't describe any exploratory analyses in the Stage 1); for the theory at stake simply state the broadest claim that is at stake given your findings (regardless of whether past research has looked at this not, which is not relevant to whether this study tests that claim). In the row for RQ3 be clear this is testing a diference of differences.
Finally, as we have talked about, justify minimal effects of interest by their relevance to the theory tested (in its scientific context) rather than by researcher resources, which have no inferential relation to what a theory predicts.
Reviewed by Nicholas Sinclair-House, 22 Dec 2021
Reviewed by Roger Giner-Sorolla, 16 Jan 2022
Evaluation round #1
DOI or URL of the report: https://osf.io/zd754/
Author's Reply, 05 Nov 2021
Decision by Zoltan Dienes, 02 Nov 2021
Before sending out for review I just want to check a few statistical matters.
1) Your N is set by a power calculation for testing against an H0 of no effect. But you wish to interpret non-signfiicant results with equivalence tests. That means you have one system of inference for asserting there is an effect, and a different one for asserting there is not an effect of interest. This can lead to contradictions, e.g. a significant test against no effect would have led to a conclusion of no effect of interest if equivalence testing alone had been done. You may wish to use a system that is more consistent. For example, you could just use hypothesis testing against no effect with high power; or you could just use equivalence testing, generalized as an "inference over intervals" such as the rule that if a 90% CI is completely within the equivalence region there is no effect of interest; if completely outside there is an effect of interest. If using inference over intervals, the relevant way to estimate N would be the N needed for the CI to fall within the equivalence region e.g. 90% of the time when there is no effect; and outside of it 90% of the time when there is the predicted effect (see https://psyarxiv.com/yc7s5/).
2) You define d = 0.2 as your minimal effect of interest. Why wouldn't d= 0.11 be of relevance to the theory being true, given you cite such an effect as one obtained in past studies and taken seriously I presume as an effect of interest? Is there really the same minimal effect of interest for all contrasts and DVs? (THat would imply that e.g. all DVs have the same reliability.) (See paper just referenced.)
3) You say main effects will be followed by Bonferroni corrected t-tests; I am not sure what these would be given df = 1 for all main effects. Specify the family of tests and what the correction will be, if you are going this way. What I actually recommend is that you pick the one contrast that best tests each theoretical claim, and stick with that; i.e. each row i nthe design table has one test aligned to the substantial theoretical claim at stake. Other tests can be exploratory and reported as such in the Stage 2.
4) There is some flexibility in drawing conclusions given a number of DVs are used. Is it possible at this point to relate different DVs to different theoretical questions, i.e. to make clear what conclusions you would draw given different patterns of outcome, and how that relates to the main theory?
5) When you say non-significant effects will be followed by equivalence tests, did you mean the interaction effects as well? Or, put another way, if you are going for inference by intervals, presumably you will use that same inferential method for interactions and main effects?