A mental health perspective to adolescents’ social media experiences
Identifying relevant experiences to the measurement of social media experience via focus groups with young people: A registered report
Recommendation: posted 15 August 2023, validated 16 August 2023
Karhulahti, V. (2023) A mental health perspective to adolescents’ social media experiences . Peer Community in Registered Reports, . https://rr.peercommunityin.org/articles/rec?id=487
In the present registered report, Hickman Dunne et al. (2023) carry out a focus group study with adolescents (n=32) aged 11 to 15 in Northwest England to improve the understanding of constructs related to social media and mental health experiences. The work is carried out as part of a long-term measure development project. The authors apply reflexive thematic analysis to explore adolescents’ social media use experiences and related motivations in the light of mental health, in addition to which the adolescents’ own views of benefits and risks are mapped out.
A particular strength of the design is the engagement of three Young Researchers who will co-facilitate the focus groups and be involved in the analysis. The research plan also meets high reflexivity and transparency criteria, and as such, can significantly contribute to future scale development as well as our general understanding of adolescents’ social media experiences.
The Stage 1 manuscript was reviewed over two rounds by five unique reviewers, one of which participated in both rounds. The reviewers’ expertise ranged from social media and technology use research to health psychology and qualitative methods. Based on careful revisions and detailed responses to the reviewers’ comments, the recommender judged that the manuscript met the Stage 1 criteria and therefore awarded in-principle acceptance.
Level of bias control achieved: Level 2. At least some data/evidence that will be used to answer the research question has been accessed and partially observed by the authors, but the authors certify that they have not yet observed the key variables within the data that will be used to answer the research question.
List of eligible PCI RR-friendly journals:
1. Borsboom, D. (2005). Measuring the mind: Conceptual issues in contemporary psychometrics. Cambridge University Press.
2. Hickman Dunne J., Black L., Banwell E., Nanda P., Anderton M, Butters L.C., Demkowicz O., Davidson B., Qualter P., Humphrey N., Jay C., and Panayiotou M. (2023). Identifying relevant dimensions to the measurement of adolescent social media experience via focus groups with young people: A registered report. In principle acceptance of Version 5 by Peer Community in Registered Reports. https://osf.io/w24ec
The recommender in charge of the evaluation of the article and the reviewers declared that they have no conflict of interest (as defined in the code of conduct of PCI) with the authors or with the content of the article.
Evaluation round #2
DOI or URL of the report: https://psyarxiv.com/erjvz/
Version of the report: 2
Author's Reply, 15 Aug 2023
Decision by Veli-Matti Karhulahti, posted 11 Aug 2023, validated 11 Aug 2023
Dear Margarita Panayiotou and co-authors,
One of the reviewers was able to fully reassess the new manuscript and your responses within the present time limits, in addition to which I have personally reviewed all revisions and materials. We both agree the manuscrip is generally ready for in-principle acceptance, but I make small final notes that can help making the wonderful plan even more wonderful.
The MS now has a clear aim to exlore mental health. However, how do the RQs connect to it? This was explained in the review replies, but it might not be evident for readers. A simple solution could be to make it explicit in the RQs, something like this (just examples):
RQ1: What are the motivations behind adolescent social media use?
—> How motivations behind adolescent social media use relate to mental health?
RQ2: What are adolescents’ social media experiences?
—> What are adolescents’ social media experiences in the light of mental health?
What are the views of adolescents of the risks and benefits associated with using social media?
—> What are adolescents' views of mental health risks and benefits associated with using social media?
Such changes could clarify and ensure that findings both answer RQs and help measure development.
On page 10 you correctly refer to QHs aiming "to disclose and pre-register our hypothetical biases" (as we framed it). I would rarely pick a single word to change, but since RTA explicitly opposes the term "bias" (Braun and Clarke 2023 10.1080/26895269.2022.2129597: "don’t mention bias", p. 4), you might just erase that one sentence, as the paragraph reads coherently without it too.
Page 12, a sentence is missing a word: "This also recognises that some socio-demographic characteristics might more easily accessible to teachers than others."
The data sharing process is clear. However, what materials produced in analysis/coding are you planning to share? As I reread the MS, I realised you're using the term "document" (e.g., "We will document the process of theme generation with reflexive notes"), but this is not discussed in the ethics/data sharing section. Will the documentation be private or public? Based on my experience, it's good to decide/plan early on what materials will be public or shared with reviewers, as codes and their thematic iteration often involve identifiers and the de-intentification process can be challenging unless researchers pursue it from the start of analysis. A good discussion of related issues can be found in Branney et al. 2023 (https://doi.org/10.1111/spc3.12728).
After considering the above brief notes, I believe we're ready for an IPA for this valuable study.
Reviewed by Jana Papcunova, 09 Aug 2023
Evaluation round #1
DOI or URL of the report: https://psyarxiv.com/erjvz/
Version of the report: 2
Author's Reply, 01 Aug 2023
Decision by Veli-Matti Karhulahti, posted 23 Jul 2023, validated 23 Jul 2023
Dear Margarita Panayiotou and co-authors,
Thank you for submitting your Stage 1 RR to PCI RR. I am delighted to deliver this decision letter exceptionally with no less than five wonderful reviews. All reviews provide valuable comments from their different positions. I agree with the reviewers that this is a promising proposal. I don’t want to unnecessarily expand the already large amount of feedback, but I must make a few notes that I believe can significantly improve the outcome at Stage 2. Especially, it’s important to take methodology seriously at Stage 1 when we can still work on it. Doing it carefully can save time and help getting results that contribute to the project in a desired way.
Although it is surely possible use RTA in this study, two reviewers already make insightful observations about the difficulties of matching RTA with the current design and goals. I agree with them and am also worried that the underlying philosophical premises – driven by measure development, multiple analysts, and robust conceptualization – are not fully in line with RTA. Essentially, RTA is a non-positivist method defined by researcher subjectivity. It is not very easy to match that with the present process and large team. With reference to Braun & Clarke’s guidelines for editorial RTA assessment:
“A research team is not required or even desirable for quality … We contend that even TA with a descriptive purpose is an interpretative activity undertaken by a researcher who is situated in various ways, and who reads data through the lenses of their particular social, cultural, historical, disciplinary, political and ideological positionings. They edit and evoke participant ‘voices’ but ultimately tell their story about the data” (2021: https://doi.org/10.1080/14780887.2020.1769238) … “themes in [RTA] are conceptualized as meaning-based, interpretative stories” (2023: https://doi.org/10.1080/26895269.2022.2129597)
I encourage you to consider one more time whether you wish to report these data — that ultimately aim at measure development — under the above RTA premises. To be clear, I fully support whatever analytic approach is chosen and it’s ok to use RTA, but in case of choosing RTA, we must ensure that the Stage 1 plan coheres with it and is doing it knowingly. You have already done a great job explaining some elements of reflexivity. But Braun & Clarke (2023) explicitly warn about “positivism creep” in RTA studies, such as aiming for “accuracy” or “assuming… line-by-line coding apply to TA without any explanation or justification” (2021: p. 345). As some reviewers point out, there are instances that imply such issues, e.g. identifying analysis as “data-driven” (p. 13) and “new themes created to represent the data more accurately” (p. 14) and “Systematic line-by-line coding to organise data” (p. 15). You also refer to saturation (p. 11), which is against RTA (2019: https://doi.org/10.1080/2159676X.2019.1704846) – since you’ve written “guided by” it’s not a major issue here but also raises questions: how saturation, which doesn’t apply to RTA, can guide RTA?
If the RTA approach is kept, it would be necessary to revise the MS systematically with these features in mind and include a separate section where epistemological, ontological, and other author/team positional premises are explicitly discussed, with a plan for integrating them in outlined semantic/latent analytic process. Alternatively, a list of justified deviations could be noted, or other TA or qualitative approaches, which are not so fully committed to researcher subjectivity, could be applied.
The section “The Appropriateness of the Registered Report Format” is ok, but I would also like to offer an opportunity to remove it and win some word space. I understand you wish to justify the use of RR in a qualitative study because it’s historically not very common, but today when we already have multiple Stage 1 and Stage 2 qualitative RRs (and a primer: https://doi.org/10.12688/openreseurope.15532.2), I don’t see it necessary. To be clear again, you can also keep this section if you wish.
I agree with the reviewers that the RQs are generally appropriate, but it’s also a bit unusual to have as much as five unique RQs! Sufficiently answering all of them in a single study -- especially considering that RTA typically leads to 2–6 themes -- can be a challenge. How do you ensure that the themes generated in the inductive process will match all five RQs? After taking into consideration the reviewers’ valuable feedback on conceptualization, please carefully assess whether fewer and more focused RQs could be the basis for analysis.
I also agree with the reviewers that the background for QHs needs more explanation and justification. Only a single previously undiscussed study (van der Wal et al., 2022) is cited to ground them, without explaining what that study says. It's ok to not commit to any specific theory in a study like this, but a reader is left thinking through what conceptual or theoretical frame do you understand e.g. “motivation” (incentive salience, SDT, etc?) and “experience” (narrative identity, phenomenology, etc?)? I would personally prefer to have a paragraph for each QH that clearly explains why do you expect this (but other solutions can work too). Also, “heterogeneity” is used a bit vaguely -- what qualitative data would not be heterogenous? Following that you already have mentioned age (this’s great!), perhaps consider the reviewers’ suggestions and expect heterogeneity to be associated with the use of different platforms, specific apps, or similar? (If that’s what you actually expect, naturally.)
Reviewers worry about the sample, and I share that. Adding to their notes, my worry is the following: because many negative experiences with technology tend to go hand-in-hand with accumulating health and life challenges, the focus groups might be selecting for participants who are doing well or at least fine – and the voices of those who are socially excluded, lonely, or otherwise uncomfortable speaking in a peer group will not be heard. This could be exactly the group whose experiences are at the core of all debates. This should be discussed in limitations. Also, I’m thinking, perhaps follow-up 1-to-1 interviews with selected individuals could mitigate this issue to some degree.
Two technical corrections about references to my own studies. Qualitative hypotheses are not discussed in the cited solo-authored 2022 article, but in our team RR: https://doi.org/10.1525/collabra.38819 . Later the same article is also cited for data anonymization guidelines; that seems to be a confusion with another article from the same year which (unlike the cited article) addresses qualitative data anonymization: https://doi.org/10.1111/bjso.12573 (I also encourage following the references, many of which can be more informative than my own paper -- e.g. Libby Bishop, Arja Kuula-Luumi, and Peter Branney’s teams have written really helpful work about qualitative data sharing and stewardship!)
One reviewer suggests considering duplication of the focus groups. It would be a wonderful addition for sure, but also I understand you may not have resources (or ethics approval) for that. If you do consider going forward with it in this study or later, online focus groups could be a fit: Flayelle et al. 2022 https://doi.org/10.1111/add.15944
Finally, I think member reflections in later parts of analysis could be very helpful: Tracy 2010 https://doi.org/10.1177/1077800410383121 (again, you can also skip if you’re not comfortable with this element)
I hope you find the reviewers’ generous comments and the above extra notes useful in your revision. Please respond to all reviewer feedback carefully. At any point, you can contact me directly if you wish to further negotiate about how to proceed most optimally, or with any other questions. I’m confident this will become a highly valuable study and help in measure development.