The authors have addressed all the comments and suggestions. Their response showcases a clear understanding of the areas that required improvement in their study design, and they have implemented changes accordingly. I commend the authors for their proactive approach to refining their work. The addition of the new section titled "The Construct of Interest and its Conceptualisation," along with supplementary materials on the study design and delivery, has offered more in-depth information about the study. In conclusion, the authors' response to the comments is comprehensive and well-executed. Their revisions have significantly strengthened the manuscript, enhancing both conceptual clarity and methodological transparency.

Jana Papcunova

I enjoyed reviewing this Registered Report about adolescent focus groups for social media measure generation. 

Note: I have no expertise in qualitative or focus group methods, so an additional review will need to be sought for this area of the manuscript. I have focused on the area of social media use, adolescence and pre-existing measurements. 

The manuscript was well written and tackled an interesting set of research questions. I felt it was lacking predominately in two areas:

1. Review of pre-existing social media measures

The social media measurement space is vast and includes a wide range of measurements that are not addiction centred (these were the only ones mentioned in the manuscript). The review on pages 3 and 4 was not extensive enough to adequately give an overview of this measurement space and provide a background for where the measure to be developed would sit (see also point 2) and what its contributions would be. There have also been recent measures developed to tap into social media experiences (see for example a new scale of social media flourishing in adolescence https://doi.org/10.3389/fdgth.2022.975557) which need to be considered. 

2. Conceptualisation of the area of measurement 

While reading the manuscript, I was not sure what the planned measure is trying to quantify. On page 3, the authors introduce that considering “social media experiences” is important, but then note that “quality” is also relevant.  Later in page 6 the authors then talk about “identifying features of social media”. On Page 8 they intend to measure “experiences, motivations and perceptions”. These are all different, and would result in very different measures with different intended uses. 

I think it is crucial for the authors to specify at what level they intend to measure social media use effects (e.g., see Adrian Meier and Leonard Reinecke review paper). It will be impossible to measure multiple levels in just one questionnaire measurement. For example, if they want to pin down social media “effects”, a measure of just experiences might be ineffectual as “ I think likes impact my mood” has a very different conceptual/methodological meaning to measuring the amount of likes received and statistically linking it to mood. Further, measuring social media features engaged with (e.g., Instagram stories), is very different to measuring activities (e.g., perpetrating cyberbullying). Motivations (e.g., I intend to go on social media to chat to my friends) are very different from actions (e.g., I went on social media X times last week to chat to my friends). Each of these levels would have potentially hundreds of potential constructs to measure, so the broad conceptualisations in the paper currently seem non-feasible. 

Minor points:

1. Page 3, “this is necessary”, I am not sure what “this” refers to

2. Page 5, very long paragraph with many separate ideas, I would recommend to split 

3. Page 5, bottom: I don’t think the authors’ bottom up approach completely avoids drawing on existing conceptualisations of social media use, as pre-existing conceptualisations are also impacting how adolescents think about and perceive their social media use. E.g., see work on social media mindsets by Angela Lee and Nicole Ellison. I think a more nuanced discussion is needed here. 

4. Page 6. “Psychometric perspective (above)”, I am not sure what is meant here.

5. Figure 1, not clear what the numbers in the figure mean at the moment. 

6. Page 10, I think more detail would be good about how a balanced sample will be recruited. For example, what prevents the authors from using specific quotas for balancing gender, ethnicity, SES and marginalised groups? 

7. Page 11, while bringing in young people might be an opportunity to minimise power dynamics, their demographics (especially age and gender) might also contribute to how young people disclose about their activities. I would have benefited from a more nuanced discussion here. 

8. Page 11, how long are the focus groups? 

9. Page 12, I did not have the permissions to review the osf document with the questions, and  as wondering whether it would make more sense for those to be put into the main text.

Signed, 

Amy Orben

Review: Identifying relevant experiences to the measurement of social media experience via focus groups with young people

I am thankful for the chance to review this RR and provide some valuable suggestions that could help enhance the ideas presented within.

The authors emphasize the need for valid and reliable instruments to understand adolescents' experiences, motivations, and perceptions of social media, and to assess the effects of social media use on adolescent mental health. The inclusion of user consultation, particularly through focus groups, is highlighted as a valuable approach to inform measure development. The RR is well-structured and provides a clear rationale for the research. The arguments presented are supported by relevant references. However, there are a few areas where further clarification and expansion would strengthen the manuscript.

1. Conceptualization. The argument for conducting focus groups as a bottom-up approach is well-justified, given the inconsistent conceptualizations of social media experience and the potential biases in existing measures. I would encourage the authors to place their undertaking within the broader area of concept explication (e.g., Chaffee, 1991, Sage). This framework involves two stages, meaning analysis and empirical analysis, which aim to clarify and define a concept or construct. By adopting this framework, the authors can systematically analyze and refine their understanding of adolescents' experiences, motivations, and perceptions of social media, as well as the effects of social media use on adolescent mental health. This approach will establish a strong foundation for developing valid and reliable instruments and contribute to advancing knowledge in the field.

2. Ensuring the protection of young adults. The authors mention that the Young Researchers (YRs) were involved in designing the study, ensuring appropriate focus group schedules, study procedures, etc. It would be helpful to provide more specific details about the role of YRs in these activities. How were they engaged in the study design process? Did they provide feedback on the study materials or were they involved in making decisions about the focus group questions? Given the prevalence of cyberbullying among adolescents (Zhu et al., 2021), it is crucial to prioritize the well-being of participants in the study. Establishing protocols for participant support and follow-up, including follow-up debriefing, is essential. By addressing potential risks and providing support, the authors can ensure ethical considerations and contribute to a positive research experience for the adolescents involved.

3. Recruitment and Sampling. It would be valuable to discuss any steps taken regarding the specific strategies to ensure the representation of diverse backgrounds and marginalized groups. How will the purposive sampling approach be implemented? Will it involve targeted recruitment within the schools or specific inclusion criteria for marginalized groups?

4. Data Collection. The manuscript provides a clear description of the focus group schedule and procedure. However, it would be beneficial to elaborate on how the roles and responsibilities will be divided among the facilitators and the Young Researcher? Additionally, the manuscript mentions the availability of post-it notes for participants to write down additional thoughts. Will these post-it notes be collected at the end of the focus groups, and if so, how will they be incorporated into the analysis?

5. Generalizability. Acknowledging the limitations of convenience sampling is crucial, including the potential for selection bias and limited generalizability of the findings. To address these limitations, the authors could consider implementing a process of "duplication" by collecting data from other cultural contexts. This approach would entail replicating the study procedure in different cultural settings, facilitating the diversification of the data. For more detailed information on this approach, refer to Karhulahti (2023) at the following link: [https://osf.io/ekm8x].

Overall, the methodology and protocol demonstrate a strong commitment to engaging young people as active partners in the research process. The approach is well-justified, and the methods employed are appropriate for achieving the research objectives. The study has the potential to contribute valuable insights into the experiences, motivations, and perceptions of adolescents regarding social media use and its effects on mental health.

Jana Papcunová

Thank you for the opportunity to review this protocol. The authors propose a qualitative study, using a series of focus groups to explore adolescent social media experience. This will ultimately inform the creation of a social media experience measure. 

Rationale and Research Questions

The authors present a clear rationale for the research, highlighting an accurate representation of the current contradictions founds within the field and the need to bring clarity to the topic. The field is sometimes led by scaremongering and assumptions, so I am very keen to see research that is adolescent-led. The authors make a good argument behind their decision to use focus groups to later inform a Delphi Study. I am pleased to see the research is situated within a larger plan of study and that decisions surrounding this process have been considered and discussed. Furthermore, it is excellent to see that young researchers have been invited on to the project as co-authors.

Five research questions are proposed. These are logical, interesting and appropriate. I appreciate that qualitative hypotheses are provided to highlight prior expectations in line with bias. Given QH1, and expectations of heterogeneity, it may be worth thinking about management of such differences within the focus group itself. I wonder if it’s worth providing opportunities for participants to further discuss any issues that they feel were pertinent but not discussed outside of the focus group?

Method and Analysis

The overall method of recruitment and procedure is clear. From my understanding, focus groups will be made up of pupils from one year group within one school. I wonder whether the authors have considered what procedure to follow if a student is not comfortable talking in a focus group in front of another particular student? Could focus group names be circulated prior to the start of the focus groups to allow students to notify the researcher of any conflicts?

The interview procedure has been very well though through. The use of flipcharts and post-it notes seem wise suggestions and I can see their use as being very valuable. I agree that RTA seems like a good analysis strategy. The authors note that an inductive approach will be used. Although I agree this suits the research aims of the paper, they need to be weary of prior biases as set out from expectations highlighted in QH2. I don’t think this is problematic, but it would be useful to have some reflection on this following data collection. That being said, I like that the authors have decided a strong, explicit process for coding to ensure theme generation is grounded in the data. 

The authors have considered the importance of safeguarding surrounding the data, and I am pleased to see this reflected in their discussion and decisions to restrict access.

I have tried to view the interview schedule but unfortunately do not have access to see this.

Minor notes

Page 6: I am not keen on the following wording: “By conducting focus groups first, we are privileging the voices of young people in the research process and using their voices to give the critical ‘on-the-ground’ perspective (Fredricks et al., 2016). ”. I understand the intent of the sentence but I do not feel that ‘privileging’ is the correct terminology here.

Page 10: It’s suggested that the focus groups will be run in June-July 2023. This will need updating to a more realistic timeframe. 

Overall

I am impressed by the thorough nature of the protocol; particularly surrounding the clarity of the research procedure and the thought processes that have driven decisions. The research appears well thought-out and I do not foresee any major limitations in the protocol. This looks to be some really useful and important research.

Dr Lisa Orchard, University of Wolverhampton

Thank you for the opportunity to review this registered report. It’s particularly great to see qualitative research being approached in this fashion. I think overall the plan is really good, although I do have a concern regarding tension between the reflexive TA method and the RR format. I’ve outlined my feedback according to the reviewer criteria provided by PCI. 

1A. The scientific validity of the research question(s)

I think it would be nice if your introduction provided more of a direct link between consulting stakeholder groups about their experience and subsequent psychometric measure development to really situate these research questions and the need for this work. I’m of the opinion that stakeholders should really be involved in pretty much any work - but why in this specific study and why now? For instance, something like examples of this being done in other fields might be appropriate. Relatedly, the sentence “although recommended approaches exist for such conceptualisation work with user consultation and are considered fundamental to the quality of a given measure, this step is rarely carried out in psychological measurement more generally” reads oxymoronically: how can they be fundamental if they are rarely carried out? 

On a more minor note, RQ3 and RQ4 feel slightly too broad to me: this could be overcome by defining what you mean by ‘motivations’ and ‘experiences’, as these terms are often used in various theoretical contexts. 

1B. The logic, rationale, and plausibility of the proposed hypotheses (where a submission proposes hypotheses)

It would be useful if you expanded more on what previous literature the hypotheses are based on, maybe split by hypothesis? I think it would also be more intuitive if the hypotheses mapped more clearly to the RQs and this was highlighted. 

1C. The soundness and feasibility of the methodology and analysis pipeline (including statistical power analysis or alternative sampling plans where applicable)

Generally the description of the methods is great and there are a couple of particularly nice touches, like the inclusion of the young researchers and the process outlined for the anonymisation of the transcripts. My main concern with the methods is around your use of reflexive thematic analysis. 

By contrast to other TA approaches, like codebook TA, reflexive TA in particular highlights the role of the researcher and the way they perceive and interpret the data as shaped by their own experiences, and is the most flexible of the TA approaches. For example, Braun & Clarke (2019) write “Assumptions and positionings are always part of qualitative research… reflexive practice is vital to understand and unpack these.” (https://www.tandfonline.com/doi/full/10.1080/2159676X.2019.1628806?casa_token=ktmGJWBjbTsAAAAA%3A_7yQvAYUybYpvOso3eW-Mmz7pbTS2qrPRE_gUxsI0pqP6I0ngQlBV47gXvuNcdAWeK3F2nasx94). In a different paper, the same authors say, “The analytic process involves immersion in the data, reading, reflecting, questioning, imagining, wondering, writing, retreating, returning. It is far from mechanical and is a process that requires ‘headspace’ and time for inspiration to strike and insight to develop.” (https://www.tandfonline.com/doi/full/10.1080/14780887.2020.1769238?casa_token=uz780JP54msAAAAA%3A-_tVAX8wVF69Uhuiz0oFgzSQ6H5mPWGZmKqUCsiwdlGntVAWuAouiAY03DQbVnOwR-fsRbilZuk) 

If you are certain it is reflexive TA which is the right method for this work, please provide more detail around this. Why are you using reflexive TA for this work, and how does it align with your goals? How will your team accomplish this necessary reflexivity, and how will your shared experiences and biases mesh together to analyse the data? If you feel like this is difficult to achieve in a registered report format, consider changing your analytical strategy to a different type of TA - perhaps coding reliability or codebook TA? - both of which, primarily the former, would lend themselves better to replication and a benchmark assessment of goals met. 

1D. Whether the clarity and degree of methodological detail is sufficient to closely replicate the proposed study procedures and analysis pipeline and to prevent undisclosed flexibility in the procedures and analyses. 

One to note specifically is that, in reflexive TA and in your outlined procedure with multiple coders it’s difficult to set standards for replication in the same way that an inter-rater reliability coefficient might be beneficial for. Please also provide more concrete details on aspects of this process: how long will JHD and EB spend familiarising themselves with the data? What will be a stopping point when the team meets to discuss their codes? Is there a pipeline for how disagreements will be resolved? When you say the wider team will provide feedback, what does that mean practically and how will you ensure everyone’s PoV is balanced? (My thoughts on this also largely tie into my above point about reflexive thematic analysis and possible mismatch with the RR format). 

A more minor point: you mention in the focus group procedure that there will be safeguarding measures - what are they specifically? 

I hope my comments were useful. As I said above, I think this will be a great study. I think one of the beauties of qualitative research is its flexibility, so, if you are certain reflexive TA is the right methodology for this format, with a few more details everything will be great :) 

Thank you for conducting this important study to explore adolescents' experiences and perceptions of social media. The authors have outlined the relevance of this research and designed a good plan to collect data through focus groups. I appreciate the use of co-production in the initiation stage of the research and involving young researchers in designing and conceptualizing the study. I would like to offer some suggestions to further strengthen the study, mainly focusing on the methodology:

1. While I understand the exploratory nature of the study and the intention to adopt a bottom-up approach, the specific focus of the measure being developed could benefit from further clarification. It would be helpful to specify what the measure will specifically gauge and how it could be used in practical applications. Though the research questions and interview schedule partially address this concern, the brief mention of social media addiction in the introduction might need more relevance to the study's overall scope.

2. It's crucial for the authors to reflect on their philosophical/theoretical positions and acknowledge any potential biases (positionality, backgrounds, etc.) (in addition to the qualitative hypotheses). Such a practice is considered essential in qualitative research.

3. The authors plan to recruit adolescents who self-identify as social media users. For clarity, they should provide more specific criteria for social media users. For example, should participants have a social media account or engage in a certain number of hours per day on social media to be eligible for the study? Also, it would be helpful to state whether this criterion is consistent across all age groups.

4. The authors plan to use reflexive thematic analysis in the study. I suggest that they explicitly mention the ways in which they are and will be reflexive throughout the research process, including data collection and analysis. Additionally, it is not entirely clear how the coding will be done. Will the two coders (and YR) code separate transcripts, or will they code the same or different ones? Why/how are multiple researchers involved in the analysis process? Do the authors aim to establish coding reliability? These aspects might be inconsistent with Braun and Clarke’s RTA. I would recommend the authors refer to the paper “Braun, V., & Clarke, V. (2021). One size fits all? What counts as quality practice in (reflexive) thematic analysis?. Qualitative research in psychology, 18(3), 328-352” and their recent book "Thematic analysis: A practical guide" by Braun and Clarke for additional guidance.

5. In the analysis, the authors also mention using content analysis to develop scale items. However, they do not elaborate on their plan for doing this.

6. It is great to see the authors' commitment to sampling participants from diverse backgrounds, considering the unique experiences they may have with social media. Anecdotally, given the rise of polarizing/extreme opinions on social media, some participants (e.g., LGBTQ+) might have extreme or difficult experiences (e.g., cyberbullying). I am curious about the safeguards the researchers have in place to protect participants if discussions become sensitive and how the participants will be debriefed after such situations.

7. It would be valuable for the authors to acknowledge the limitations of their research or the scale developed through this research. For instance, they could consider mentioning that the measure might not be valid for other cultures outside the UK.

Overall, the study holds great potential, and I encourage you to proceed with your research after considering my above points. Best of luck!