Developmental Coordination Disorder Diagnosis as Part of Evolving Self-Concepts

Dear Marta Topor and co-authors,
 
Thank you for submitting the revised Stage 1 with careful changes. This version is now very close to in-principle acceptance, thus I invite you to format the next version as “clean” with a separate document attached to show track changes. I have a few final comments for your consideration, most of which are very minor, but worth addressing now because after Stage 1 this part of the manuscript will be locked.
 
- You have chosen codebook TA to accommodate a realist epistemology with a reflexive approach -- this is good choice. As one more clarification, I would ask you to add an explicit note regarding *how* the codebook will be applied. As Braun & Clarke (2022, p. 7) point out, the codebook philosophy encompasses numerous approaches like matrix, framework, network, and template analyses; are you committing to one of these? I am asking because in the response letter you refer to template analysis (King 2012), but this is not mentioned in the manuscript. I just want to double check to avoid misunderstandings at Stage 2. To be clear, you *don’t* have to commit to any of these existing approaches, but you can also follow your own codebook/instructions. In such case, I would kindly ask you to create tentative codebook/instructions for the two coders and add these as a supplement so that we have them registered at Stage 1. To give an example, in a recent template analysis study we preregistered the following instructions based on one guiding RQ for two coders: https://osf.io/umvsy Note that your instructions can be entirely different and you can recreate a codebook as you work, but having some formalized plan at Stage 1 will allow showing how/what changes occurred, if any (it already reads “coders will be instructed to identify codes…” but let’s make these instructions visible to reach the highest levels of transparency!).
- Minor comment: Your QH1 states: “Participants who received a late DCD diagnosis will have different feelings towards this experience depending on how much time has passed…” This is good, but how will you tackle these temporal differences in practice? For instance, will each participant’s current state be situated in Murphy and LeVer’s stage model of ADHD? I note this because I’ve once had an RQ like this and found it very difficult to analyze/report in practice. If I may suggest (ignore this in case it doesn’t feel right), one part of the codebook/instructions, if you decide to have that, could be to instruct the coders to evaluate which stage/s (a–f) the participant represents, assuming the stage model of ADHD applies here. [You may skip this entire comment if you’re confident that your analysis will produce results that address the temporal dimension without further additions]
- Minor comment: you’re using the words “self-identity”, “self-concept” and “self-perception” on several occasions in the manuscript. Please double check if all these terms are intentionally used with separate meanings or could one be dropped to increase consistency? (I’m recalling Roy Baumeister’s lengthy review “The Self” from 1998 -- again, you can rebut this comment in case all three terms are intentional and needed).
- Minor comment: I would encourage to rephrase this part -- “themes should emerge from the participants own words rather than the researchers’ preconceptions and beliefs” (351). You can keep it as is, but consider e.g., “To investigate the participants’ experiences through their own words, a semantic-realist approach to the data is justified” (for a more neutral tone)
- This last comment does not entail any action from your part at the moment, but with reference to earlier reviewer feedback, please allow me to remind to explicitly communicate at Stage 2, in the Discussion or other section, how the results will guide practitioners. This is promised twice, but not yet explained *how* (it's ok as long as it becomes clear at Stage 2).
 
Please do not hesitate to contact me for any questions in case some of the above feedback is unclear. I look forward writing an IPA for the next version!
 
Veli-Matti Karhulahti

Dear Marta Topor and co-authors,
 
Thank you for submitting the revised Stage 1 version of your manuscript. I have reviewed the revised manuscript file as well as the responses to the reviewer feedback, based on which I did not consider it necessary to initiate a new round of external reviews. However, I personally made note of points that are necessary to revise before IPA. I start with minor points. 
 
1. Line 102: “However, within the RR context, hypotheses are normally discussed as *quantitative* hypotheses” --> I recommend using the term “testable” instead of “quantitative,” as the idea of non-testing is most important in the present context, I believe. 
2. Line 108: the reference to our work here is incorrect; we have two Stage 1 RRs in the PCI and the qualitative one is “Phenomenological Strands for Gaming Disorder and Esports Play: A Qualitative Registered Report”
3. Line 132: I like the new RQ2 and I agree the link to RQ1 is now clearer. Yet to make it even more explicit, I suggest integrating the link to emotional findings, e.g. along the following lines: “How does self-identity change following the emotional events of a late DCD diagnosis…”
4. You have now justified the use of OSF for data sharing, which is totally ok. Nevertheless, please allow me to add a bit of context, which was perhaps the intention of the reviewer but was left unspecified. Sharing qualitative health data can be difficult and comes with unique risks; anonymizing human experiences takes a lot of time and requires carefully assessing the risk-benefit ratio i.e., what parts of the text should be shared for the data to be useful for other researchers and what should be erased to protect the participants. Due to these challenges, it is common that expert repositories are collaborated with so that they can assist with de-identification and, as is often the case, control data access so that only people with scientific interests can reuse the data. These services are not provided by the OSF. Naturally, you have the full right to share the data via OSF following the ethical principles outlined, but do not hesitate to contact me if you wish to further discuss the plan for and potential issues related to data sharing.
5. You are using the ADC as an inclusion criterion, which is ok. However, I am thinking, as you are also inquiring about official clinical diagnosis in the questionnaire, wouldn’t that be the gold standard? I.e., wouldn’t someone with an official diagnosis meet a sufficient alternative inclusion criterion, regardless of what their ADC score is? Of course, it’s likely that the participants meet both, but in case of false negative outcomes from ADC, it seems that a clinical diagnosis should suffice, too. What do you think?
6. Line 241: You mention “neurodevelopmental, neurodegenerative, or psychiatric” conditions as exclusion criteria but only survey about the first one (line 245). Unless otherwise justified, please remove neurodegenerative and psychiatric conditions as exclusion criteria (this would also help you to find participants, and as I see it, psychiatric conditions such depression are so common that there is no reason to exclude participants with them if they are willing to participate). 
 
Finally, I have three more general points that need attention. I try to be as clear as possible, so this takes a bit of explaining (apologies if I sound too instructive below, but I think these are discrepancies that could be criticized by future readers unless we clarify them here and now). 
 
- There’s a lot of conversation around Braun and Clarke’s TA across fields these days, and the authors themselves have been very actively reconceptualizing their method into “reflexive” TA – a position that was not clearly present in their 2006 paper yet (and this paper is the basis of your MS). In the 2006 paper, the authors indeed discuss many possible ways of doing (all kinds of) TA, including realist approaches, but in the more recent papers they have divided TA into “reliability”, “codebook” and “reflexive” schools (many of the which they do *not* represent -- historically, there are likely more than a hundred different types/schools of TA). In Braun and Clarke's view, the “reflexive” school (which they have now coined and represent explicitly) is strongly tied to researcher subjectivity instead of realist or (as they call it) neopositivist epistemologies that assume there to be one “real truth”. Therefore, as you since line 340 note that the study is based on realist epistemology and “real truth” but elsewhere mention that reflexive TA is followed, there seems to be a conflict between these positions (also recall that you dropped the second coder because in reflexive TA there isn’t necessarily one truth but both coders could’ve been correct!). In summary, it would seem logical for this study to be carried out with an epistemology that matches reflexive TA (of if not, *why* is realist epistemology more optimal for your RQs?). On the other hand, I wouldn’t want to enforce anyone to assume an epistemological position that they do not actually hold -- so I leave this comment to be discussed in your team (please also consider rephrasing “emerging” themes; in reflexive TA themes are constructed subjectively rather than them emerging objectively, see e.g., Braun and Clarke 2006; Table 2 point 15). Again, I am sorry to point at this issue in such detail, but as this is one of the first qualitative RRs in the world, I must ensure that these meta-scientific issues, which are very much discussed in the qualitative domain at the moment, are not left unresolved. For further reading, see e.g., 
 
Virginia Braun & Victoria Clarke (2019) Reflecting on reflexive thematic analysis, Qualitative Research in Sport, Exercise and Health, 11:4, 589-597, DOI: 10.1080/2159676X.2019.1628806
Virginia Braun & Victoria Clarke (2021) One size fits all? What counts as quality practice in (reflexive) thematic analysis?, Qualitative Research in Psychology, 18:3, 328-352, DOI: 10.1080/14780887.2020.1769238
 

(if you choose to stay with reflexive TA, please use "reflexive TA" systematically in the manuscript every time you refer to TA)

- Related to the above, about the sample size. In brief, I do not believe that N=5 would be enough for this TA study. In fact, this is also suggested by Braun and Clarke; I don’t have their 2013 book with the sample size discussion at hand, but they summarize the point online as follows: ”generally speaking 5 interviews (however long and detailed) is probably a bit too small for a TA. This is because TA focuses on the identification of patterns *across* data… 6-10 (rich and detailed) interviews [fit] for a small TA project e.g. UK undergraduate” (https://www.thematicanalysis.net/faqs/). To ensure the production of themes across data, I would suggest setting the minimum to 10 interviews (which is still not much for TA but could be sufficient, trusting on your team’s assessment). That is not to say that less data could not be informative at all; however, with less than 10 interviews other methods would be more suitable for analysis, such as interpretive phenomenological analysis or case study approaches. TA is not idiographic, i.e., not focused the analysis of individuals per se but rather the themes across individuals [on line 174 it is noted that “thematic analysis has an ideographic focus”, what does that mean and where does it come from?]
- Finally, regarding the use of inductive coding, please allow me to point at possible issues with this specific choice. I quote from Braun & Clarke 2006 again, as that is the method you've chosen: “Inductive analysis is therefore a process of coding the data without trying to fit it into a preexisting coding frame, or the researcher’s analytic preconceptions.” (83). While there are many ways of doing inductive coding, you currently highlight also working with strong analytic preconceptions, i.e. clear RQs with clear QHs: "Initial codes will be noted for ideas that fall in line with our hypotheses as well as novel insights specific to DCD." To be safely on the inductive side, I would suggest not involving your hypotheses in the coding. Coding through the RQs works well, but you might additionally wish to explain *how* the coder is instructed to interpret the data through the RQs? Do you trust their intuition, or will there be mutually agreed guidelines/limits regarding what counts as a reflection of RQ1-RQ2? 
 
I hope these comments will further help your team to make this as good study as possible. Needless to say, I can be contacted directly if some of the feedback is unclear or if you have concerns regarding how to solve some issues.
 
Best wishes,
Veli-Matti Karhulahti
 
 
 

Dear Marta Topor and co-authors,
 
Thank you for submitting your Stage 1 manuscript. It is a great to pleasure to see more qualitative work choosing the RR format, and I am confident that this promising manuscript will pave the way for more such work in the future. I have now received all three reviews, collectively representing research expertise of qualitative methods, developmental psychology, and developmental coordination disorder. The reviews are very encouraging but also identify points that require revision. Below, I list some of my own feedback and selectively comment on the reviewers’ points to facilitate your revisions process as much as possible.
 
1. Most of the requested revisions concern the introduction. One reviewer is concerned whether the research questions are optimally constructed, and another reviewer suggests previous literature that should be taken into consideration to ensure the findings contribute to cumulative science. I agree and encourage you to reconsider the RQs after assessing 
 
a) what would be theoretical or other links between RQ1 and RQ2 (or alternatively justify why two not-so-strongly connected RQs need to be answered together), and 
b) how does the suggested literature affect (if it does) what you consider the most relevant RQs to ask at the current state of the field?
 
I will additionally comment briefly on the RQ2. With the premise that the condition has become part of the participants’ intrinsic self, as you meantion on page 5 with reference to biographical illumination theory, it would be good further specify what “through the lens” means in this context. For instance, although you address “past, present, and future” in RQ2, the following QH2 expects that “participants will present with both positive and negative accounts [of the impact].” Are you interested in how the chronology manifests (“past, present, and future” in RQ2), or about the types of impact of the diagnosis (“positive and negative accounts” in QH2)? You also mention positive and negative feelings in your QH1. Unless you are specifically interested in the positive/negative division, other concepts might better describe your predispositions. 
 
2. One reviewer also recommends merging the QHs in the rest of the text. I should disclose my own position here, as I am a co-author in the study which has previously used QHs, and I assume you are following that example. I will let you choose whether you wish to follow the reviewer’s advice, or if you wish to keep QHs. Both are viable options when well executed. Nevertheless, if you choose to the keep the QHs, please follow the reviewer’s advice and briefly explain to readers (who may not know how QHs work) that they will not be tested but they rather serve to disclose biases and preconceptions.
 
3. All three reviewers note that the sample size section needs work. First, instead of a hard sample size number, I believe a range works better in the RR format. One reviewer points out that N=15 is already a difficult-to-reach size considering that you exclude all participants with other diagnosis, whereas another reviewer is worried that N=15 might not be enough to answer your RQs properly. As reflexive thematic analysis does not operate with saturation, I suggest 
 
a) Choosing a minimum sample size that allows the study to be carried out and a maximum that would e.g., be defined by your resources.
b) Reflecting inside the team e.g., after each 5 interviews what the data tentatively feel like against the RQs, then choosing to stop/continue (you may create brief reflection documents of these steps), 
c) Include a small plan for alternatives, e.g., how would it affect your analysis if you cannot find as many participants as would be optimal but still meet the minimum sample size.
 
I also agree with one of the reviewers who would like the recruitment process to be elaborated. In what order will different recruitment locations be approached and how (most importantly, will potential participants be approached personally or do they contact you)? I can see some of these issues are noted in the COREQ list, but this information is critical enough to be included in the fixed Stage 1 main text. For the record, I also note that reflexive TA does not recommend COREQ, but I also believe any such checklist helps to transparently think about different aspects one's study, and there is no reason to change or remove COREQ. 
 
4. The last bigger topic is the method/analysis. You have chosen reflexive TA, which has a very strong non-positivist foundation. You have done an excellent job in opening your positionality via relevant statements, and you do clearly mention reflexivity as part of your work. However, there are some components in the plan that do not seem to fit reflexive TA. 
 
a) Reflexive TA generally does not recommend the use of multiple coders, as the related reliability checks and comparisons are considered positivistic and epistemologically problematic. That said, I am personally ok with you using of two coders, but with Braun & Clarke’s framework as a basis, you should rather explain, for instance, how the different outcomes of two coders are reported with respect to their two different perspectives instead of pursuing accurate interpretation via synthesis. To be clear, you can produce a synthesis, but in such case, you should explain how the diverse views manifest in it. From the point of reflexive TA, the different views of both coders can be accurate interpretations. Braun & Clarke have come to highlight these issues in several recent papers. For simplicity, I refer to their website which lists such references topic by topic:  https://www.thematicanalysis.net 
b) As the reviewers points out, there is lacking information regarding what type of reflexive TA you are planning to use. Even if you apply the 6-step process described in the original Braun & Clarke (2006), you need to make many decisions regarding the level of codes, deductive/inductive premises, and epistemology (e.g., my above note about merging two coder positions). At the same time, I feel it is important to stress that qualitative research always comes with flexibility and not all details can be perfectly planned and followed. I suggest you carefully plan within the team as many details as possible, and if you need to make changes at Stage 2, I will do my best to provide you with the required flexibility and we will simply document all changes transparently, when applicable. You do not have to be worried that you will be “locked” in your decisions, but I am confident that thinking about those decisions carefully beforehand will improve the study. 
c) One reviewer notes that the validity and reliability section does not include the latter, which is not entirely true as you do explain e.g., member checks (and I agree that your plan to do member checks by external community members is well suitable in this design). However, the reviewer is correct that if you wish to address reliability (which is not straightforward in reflexive TA), there are many options to do that and one of the reviewer’s own papers provides a good overview of the topic (Syed, M., & Nelson, S. C., 2015. Guidelines for establishing reliability when coding narrative data. Emerging Adulthood, 3.6, 375-387). From a reflexive TA viewpoint, perhaps the best approach would be to pursue negotiated consensus about what themes are most relevant to be reported as the results of this study. This would allow both coders to keep their unique epistemic positions (documented by their separate coding materials), but still make it visible how you end up reporting what you will. Related to this, the coders might wish to integrate their subjective reflections directly in the shared coding documents (analysis logs), which I believe would be more pragmatic for you to do and the readers to interpret, versus a general post-reflexivity statement. Again, both options are viable, and you may also stick with the latter if you prefer. 
 
5. Two smaller notes. 
 
a) You plan to collect a lot of information with the questionnaire, but as one reviewer comments, it is not clear how this information is utilized. I should also add that those data do not seem to be part of the data availability statement, so please clarify what role they have in the study. It is ok to collect such data e.g., for purposes beyond the present RR, but this should be explained. I also understand that some of the data may not be suitable for sharing, and you can state that explicitly in the data availability statement.
b) If you still plan to exclude participants with all other diagnoses except for DCD, please explain how the presence of those other diagnoses will be determined. For instance, the questionnaire seems to have multiple items, which may all yield data that could be a basis for exclusion. 
 
In addition to the above, please provide point-by-point responses to the reviewers’ respective feedback. Needless to say, if you disagree with some the requested revisions, you are free to justify alternative choices. Do not hesitate to contact me if something is unclear. I look forward to reading the next version, based on which I will see if another external review round is needed. I also remind that you have the option to start data collection before IPA by reducing bias control. 
 
Sincerely, 
Veli-Matti Karhulahti