Recommendation
based on reviews by Kathryn Bates and 1 anonymous reviewer
DOI or URL of the report: https://github.com/StefanVermeent/liss_wm_profiles_2023/blob/submission3/manuscript/stage2_tracked.pdf
Version of the report: stage2_tracked.pdf
Recommender
1. The two reviewers are satisfied with this revised manuscript. Lastly, I agree with the second reviewer that it would be better if the authors summarize the main findings in the conclusion section. I would like the authors to take this point into consideration for the final step. Thank you.
RESPONSE: We have now added a brief summary of our main findings to the conclusion section:
"Our psychometric investigation yielded inconclusive evidence for associations between adverse experiences in adulthood and WM capacity and updating ability: Differences in abilities were not significantly different from zero, yet also not negligibly small."
, posted 07 Oct 2024, validated 09 Oct 2024The two reviewers are satisfied with this revised manuscript. Lastly, I agree with the second reviewer that it would be better if the authors summarize the main findings in the conclusion section. I would like the authors to take this point into consideration for the final step. Thank you.
Thank you for reporting the Bayes Factor. I believe that the manuscript is much improved through the revision. I recommend the acceptance of the manuscript.
, 01 Oct 2024Thank you to the authors for the thorough response to the comments. The additional definitions and detail in the methods and analysis are helpful. The results are clearly presented with any deviations from preregistered analysis explained. The discussion is very balanced, especially in light of the inconclusive evidence, the future research suggestions are concrete and will be helpful for the research community. A sentence in the conclusion restating the main findings would be helpful for the reader. I recommend this article be accepted.
DOI or URL of the report: https://github.com/StefanVermeent/liss_wm_profiles_2023/blob/master/manuscript/manuscript.pdf
Version of the report: manuscript.pdf
, posted 01 Aug 2024, validated 01 Aug 2024I am really pleased that you have submitted your Stage 2 manuscript. Once again, I would like to commend you for your honesty in reporting the deviations that occurred during the process of the study. This is an ideal attitude and action of researcher which is also an important fundamental aspect of Registered Reports.
Now, the two experts who were responsible for the Stage 1 peer review have also participated in this stage. They have pointed out some minor things which, if addressed, will greatly increase the quality of the manuscript.
The first reviewer suggests additional analyses. I am not sure if you can draw a new separate sample from the panel, but certainly this addition would be useful as a validation of the results. Bayes factors may also be useful because of the potential to increase information, but it should be made clear that these are post hoc and unregistered information. Especially when two methods with different statistical philosophies are used in parallel, cherry-picking can be possible through situations such as the Jeffreys-Lindley paradox, which can create certain risks to the original hypothesis testing. It would be very important to clearly distinguish what was and was not registered. And discussion should be carefully nuanced. Yet, please note that these additional analyses are optional and left to the author's discretion (i.e. performing them is not a prerequisite for the recommendation).
The second reviewer noted a terminology issue. Normally, this type of methodological change in Stage 2 would be a risk for the RR, but addressing this reviewer's point would not be a problem as it does not affect the hypothesis, but rather a necessary step to avoid an illusory perception of causality by the reader.
I sincerely look forward to your consideration of these comments and to your submission of a revised manuscript.
Thank you for submitting the Stage 2 manuscript of your Registered Report. While there were some deviations from the original protocol, I do not consider these to be major concerns. However, I would like to suggest two additional analyses to potentially strengthen the manuscript:
1. Given that the current results appear inconclusive, I suggest calculating Bayes factors for the main analyses. This would provide a more nuanced understanding of the evidence for the hypotheses under investigation.
2. As I understand that your database contains a larger sample, I propose conducting a replication analysis using an additional 800 participants from your existing dataset. This would enhance the robustness and generalizability of your findings.
Please note that you are not obligated to conduct all of these analyses. I encourage you to use your discretion and perform only those analyses that you deem necessary and beneficial to your study. Your expertise in the subject matter will guide you in determining which of these suggestions, if any, would most effectively bolster your research.
These supplementary analyses, if undertaken, would significantly contribute to the comprehensiveness and reliability of your study. I look forward to reviewing the revised manuscript and the rationale behind any additional analyses you choose to include.
Sincerely,
, 16 Jul 2024 The authors have thoroughly addressed the comments from the reviewers. I appreciate the extra detail in the methods and data analysis.
I think the discussion is very balanced, especially given the lack of evidence in line with hypotheses. My only comment here is that the conclusion would benefit with a sentence repeating the main findings.
I just have one concern, and a minor point:
1. I still think the terms lowered, intact, and enhanced are problematic. In their response, the authors point out that using language such as predicted would imply causality as is not appropriate in this case, but this is the same point about the lowered, intact and enhanced terminology. “Lowered” implies that adversity caused WM to be lower: but this cannot be determined from a statistically significant negative association between adversity and working memory. I do not want to be pedantic, but I think this language point is important because “lowered” implies an experimental manipulation that led to lower WM or that the study tested how adversity lowered WM, but the negative association doesn’t confirm this. I recommended changing the terminology to “lower” and “higher” WM, and “middle” or “average” or something similar for the “intact” group. I’m open to a rebuttal here but I do think this could be something easily misinterpreted by the reader.
2. Table 2 is a helpful addition but it is not clear what is in the “Statistic” column, especially with reference to education, are they percentages? Are the threat etc. rows mean or sum? Please add this info.
