Recommendation

Adversity and working memory: Nuanced effects underpinned by rigorous methodology

ORCID_LOGO based on reviews by Kathryn Bates and 1 anonymous reviewer
A recommendation of:

Inconclusive evidence for associations between adverse experiences in adulthood and working memory performance

Abstract

EN
AR
ES
FR
HI
JA
PT
RU
ZH-CN
Submission: posted 25 June 2024
Recommendation: posted 18 October 2024, validated 21 October 2024
Cite this recommendation as:
Yamada, Y. (2024) Adversity and working memory: Nuanced effects underpinned by rigorous methodology. Peer Community in Registered Reports, 100859. 10.24072/pci.rr.100859

This is a stage 2 based on:

Working memory performance in adverse environments: Enhanced, impaired, or intact?
Stefan Vermeent, Anna-Lena Schubert, Meriah L. DeJoseph, Jaap J. A. Denissen, Jean-Louis van Gelder, Willem E. Frankenhuis
https://github.com/StefanVermeent/liss_wm_profiles_2023/blob/master/manuscript/manuscript.pdf

Recommendation

Adverse environments involving threat, uncertainty, deprivation, and stress have been shown to have significant impacts on cognition and development. In this Stage 2 manuscript, Vermeent et al. (2024) adhere to their Stage 1 protocol, investigating the effects of adversity on working memory (WM) using a comprehensive, psychometric modeling approach. The authors aimed to clarify seemingly contradictory findings from previous research: The evidence for working memory capacity impairments in adverse environments versus the possibility that adversity might enhance specific aspects of WM, such as updating ability. Moreover, they examined the effects of distinct types of adversity—neighborhood threat, material deprivation, and unpredictability—on WM performance.
 
The results of the study were, overall, inconclusive: the authors did not find consistent associations between adversity and either WM capacity or WM updating ability. Despite using a large sample and employing latent variable modeling, the study did not reveal significant effects that were either clearly positive or negative for any type of adversity examined. In addition, no evidence for equivalence to zero associations was found. The lack of clear associations suggests that the relationship between adversity and WM is likely more complex than previously thought.
 
As with the Stage 1 evaluation, this manuscript has undergone a rigorous peer review process at Stage 2. The reviewers included specialists in child and youth cognitive development. The constructive feedback from the reviewers ensured that the pre-registered protocol was followed accurately, deviations were reported appropriately, and all concerns raised were addressed satisfactorily. These processes helped to refine Vermeent et al.'s methods and confirm that the planned analysis was followed. Despite the inconclusive results, this study makes a significant contribution to our understanding of the subtle effects of adversity on working memory by providing transparent and rigorous findings that add valuable data to the field. Therefore, I am fully confident that this manuscript is suitable for Stage 2 recommendation.
 
URL to the preregistered Stage 1 protocol: https://osf.io/dp7wc
 
Level of bias control achieved: Level 3. At least some data/evidence that was used to the answer the research question had been previously accessed by the authors (e.g. downloaded or otherwise received), but the authors certify that they had not yet observed ANY part of the data/evidence until after Stage 1 in-principle acceptance.
 
List of eligible PCI RR-friendly journals:
 
 
References
 
1. Vermeent, S., Schubert, A.-L., DeJoseph, M. L., Denissen, J. J. A, van Gelder, J.-L. & Frankenhuis, W. E. (2024). Inconclusive evidence for associations between adverse experiences in adulthood and working memory performance [Stage 2]. Acceptance of Version 3 by Peer Community in Registered Reports. https://github.com/StefanVermeent/liss_wm_profiles_2023/blob/master/manuscript/stage2_tracked.pdf
Conflict of interest:
The recommender in charge of the evaluation of the article and the reviewers declared that they have no conflict of interest (as defined in the code of conduct of PCI) with the authors or with the content of the article.

Evaluation round #2

DOI or URL of the report: https://github.com/StefanVermeent/liss_wm_profiles_2023/blob/submission3/manuscript/stage2_tracked.pdf

Version of the report: stage2_tracked.pdf

Author's Reply, 10 Oct 2024

Recommender

1. The two reviewers are satisfied with this revised manuscript. Lastly, I agree with the second reviewer that it would be better if the authors summarize the main findings in the conclusion section. I would like the authors to take this point into consideration for the final step. Thank you.


RESPONSE: We have now added a brief summary of our main findings to the conclusion section:

"Our psychometric investigation yielded inconclusive evidence for associations between adverse experiences in adulthood and WM capacity and updating ability: Differences in abilities were not significantly different from zero, yet also not negligibly small."

Decision by ORCID_LOGO, posted 07 Oct 2024, validated 09 Oct 2024

The two reviewers are satisfied with this revised manuscript. Lastly, I agree with the second reviewer that it would be better if the authors summarize the main findings in the conclusion section. I would like the authors to take this point into consideration for the final step. Thank you.

Reviewed by anonymous reviewer 1, 07 Sep 2024

Thank you for reporting the Bayes Factor. I believe that the manuscript is much improved through the revision. I recommend the acceptance of the manuscript. 

Reviewed by ORCID_LOGO, 01 Oct 2024

Thank you to the authors for the thorough response to the comments. The additional definitions and detail in the methods and analysis are helpful. The results are clearly presented with any deviations from preregistered analysis explained. The discussion is very balanced, especially in light of the inconclusive evidence, the future research suggestions are concrete and will be helpful for the research community. A sentence in the conclusion restating the main findings would be helpful for the reader. I recommend this article be accepted.


Evaluation round #1

DOI or URL of the report: https://github.com/StefanVermeent/liss_wm_profiles_2023/blob/master/manuscript/manuscript.pdf

Version of the report: manuscript.pdf

Author's Reply, 06 Sep 2024

Decision by ORCID_LOGO, posted 01 Aug 2024, validated 01 Aug 2024

I am really pleased that you have submitted your Stage 2 manuscript. Once again, I would like to commend you for your honesty in reporting the deviations that occurred during the process of the study. This is an ideal attitude and action of researcher which is also an important fundamental aspect of Registered Reports.

Now, the two experts who were responsible for the Stage 1 peer review have also participated in this stage. They have pointed out some minor things which, if addressed, will greatly increase the quality of the manuscript.

The first reviewer suggests additional analyses. I am not sure if you can draw a new separate sample from the panel, but certainly this addition would be useful as a validation of the results. Bayes factors may also be useful because of the potential to increase information, but it should be made clear that these are post hoc and unregistered information. Especially when two methods with different statistical philosophies are used in parallel, cherry-picking can be possible through situations such as the Jeffreys-Lindley paradox, which can create certain risks to the original hypothesis testing. It would be very important to clearly distinguish what was and was not registered. And discussion should be carefully nuanced. Yet, please note that these additional analyses are optional and left to the author's discretion (i.e. performing them is not a prerequisite for the recommendation).

The second reviewer noted a terminology issue. Normally, this type of methodological change in Stage 2 would be a risk for the RR, but addressing this reviewer's point would not be a problem as it does not affect the hypothesis, but rather a necessary step to avoid an illusory perception of causality by the reader.

I sincerely look forward to your consideration of these comments and to your submission of a revised manuscript.

Reviewed by anonymous reviewer 1, 31 Jul 2024

Thank you for submitting the Stage 2 manuscript of your Registered Report. While there were some deviations from the original protocol, I do not consider these to be major concerns. However, I would like to suggest two additional analyses to potentially strengthen the manuscript:

1. Given that the current results appear inconclusive, I suggest calculating Bayes factors for the main analyses. This would provide a more nuanced understanding of the evidence for the hypotheses under investigation.


2. As I understand that your database contains a larger sample, I propose conducting a replication analysis using an additional 800 participants from your existing dataset. This would enhance the robustness and generalizability of your findings.


Please note that you are not obligated to conduct all of these analyses. I encourage you to use your discretion and perform only those analyses that you deem necessary and beneficial to your study. Your expertise in the subject matter will guide you in determining which of these suggestions, if any, would most effectively bolster your research.

These supplementary analyses, if undertaken, would significantly contribute to the comprehensiveness and reliability of your study. I look forward to reviewing the revised manuscript and the rationale behind any additional analyses you choose to include.

Sincerely,

Reviewed by ORCID_LOGO, 16 Jul 2024

 The authors have thoroughly addressed the comments from the reviewers. I appreciate the extra detail in the methods and data analysis.
 
I think the discussion is very balanced, especially given the lack of evidence in line with hypotheses. My only comment here is that the conclusion would benefit with a sentence repeating the main findings.
 
I just have one concern, and a minor point:
1.     I still think the terms lowered, intact, and enhanced are problematic. In their response, the authors point out that using language such as predicted would imply causality as is not appropriate in this case, but this is the same point about the lowered, intact and enhanced terminology. “Lowered” implies that adversity caused WM to be lower: but this cannot be determined from a statistically significant negative association between adversity and working memory. I do not want to be pedantic, but I think this language point is important because “lowered” implies an experimental manipulation that led to lower WM or that the study tested how adversity lowered WM, but the negative association doesn’t confirm this. I recommended changing the terminology to “lower” and “higher” WM, and “middle” or “average” or something similar for the “intact” group. I’m open to a rebuttal here but I do think this could be something easily misinterpreted by the reader.
2.     Table 2 is a helpful addition but it is not clear what is in the “Statistic” column, especially with reference to education, are they percentages? Are the threat etc. rows mean or sum? Please add this info.

User comments

No user comments yet