Recommendation
Over the last decade, a growing body of evidence has revealed potential benefits of unconditional cash transfers (UCTs) on a variety of outcomes, including self-reported happiness and life satisfaction (Haushofer & Shapiro, 2016), economic and financial well-being (Blattman et al., 2013; Baird et al., 2018) and educational attainment (Baird et al., 2016). Although the effects of UCTs do not always out-perform rigorous control conditions (Whillans & West, 2022), these findings prompt the question of whether the alleviation of poverty via UCTs can also influence cognitive processing and performance.
In the current study, Szaszi et al. propose to analyse the results of a previous randomised trial of UCTs by Blattman et al. (2017) to test whether a $200 lump sum administered to a sample of young men in Liberia carries both short- and long-term benefits for a range of executive functions, including attention, response inhibition, and working memory capacity.
The Stage 1 manuscript was evaluated over two rounds of in-depth review. Based on detailed responses to the reviewers' comments, the recommender judged that the manuscript met the Stage 1 criteria and therefore awarded in-principle acceptance (IPA).
URL to the preregistered Stage 1 protocol: https://osf.io/k56yv
Level of bias control achieved: Level 2. At least some data/evidence that will be used to answer the research question has been accessed and partially observed by the authors, but the authors certify that they have not yet observed the key variables within the data that will be used to answer the research question AND they have taken additional steps to maximise bias control and rigour.
List of eligible PCI RR-friendly journals:
References
DOI or URL of the report: https://osf.io/nwsjg/?view_only=1781fb681edc4cdeb61287172cd14ba2
Version of the report: v1
Dear Dr. Chris Chambers,
We are pleased to submit a revision of our manuscript “Does alleviating poverty increase cognitive performance? Short and long term evidence from a randomized controlled trial” to PCI RR.
We would like to thank again for the constructive comments and helpful suggestions. Below you can find a point-by-point response to all comments in bold.
To support the review process, we have submitted two versions of the updated manuscript to the OSF page of the project. One pdf with the final text, and one docx, where the changes are tracked.
We look forward to your comments.
Kind regards,
Barnabas Szaszi on behalf of the author team
, posted 31 Jan 2022The two reviewers from the previous round have now evaluated the revised manuscript. The comments are broadly positive, and most queries have been answered. There are, however, some remaining issues requiring clarification, mainly concerning the planned outcomes and how they will be reported, but also resolving some straightforward methodological points. Provided you are able to address these issues comprehensively in a final revision and point-by-point response, in-principle acceptance should be forthcoming without requiring further in-depth Stage 1 review.
, 21 Jan 2022, 06 Jan 2022Does alleviating poverty increase cognitive performance? Short and long term evidence from a randomized controlled trial
PCI Registered Report Peer-review, Revision #2
I have now conducted a review of this revised manuscript and thank the authors for their overall attentiveness to my comments. Whilst the manuscript has improved considerably, and the majority of my comments have been addressed, some are still outstanding and require attention. Of main concern is the number of planned analyses in the report, with the authors proposing a primary analysis, a multi-verse analysis, and a mediation analysis - will all of these analyses be reported in one manuscript, supplementary materials, and/or multiple manuscripts? Are the authors aware that a Stage 1 manuscript cannot change in the revisions of a Stage 2 manuscript, meaning that word count could not be cut down post-hoc? Also, there is some inconsistency with regards to the described Robustness Regions, which needs to be addressed. I now outline the concerns addressed, those outstanding, and some minor comments relating to proofreading.
Concerns addressed:
My first main concern related to the terminology of ‘cognitive functioning/performance’ used in the manuscript when the executive functions of inhibition, shifting, switching, and working memory have been measured, and the aggregated index of these measures into a primary outcome of cognitive functioning. The reviewers have now addressed this concern by revising the terminology and including an exploratory multi-verse analysis which can tease apart performance differences in the different tasks (as well as still conducting the primary analysis on cognitive functioning).
A second concern was regarding the potential for procedural flexibility with a stopwatch being used rather than computerised tasks of cognitive performance. The authors have confirmed that this was due to limits on technological resources in the testing environment and have included this clarification in the revised manuscript. Indeed, that this study was conducted in the field is a strength to the methodological design.
A third concern was that additional details were required to ensure that the methodology is replicable, with synergy between the open materials and the reported measures. The authors have now attentively revised the manuscript so that this is clear. The materials and analysis scripts are uploaded to the OSF.
Concerns raised in revisions and/or outstanding:
1. My main concern regards the number of planned analyses in this Stage 1 manuscript - a primary analysis, multi-verse analysis, and mediation analysis are proposed. Will all of these analyses be reported in one manuscript, supplementary materials, and/or multiple manuscripts? Relatedly, for the multi-verse analysis there are a total of 17 dependent variables (Arrow switching – accuracy; Arrow switching- RT; Arrow switching- Inverse efficiency index; Arrow inhibition – accuracy; Arrow inhibition - RT; Arrow inhibition - Inverse efficiency index; Arrow attention – Accuracy; Arrow attention – RT; Arrow attention - Inverse efficiency index; Arrow tasks response time index; Arrow tasks accuracy index; Forward Digits – accuracy; Backward digits - accuracy; Digit span index; Maze – accuracy; Maze - total completion time; Executive function index). Here you plan to analyse multiple indices of the same task (e.g., accuracy and reaction time): could you not focus on one index for each, thus reducing the number of dependent variables being entered into this analysis? I believe this would make your analyses more stringent, allowing you to better assess support for your hypotheses.
2. Referring back to a comment from my initial review, from the Introduction and Abstract it is not clear that this is a secondary data analysis/re-analysis of Blattman et al. (2017). This should be clarified from the outset and would only require a minor amendment (for example, the sentence “To do so, we leverage a cash transfer-based poverty alleviating program” could be extended to state “by analysing pre-existing data from Blattman et al.”).
3. Please check consistency between the reference to the mini-meta analysis used to inform power calculations on Page 12 and the later description on Page 16. Page 12 states that you will “report Robustness Regions for each Bayes factor with two extreme priors (b = 0.09, b = 1.57), with the half of smallest and twice of the largest effect sizes from the mini-meta analysis described below”, however Page 16 then states that you will “repeat all the analysis with three different priors: the effect size used in the primary analysis (b=0.34), as well as the smallest (b=0.18) and the largest (b=0.79) effect sizes from the mini meta-analysis described above”. Note that the first mention states, “half of the smallest” and “twice of the largest” (which I query separately below), but then later you state “smallest” and “largest” whilst referring to different values.
There are also a few grammatical and issues of sentence structure, as follows:
1. Should ‘registered report’ not be capitalised (Registered Report)?
2. Should there be hyphens in “short and long term” throughout (short- and long-term)?
3. I don’t think the word ‘approximately’ should be abbreviated in the Abstract.
4. Abstract: “we will use an experimental setting to, and test…”. What do you mean by this? The data are already collected so this study represents a re-analysis of pre-existing data (see my related comment above). Do you mean that you will reanalyse an experimental manipulation (cash transfer) implemented by Blattman et al. 2017?
5. Page 3, “as compared with”, please remove “as” to aid readability.
6. Page 3, please correct “Whicherts” to “Wicherts”.
7. Page 5, “by doing these” – do you mean “by doing this”?
8. Page 5, “and the focused their paper on how therapy and unconditional cash transfers should affect criminal and violent behavior.” – please revise this sentence paying attention to “and the focused their”.
9. Page 8, “(as well as 28 percent into therapy only (n = 277), and 25 percent into the joint treatment arm (n = 249)).” Add to this “not analysed here” to clarify that these data are not analysed in the present study.
10. Page 8, “As reported in detail in Blattman et al. 11, the treatment is largely balanced along the covariates”. Can you add “reported below” to this sentence so that the reader knows that these covariates are described later? Without this I am left wondering what the covariates are.
11. Is there a reference for the ‘arrows tasks’? Is this an established task?
12. Page 11, “All task materials are available in the Appendix” only needs to be stated once (which it is on Page 10). Also, rather than referring to the Appendix, is it not best to refer to the Open Science Framework project page with the direct OSF link?
12. Page 12, “with the half of smallest and twice of the largest effect sizes from the mini-meta analysis described below”. The terms “half of the smallest” is very difficult to follow – could this be clarified? Please refer back to my major point 3 when revising this.
13. Page 12, “an intention-to-treat Bayesian regression analysis in the short term and in the long term separately” – can you clarify “short term” and “long term” here by including the term ‘phase’ (i.e., “in the short term and in the long term phase separately”).
14. Page 13, Could you include a rationale for the control characteristics included in the model – are these all found to influence cognitive performance?
15. Page 15, “The results showed the rates of misleading evidence were < 1% both of the hypotheses as well.”. This sentence doesn’t read quite right – “for both of the hypotheses”?
Dr Charlotte Pennington
DOI or URL of the report: https://osf.io/ekp3u/?view_only=1781fb681edc4cdeb61287172cd14ba2
, posted 22 Aug 2021Two reviewers have now assessed the Stage 1 manuscript and I have also read it myself. As you will see, the reviews are very constructive and find substantial merit in the proposal while also raising a number of concerns to address, including the level of methodological detail, the clarity and justification of the rationale and key design components, potential methodological confounds that could obscure the intepretation of the outcomes, uncaptured risk of bias through procedural flexibility or prior data observation, and presentation structure. I agree with these assessments -- I found this to be a promising Stage 1 submission that should be suitable for in-principle acceptance once the issues raised by the reviewers are fully addressed. In revising, please ensure that all OSF materials (components and code) are available to the reviewers.
, 06 Aug 2021Please see the attached PDF.
Download the review, 18 Aug 2021