Does alleviating poverty increase cognitive performance? Short and long term evidence from a randomized controlled trial

PCI Registered Report Peer-review, Revision #2

I have now conducted a review of this revised manuscript and thank the authors for their overall attentiveness to my comments. Whilst the manuscript has improved considerably, and the majority of my comments have been addressed, some are still outstanding and require attention. Of main concern is the number of planned analyses in the report, with the authors proposing a primary analysis, a multi-verse analysis, and a mediation analysis - will all of these analyses be reported in one manuscript, supplementary materials, and/or multiple manuscripts? Are the authors aware that a Stage 1 manuscript cannot change in the revisions of a Stage 2 manuscript, meaning that word count could not be cut down post-hoc? Also, there is some inconsistency with regards to the described Robustness Regions, which needs to be addressed. I now outline the concerns addressed, those outstanding, and some minor comments relating to proofreading. 

Concerns addressed: 

My first main concern related to the terminology of ‘cognitive functioning/performance’ used in the manuscript when the executive functions of inhibition, shifting, switching, and working memory have been measured, and the aggregated index of these measures into a primary outcome of cognitive functioning. The reviewers have now addressed this concern by revising the terminology and including an exploratory multi-verse analysis which can tease apart performance differences in the different tasks (as well as still conducting the primary analysis on cognitive functioning).

A second concern was regarding the potential for procedural flexibility with a stopwatch being used rather than computerised tasks of cognitive performance. The authors have confirmed that this was due to limits on technological resources in the testing environment and have included this clarification in the revised manuscript. Indeed, that this study was conducted in the field is a strength to the methodological design. 

A third concern was that additional details were required to ensure that the methodology is replicable, with synergy between the open materials and the reported measures. The authors have now attentively revised the manuscript so that this is clear. The materials and analysis scripts are uploaded to the OSF.

Concerns raised in revisions and/or outstanding:

1.       My main concern regards the number of planned analyses in this Stage 1 manuscript - a primary analysis, multi-verse analysis, and mediation analysis are proposed. Will all of these analyses be reported in one manuscript, supplementary materials, and/or multiple manuscripts? Relatedly, for the multi-verse analysis there are a total of 17 dependent variables (Arrow switching – accuracy; Arrow switching- RT; Arrow switching- Inverse efficiency index; Arrow inhibition – accuracy; Arrow inhibition -  RT; Arrow inhibition - Inverse efficiency index; Arrow attention – Accuracy; Arrow attention – RT; Arrow attention - Inverse efficiency index; Arrow tasks response time index; Arrow tasks accuracy index; Forward Digits – accuracy; Backward digits -  accuracy; Digit span index; Maze – accuracy; Maze - total completion time; Executive function index). Here you plan to analyse multiple indices of the same task (e.g., accuracy and reaction time): could you not focus on one index for each, thus reducing the number of dependent variables being entered into this analysis? I believe this would make your analyses more stringent, allowing you to better assess support for your hypotheses. 

2.       Referring back to a comment from my initial review, from the Introduction and Abstract it is not clear that this is a secondary data analysis/re-analysis of Blattman et al. (2017). This should be clarified from the outset and would only require a minor amendment (for example,  the sentence “To do so, we leverage a cash transfer-based poverty alleviating program” could be extended to state “by analysing pre-existing data from Blattman et al.”).

3.       Please check consistency between the reference to the mini-meta analysis used to inform power calculations on Page 12 and the later description on Page 16. Page 12 states that you will “report Robustness Regions for each Bayes factor with two extreme priors (b = 0.09, b = 1.57), with the half of smallest and twice of the largest effect sizes from the mini-meta analysis described below”, however Page 16 then states that you will “repeat all the analysis with three different priors: the effect size used in the primary analysis (b=0.34), as well as the smallest (b=0.18) and the largest (b=0.79) effect sizes from the mini meta-analysis described above”. Note that the first mention states, “half of the smallest” and “twice of the largest” (which I query separately below), but then later you state “smallest” and “largest” whilst referring to different values. 

There are also a few grammatical and issues of sentence structure, as follows:

1. Should ‘registered report’ not be capitalised (Registered Report)? 

2. Should there be hyphens in “short and long term” throughout (short- and long-term)? 

3. I don’t think the word ‘approximately’ should be abbreviated in the Abstract. 

4. Abstract: “we will use an experimental setting to, and test…”. What do you mean by this? The data are already collected so this study represents a re-analysis of pre-existing data (see my related comment above). Do you mean that you will reanalyse an experimental manipulation (cash transfer) implemented by Blattman et al. 2017? 

5. Page 3, “as compared with”, please remove “as” to aid readability. 

6. Page 3, please correct “Whicherts” to “Wicherts”. 

7. Page 5, “by doing these” – do you mean “by doing this”? 

8. Page 5, “and the focused their paper on how therapy and unconditional cash transfers should affect criminal and violent behavior.” – please revise this sentence paying attention to “and the focused their”.

9. Page 8, “(as well as 28 percent into therapy only (n = 277), and 25 percent into the joint treatment arm (n = 249)).” Add to this “not analysed here” to clarify that these data are not analysed in the present study. 

10. Page 8, “As reported in detail in Blattman et al. 11, the treatment is largely balanced along the covariates”. Can you add “reported below” to this sentence so that the reader knows that these covariates are described later? Without this I am left wondering what the covariates are. 

11. Is there a reference for the ‘arrows tasks’? Is this an established task? 

12. Page 11, “All task materials are available in the Appendix” only needs to be stated once (which it is on Page 10). Also, rather than referring to the Appendix, is it not best to refer to the Open Science Framework project page with the direct OSF link? 

12. Page 12, “with the half of smallest and twice of the largest effect sizes from the mini-meta analysis described below”. The terms “half of the smallest” is very difficult to follow – could this be clarified? Please refer back to my major point 3 when revising this. 

13. Page 12, “an intention-to-treat Bayesian regression analysis in the short term and in the long term separately” – can you clarify “short term” and “long term” here by including the term ‘phase’ (i.e., “in the short term and in the long term phase separately”). 

14. Page 13, Could you include a rationale for the control characteristics included in the model – are these all found to influence cognitive performance? 

15. Page 15, “The results showed the rates of misleading evidence were < 1% both of the hypotheses as well.”. This sentence doesn’t read quite right – “for both of the hypotheses”? 

Dr Charlotte Pennington

Please see the attached PDF.