Dear authors

As you can see, the final reviewer has now recommended in-principle acceptance of your Stage 1 RR manuscript. However, I notice that there are still some issues to be fixed before I can grant IPA. Specifically, under Analysis Plan 1.2 you list the same hypothesis twice. There may be other issues that I might have missed, so please carefully check the manuscript.

Moreover, looking through the Appendix with the survey materials it occurred to me that the survey is considerably longer than one would expect from reading your Methods, probing aspects such as colour associations, emotion type, etc, and the Goldsmiths Musical Sophistication Index. Given this is therefore a considerably richer data set than strictly needed to address your preregistered hypotheses, some caution is advised. Obviously, it is perfectly fine to collect additional measures for further exploratory analyses, which can possibly be included in Stage 2 as long as they are explicitly labelled. It is also sensible to limit the preregistration to clear-cut predictions you can make now.

However, given the large amount of additional information it is important to avoid the scenario where the Stage 2 article consists mostly of exploratory analyses. This would in my view undermine the purpose of conducting a Registered Report in the first place. One could also argue that the methods description should contain the necessary details of all the measures being collected. I apologise for not spotting this earlier (but I am somewhat reassured by the fact that the reviewers didn't either).

Given the late stage and the fact that the preregistered study itself is now quite straightforward, I don't think this requires an extensive revision. For the Goldsmiths questionnaire, a short sentence with the appropriate citation explaining its inclusion will suffice. For the other measures in your actual survey, you could possibly even keep that very succinct, simply stating that the survey contains other measures, briefly list what they are, and that they are collected for future hypothesis generation etc but will not be presented in the Stage 2 article. If you plan on exploratory analyses of any of these data, more detail about them should however be provided in the Methods. Even then, I don't think this calls for extensive rewrites - these are simple survey questions that can be described easily.

Keep in mind that all exploratory analyses in Stage 2 will be subject to peer review and could possibly constitute grounds for rejection.

As always, please contact me directly if you have any questions or comments on this.

Best wishes
Sam Schwarzkopf

Dear authors

Your revised RR Stage 1 manuscript has now been reviewed by the same two previous reviewers. Their comments are generally positive but as you see one reviewer still has several substantial concerns that I would ask you to address in another round of review. Some aspects are probably judgement calls, in particular the question of the effect size of interest and the resulting necessary sample size. As suggested by the reviewer, I may make a final decision about granting in-principle acceptance based on that in the next round.

Both reviewers also raised concerns about the statistical approach and it took me a while to figure out why. In section 1.2 "Analysis Plan" you still refer to paired t-tests instead of the new analysis approach you are now using. (I stared at the relevant page for some time without noticing it, so I thank the reviewers and Chris Chambers for pointing it out!) I assume this is a mistake and can be fixed easily. But please go through the whole manuscript carefully to ensure all parts are up-to-date as this is the version that will define your preregistered methodology.

Best wishes
Sam

Dear authors

Your Stage 1 RR manuscript has now been reviewed by three experts in the field. You can read their detailed comments on the PCI:RR website. Based on their comments, the manuscript is not yet ready for in-principal acceptance by PCI:RR.

Please submit a revised version of the manuscript, including one with tracked changes, and a point-by-point response to each reviewer's comments. I summarise the main points here, and also flag up a small but important typographic error.

Rationale:

I concur with the reviewers' concerns that the rationale and the whole premise of the study lacks clarity. For example, it should be explained early in the manuscript what is meant by "visual density", how this relates to visual imagery, and why you expect that to depend on music.

Statistical approach:

Moreover, all reviewers raise issues with the statistical approach that parallel my own comments from pre-screening. The series of t-tests does not seem to accurately reflect the hypotheses. An effect across all cultures could be reflected in a significant main effect in an omnibus test. But as you see, reviewer Juan David Leongómez​ has an alternative suggestion and even provided some example code to be used for a potentially better-suited approach. I suggest you develop these proposals further - the reviewer offered to assist with this but of course you may wish to also draw upon the expertise of others.

Sample size:

One reviewer calculates a sample size for your expected effect size of n=41 rather than 14, which makes me wonder if the 14 you've used is a typo that permeated through the manuscript? All reviewers raised a concern with this small sample size. You seem to expect a relatively large effect size that is perhaps justified for the arousal and visual density ratings, but for any difference between cultures I would expect this to be a lot smaller. As mentioned by several reviewers, given the fact you are planning online data collection, it should be entirely feasible to collect a much larger sample than n=14?

Typographic errror:

Finally, there is still an error I already flagged up during prescreening: Under hypothesis 1.2.2 you talk about 'arousal ratings' instead of visual density judgements.

Note on one review:

Judging by their comments, one reviewer may have commented on the originally submitted version of the manuscript rather than the revision after my pre-screening. So some of their points may no longer be relevant. But it is also possible that I missed issues in my (hasty) pre-screening. Please check the comments carefully and respond to each point by this reviewer to clarify.

Best regards
Sam Schwarzkopf

Dear authors

Thank you for your submission of your manuscript to this scheduled RR review. We regularly prescreen RR submissions to ensure they are ready for review. In this case, there are several issues that need to be addressed before I can invite the scheduled reviewers.

First up, let me apologise that I am writing this in some haste. This is the first scheduled review I am handling and the tight schedule means that I want to get these comments to you as soon as possible so you can redraft the submission in time for the reviewers to conduct their review. In your revision, please include a quick response to each of my points but I won't ask you to write long-winded prose here - bullet point replies will suffice. However, please include a version of the revised manuscript with tracked changes on the sysyem though to facilitate the screening.

1. Hypotheses: Please ensure that all hypotheses listed in the text match those in your Design Table. Also consider if the statistics used to test the hypotheses are appropriate. Your text describes the hypotheses as testing for correlations, but your statistical approach does not explicitly test for correlations. I appreciate that you could informally phrase the research questions this way but the hypotheses should match and explicitly describe the analysis used. In the same vein, please also spell out your inference plan in the main text to match what is explained in the Design Table.

2. Statistical approach: You propose a series of three paired t-tests for each cultural group (plus possible equivalent tests). Consider using an omnibus test to test the significance of the main effect of tempo (irrespective of group) and any interaction (in case the effect differs by culture). I believe this would be more sensitive. Alternatively, please provide a clear justification for the approach adopted here. You also write that the use of Likert scales mean that the parametric statistical tests are inadequate. You may consider using non-parametric alternatives like a Kruskal-Wallis and Wilcoxon signed-rank tests. However, I also appreciate that parametric statistics are commonly used for such ordinal data and this use may not necessarily pose a major problem. Nevertheless, if possible I would seek out the advice nof a statistical expert.

3. Exploratory analyses: In a RR Stage 1 submission, all planned tests should be formulated as preregistered hypotheses with a detailed statistical analysis plan, supported by power analysis. The RR format does not preclude carrying out exploratory analyses at Stage 2, provided they will be labelled explicitly. However, this does not belong in the Stage 1 manuscript. Please remove the Exploratory Analyses section or add these tests as fully preregistered hypotheses to the manuscript and Design Table, with appropriate power analysis. This same point applies to any comparison between solo and group excerpts. It is fine to describe these differences in your pilot data, but if you plan to conduct this comparison you will need to add specific hypotheses for this supported by power analysis, or only add these as exploratory analyses in Stage 2 but not mention them in Stage 1.

4. Power analysis: On this note, you state in the Power Analysis section on page 6 that the minimally interesting effect size is d=0.04 but later on and in your Design Table you power the experiment for d=0.4. I assume this first instance is a typo? The power analysis is based on a conventional alpha=0.05 (Bonferroni corrected for number of hypotheses) but 95% power. This is fine and the decision is yours, but please note that many journals accepting RRs for publication have specific power requirements, most commonly 90% power with alpha=0.02.

5. Missing methods details:

a) Please add details about the visual stimuli used in the experiment, such as the dimensions of the patterns, how they are generated, etc. The methods should make independent replication possible. Since your data collection is online, I appreciate that some parameters (like visual angle or luminance) are outside your control - but at least relatively parameters can be reported in detail. My suggestion is also to add a figure showing the texture patterns.I may be missing something but I also do not follow how X2+1 results in 31 lines (assuming X is the density scale variable X=5 in this case?)

b) Under Exclusion Criteria please explain what you mean by participants who "most associate with the culture of the country" and how this is determined.

c) Under Materials it is uncear to me why there are 24 excerpts. With 6 excerpts (one solo one group, from each of the 3 countries) and two tempos, aren't there only 12 excerpts?

d) Also under Materials, please rephrase the first sentence to clarify that this is effectively an online survey participants "will fill out".

6. Error in Analysis Plan 1.2: Please remove the "emotion categories" from the first sentence in this section as this refers to potential exploratory analyses but is not preregistered. Also, under 1.2.2 Temporal-density correlation you state that arousal ratings are being collected instead of visual density.

7. Phrasing of hypotheses: In the text you refer to the hypotheses as testing the effect of tempo "changes". Technically the experiment compares stimuli with different tempos not specifically the change in tempo. This is a minor point and you may want to keep it as is - just food for thought.

I look forward to receiving your revised manuscript in due course.

Best wishes
Sam Schwarzkopf