In general, the proposed approach is careful, clear, and well thought out (much as one might expect given the authors' interest in pre-reistration). I was especially excited to see the authors commit to tracking down unpublished data, as this is one of the most tedious but important steps.

My own expertise is not in psychometric meta-analysis, but looking over documentation for the R package the authors plan to use, I am confident that the analysis will be carried out correctly. Unfortunately, I am not aware of the exact ways in which psychometric MA interacts with publication bias, which I do consider to be my area of expertise. It is my understanding that the primary issue is that low reliability may correlate with publication status or sample size and, therefore, one might incorrectly conclude that bias exists when using typical correction methods. An obvious response to this would be to report on this correlation, which I hope the authors will do for any meta-analytic dataset they are interested in.

A further exploratory analysis could also be proposed in which reliability-corrected effect sizes are analyzed (as in Fig 3 here doi:10.1177/2515245919885611) using standard publication bias correction methods and the kind of sensitivity analysis I have recommended in "Correcting for bias in psychology: A comparison of meta-analytic methods." The issue here, of course, is that this kind of analysis is, to my knowledge, not studied in simulation. That would make it difficult to draw strong conclusions in the event results were difficult to interpret. This exercise might still be very useful.

On a similar point, the authors write, "We also conducted a sensitivity analysis (Mathur & VanderWeele, 2020) with the use of cumulative meta-analysis." I wasn't familiar with Mathur and Vander Weele's work, but in looking over the paper, the authors' sentence doesn't immediately make it clear how they will deal with publication bias via sensitivity analysis. I believe they're referring to section 4.1, and if so, this approach doesn't seem to provide a corrected meta-analytic method, but a sense of whether or not the true effect could be zero and simply inflated by publication bias. In my own work, I prefer to focus on producing corrected estimates as I beleive they make for more useful and impactful meta-analyses. However, if the authors feel that this method meets their needs and will provide useful info for future researchers, I am completely in support of its use here. 

I noticed two other points on which I think clarification would be useful:

1. When results are only in the form of regression coefficients, how will the authors deal with multiple regression models? From what I can see in the supplement, the regression coefficients for which there is a plan seem to come from single-predictor models. The paper, "Concealed correlations meta-analysis: A new method for synthesizing standardized regression coefficients" may be a good resource. 

2. Will multiple coders be used per retrieved study? Will inter-coder reliability be reported? 

This is an ambitious project and I really think the authors should be commended for their rigorous approach!

Evan Carter

The proposed research is an interesting and important topic (personal factors and their impact on group creativity). Possible moderators are well-described and many aspects of their meta-analytic procedures are well done and clear. The theory is laid out well and explicitly.

There are serious problems with the proposal, however. Replication would be difficult because there are problems of confusion/clarity mostly dealing with measurement/operationalization issues. The biggest problem I see in this proposal is the authors are very unclear on how to operationalize the key components of the study, namely team creativity and personal factors of the individuals within the teams. The obvious problem is measuring individual personality and cognitive data per individual team member and then measuring creativity at the group level. The researchers certainly have an answer to this but, as far as I can see, they do not make that clear anywhere in this proposal. As someone very familiar with personality and creativity research I need them to spell out the mechanics of individual versus team level measurements. For example, let’s say a team has 10 people. That is an N of 10 on personal factors. But team creativity (fluency, originality, etc) of the team and has an N of 1 (one team). So how do you correlate across levels of analysis?

Similarly, in their Introduction they confuse/conflate research at the individual and group levels. For instance, Table 1 title says “personal factors in creative groups” but then the researchers review studies that do not deal with group creativity (e.g. Furnham, Batey, King, etc). Other studies in the table are clearly at the group level (Bechtold et a., 2010). So these seem to be confounded in this table. Similarly, in the section on “Relationships between Personal Factors and Creative Activity Characteristics” (1st paragraph), they say there is a debate regarding relationships between personal constructs and group creativity and then cite research that was not at all group-based (e.g., Feist, 1998).

The lack of clarity about group continues in the Design section (p. 8). Creativity outcomes are described as “number of ideas generated,” “originality of these ideas,” and “usefulness of the ideas” without specifying group and without operationalizing originality and usefulness.

Other scholars have been more clear about these problems.  As Litchfeld et al (2017) discuss in their chapter, team personality (and creativity) must spell out whether it is measured via the composition or compilation method. The current proposal does neither. Even so, in my mind, the composition method has a problem, since it derives a team-level score from either the mean or variance of that trait. The mean without variance within a group can be very misleading. A mean of 50, for example, could be derived from a team that varies little or a lot around that mean of 50. Yet, substantively, group with a little or a lot of variance are different groups. The variance without the mean is better since you can have high and low heterogeneity groups on a personality dimension. But this distinction is not made in the proposal.

Coursey et al (2010) also discuss the dynamic and potentially synergistic effects of individuals working in groups and distinguish between the additive, contingent, and configural approaches. And they also explicitly discuss “aggregate Openness” for instance when discussing personality and group creativity. This kind of discussion is missing in the proposed study and is needed to clarify how person factors and group creativity are operationalized.

The same problems that exist in operationalizing personality also exist for team creativity. But the authors never even address this question. Is it team creativity via composition or compilation method? Does the team get one score or many on each creativity outcome (e.g., originality, fluency)?

For the most part, the hypotheses are meaningful. Publication status, however, is bit obvious and well-established in the meta-analytic literature, namely the larger effect for published versus unpublished studies. I am not sure that adds anything to the study or the literature.

In keywords for the literature search, I don’t see “brainstorming.” They say in their Intro that is synonymous with group creativity,  so it is surprising that is not included.

The authors do a good job of avoiding problems associated with Null Hypothesis Significance Testing (NHST), but met-analyses generally do.  I am not sure, however, there is need to use their “meaningful” criterion of r > .10 since effect sizes have their own more established “rules of thumb” for small, medium, and large effects (see Cohen). But this is not a critical issue.

As a meta-analysis, the ethical issues are minimal to none and there is no untoward conflicts or problems with this study. No IRB is required.