The role of metacognition in how children test surprising claims

Thank you for the opportunity to review this registered report. The topic is both interesting and timely and has come together quite well. My main concern at this stage concerns clarity and cohesion in how the major concepts (scientific reasoning, uncertainty, and metacognition) and, more importantly, the relationships between them are presented. There are several places where the authors' thought process in connecting these ideas is very difficult to follow. However, I expect it can all be addressed in revision, and hope my comments on specific points of confusion will be helpful.

Major Comments:

104-106 — The claim made in this sentence is sensible, but it is not well matched to or explained by the preceding paragraph. In fact, at a superficial level, it almost suggests the opposite, that how children explore uncertainty is unrelated to their ability to explicitly report it. Possibly, an explanation of the distinction the authors seem to be drawing between “ability to reflect on uncertainty” and “explicit awareness of uncertainty” would help the flow here. 

127-128 — Echoing my previous comment, while the literature presented in the introduction clearly speaks to the claim the authors are making, they are not direct evidence for metacognition playing a role in information search — thus, the authors need to explain the logic of the connection. 

160 — While I agree intuitively with all the statements in this paragraph, I am really struggling to follow the argument as it is currently presented:
-“The ability to accurately report feelings of uncertainty” would seem to refer to a performance limitation, making an accurate explicit response. However, the next sentence suggests that in fact (1) refers to children’s feelings of uncertainty, while the sentence after that goes back to “ability to report.” The authors should take time to address how the difference between performance and competence is relevant to their ideas.
- “Identifying the most likely explanation for why the claim is wrong” seems oddly unmetacognitive, given the focus of the paper?
- The prediction “this effect holds when also controlling for the ability to identify an efficient test” has not been set up by the preceding text, so its importance is unclear. 

207— Perhaps I am simply forgetting something from previous stages of review, but why isn’t there an analysis to see if whether or not children provided a plausible reason for skepticism was related to whether they provided a targeted empirical test?

227 — The “Select question” also helps to control for age-related differences in children’s ability to respond to a generation question, regardless of their scientific reasoning. This strength could also be mentioned here. 

300 — Perhaps I have missed something — but I don’t see why “the yellow thing is pink” is “something to be less sure of”? Given that it is objectively false, shouldn’t children be ‘sure’ it is ‘not true’? 

348 — Given the level of detail with which the rest of the presented, I think it would be consistent and helpful to include one example of the three options on the Selection Task in this section (or potentially as an additional figure). 

373 — I am surprised that so few children (less than 30%) indicated they wanted to find out if the claim was true, given that these claims were selected to be surprising. Given that the design question was only asked for children who said yes, that means that the N’s in several analyses were much lower than the target 175. Can the authors comment on this and the influence on the interpretability of their results. 

375 — For the sake of clarity (even at the expense of brevity) it might be a good idea to replace “T1” T3” and so on, with “Experimental Test Question” “Selection Task Question 2” and so on. 

410-422 — Did the trials differ in how children’s responses would relate to this coding? For example, in the trial described earlier, all three objects need to be lifted to assess the claim that “the small one is the heaviest” — and there wouldn’t seem to be a way of proposing “too much exploration” as it is explained here.

Results & Discussion — I am unsure of what to make of the fact that younger children were less skeptical of the surprising claims than older children, given that the claims were selected to be surprising. I am very open to hearing the argument against this concern: but I worry it undermines the interpretation of children’s later responses as a measure of how testing uncertain claims changes with age if there’s evidence that different ages were not experiencing equal uncertainty about those claims (and even more so in whether they saw them as true or false). 

594 — What is the 74% capturing? How does it differ from the 30% reported in 373?

689 — The distinction between “understanding why you are skeptical” and “skills to reason scientifically” strikes me as odd. There are strong proponents (D. Kuhn, for example) of the idea that mature scientific reasoning -requires- mature metacognition, for exactly this reason. What alternative to that are the authors proposing? 762-764— Since this section and previous sections of the Discussion do not include results on children’s ability to seek out evidence, this statement is premature here. 

774-778 — Perhaps I have significantly misunderstood something, but isn’t improvement in scientific reasoning what the authors are claiming the difference between older and younger children’s performance is capturing? 

812-832 — I think the claims made in this paragraph are impressive and plausible, however, how does this square with the findings (for example from Lapidow et al 2022) that children’s exploration of uncertainty does not rely on their ability to reflect on their uncertainty? Furthermore, how do we separate this from a generalized “ability to articulate”? That is, if I don’t know how to verbally express “I have uncertainty about what’s in the box because I haven’t seen what’s inside it,” isn’t it very likely that I also don’t know how to verbally express “I would look inside the box” in response to the experiment question? 

Minor Comments:

79-82 — This sentence is oddly worded and needs to be simplified. 

150 — What were the ages of the children/age differences in Cottrell et al., 2022? 

289 — I expect this will be made clear below, but could the authors include whether or not the prompt, belief question, etc were also included on the Select Task trials?

This is a Stage 2 review of the manuscript Knowing why: Children’s reflection on their own uncertainty about an adult’s surprising claim increases their tendency to efficiently test that claim.

The study appears to have been conducted as planned, with a total of 174 children in the sample, as predicted. The Introduction is the same, and the hypotheses proposed were followed, with a tweak to the third hypothesis that with increasing age, children will be more likely to suggested targeted empirical tests for a claim. The original prediction did not qualify this prediction but the update added “and that this effect holds also when controlling for their ability to identify an efficient test when provided with multiple options.” The fact that this tweak was added at this stage may need to be added to the paper.

The sample is clearly described. It was a group of children from generally high socioeconomic families, with the overwhelming majority (~83%) coming from a family where at least one parent with a Bachelor’s degree, and about 75% from families with income above the national Norwegian median.

Data were largely coded as described, with high agreement and Cohen’s K. Because of a large number of children who did not directly suggest testing the claim but indicated interest in exploring it, a few new variables were coded for exploratory analyses, categorizing children by whether they wanted to test the claim, whether they  proposed efficient tests or not, or never wanted to explore. Only a handful of children were excluded from the study, and the reasons for these exclusions are detailed. 

In the Results, the authors are clear about which analyses were planned and which are exploratory. I think it might be helpful to clarify what is meant by Block, as that term is only used in the Results section right now.

In testing Hypothesis 1, the effect of age was actually the reverse effect predicted, and exploratory analyses detailed that changing the order of introduction of variables into the model led to an Age x Belief interaction, such that older children are more uncertain when believe a claim and more certain when rejecting a claim, and younger children show the opposite pattern.

In Hypothesis 3, the original proposed outcome variable, the total number of times a child suggested an efficient test strategy, led to a model that supported the prediction of the ability to target empirical tests increasing with age. However, as noted above, this is where additional outcome variables were created and tested, to allow for the inclusions of a larger subset of the participants. Exploratory analyses note that the same pattern held even when using the newly-created variables.

Hypothesis 4 was not supported as planned regarding the effect of prompting, though exploratory analyses provided partial support.

The exploratory analyses are clearly identified as such, and seem reasonable given the restrictions of the proposed coding scheme. They also allow for more nuanced understanding of the data. 

Table 1 adds a column to note which of the confirmatory hypotheses were supported, and which were not, and summarizes some of the exploratory analyses, for a clear overview.

The Discussion clearly walks through the findings and offers solid explanations for the findings. The final conclusions do not differentiate the preregistered analyses from the exploratory ones, though the rest of the paper, from the Abstract, to the Results, to the main part of the Discussion, makes this point clearly.

Overall, I think the study is well-done, in accordance with the preregistered plan, and the analyses and findings are clearly described.