Thank you for the opportunity to review this Stage 1 Registered Report. To contextualise my review, I am a metascientist and psychologist interested in open science and improving methods. I have no experience conducting systematic reviews. I am currently working as data manager for one big team science project using ESM, but this is my only experience of ESM research and data. I look forward to reading the reviews from those more experienced in these areas! In general, the Stage 1 Registered Report seems to me to be very clear and well written. I have a few relatively small suggestions for improvement below.

Abstract: the abstract currently feels quite long, and could be made more concise. It might be helpful to at this point add placeholders for certain information that you do not yet have (e.g. the number of included records, a summary and an explanation of the main result, and the implications of the review) as I’m not sure how much editing is permitted at Stage 2 (feel free to ignore if the abstract is free to be edited as much as desired at Stage 2).

Search dates: I don’t believe that the focus on only 2023 is currently well-justified. If you want the most recent work, would it make sense instead to start 1 year before the date of the first search going up to the date of the first search? Is there a contingency plan to broaden the search if there are not enough articles in this time period? Is there a minimum sample size of articles that would be deemed sufficient? Is there a maximum that can be analysed due to capacity or the authorship team? You say that citation tracking will not be carried out as there would be few additional records due to the time period – would this not just mean that it would be very quick to do, and therefore still worth doing?

Data synthesis: how many reviewers will synthesise the results? Describe how the reliability of the decisions will be assessed if only one reviewer will be involved. In order to avoid bias more than one reviewer should contribute to this process. (Topor et al., 2020)

I always sign my reviews,

Priya Silverstein

The manuscript "Mapping methodological variation in experience sampling research from design to data analysis: A systematic review " is well-written and aims to identify the amount of transparency in reporting - focusing on ESM studies published within one year.

I only have a few minor comments.

It would be beneficial if the concept of adaptivity is better explained. Neither figure 1 nor the first time the concept is mentioned in the text establishes this concept thoroughly. However, adaptivity is one of the criteria to assess transparency and methodological variation. I could (at least in the SOM) envision a table, providing a non-exhaustive overview of adaptivity - and frankly I am not 100% I know what you mean by adaptivity.

Given the amount of ESM studies, the restriction to one year and 150 randomly selected studies is sensible. I do though wonder whether one could apply either a stricter exclusion criterion or consider even two separate papers - published jointy.

My reasoning is that ESM is used in various fields, including clinical psychology / psychiatry. I am curious whether the transparency differs (is lower/higher?) in ESM studies conducted on patients. Here I would expect very little data sharing, and even a too concise data analysis section. Some PSA members have written a book chapter on how to improve open science in clinical psychology (see https://link.springer.com/chapter/10.1007/978-3-031-04968-2_19) as this field of psychology might be behind cognitive psy. Note, I do not have data, only own collaboration with researchers from both fields. Transparency with resepct to describing the patient population is high, but method section are often less detailed in clinical psy papers, including ESM studies. That brings me to another point worth to look at in your analysis: where is the article published.
In the advent of OSF and alike, SOMs are a no brainer, and necessary, as some journals have word limitations! This should be taken into account, not least if the ESM is published as "short report" with a e.g., 4000 word limit. That does severly hamper full transparency.

With 150 articles, it should be doable to check the journal requirements / restrictions if any. Thanks

Very minor: regarding figure 2, software should be software and version

study design may include hardware, some ESM software is only for android phones, other researchers hand out mobile devices to participants (not uncommon in clinical studies). This has IMO an effect on compliance. Please note that I have not checked whether anybody has investigated this.

line 391: "... through MS forms" - are you refering to having used a microsoft survey tool? Following FAIR principle, this acronym may not be understood in 10 years, who knows.

I am not sure you can score a non-reporting as 0 (absent), as there might be absent due to N/A, and absent due to lack of transparency (i.e. the study design or analysis clearly implies that this and that has been done)

I wish you good luck with the systematic review and very much look forward to learnig about your findings.