Thank you for the invitation to review the Stage 1 report for "Finding the right words to evaluate research: An empirical appraisal of eLife’s assessment vocabulary." It was a pleasure to read this report. It's a fantastic proposal, and I only have minor points to make.

What went well

This stage 1 report was encompassing and provided a fantastic introduction to the material. I've followed eLife's changes, but was not aware of the details that form the basis of this paper. I felt like the authors shared their expertise generously and that I learned why this work is important. It's of course fantastic to see a paper on communication be this strong in communicating itself.

I also very much enjoyed the dynamic document available to highlight what analyses will be included. I wish every manuscript that crosses my inbox was as rigorous as this. It made reviewing a breeze and I felt confident in understanding what's going to happen.

What could be better

My only minor connotations are the following:

• This is called a "psychometric study" - that's a bit of a stretch given that there is no psychometric modelling happening. I'd suggest to drop the psychometric altogether.
• The introduction highlights the potential confusion around the current vocabulary eLife uses - is there any anecdotal evidence that indicates confusion goes beyond the author team? I read the references, and I think that's sufficient evidence to take this step, but it would be good to know whether in this specific instance that confusion is already observed
• Continuing on that note - it might be helpful to include some context on why this specific eLife vocabulary was included to begin with. I imagine they deliberately chose this, and had certain considerations. It would strengthen the final report to have this, and if that is not already public, it might be worth reaching out to Mike Eisen (or someone else from eLife) for some comments on the vocabulary. This will help ground the reader and may provide you with worthwhile information and stepping stones to potentially improve the language in practice at eLife (if that's something you'd like)

What to watch out for

---
Warmest regards,
Dr. Chris Hartgerink

I’m excited to review this MS, as it tackles a meta-scientific issue toward which I have great personal curiosity and interest. The work is clear and well-organized. I’m not an expert in statistics so I leave in-depth commentary on that area to other reviewers and recommenders (with a few exceptions). I’m an interdisciplinary researcher with a background in philosophy/theory and qualitative work; this info hopefully helps interpreting my feedback. The comments come in no particular order but I list them to make it all easier to read. 
 
1. The premise of the MS is as follows: “Our understanding (based on eLife’s New Model, 2022) is that eLife intends the common vocabulary to represent different degrees of each evaluative dimension on an ordinal scale” (p.2). This is later paraphrased on page 5: “The success of eLife assessments will depend (in part) on whether readers interpret the common vocabulary in the manner that eLife intends.” As I read it, the MS is based on the authors’ guess about what “eLife intends” in their recent release. I don’t know the authors’ positionalities (e.g., if some are or have been affiliated to eLife) but to me it would be obvious to *talk* to eLife as a Pilot #1 interview and simply ask what they intended (instead of guessing). This would make the foundations of the study stronger. Open dialogue could correct misunderstandings and save time.
2. Related to the above, the MS correctly points out how the significance and support domains involve many dimensions. The authors mention breadth/scope and degree, but personally I see many more. E.g., a study could be “valuable” for pragmatic reasons in a subfield and immediately save human lives (vaccines etc.) and this feels incomparable to theoretical development or other contributions to cumulative models or science at large. Likewise, “support” seems to conflate various dimensions related to methodology; e.g., would larger effect sizes or statistical power contribute to “more support”, and how would e.g., qualitative evidence or clinical case studies be assessed in this domain? On the other hand, if “support” here simply refers to a general commitment to established methodology (as is implied by “exemplary use of existing approaches”), this doesn’t seem to have much to do with “support” but rather methodological rigor (e.g., a correctly reported patient case study of N=1 could have “exceptional strength of support”). This goes back to my #1: it would be important to know what the intended areas of evaluation really are -- and whether they are intended to operate purely as continuous ordinal scales -- before developing competing vocabulary.
3. I really like the Box 1 example, as it demonstrates the practical problems related to the current eLife model. Indeed, while the chosen words now have clear definitions and meaning, the full description additionally uses various undefined evaluations such as “huge amount of data”, “very interesting observations”, “well supported by the data”, and a promise to “sharpen our understanding.” That is, while the reader could pick up the meanings of one or two now-defined terms, most of the semantics remain vague and undefined. If the vocabulary is meant to operate as a continuous ordinal scale, the hermeneutic benefits over simple numeric ratings seem to disappear in actual use. This comment is perhaps directed more to eLife than the authors, but could be useful to reflect on in the MS.
4. It’s not stated who the participants are. It’s noted that the data comes via Qualtrics and “our sample are more likely to be undergraduate students” (p. 13). It would be good to clarify this aspect of the data.
5. There’s one hypothesis in the MS, expecting a higher accuracy for the alternative wordings. That’s quite simple, but I have two notes (naturally ignore some of this if it feels not right). First, I think it always helps the reader when hypotheses are framed in such way that: *why* something expected, *what* is expected, and *where* do the results lead us (“Because X, we expect Y. If Y, that’s interesting because Z). Currently, the hypothesis is kind of “hidden” in a wall of text. Second, I would really like to see something in the *where* part (“theory that could be wrong”). It’s written in the final table that this is an applied RQ, but they also have implications and it would be important to spell out these implications at Stage 1 (especially in the final table, which serves as a reference at Stage 2). 
6. Page 12 says there are no planned analyses for RQ3. Although I strongly support exploratory and descriptive work, I would also like to see a bit more detail here (skip if it doesn’t feel right). Another note that may not be directly relevant: it feels the auxiliary hypotheses behind “To what extent do different phrases used to describe scientific research elicit overlapping interpretations” is that everyday terms are expected to “elicit” certain interpretations, and this is what eLife is trying to catch. However, as I see it, the purpose of the new eLife vocabulary was to give *new technical definitions* to these terms and not to try catching their organically eliciting universal features (I can be wrong). It could be a useful enterprise for any journal to build a glossary for assessment terms, but that seems to be a different effort vs trying to capture the “true” meaning of everyday terms (see my #8 too).
7. I list a number of small technical comments. 
           a.     There’s only one attention check. I know there are many opinions about this and no single correct solution, but subjectively I like giving participants the opportunity to correct their mistake once with a follow-up attention check if they fail (some people have naturally low attention by neurodiversity). Alternative checks could be used too.
           b.     Page 8: “ignoring exclusions” -- for this non-native English speaker it’s not very clear, perhaps just with or without exclusions?
           c.     In the inclusion criteria, me coming from Finland, I don’t know what “A-levels” are.
8. Last, I briefly return to the RQs. As per Wittgenstein, words gain meaning through language games, and one and the same word can have multiple meanings depending on the context. Here too, different use contexts could yield different messages, as “evidence” remains relative to a “claim.” Sadly I’m not a linguist nor semiotician, but I would think there’s existing evidence/theory in these fields to which the present findings could contribute (or learn from). The design nicely borrows from earlier studies on probabilistic statements, but I’m also reminded of e.g., Manfred Krifka’s and Teun van Dijk’s work, which could be informative at least at Stage 2. I don’t have the topic expertise to be able to pinpoint what specific model/theory would be helpful in this context, but I would love to see further interdisciplinary bridges in future investigation. Again, just skip this if it’s not helpful (hopefully another reviewer is a topic expert). 
9. Although significance and support are rated independently, would it really be possible for a paper to have e.g., “landmark” significance and “inadequate” support? (Feeling the future?) In reality it seems the two domains are somewhat artificial and not necessarily worth separating, and if so, it could be worth rethinking how to frame this study to maximize its application value for the academic world and its evolving journals.
 
I hope some of my comments are useful; just ignore those that aren’t. If something feels unclear or unfair, I can be contacted directly. Best wishes for revising this important paper that will likely yield compelling evidence,
 
Veli-Matti Karhulahti

The study aims to assess the perception of vocabulary describing studies used by eLife and an alternative set of descriptions which is supposed to improve on the descriptions used by eLife. The study is practical and well designed. I have just a few comments:

1) Readers who see a description of a study usually probably read just one description used for a single study. The experiment, however, has the participants evaluate the whole set of descriptions as well as its alternatives. The comparison of the different descriptions is thus much easier and more likely. It is possible that one set of descriptions is easier to order while the other is easier to interpret when seen alone. As an absurd example, it is possible to imagine vocabulary “100% support”, “75% support”, etc., which would be ordered correctly, everyone would rate the support similarly on the scale used in the study, but it would be probably useless in practice because it would be hard to know what the percentages mean when taken out of the context of the current study.

2) The strength of support has 6 levels in the eLife vocabulary, but just 5 levels in the proposed one. How is this taken into account in the analysis?

3) The answer scales should probably show percentages, otherwise people might not know how to interpret them (it is mentioned just once in the instructions that the participants should answer in percentages). Another question related to the scales is whether they even make sense. That is, what does it mean that a study is 80% important or that it provides 30% strength of support? I would have little idea what these statements mean. It is possible that this will not be a problem if people use the scales consistently, but that seems like something that should be established or at least discussed.

4) It should be possible to come up with a way to analyze the research aim 3. For example, the ideal scale should probably be interpreted to have the subsequent levels equidistant and cover the whole scale (0-25-50-75-100). It is then possible to use a difference between this ideal interpretation and the participants’ interpretations to assess the vocabularies. Similarly, for research aim 1, it is mentioned what will be reported, but not what will be then interpreted. In the simulation, IQR is used for interpretation, but it is not mentioned in the article itself.

5) Is it correct that the example on p. 19 yields an odds ratio of 2? It seems to me that the odds would be 2, but odds ratio would be (0.2/0.8)/(0.1/0.9) = 0.25/0.11 = 2.27. But, I might be mistaken. 

Signed,

Štěpán Bahník

Comments on the introduction

Perhaps I am old-fashioned but it might have been nice to turn-on line-numbering on your manuscript to make it easier for reviewers to refer to specific lines. One for next time perhaps :)

I think it would be useful to mention to readers that making peer review reports publicly available alongside the published paper, on a MANDATORY basis (for all research papers in that journal), has been practiced at some journals for over TWENTY years now. Examples include BMC Medicine since the start in 2003 & Atmospheric Chemistry and Physics . The intention of this is not to promote any particular brand or publisher, but rather to help the readers of this research understand that in some communities this practice is very far from ‘new’ or ‘untested’, it is NOT a new innovation. Publicly available peer review reports are a tried and tested system.

Merely stating a “growing number of journals” is true but in my opinion is insufficient context. 

I would also quibble with representing the entirety of the new model eLife has chosen as to “go even further”. Specifically, abandoning ‘accept/reject’ is not on the same plane as publishing the peer review reports. By abandoning ‘accept/reject’, eLife are not going further in the direction of transparency, but rather they travel further on a different plane, that of egalitarianism(?) 

“These improvements”  - I’m not an eLife basher. But is it not a little bit subjective to proclaim the changes are definitely improvements? I’d just call them changes to the eLife publishing model/process for the sake of objectivity.

“their success will depend on accurate communication with readers” hmmm… this is a little bit debatable. What do you, the authors of this manuscript define as ‘success’ for eLife? What does the eLife leadership team define as ‘success’ for eLife? What do authors who have published with eLife both before and after the changes see as ‘success’ for eLife?

Success if mentioned at all needs to be defined. Success may be measured or assessed differently by different stakeholders. 

It seems quite a bold assertion to say that success (however defined) depends on accurate communication (of what exactly?) with readers. I would say there have been many journals that have been commercially successful despite having very poor communication with readers. Likewise, taken from a different viewpoint of ‘success’ there are lots of small not for profit run journals that have been tremendously successful from the point-of-view of publishing rigorous reproducible robust research, consistently highly relevant and thought-provoking to those interested in a small discipline – these would all be considered ‘failures’ from a narrow financial profit/surplus perspective – why did the journal not publish more, why did the journal not make more revenue? Thus that line: 

“their success will depend on accurate communication with readers” 

Needs a whole lot more explanation and definition for me to able to understand it, let alone possibly agree it could be true. I can’t approve it until I understand exactly what the authors mean here. If it is not essential to the work, perhaps just strike it out? Talk about eLife’s future instead perhaps – future is more neutral and less measure/viewpoint dependent?

Linguistic pedantry: “...used in eLife assessments is perceived as intended by potential readers”

I think as both the assessments are plural and the readers are plural then the perception of those assessments by readers should also be plural? "is perceived" -> "are perceived" .

Have the authors considered whether eLife’s common vocabulary might be _intentionally_ ambiguous and thus not always necessarily so transparently easy to order on a scale? I don’t know if that is the case, but if it really were so easy to number the terms why wouldn’t eLife just use a 0 to 10 number scale? Is ambiguity in assessment not ‘romantic’ or some such, a specific and deliberate avoidance of ruthless and inhuman clarity?

On the suggested alternative vocabulary, have you thought about suggesting “no importance” ? Given the prevalence of AI-generated texts and fraud in the publishing system,  I can clearly see times in which a reviewer will want & need for the sake of integrity to reach for “no importance” or “zero importance” to accurately represent the utter rubbish they have read. I say this even despite footnote 1. Yes, eLife will likely be able to filter out some of the trash before sending for peer review, but perhaps not all of it.

I am certainly not an expert in the science of misperception. Far far from it – I have no research experience in this area.  However I do think it is naïve to think that simple English language words & phrases will ever ever be necessarily interpreted exactly/identically the same by people from e.g. different cultures and different parts of the world. Language and meaning doesn’t always have absolute precision, especially across different cultural contexts. 

The word ‘tabled’ is a classic & extreme example of this. A British person would probably think it is an idea on the agenda for discussion, whilst an American might think it has been postponed or cancelled – despite being the exact same word, and same spelling and written in “English”. It has two very different interpretations by two different communities of readers - both of whom are fluent in "English".

Regarding aim Two

The authors themselves allude to this but what if there isn’t necessarily a single unidimensional ranking for strength of support. What if instead there were three or four dimensions upon which a manuscript could be assessed. eLife seems to only band it into two: ‘significance’ and ‘Strength of support’ – perhaps this is where an issue might lie?

Comments on the proposed methodology

As far as I know, eLife was not created or intended for _just_ the UK and USA demographic, neither on the author-side nor the reader-side. The authorship profile in this journal certainly is more diverse than just this, and is publicly available data (e.g. at Lens: https://www.lens.org/lens/search/scholar/analysis?p=0&n=10&s=date_published&d=%2B&f=false&e=false&l=en&authorField=author&dateFilterField=publishedYear&orderBy=%2Bdate_published&presentation=false&preview=true&stemmed=true&useAuthorId=false&sourceTitle.must=eLife  )

As I have alluded to earlier, I think it is important to account for cultural/geographical variance in the interpretation of English-language words and phrases. Unless I missed it, I see nothing in the method that would prevent the participants from comprising 100% of e.g. people born and raised in the USA. Which would be useful in some respects (some data better than no data), but not a global view, and eLife is globally read and diversely authored-in (not all countries e.g. zero Zambia/Chad/Sudan-based institutional author affiliations so far) as far as I know.

The WEIRD (Western, educated, industrialized, rich and democratic) bias is well known in psychology and I’m surprised I haven’t seen this registered report do more to ameliorate it (okay, maybe the E is needed in this case given the high-level content, but the potential for WIRD bias still needs to be acknowledged and ameliorated). 

I see the authors refer to it as a "convenience sample" in the limitations section, and I acknowledge that. But I think just a little bit more effort and a slightly more complex design would significantly improve the validity and robustness of this research.

Ideally, I would like to see the participant sample stratified either by where existing eLife readers are known to come from (if that data is available) OR by where English language scientific articles are produced from, for which data is most certainly available e.g. https://en.wikipedia.org/wiki/List_of_countries_by_number_of_scientific_and_technical_journal_articles  The participant sample absolutely must comprise at least some people living in these major consumer&producer (of science) countries: China, US, UK, India, Germany, Italy, Japan, Canada, Russia, France, Australia, Spain, South Korea, Brazil, Iran, Netherlands, Turkey, Poland, Indonesia, Switzerland… 

Given the stratification required and the chance that country/culture has an effect on perception of English-language words, I suspect the minimum sample size will also need to be enlarged to accommodate the ability to compare perceptions between countries, which clearly won’t be statistically meaningful to do if only 300 participants (all countries) are sampled.

It might also be interesting to consider, for readers whose first language is NOT English (whilst also being fluent in English), how exactly do they choose to read review reports at eLife ? A small proportion for instance may opt to machine-translate from the English report into their first language, and _then_ read the report. One must perhaps examine how the two suggested vocabularies are machine-translated into other languages and if any peculiarities arise from that. Facebook machine translation once translated ‘good morning’ into ‘attack them’ leading to a Palestinian man being arrested (see https://www.theguardian.com/technology/2017/oct/24/facebook-palestine-israel-translates-good-morning-attack-them-arrest ). Language, translation, and perception is hard to get right.

Comments on the proposed analysis plan

The analysis plan needs to factor-in country/culture of the study participant and analyze potential differences of interpretation between source countries. Not controlling for this potential variance undermines the usefulness of the proposed design.