Recommendation
DOI or URL of the report: https://osf.io/n7ga6
Version of the report: 3
Dear Dr. Fillon,
thank you very much for your fast reply and support in further strengthening our manuscript. We have worked on all mentioned passages and included your suggestions almost identically.
We are looking forward to your feedback and hope to have incorporated everything correctly.
While you suggested PsyArxiv, we stayed with osf for now. If you want us to migrate the project, please let us know to discuss this option.
Best regards and have a nice afternoon
Leopold Roth and Tassilo Tissot
, posted 19 Jun 2024, validated 19 Jun 2024Dear Authors. I went over the draft, and you will find below all of my comments to help you improve the text, before I can provide an IPA. Please treat my comments with critical thinking, and adjust them if needed.
Abstract
in countries not yet included in this research
Included in this study
we will examine whether lower effort moralization is observed as a function of age (including non-linear terms).
we will examine whether lower effort moralization depends on participant’s age.
Introduction
p.8
The ideological debate about the lack of qualified workforce and specifically amongst younger potential employees
of objective reasons
These reasons seems not to be “objective”. I would say structural or systemic.
An effect that persists even if the effort is not productive (Celniker et al., 2023)
This effect persists…
Regarding the above-described debates, we hypothesize that younger individuals show less effort moralization of ineffective labor – not judging higher, ineffective effort as a sign of higher morality.
The effort moralization effect can inform the debate around the lack of qualified and willing workforce. We hypothesize that younger individuals indicate less effort moralization of ineffective labor. Younger individuals do not judge ineffective effort as a sign of higher morality based solely on the effort.
This offers
Replicating the effort moralization effect offers
This is crucial to select romantic partners (Brown et al., 2022; Chan, 2023; Oda & Hayashi, 2020)
I would delete this part, keeping only the “Moral judgement is crucial in cooperation settings (Celniker …)
The estimation of future moral behavior is by nature not trivial, which is illustrated by the multitude of models and measures around moral foundations, moral identity, virtue, or similar ideas (Aquino & Reed, 2002; Haidt & Joseph, 2008; Ruch et al., 2010; Schlenker, 2008). Still, most individuals depend on approximations of character virtue through observation in daily life. While individuals rely on a variety of cues for this purpose, including facial and body expressions (Horberg et al., 2013), stereotypical appearance (Grizzard et al., 2018), or religious beliefs (Gervais, 2011), one of the main signals for inferring the morality of others remains behavioral observation (Mickelberg et al., 2022; Pizarro & Tannenbaum, 2012). Naturally, these require some sort of quantification to tell, how moral a person is, based on mostly trivial actions.
Suggestion:
To attribute morality to others, most individuals depend on approximations of character virtue through observation in daily life. While individuals rely on a variety of cues for this purpose, including facial and body expressions (Horberg et al., 2013), stereotypical appearance (Grizzard et al., 2018), or religious beliefs (Gervais, 2011), one of the main signals for inferring the morality of others remains behavioral observation (Mickelberg et al., 2022; Pizarro & Tannenbaum, 2012). Observations lead to create inferences about the morality of a person based on trivial actions.
One phenomenon, that has raised scientific psychology’s interest in recent years is the observation that people appear to use effort invested in given tasks, as information on the morality of agents, summarized as effort moralization effect.
This does not flow well with what is above so I suggest deleting it.
While the core idea likely follows
The core idea of the moralization of effort is a heuristic…
Also, put a point between "performance" and "the focal interest"
p. 12
This was still observed when the behaviors were not successful
The effect of effort on moral judgement was still observed when…
I also wonder if the term “successful” is the good one. Maybe the behavior was not predicted? The behavior was not expected?
p.13
As described above,
Not useful, delete
p. 14
economically redundant labor, such as universal basic income
You don’t need the coma
It is plausible to assume that recent movements such as the ‘great resignation’, ‘quiet quitting’, etc. represent responses to such resistance, fueled by a generational shift in work values and changing perceptions of (necessary) effort among younger generations.
You don’t need the coma before fueled
Such differences may provide
You don’t need the may (it is already in the sentence before)
p.15
You need to put a dot in the big sentence. I propose to put it at the end of mexico as follow:
research (Germany and Mexico1)
namely Germany and Mexico. Testing the effect in different countries will inform the generalizability of the effect.
Use “found” or “indicate” instead of demonstrated.
p.16
was computed, using the pwrss
You don’t need the coma
Because we were conducting the study in Germany and Mexico,
You can begin by “We translated the vignettes in German and Spanish
by two independent translators, one of whom was one of the two authors of this paper
By two independent translators, including one author of this paper.
p.21
To test whether the effect of effort moralization was replicated
To test the signal consistency of the effort moralization effect between the original study and the replication
Please ensure that if you delete citations, you do as well in the reference list. You can also use recitework to help you. Upload a clean version to PsyArxiv and come back to me once it is done.
Best regards,
Adrien Fillon
DOI or URL of the report: https://osf.io/63xws
Version of the report: 2
, posted 05 Jun 2024, validated 05 Jun 2024Dear Authors,
I now received reviews from two reviewers, who were positive regarding your manuscript, and satisfied with the changes you made.
They both asked for improvement in the writing of the introduction, and in the flow of the article. I think that they are easily addressable. I look forward to receiving a revision.
Best regards,
Adrien Fillon
Geneva, May 7th 2024
Review of PCI RR – Second Round of Stage 1
Is it Worth the Hustle? A Multi-Country Replication of the Moralization of Effort Effect and an Extension to the Increasing Aversion to Bullshit Jobs - second round
I am happy to see a revision of this paper. It is much improved. Sorry for missing that the data was simulated in round 1, my fault.
I want to point to you a Twitter thread regarding older people saying that now, young people do not want to work anymore, by Paul Fairie: link
If not useful, certainly funny.
I still have some observations about the writing:
1. I still think you should be more to the point in the introduction and tell your reader immediately that you want to replicate the EME (which is the primary thing you do) and test whether age moderates it, rather than having this overly complicated introduction. For instance, the sentence ‘The current study aims to replicate and extend the original findings by Celniker et al. (2023), specifically Study 6’ should be at p.6 and not at p.10, perhaps replacing the useless reference to Aristotle. Sorry for insisting! (Actually I am not sorry at all).
2. The text is still full of unnecessary commas, including the very sentence I copy-edited last time. Another example: ‘Naturally, these require some sort of quantification to tell, how moral a person is, based on mostly trivial actions’. Only the comma after ‘naturally’ is needed. The others separate verb and subject and are mistakes. Again, sorry for insisting (not sorry).
Please see below an updated version of my evaluation of this paper following the PCI RR guidelines.
Does the research question make sense in light of the theory or applications? Yes
Is it clearly defined? Yes.
Where the proposal includes hypotheses, are the hypotheses capable of answering the research question? Yes
Is the protocol sufficiently detailed to enable replication by an expert in the field, and to close off sources of undisclosed procedural or analytic flexibility? Yes
Is there an exact mapping between the theory, hypotheses, sampling plan (e.g. power analysis, where applicable), preregistered statistical tests, and possible interpretations given different outcomes? Yes
For proposals that test hypotheses, have the authors explained precisely which outcomes will confirm or disconfirm their predictions? Yes.
Is the sample size sufficient to provide informative results? Yes.
Where the proposal involves statistical hypothesis testing, does the sampling plan for each hypothesis propose a realistic and well justified estimate of the effect size? Yes
Have the authors avoided the common pitfall of relying on conventional null hypothesis significance testing to conclude evidence of absence from null results? Yes
Where the authors intend to interpret a negative result as evidence that an effect is absent, have authors proposed an inferential method that is capable of drawing such a conclusion, such as Bayesian hypothesis testing or frequentist equivalence testing? Yes
Have the authors minimised all discussion of post hoc exploratory analyses, apart from those that must be explained to justify specific design features? Maintaining this clear distinction at Stage 1 can prevent exploratory analyses at Stage 2 being inadvertently presented as pre-planned. Yes, it seems so.
Have the authors clearly distinguished work that has already been done (e.g. preliminary studies and data analyses) from work yet to be done? Yes
Have the authors prespecified positive controls, manipulation checks or other data quality checks? Yes, because they follow the original paper.
If not, have they justified why such tests are either infeasible or unnecessary? Is the design sufficiently well controlled in all other respects? NA
When proposing positive controls or other data quality checks that rely on inferential testing, have the authors included a statistical sampling plan that is sufficient in terms of statistical power or evidential strength? Yes.
Does the proposed research fall within established ethical norms for its field? Regardless of whether the study has received ethical approval, have the authors adequately considered any ethical risks of the research? Yes
Good job!
Best of luck with the remainder of this project.
Ignazio Ziano (University of Geneva) – Ignazio.ziano@unige.ch
DOI or URL of the report: https://osf.io/fnj73
Version of the report: 1
, posted 04 Apr 2024, validated 04 Apr 2024Dear authors,
First, I would like to thank you for this first draft. This is one of the best I read as a recommender from PCI-RR, in terms of completeness of information, addition of Qualtrics files, script and dummy data.
I received 3 reviews, and all reviewers signed them. I will therefore use their names when pointing to arguments.
All reviewers (and I) agreed on the following:
The introduction is too long and too short: too long on unrelated topics, and too short (and with a lack of precision) on the actual topic. Both Jared Celniker and Ignazio Ziano provided examples and suggestions.
There is room for improvement in the method, especially for detailing the sample used, the power analysis (see Jared Celniker comments here), and the tests (as Jared pointed out, I don’t understand the need for a two sided t-test, and as Ignazio pointed out, you need to improve correction for multiple testing). Please be very clear regarding the relationship between sample size, power, and effect size of interest. I add that, in complement to Cronbach’s Alpha, I would like to see a McDonald’s Omega, see here: https://journals.sagepub.com/doi/full/10.1177/2515245920951747.
Jared Celniker and Michael Inzlicht asked for details regarding age. I think that when looking at the R script, things are clearer. The confusion lies in the method section, as you want a sample with a balanced distribution, while you will use age as a continuous variable. Please State that explicitly to avoid confusion and explain in the introduction why the use of age as a continuous variable can help understand better the theory tested (See Michael’s review).
In additions to the typos found by reviewers, I would like to add that page 16 the “after” is truncated, page 17 is blank, and table 5 the footnote 2 is not displayed, at least on my version. In the introduction, please avoid the strong terms as “demonstrated” as researchers and findings indicate support for a theory, they don’t demonstrate anything.
Now regarding disagreement, both Jared Celniker and Ignazio Ziano used the PCI-RR guideline for reviewing, and they mostly don’t agree with each other’s. Based on the details of their reviews, we can understand why they don't and how authors can overcome these problems. I strongly suggest the authors to firstly and extensively answer Ignazio’s review, as he provided several suggestions for improvement, before completing with answers to Jared Celniker and Michael Inzlicht.
I am looking forward to reading your revision,
Best regards,
Adrien Fillon
, 30 Mar 2024