Close printable page
Recommendation

Are people who exert more effort in a task seen as more moral?

ORCID_LOGO based on reviews by Jared Celniker, Ignazio Ziano and Michael Inzlicht
A recommendation of:

Is it Worth the Hustle? A Multi-Country Replication of the Effort Moralization Effect and an Extension to Generational Differences in the Appreciation of Effort

Abstract

EN
AR
ES
FR
HI
JA
PT
RU
ZH-CN
Submission: posted 18 January 2024
Recommendation: posted 21 June 2024, validated 21 June 2024
Cite this recommendation as:
Fillon, A. (2024) Are people who exert more effort in a task seen as more moral?. Peer Community in Registered Reports, . https://rr.peercommunityin.org/articles/rec?id=673

Recommendation

This study seeks to understand cultural and age differences in the effort moralization effect, a phenomenon in which people who put more effort into a task are considered more moral, regardless of the quality or the morality associated with the task. This is shown in common phrases such as the “great resignation” or “quiet quitting”, which are mostly used against younger members of the population, in particular generation Z.
 
Tissot and Roth (2024) propose to conduct a replication of a study from Celniker et al. (2023) which found evidence for this effect, with new samples from Mexico and Germany to test potential cultural differences. They will also test the effect of age on the effort moralization effect. Therefore, the study will be a quantitative analysis.
 
The authors included an adequate power analysis, alternatives for non-supported hypotheses, and filtering to ensure a high quality of data collection. They already provided an R script and dummy data to ensure the quality of the analysis.
 
The Stage 1 manuscript was evaluated over three rounds of in-depth review. Based on ​detailed responses to reviewers’ and the recommender’s comments, the recommender judged that the manuscript met the Stage 1 criteria and therefore awarded in-principle acceptance.​​​
 
URL to the preregistered Stage 1 protocol: https://osf.io/tvgw2
 
Level of bias control achieved: Level 6. No part of the data or evidence that will be used to answer the research question yet exists and no part will be generated until after IPA.
 
List of eligible PCI RR-friendly journals:
 
 
References
 
1. Celniker, J. B., Gregory, A., Koo, H. J., Piff, P. K., Ditto, P. H., & Shariff, A. F. (2023). The moralization of effort. Journal of Experimental Psychology: General, 152, 60–79. https://doi.org/10.1037/xge0001259
 
2. Tissot, T. T. & Roth, L. H. O. (2024). Is it Worth the Hustle? A Multi-Country Replication of the Effort Moralization Effect and an Extension to Generational Differences in the Appreciation of Effort. In principle acceptance of Version 4 by Peer Community in Registered Reports. https://osf.io/tvgw2
Conflict of interest:
The recommender in charge of the evaluation of the article and the reviewers declared that they have no conflict of interest (as defined in the code of conduct of PCI) with the authors or with the content of the article.

Reviews

Evaluation round #3

DOI or URL of the report: https://osf.io/n7ga6

Version of the report: 3

Author's Reply, 20 Jun 2024

Download tracked changes file

Dear Dr. Fillon,

 

thank you very much for your fast reply and support in further strengthening our manuscript. We have worked on all mentioned passages and included your suggestions almost identically.

We are looking forward to your feedback and hope to have incorporated everything correctly.

While you suggested PsyArxiv, we stayed with osf for now. If you want us to migrate the project, please let us know to discuss this option.

 

Best regards and have a nice afternoon

 

Leopold Roth and Tassilo Tissot

Decision by ORCID_LOGO, posted 19 Jun 2024, validated 19 Jun 2024

Dear Authors. I went over the draft, and you will find below all of my comments to help you improve the text, before I can provide an IPA. Please treat my comments with critical thinking, and adjust them if needed.

Abstract

in countries not yet included in this research

Included in this study

we will examine whether lower effort moralization is observed as a function of age (including non-linear terms).

we will examine whether lower effort moralization depends on participant’s age.

Introduction

p.8

The ideological debate about the lack of qualified workforce and specifically amongst younger potential employees

of objective reasons

These reasons seems not to be “objective”. I would say structural or systemic.

An effect that persists even if the effort is not productive (Celniker et al., 2023)

This effect persists…

Regarding the above-described debates, we hypothesize that younger individuals show less effort moralization of ineffective labor – not judging higher, ineffective effort as a sign of higher morality.

The effort moralization effect can inform the debate around the lack of qualified and willing workforce. We hypothesize that younger individuals indicate less effort moralization of ineffective labor. Younger individuals do not judge ineffective effort as a sign of higher morality based solely on the effort.

This offers

Replicating the effort moralization effect offers

This is crucial to select romantic partners (Brown et al., 2022; Chan, 2023; Oda & Hayashi, 2020)

I would delete this part, keeping only the “Moral judgement is crucial in cooperation settings (Celniker …)

The estimation of future moral behavior is by nature not trivial, which is illustrated by the multitude of models and measures around moral foundations, moral identity, virtue, or similar ideas (Aquino & Reed, 2002; Haidt & Joseph, 2008; Ruch et al., 2010; Schlenker, 2008). Still, most individuals depend on approximations of character virtue through observation in daily life. While individuals rely on a variety of cues for this purpose, including facial and body expressions (Horberg et al., 2013), stereotypical appearance (Grizzard et al., 2018), or religious beliefs (Gervais, 2011), one of the main signals for inferring the morality of others remains behavioral observation (Mickelberg et al., 2022; Pizarro & Tannenbaum, 2012). Naturally, these require some sort of quantification to tell, how moral a person is, based on mostly trivial actions.

Suggestion: 

To attribute morality to others, most individuals depend on approximations of character virtue through observation in daily life. While individuals rely on a variety of cues for this purpose, including facial and body expressions (Horberg et al., 2013), stereotypical appearance (Grizzard et al., 2018), or religious beliefs (Gervais, 2011), one of the main signals for inferring the morality of others remains behavioral observation (Mickelberg et al., 2022; Pizarro & Tannenbaum, 2012). Observations lead to create inferences about the morality of a person based on trivial actions.

One phenomenon, that has raised scientific psychology’s interest in recent years is the observation that people appear to use effort invested in given tasks, as information on the morality of agents, summarized as effort moralization effect.

This does not flow well with what is above so I suggest deleting it.

While the core idea likely follows

The core idea of the moralization of effort is a heuristic…

Also, put a point between "performance" and "the focal interest"

p. 12

This was still observed when the behaviors were not successful

The effect of effort on moral judgement was still observed when…

I also wonder if the term “successful” is the good one. Maybe the behavior was not predicted? The behavior was not expected?

p.13

As described above,

Not useful, delete

p. 14

economically redundant labor, such as universal basic income

You don’t need the coma

It is plausible to assume that recent movements such as the ‘great resignation’, ‘quiet quitting’, etc. represent responses to such resistance, fueled by a generational shift in work values and changing perceptions of (necessary) effort among younger generations.

You don’t need the coma before fueled 

Such differences may provide

You don’t need the may (it is already in the sentence before)

p.15 

You need to put a dot in the big sentence. I propose to put it at the end of mexico as follow:

research (Germany and Mexico1)

namely Germany and Mexico. Testing the effect in different countries will inform the generalizability of the effect.

Use “found” or “indicate” instead of demonstrated.

p.16 

was computed, using the pwrss

You don’t need the coma

Because we were conducting the study in Germany and Mexico,

You can begin by “We translated the vignettes in German and Spanish

by two independent translators, one of whom was one of the two authors of this paper

By two independent translators, including one author of this paper.

p.21

To test whether the effect of effort moralization was replicated

To test the signal consistency of the effort moralization effect between the original study and the replication

 

Please ensure that if you delete citations, you do as well in the reference list. You can also use recitework to help you. Upload a clean version to PsyArxiv and come back to me once it is done.

 

Best regards,

Adrien Fillon

Evaluation round #2

DOI or URL of the report: https://osf.io/63xws

Version of the report: 2

Author's Reply, 18 Jun 2024

Decision by ORCID_LOGO, posted 05 Jun 2024, validated 05 Jun 2024

Dear Authors,

I now received reviews from two reviewers, who were positive regarding your manuscript, and satisfied with the changes you made.

They both asked for improvement in the writing of the introduction, and in the flow of the article. I think that they are easily addressable. I look forward to receiving a revision. 

Best regards,

Adrien Fillon

Reviewed by , 07 May 2024

Geneva, May 7th 2024

Review of PCI RR – Second Round of Stage 1

 Is it Worth the Hustle? A Multi-Country Replication of the Moralization of Effort Effect and an Extension to the Increasing Aversion to Bullshit Jobs - second round

 

I am happy to see a revision of this paper. It is much improved. Sorry for missing that the data was simulated in round 1, my fault.

I want to point to you a Twitter thread regarding older people saying that now, young people do not want to work anymore, by Paul Fairie: link

If not useful, certainly funny.

I still have some observations about the writing:

1.      I still think you should be more to the point in the introduction and tell your reader immediately that you want to replicate the EME (which is the primary thing you do) and test whether age moderates it, rather than having this overly complicated introduction. For instance, the sentence ‘The current study aims to replicate and extend the original findings by Celniker et al. (2023), specifically Study 6’ should be at p.6 and not at p.10, perhaps replacing the useless reference to Aristotle. Sorry for insisting! (Actually I am not sorry at all).

2.      The text is still full of unnecessary commas, including the very sentence I copy-edited last time. Another example: ‘Naturally, these require some sort of quantification to tell, how moral a person is, based on mostly trivial actions’. Only the comma after ‘naturally’ is needed. The others separate verb and subject and are mistakes. Again, sorry for insisting (not sorry).

Please see below an updated version of my evaluation of this paper following the PCI RR guidelines.

Does the research question make sense in light of the theory or applications? Yes
Is it clearly defined?  Yes.
Where the proposal includes hypotheses, are the hypotheses capable of answering the research question? Yes
Is the protocol sufficiently detailed to enable replication by an expert in the field, and to close off sources of undisclosed procedural or analytic flexibility? Yes
Is there an exact mapping between the theory, hypotheses, sampling plan (e.g. power analysis, where applicable), preregistered statistical tests, and possible interpretations given different outcomes? Yes 
For proposals that test hypotheses, have the authors explained precisely which outcomes will confirm or disconfirm their predictions? Yes.
Is the sample size sufficient to provide informative results? Yes.
Where the proposal involves statistical hypothesis testing, does the sampling plan for each hypothesis propose a realistic and well justified estimate of the effect size? Yes
Have the authors avoided the common pitfall of relying on conventional null hypothesis significance testing to conclude evidence of absence from null results? Yes
Where the authors intend to interpret a negative result as evidence that an effect is absent, have authors proposed an inferential method that is capable of drawing such a conclusion, such as Bayesian hypothesis testing or frequentist equivalence testing? Yes
Have the authors minimised all discussion of post hoc exploratory analyses, apart from those that must be explained to justify specific design features? Maintaining this clear distinction at Stage 1 can prevent exploratory analyses at Stage 2 being inadvertently presented as pre-planned. Yes, it seems so.
Have the authors clearly distinguished work that has already been done (e.g. preliminary studies and data analyses) from work yet to be done? Yes 
Have the authors prespecified positive controls, manipulation checks or other data quality checks? Yes, because they follow the original paper. 
If not, have they justified why such tests are either infeasible or unnecessary? Is the design sufficiently well controlled in all other respects? NA
When proposing positive controls or other data quality checks that rely on inferential testing, have the authors included a statistical sampling plan that is sufficient in terms of statistical power or evidential strength? Yes.
Does the proposed research fall within established ethical norms for its field? Regardless of whether the study has received ethical approval, have the authors adequately considered any ethical risks of the research? Yes


Good job!

Best of luck with the remainder of this project.

Ignazio Ziano (University of Geneva) – Ignazio.ziano@unige.ch

 

Reviewed by , 03 Jun 2024

Evaluation round #1

DOI or URL of the report: https://osf.io/fnj73

Version of the report: 1

Author's Reply, 03 May 2024

Decision by ORCID_LOGO, posted 04 Apr 2024, validated 04 Apr 2024

Dear authors,

First, I would like to thank you for this first draft. This is one of the best I read as a recommender from PCI-RR, in terms of completeness of information, addition of Qualtrics files, script and dummy data.

I received 3 reviews, and all reviewers signed them. I will therefore use their names when pointing to arguments.

All reviewers (and I) agreed on the following:

The introduction is too long and too short: too long on unrelated topics, and too short (and with a lack of precision) on the actual topic. Both Jared Celniker and Ignazio Ziano provided examples and suggestions.

There is room for improvement in the method, especially for detailing the sample used, the power analysis (see Jared Celniker comments here), and the tests (as Jared pointed out, I don’t understand the need for a two sided t-test, and as Ignazio pointed out, you need to improve correction for multiple testing). Please be very clear regarding the relationship between sample size, power, and effect size of interest. I add that, in complement to Cronbach’s Alpha, I would like to see a McDonald’s Omega, see here: https://journals.sagepub.com/doi/full/10.1177/2515245920951747.

Jared Celniker and Michael Inzlicht asked for details regarding age. I think that when looking at the R script, things are clearer. The confusion lies in the method section, as you want a sample with a balanced distribution, while you will use age as a continuous variable. Please State that explicitly to avoid confusion and explain in the introduction why the use of age as a continuous variable can help understand better the theory tested (See Michael’s review).

In additions to the typos found by reviewers, I would like to add that page 16 the “after” is truncated, page 17 is blank, and table 5 the footnote 2 is not displayed, at least on my version. In the introduction, please avoid the strong terms as “demonstrated” as researchers and findings indicate support for a theory, they don’t demonstrate anything.

Now regarding disagreement, both Jared Celniker and Ignazio Ziano used the PCI-RR guideline for reviewing, and they mostly don’t agree with each other’s. Based on the details of their reviews, we can understand why they don't and how authors can overcome these problems. I strongly suggest the authors to firstly and extensively answer Ignazio’s review, as he provided several suggestions for improvement, before completing with answers to Jared Celniker and Michael Inzlicht.

I am looking forward to reading your revision,

Best regards,

Adrien Fillon

 

Reviewed by , 04 Apr 2024

Reviewed by , 27 Mar 2024

Reviewed by ORCID_LOGO, 30 Mar 2024

1A. Scientific validity of the research question
 
I believe that the authors’ questions are mostly valid. They are mostly looking to replicate the “moralization of effort” effect. Secondary to this is to examine the replicability/generalizability of this effect on countries different from the original ones tested (e.g., USA, France, South Korea). In addition, they examine if age moderates the moralization of effort effect. While I see the value in this replication, the authors do not justify why they chose the countries of Germany, Mexico, Netherlands, and South Africa. I suppose they could make a WEIRD argument, but my bet is that these data will come from wealthy students even in the developing nations, opening questions about how much of a generalization this is. I am also unclear about the validity of the age prediction. I mean, sure, go ahead and examine this as it would be an interesting and free thing to examine. However, I do not believe it is a direct test of whether younger people are less willing to exert effort and work than older people. For that, the authors could examine actual effort willingness or even simply ask them about their values. The analysis here only gets at this indirectly, asking if young people moralize effort to the same degree as older people. Still interesting but not an examination of work ethic, per se. 
 
I have another comment about the introduction. To put in plainly, I found it strange. There is a rich tradition in social psychology of person perception, whereby the field has tried to understand how we perceive and appraise other people. Instead of placing this study in this context, the authors talk about physiognomy, race science, and face perception, that has nearly nothing to do with these studies. My advice to the authors is to use the same framing as the Celniker paper, but to clearly frame this as a replication of that paper and then say why this replication is important, critical, etc.
 
1B. Logic and rationale of proposed methods 
 
The hypotheses are logical, rational, and sound. I believe the hypotheses are worthy of investigation (i.e., examining the replicability of the moralization of effort effect).
 
1C. Soundness of the methods.
 
Methods are sound. However, I have two related questions. First, the authors list 8 dependent variables, yet their Bonferroni correction indicates 5 family of tests. Can the authors explain this discrepancy. Second, and related, I imagine that there are different hypotheses for some of these dependent variables. What are they? Do the authors predict, for example, that “quality of work” will change via effort framing? I assume not, as the vignette makes clear that work quality is the same. So, some clarification needed here.
 
1D. Outcome neutral conditions.
 
The authors include some manipulation checks, but unfortunately, they don’t always do a good job of labeling them as such (see my second point above). The authors should clarify these manipulation checks and why they are important.
 
I don’t know where to say this, so I will say it here. There are spots where the authors were not sufficiently careful, leading to typos and errors. For example, on p. 7 they write “integer” but perhaps they mean integrity; p. 17 is a blank page, etc. There are few other errors like this. The authors should read their manuscript carefully to correct these errors.
 
I sign all my reviews,
 
Michael Inzlicht