COVID-19 has been suspected to have long-lasting effects on cognitive function. The SARS-CoV-2 virus may enter the central nervous system (Frontera et al., 2020; Miners, Kehoe, & Love, 2020), explaining the observed detrimental effects of COVID-19 on verbal planning and reasoning (Hampshire et al., 2021; Wild et al., 2021), executive function (Hadad et al., 2022), and long-term memory (Guo et al., 2022). In particular, Guo et al. (2022) used verbal item recognition and non-verbal associative memory tasks. Weinerova et al. (2024), in the current study, propose to conduct a replication of Guo et al. (2022), but specifically, to disentangle the effect of COVID-19 infection status on both memory type (item vs. associative) and stimulus modality (verbal vs. non-verbal). Furthermore, Weinerova et al. (2024) propose to analyze cognitive function based on vaccination status before infection to provide a critical test of the potential protective effects of vaccination on cognitive function.

Data collection has been completed with 325 participants after exclusion criteria were applied (COVID group N = 232, No COVID group N = 93). Simulations assuming an effect size observed in Guo et al. (2022), a Bayesian t-test comparing the groups, and a Bayes Factor of 6 indicated that N = 320 is sufficient to detect an effect on 79% of simulations. The main analyses will be conducted using a Bayesian ANCOVA that allows for the inclusion of control variables such as age, sex, country, and education level. Both accuracy and reaction times from the item and associative recognition tasks will be analyzed as the dependent variables. In one analysis, vaccination status will be included as a between-subjects factor, to understand whether vaccination status at the time of infection influences subsequent cognitive function. 

It is important to note that participants were recruited through long-COVID Facebook groups and clinics. Therefore, the results must be interpreted carefully to avoid generalizing to all COVID-19 infections. The data are part of a larger longitudinal study, and the current pre-registration applies only to the baseline timepoint for a cross-sectional analysis. The remaining longitudinal data collection is ongoing and is not part of the current pre-registration.  

The study plan was refined after one round of review, with input from three external reviewers who all agreed that the proposed study was well-designed and scientifically valid. The recommender then reviewed the revised manuscript and judged that the study met the Stage 1 criteria for in-principle acceptance (IPA).

 

URL to the preregistered Stage 1 protocol: https://osf.io/tjs5u (under temporary private embargo)
 
Level of bias control achieved: Level 3. At least some data/evidence that will be used to the answer the research question has been previously accessed by the authors (e.g. downloaded or otherwise received), but the authors certify that they have not yet observed ANY part of the data/evidence.
 
List of eligible PCI RR-friendly journals:

 

 

 

1. Frontera, J., Mainali, S., Fink, E.L. et al. Global Consortium Study of Neurological Dysfunction in COVID-19 (GCS-NeuroCOVID): Study Design and Rationale. Neurocrit Care 33, 25–34 (2020). https://doi.org/10.1007/s12028-020-00995-3

2. Guo, P., Benito Ballesteros, A., Yeung, S. P., Liu, R., Saha, A., Curtis, L., Kaser, M., Haggard, M. P. & Cheke, L. G. (2022). COVCOG 2: Cognitive and Memory Deficits in Long COVID: A Second Publication From the COVID and Cognition Study. Frontiers in Aging Neuroscience. https://doi.org/10.3389/fnagi.2022.804937  

3. Hadad, R., Khoury, J., Stanger, C., Fisher, T., Schneer, S., Ben-Hayun, R., Possin, K., Valcour, V., Aharon-Peretz, J. & Adir, Y. (2022). Cognitive dysfunction following COVID-19 infection. Journal of NeuroVirology, 28(3), 430–437. https://doi.org/10.1007/s13365-022-01079-y  

4. Hampshire, A., Trender, W., Chamberlain, S. R., Jolly, A. E., Grant, J. E., Patrick, F., Mazibuko, N., Williams, S. C., Barnby, J. M., Hellyer, P. & Mehta, M. A. (2021). Cognitive deficits in people who have recovered from COVID-19. EClinicalMedicine, 39, 101044. https://doi.org/10.1016/j.eclinm.2021.101044

5. Miners, S., Kehoe, P. G., & Love, S. (2020). Cognitive impact of COVID-19: looking beyond the short term. Alzheimer's research & therapy, 12, 1-16. https://doi.org/10.1186/s13195-020-00744-w 

 

6. Weinerova, J., Yeung, S., Guo, P., Yau, A., Horne, C., Ghinn, M., Curtis, L., Adlard, F., Bhagat, V., Zhang, S., Kaser, M., Bozic, M., Schluppeck, D., Reid, A., Tibon, R. & Cheke, L. G. (2024). Changes in memory function in adults following SARS-CoV-2 infection: findings from the Covid and Cognition online study. In principle acceptance of Version 2 by Peer Community in Registered Reports. https://osf.io/tjs5u

7. Wild, C. J., Norton, L., Menon, D. K., Ripsman, D. A., Swartz, R. H. & Owen, A. M. (2022). Disentangling the cognitive, physical, and mental health sequelae of COVID-19. Cell Reports Medicine, 3, 100750. https://doi.org/10.1016/j.xcrm.2022.100750 

Comparative research is how evidence is generated to support or refute broad hypotheses (e.g., Pagel 1999). However, the foundations of such research must be solid if one is to arrive at the correct conclusions. Determining the external validity (the generalizability across situations/individuals/populations) of the building blocks of comparative data sets allows one to place appropriate caveats around the robustness of their conclusions (Steckler & McLeroy 2008).

In the current study, Alley and colleagues (2023) tackled the external validity of comparative research that relies on subjects who are either university students or participating in experiments via an online platform. They determined whether data from these two types of subjects have measurement equivalence - whether the same trait is measured in the same way across groups.

Although they use data from studies involved in the Many Labs replication project to evaluate this question, their results are of crucial importance to other comparative researchers whose data are generated from these two sources (students and online crowdsourcing). The authors show that these two types of subjects do not often have measurement equivalence, which is a warning to others to evaluate their experimental design to improve validity. They provide useful recommendations for researchers on how to to implement equivalence testing in their studies, and they facilitate the process by providing well annotated code that is openly available for others to use.

After one round of review and revision, the recommender judged that the manuscript met the Stage 2 criteria and awarded a positive recommendation.

 

Level of bias control achieved: Level 2. At least some data/evidence that was used to answer the research question had been accessed and partially observed by the authors prior to Stage 1 IPA, but the authors certify that they had not yet observed the key variables within the data that were used to answer the research question AND they took additional steps to maximise bias control and rigour.

 

 

• Advances in Methods and Practices in Psychological Science
• Collabra: Psychology
• F1000Research
• Peer Community Journal
• PeerJ
• Royal Society Open Science
• Studia Psychologica
• Swiss Psychology Open

 

 

1. Pagel, M. (1999). Inferring the historical patterns of biological evolution. Nature, 401, 877-884. https://doi.org/10.1038/44766

 

2. Steckler, A. & McLeroy, K. R. (2008). The importance of external validity. American Journal of Public Health 98, 9-10. https://doi.org/10.2105/AJPH.2007.126847

 

3. Alley L. J., Axt, J., & Flake J. K. (2023). Convenience Samples and Measurement Equivalence in Replication Research [Stage 2 Registered Report] Acceptance of Version 2 by Peer Community in Registered Reports​. https://osf.io/s5t3v

Traditional Likert-type items are commonly used but can elicit response bias. An alternative approach, the forced-choice question, required respondents to rank order all items. Forced-choice questions boast some advantages but required advanced item response theory analysis to generate scores which are comparable across individuals and to evaluate the properties of those scales. However, there has been limited discussion of how to test differential item functioning (DIF) in these scales. In a previous study, Lee et al. (2021) proposed a method for testing DIF.

 

Here, Plantz et al. (2024) explore the implications of incorrect specification of anchors in DIF detection for forced choice items. The study proposes to use a Monte Carlo simulation which manipulates sample size, equality of sample size across groups, effect size, percentage of differentially functioning items, analysis approach, anchor set size, and percent of DIF blocks in the anchor set. This study aims to answer research questions about the type I error and power of DIF detection strategies under a variety of circumstances, both evaluating whether the results from Lee et al. (2021) generalize to misspecified models and expanding to evaluate new research questions. Results of this study will provide practical implications for DIF testing with forced-choice questions. An important limitation of the study is that it does not explore non-uniform DIF, only uniform DIF. Additionally, as with all simulation studies not all results can only apply to conditions which are simulated and so rely on the realistic selection of simulation conditions. The authors have selected conditions to match reality in circumstances where data is available, but relied on previous simulations in cases when data is not available. 

 

This Stage 1 manuscript was evaluated over two rounds of review by two reviewers with expertise in psychometrics. An additional round of review was completed by the recommender only. Based on the merits of the original submission and responsiveness of the authors to requests from the reviewers, the recommender judged that the manuscript met the Stage 1 criteria and therefore awarded in-principle acceptance (IPA).​

 

URL to the preregistered Stage 1 protocol: https://osf.io/p8awx
 
Level of bias control achieved: Level 6. No part of the data or evidence that will be used to answer the research question yet exists and no part will be generated until after IPA.
 
List of eligible PCI RR-friendly journals:
 

 

1. Lee, P., Joo, S.-H. & Stark, S. (2021). Detecting DIF in multidimensional forced choice measures using the Thurstonian Item Response Theory Model. Organizational Research Methods, 24, 739–771. https://doi.org/10.1177/1094428120959822

 

2. Plantz, J. W.,  Brown, A., Wright, K. & Flake, J. K. (2024). Detecting DIF in Forced-Choice Assessments: A Simulation Study Examining the Effect of Model Misspecification. In principle acceptance of Version 3 by Peer Community in Registered Reports. https://osf.io/p8awx

We may not believe what our memory tells us: Memory may deliver a compelling recollection we believe did not happen (we know we were not there at the time); and we may know an event happened that we fail to remember (even for when we were not drunk!). That is, there can be distrust in remembering and distrust in forgetting. Previous work by the authors has looked at this through a signal detection lens, reporting in separate studies that people who have distrust in remembering have either a high or low criterion for saying "old" (Zhang et al, 2023, 2024a). A plausible explanation for these contrasting results is that the criterion can either be the means by which false memories are generated enabling the distrust (low criterion); or rather, in conditions where accuracy is at stake, the means for compensating for the distrust (high criterion).

 

In the current study by Zhang et al (2024b), participants will be incentivised to be as accurate as possible, and in a memory test given feedback about commission errors or, in another group, ommission errors. As a manipulation check, the authors will test that the feedback increases distrust in remembering or distrust in forgetting, respectively, compared to a no feedback control group. Crucially, the authors hypothesize that people will adjust the criterion to say "old" in a compensatory way in each group. The study uses inference by intervals to provide a fairly severe test of this hypothesis.

 

The Stage 1 manuscript was evaluated over multiple rounds of in-depth review. Based on detailed responses to the reviewers' and recommender's comments, the recommender judged that the manuscript met the Stage 1 criteria and awarded in-principle acceptance (IPA).

 

URL to the preregistered Stage 1 protocol: https://osf.io/x69qt
 
Level of bias control achieved: Level 6. No part of the data or evidence that will be used to answer the research question yet exists and no part will be generated until after IPA. 
 
List of eligible PCI RR-friendly journals:

 

 

 

1. Zhang, Y., Qi, F., Otgaar, H., Nash, R. A., & Jelicic, M. (2023). A Tale of Two Distrusts: Memory Distrust towards Commission and Omission Errors in the Chinese Context. Journal of Applied Research in Memory and Cognition. https://doi.org/10.1037/mac0000134

 

2. Zhang, Y., Otgaar, H., Nash, R. A., & Rosar, L. (2024). Time and memory distrust shape the dynamics of recollection and belief-in-occurrence. Memory, 32, 484–501. https://doi.org/10.1080/09658211.2024.2336166

 

3. Zhang, Y., Otgaar, H., Nash, R. A., & Li, C. (2024b). The effects of memory distrust toward commission and omission on recollection-belief correspondence and memory errors. In principle acceptance of Version 5 by Peer Community in Registered Reports. https://osf.io/x69qt

 

One of the hallmarks of cognitive control is the ability to flexibly update attention and action when goals change. The prefrontal cortex has long been identified as important for such updating, but much remains to be understood about the anatomical and temporal mechanisms that support cognitive flexibility within prefrontal networks. In the current study, Jackson et al. (2024) build upon insights from recent transcranial magnetic stimulation (TMS) and neuroimaging studies to investigate the critical role of prefrontal cortex for updating goals and selecting behaviourally-relevant stimuli.

 

To measure updating, the authors deploy an attentional switching paradigm in which participants selectively attend to one feature of a novel object (colour or form) while ignoring the other feature. On each trial, a symbol (called a rule cue) indicates whether to attend to the colour (green or blue) or to the form (X or non-X) of the upcoming object. By mapping each stimulus response to a separate button press (two buttons for the two colours; two buttons for the two features), the authors can then categorise different types of behavioural errors – focusing especially on attending incorrectly to the task-irrelevant feature (rule error) vs. applying the correct rule but failing to correctly identify the task-relevant feature (stimulus error). If disruption of a specific cortical region causes a selective increase in one type of error, then this would indicate that the stimulated region is important for either rule processing or stimulus processing.

 

The proposal includes a number of key features that add depth and rigor to the investigation. First, to probe the anatomical specificity of cognitive control, the authors will contrast the effect of TMS delivered to different prefrontal regions that reside within different networks and may have divergent roles in cognitive control: the dorsolateral prefrontal cortex (dlPFC, part of the multiple-demand network) and dorsomedial prefrontal cortex (dmPFC, part of the default mode network). Moreover, unlike many previous TMS studies, the authors will use electric field modelling to normalise cortical stimulation strength between regions, enabling a more controlled anatomical comparison. Second, since the task involves responding to a rule cue and then selectively attending to a task-relevant feature, it is likely that a particular brain region could be selectively critical at a specific time – for instance, if dlPFC were important for rule processing then it should only be necessary shortly after (or around) presentation of the rule cue. To capture the temporal specificity of cortical involvement, the authors will apply a short burst of TMS at different times, beginning either +150ms after the cue or +700ms during stimulus processing. In a preliminary study, the authors used magnetoencephalography (MEG) in combination with the same behavioural task and multivariate pattern analysis (MVPA) to identify these epochs for TMS. Finally, the experiment includes a range of additional control conditions and quality checks to rule out alternative explanations of potential findings, such as TMS impairing perception of the rule cue rather than implementation of the rule, and the effect of peripheral TMS artefacts. Overall, the study promises to reveal a range of intriguing new insights into the timecourse and anatomical specificity of cognitive updating, with implications for theories of prefrontal cortical function.

 

The Stage 1 manuscript was evaluated over one round of in-depth review. Based on detailed responses to the reviewers' and recommender’s comments, the recommender judged that the manuscript met the Stage 1 criteria and therefore awarded in-principle acceptance (IPA).

 

URL to the preregistered Stage 1 protocol: https://osf.io/7pxjz (under temporary private embargo)

 

Level of bias control achieved: Level 6. No part of the data or evidence that will be used to answer the research question yet exists and no part will be generated until after IPA.

 
List of eligible PCI RR-friendly journals:

 

 

 

Jackon, J. B, Runhao, L., & Woolgar, A. (2024). Causal dynamics of task-relevant rule and stimulus processing in prefrontal cortex. In principle acceptance of Version 2 by Peer Community in Registered Reports. https://osf.io/7pxjz