Since the replication crisis and several scandals that uncovered cases of misinterpretation of statistical testing and results [45, 59, 60], great efforts have been made to increase the number of published replications and improve the methodologies that are used for replicating studies [19]. Still, some researchers consider that replications are only useful for the purpose of confirming the replicability of research findings under specific circumstances [11], rather than to benefit the advancement of knowledge on a broader scale (generalizability to different contexts, contemporaneity of the research, and moderator effects) to enrich science [62].

Making use of the opportunities associated with the Internet revolution that were emphasized by the early pioneers of remote research [3, 32, 37, 38], we have witnessed the development of several “Big Team” efforts [7] and uses of crowdsourcing (the Hagen Cumulative Science Project [15] and the Psychological Science Accelerator [31]) to conduct replication studies in different laboratories around the world. These initiatives are beneficial to science, as they test and increase the generalizability of the results to different populations and contexts. Replication is crucial for distinguishing between true effects and chance findings in scientific research, particularly in fields with low base rates of real effects [57]. This is where remote replications offer a distinct advantage: they permit larger, more diverse participant pools and greater geographical and cultural variation, which help in assessing the generalizability of results. Remote experiments, for instance, permit easier replication and testing across different contexts, contributing to the evaluation of the robustness of findings. In this way, remote studies and Internet-based research play a significant role in improving the reliability of scientific knowledge.

Switching to Internet-based experiments that were originally conducted in the laboratory can be taken a step further. Evidence from various fields, for example, research on music perception and cognition [16], has shown that concerns about switching from laboratory to Internet-based experiments are often unwarranted, as results are often comparable. However, it remains fundamental to adopt rigorous methodologies in Internet-based research to ensure the reliability and comparability of results. Few concerns have mostly been focused on the reliability of data collection, and the lack of complete control of the experimental conditions [17]. For instance, issues such as repeated use of the same MTurk samples, internal validity concerns due to conditional dropout, and the potential for increased error variance have been highlighted [5, 34, 37, 38]. These challenges can lead to concerns about the quality of data obtained online, but it is important to note that Internet-based research has a long-standing tradition of rigorous methodologies [2, 40, 41, 43], predating the massive use of MTurk samples. In Internet-based research, several techniques have been implemented to enhance internal validity by addressing dropout issues. For instance, the multiple-site entry technique [39] was used to monitor sample effects and differing dropout patterns. Implementing seriousness [1] and quality control [12] checks improve data quality. By employing these methods, researchers can better ensure data integrity and enhance the reliability of findings in remote studies. Moreover, psychological research has made significant strides in improving techniques for conducting remote studies, particularly in the aftermath of COVID-19 pandemic, which accelerated the development of best practices for ensuring data reliability in online environments [46].

The increase in the number of Internet-based studies has prompted the development of rigorous research methodologies [40, 41] that provide Internet-based replication projects a great opportunity. The advantages of replicating experiments in Internet-based environments are several. From a practical perspective, subject pool diversity, lower costs, and lower hurdles to run studies make the theory-experiment cycle very smooth.

From an educational perspective, introduction to Open Science practices training and replication studies have become a part of basic research training, which would benefit from complete and accessible materials [24, 25, 61]. Moreover, meta-analyses are essential for advancing research and serving educational purposes. They help to determine whether an effect exists and reduce uncertainty, but their execution is often hindered by difficulties in accessing data from multiple studies. While the growing interest in Open Science is gradually easing this process, digitalized data from remote studies also play a crucial role in facilitating broader access to previous studies [22].

From a strictly methodological perspective, criticisms of replication studies have centered on concerns regarding internal validity, as noted by Feest [11], who pointed to overlooked confounding variables and the potential misidentification of relevant causal features of stimuli. Remote experiments offer the opportunity to easily make experimental materials available and thus facilitate simplified reproducibility and mitigate threats to internal validity [1, 12, 39]. It is important to note that repeated investigations of the same phenomenon inherently result in heterogeneity of effect sizes, which can be attributed to random variation across studies [23]. However, this variability should be valued, as it contributes to the generalizability of findings [37, 38].

Machery [29] emphasized the importance of sample heterogeneity in replication studies and ensuring that sample characteristics are clearly reported. While the criticism of replication studies’ internal validity [11] can be debated, Internet-based research offers advantages that address these concerns. The transparency afforded by making materials fully available online allows researchers to detect confounding factors that may have been missed in the original studies and provide additional information about the potential sources of random variation across studies. Furthermore, Internet-based studies permit recruitment of more heterogeneous samples, addressing Machery’s concerns [29] of sample diversity. Online platforms also facilitate techniques, such as Reips’ multiple site entry technique [39], which supports diverse data collection and sample characteristic reports.

To summarize, Internet-based (remote) replication studies can be easily implemented with access to large, heterogeneous samples and promote transparency through materials that are readily available for replication efforts. Additionally, they offer valuable learning opportunities for students and researchers, fostering broader engagement in replication efforts.

Replication projects about cognitive effects are fundamental. They help in developing the understanding of theories, assessing the methodological characteristics of previous studies, and identifying the conditions that moderate or limit the generalizability of specific findings. The cognitive effect “mental accounting” is defined as the human tendency to sort resources into mental categories, each characterized by different subjective values denoted as x units that impact decision-making processes [56]. When applied to money, mental accounting refers to the tendency to categorize financial resources differently based on the mental accounts they are assigned to, and the subjective value associated with each account.

In this article, we integrate two goals: (1) to provide evidence emphasizing the importance of systematic replication studies in advancing science, and (2) to illustrate how Internet-based methodologies support replication studies through methodological rigor and the facilitation of diverse participant recruitment, thereby broadening the generalizability of research findings.

We conducted three remote replication experiments on the mental accounting effect [21], adopting them as empirical examples to evaluate the strengths and weaknesses of exact versus conceptual replication approaches online to increase the likelihood of replicating cognitive effects.

Experiment 1 was conducted to evaluate whether the mental accounting effect can be replicated by using the original ticket and loss price (10$). To pursue the goal of adapting the experiment to contemporary times, we explored the role of purchase medium impact on individual decisions.

Experiment 2 was conducted to evaluate whether the mental accounting effect can be replicated with an effect size similar to the original study, specifically adapting the monetary values of a theater ticket and associated losses to contemporary standards (40€).6 Additionally, we varied the experiment by manipulating the mental account variable within-subjects, to understand whether the reduced variability associated with individual differences could play a role in the effect size of the mental accounting effect.

Experiment 3 was conducted to check if the results of Experiment 2 were not artifacts of a within-subjects design (e.g., participants contrasting the conditions artificially – demand effect) but rather reflected the importance of adapting the experiment to contemporary times.

The current study tested whether adapting the experimental context to the current historical context (i.e., monetary value, modern purchase medium) increased the likelihood of replicating a cognitive effect with a similar effect size to the original study. We described the magnitude of the effect observed in the main analyses, which we assessed by comparing the differences in key outcomes between conditions in our study and those reported in the original study. To provide empirical evidence, we conducted three remote experiments testing the mental accounting effect based on Kahneman and Tversky’s theater ticket paradigm [21].

Replicating the experiments using Internet-based methodologies was a step forward compared with the original experiment conducted in a laboratory setting. We observed that conducting the experiments remotely was advantageous because it allowed us to reach a large and heterogenous sample [39], and create easily reproducible materials [38], while adapting the replication paradigms to contemporary contexts and methodologies. These adaptations help ensure that replication studies remain relevant and robust, addressing the evolving demands of scientific inquiry [35]. Due to the ease of sharing links to remote experiments, using various recruitment platforms or services that allow pre-screening parameters (e.g., Prolific) can significantly enhance the heterogeneity of samples compared to traditional laboratory settings. The samples recruited in the three experiments had similar median ages—25, 20, and 25 for Experiments 1, 2, and 3, respectively, but were characterized by relatively broad age ranges, which contributes to the generalizability of the findings. In this study, we intentionally utilized multiple recruitment platforms to increase sample diversity and be able to invesigate robustness across samples via this multiple site entry technique [39]. Notably, in Experiments 2 and 3, where we collected additional demographic information such as occupation status and country of residence, we demonstrated the feasibility of reaching participants from varied backgrounds. Unlike laboratory-based experiments, which often rely on participants affiliated with the institution or require time-consuming coordination with external labs, conducting these experiments remotely allowed us to recruit participants from four platforms in Experiment 2, five platforms in Experiment 3, and eight platforms in Experiment 1, which is considered beneficial to increase the sample’s diversity.

In response to Feest’s concerns about internal validity [11]—i.e., the risk of overlooked confounding variables and misidentification of relevant causal features—we found that combining a process-oriented approach with remote replications offered several advantages. The process-oriented approach allowed us to identify the monetary value of the ticket price and monetary loss as an influential factor in the tested effect. Additionally, conducting remote replications facilitated the online sharing of easily reproducible experimental materials, offering future researchers the opportunity to further investigate the effects or explore potential additional factors more easily. Thus, remote replications support greater transparency and broader scrutiny of experimental materials.

Literature has shown that direct comparisons between data collected in laboratory and Internet-based settings yield similar results for replicating and testing cognitive effects [51], even for time-sensitive tasks, if best practices are being followed [13]. Additionally, researchers observed that results and variances from remote experiments are comparable to those obtained from laboratory-based studies [49]. The abovementioned advantages of remote experiments, coupled with the high data quality provided by rigorous online methodologies, support the argument that exact replications do not need to be confined to laboratory settings when replicating original studies conducted in such environments. Indeed, exact replication should maintain the main components (same experimental designs, main statistical analyses, and key stimuli) of the original experiments [18], rather than striving for a (practically unattainable) full replication of the original experiment’s conditions. Our study contributes to the body of knowledge by showing that the mental accounting effect is replicable using Internet-based research methodologies.

Our step-by-step approach in developing replication studies of relatively older theories led us to focus on minor adjustments when conducting the conceptual replications. This prevented premature conclusions about the inability to replicate the cognitive effect with a similar effect size to the original results in Experiment 1. Experiment 2, utilizing a within-subjects design and adjusting the ticket prices to contemporary standards, successfully reproduced the original results with a similar effect size to the original study. Similarly, in Experiment 3, adapting the price of the theater ticket to contemporary times but adopting a between-subjects design, resulted in a general replication of the original findings. Results from Experiment 3 suggest that adapting the purchase medium to the current times may play a role; however, these findings should be interpreted with caution, as they were observed only in this experiment.

The results of the three experiments align with prior research, indicating that the cognitive processing of mismatch among real-life prices and prices in the experiment might alter participants’ decisions [33]. Consistent with Hypothesis 3, the lack of replicated results with a similar effect size to the original study in Experiment 1, and the outcomes of Experiments 2 and 3 suggest that for replication studies of relatively older experiments involving money, it is recommended to consider factors such as inflation and the role of cognitive processing mismatch between the real world prices and prices used in the experiments [33].

The mental accounting of money effect was replicated (Hypothesis 1), in line with the original study from Kahneman and Tversky [21]. However, further studies are needed to clarify whether and how specific prices affect this cognitive effect. The extent of sensitivity to the context impacting replication studies [58], or the mismatch among real-world prices and the prices considered in the experiment [27] may impact the effect size of the replicated cognitive effects. We observed that participants who associate higher perceived prices with the theater (i.e., they attribute an increased monetary value to the experience) are more likely to buy a ticket, independently from the experimental conditions. This aspect was not part of the experimental manipulation, and it was not part of the original experiment, but it might be inspiring for future research. The price attributed to the theater ticket (and the monetary loss) is highly relevant, and studying in depth the relationship between monetary prices and cognitive effects is valuable for research in cognitive psychology. This includes examining the cognitive effect’s influence on price perceptions [4] and the potential impact of price on cognitive effects [26]. Considering participants’ attitudes toward ticket prices and current theater pricing trends, we adjusted the ticket price to 40€ in the conceptual replications. However, this adjustment resulted in participants losing multiple bills in the bill conditions of the two conceptual replications, as no 40€ bill exists. Although our results do not indicate a significant impact on participants’ decisions, future research should investigate this confound. Referring to the importance of tailoring paradigms to specific contexts, future studies could explore adapting the currency used in experiments to participants’ geographic locations to evaluate whether any differences emerge. In this study, we prioritized increased generalizability and sample heterogeneity by using the same currency across all experiments, regardless of participants’ locations. This decision ensured comparability across conditions, particularly in relation to the cognitive effects we were testing, which relied on the relative comparison of values (e.g., 40€) rather than on specific purchasing power. Moreover, we controlled for participants’ maximum willingness to pay for a ticket in our binary logistic regression analysis to mitigate concerns about individual differences in price perception. Future process-oriented replications that focus on cultural differences and variations in purchasing power could further investigate these aspects.

We observed that the purchase medium did not have a significant effect on participants’ willingness to pay for a ticket, and thus, Hypothesis 2 was not confirmed. We propose that, despite the recognized influence of modern purchase mediums on purchase intentions [10], it is the richness of the purchase medium’s description [30] that significantly impacts purchase intentions. Thus, only changing the purchase medium was not sufficient to observe significant differences in the purchase intentions.

We conclude that the mental accounting effect is replicable when appropriately adapted to contemporary contexts and circumstances. Moreover, we emphasize the importance of process-oriented replication projects to better understand the fundamental components of experimental paradigms used to test cognitive theories, such as mental accounting. Notably, Internet-based methodologies offer significant advantages for replication studies, including the ability to easily share experimental materials and access large samples with ease. The primary objective of replication studies should be to evaluate the robustness of the cognitive effect under investigation, which can be achieved through rigorous methodologies and the inclusion and comparison of several different participant samples.

Supplementary materials, including Appendices A and B, are available online at: https://osf.io/s3a8b/.