Reproducibility Standards in Economics

Project Overview¶

Project Summary¶

The Economics profession needs to catch up to other technical fields in software development, reproducibility practices, and “exchangeability” of results.

To that end, Econ-ARK has been working for several years on the REMARK project, a set of standards and tools for reproducibility for computational modeling in economics. REMARKs are self-contained and complete projects whose contents should be executable by anyone on any modern computer (local or cloud), so long as the platform has the necessary hardware (generically described)^[1]. A critical aspect of REMARKs is the emphasis on clear documentation, testing procedures, and standardized metadata to ensure that research outputs are reproducible, understandable, reusable, and securely attributable to their true authors.

The design specs of the REMARK standard have been crafted with the collaboration of the editor of a projected journal that would require all submissions to abide by the REMARK standard.

While we have solved most of the computational challenges (using the blossoming ecosystem of tools including Docker containers, version control, etc.), one piece of the infrastructure needed to complete the specification is still lacking: A robust, reproducible, and portable standard for the production of the text of the paper (or other research product) that can directly integrate reproducible content.

The project sponsored by FOSSProF allowed us to hire an open-source contractor, Curvenote, to fill some crucial gaps in the infrastructure necessary to translate the standard medium of technical writing, $\LaTeX$, to the new world of lightweight, reproducible content. The bulk of the other FOSSProF funding allowed some JHU PhD students to create new examples of REMARKs that use these tools.

A brief history of REMARK¶

The REMARK project started as a means to enhance the sharing and reproducibility of research that utilized the Econ-ARK’s HARK toolkit. As the development of HARK was guided mainly by active research, it was essential to expand the codebase and integrate the code with the documentation and drafting of academic manuscripts. In economics, however, the standard practice is to treat the research manuscript and code as entirely separate entities, with the code being used as a tool to generate results that are posted in the manuscript but then relegated to a .zip file attached to the published paper. Because there are no standards for the contents of these zip files, there is no real guarantee that the code can be run by anyone other than the original author(s) on the original computer(s) on which it was developed with the exact software configuration on that computer at the time the final version of the code was zipped. This practice severely limits scholars’ ability to reproduce and build upon previous work.

Recognizing the need to integrate scientific software development with the publication of research, Econ-ARK began working on the REMARK standard to make the models and results from the HARK toolkit easy to share. But as the REMARK project grew, it became clear that such a lightweight standard for reproducible computational results could be helpful for the broader Economics community (and perhaps more broadly in the scientific computing community).

Target Audience¶

The original intended users of REMARKs were students, researchers, and practitioners of economics who would benefit from reproducibility of economic modeling results. However, nothing about the project limits the scope of its use to economics; there is no reason the standard could not be used in any field in which computational results need to be reproducible. As we have worked on the REMARK standard, we have come to realize that there is a broader need for reproducible research practices in many fields where computational results are important to the substantive conclusions of the research.

Code Repository¶

This project is a collaboration between the Econ-ARK team and Curvenote. The REMARK standard itself can be found at the REMARK GitHub repository, but much of the code generated during this project has already been integrated into the MyST project.

Project Activities and Progress¶

Work Completed¶

The primary goal of this project was to integrate the MyST Markdown tools into the REMARK project to expand the range of tools available for the standard. To do this, we identified three existing REMARKs that highlighted gaps in the open-source tooling for typesetting and integrating reproducible results directly into a REMARK. Through these examples, we prioritized our contractors, Curvenote (who are core MyST developers and members of the steering council), and identified efficient ways to integrate MyST into REMARKs.

We also worked with some Johns Hopkins Economics PhD students to produce REMARKs of their projects, providing feedback on the REMARK standard and showing how the REMARK standard can produce high-quality, reproducible research.

Lastly, we gave a presentation of the REMARK project at the 30th International Conference for Computation in Economics and Finance, where we engaged with many students and researchers interested in open source software for reproducible research and advocated for the adoption of REMARKs as a standard for reproducible research in economics. (The project lead for the REMARK project is the incoming president of the sponsor of the conference, the Society for Computational Economics.

Collaboration with Myst/Curvenote¶

Throughout the project, we used the following GitHub issue to track our progress:

• REMARK #152: Improvements to MyST Markdown

This issue tracks the many contributions to the MyST project that were supported by this grant. A primary focus of this work was the extension of the MyST ecosystem to support the translation of documents written with the $\LaTeX$ mathematical typesetting language, which produces static documents (PDF’s), into MyST markdown which is designed to incorporate dynamically executable content.

This was necessary because among economists (and in many other technical fields), the use of $\LaTeX$ for production of academic papers is ubiquitous, and many researchers are deeply invested in existing $\LaTeX$ documents and workflows.

Student generation of REMARKs¶

We also used this grant to support the work of three Johns Hopkins University PhD students, John Green, Adam Edwards, and Ashish Kumar, in the Department of Economics, for the development of REMARKs based on papers relevant to their dissertation research.

Engaging with students to produce REMARKs allowed us to gather feedback on the standard and to improve the documentation and tools available to researchers. Moreover, it will allow these students to produce high-quality, reproducible, and portable research that can be used as part of their job market and portfolio of materials. We hope this will lead to more visibility and exposure for their research.

Presentation of REMARK project at CEF 2024¶

At the 30th International Conference for Computing in Economics and Finance in Singapore, we gave a presentation on the REMARK project and engaged with many students and researchers interested in open source software for reproducible research.

The presentation consisted of introducing the REMARK standard, demonstrating the capabilities of REMARKs through a live interactive presentation, and discussing the benefits of reproducibility in economics.

After the presentation, we received feedback from researchers who were very interested in the REMARK standard and were eager to introduce their students to the newly developed tools. We would love to take this presentation and workshop on the road and introduce REMARKs to the next generation of economists.

Technical Milestones¶

A significant milestone of this project was the expansion of MyST compatibility with the $\LaTeX$ syntax. Many existing REMARKs are written in $\LaTeX$, and it is critical to ensure backward compatibility and the ability to refresh existing REMARKs with new MyST capabilities, including rich cross-references, hover previews, and (most importantly) integrated computation. Although we have not yet reached the goal of complete backward compatibility, we have made considerable progress in this area. We welcome contributions from the open source community to help us reach this goal.

Second, we started work on a remark command-line-interface tool that can be used to generate REMARK templates and check them against the REMARK standard. This tool also includes functionalities for building environments and running reproducibility scripts.

Third, we have continued to expand the catalog of existing REMARKs, including several new REMARKs of student projects and active research projects both within and outside Econ-ARK.

Challenges and Solutions¶

The project has gone quite smoothly, thanks to the deep expertise and strongly aligned goals of the REMARK project with the Myst project. The principal challenge is simply that making use of the new technologies requires an investment of time, which is the scarcest resource among busy scholars.

Outcomes and Impact¶

Project Impact¶

The overarching objective of REMARK is to improve the reliability and trustworthiness of scientific findings within fields heavily reliant on computational methods. By promoting the use of standardized tools and workflows, the REMARK project aims to make computational research in economics and social sciences more transparent and reproducible.

REMARK encourages the adoption of best practices in software development, documentation, and manuscript preparation within the economics community, which has traditionally lagged behind other fields in which computational results are first-class scholarly contributions. Our goal is to facilitate knowledge sharing and collaboration, reduce duplication efforts, and accelerate the dissemination of knowledge within the field.

Community Engagement¶

Our team primarily engaged with the main developers of the MyST project through consultants at Curvenote. We set up a weekly meeting where we discussed our needs for the REMARK project and suggestions for improving the MyST project itself. The team at MyST/Curvenote is deeply experienced and is closely connected with the open science and publishing communities.

Additionally, we advocated for the use of reproducibility standards at the 30th International Conference for Computing in Economics and Finance and at the OpenSource.Science Economics interest group. We met with many students and researchers interested in open source software for reproducible research and advocated for the adoption of REMARKs as a standard for reproducible research in Economics.

Sustainability and Future Plans¶

Renaming the project¶

The need for the REMARK standard arose from our desire to make research conducted as part of the Econ-ARK project easily reproducible and shareable -- an origin that is transparently signaled by the fact that the acronym is derived from “R[eplications]/[eproductions] and Explorations Made using ARK.”

Now that the REMARK standard is completely independent of the Econ-ARK toolkit, this name is inappropriate, because it will mislead people into thinking they must be using the Econ-ARK toolkit if they want to create a reproducible software object using these tools.

To address this issue, we have discussed internally the need to rename the project, with the current leading candidate name “SCI-PASS”, which stands for “Scholarly Communication Infrastructure for Publishing and Archiving Scientific Software.” This name change captures our current ambition of establishing a universal and inclusive standard for reproducible scientific software.

Renaming REMARK is not a cosmetic change; it is a signal of our ambition to build a robust standard for reproducible computational research of any kind. The near-term evolution of the project will be guided by our interactions with the editor of the prospective journal mentioned above, as we develop it into submission standard for scientific and technical journal articles. We will continue actively engaging with the scientific community to improve and promote the standard and foster collaboration, including establishing communication channels with existing journal editors, journal data editors, and others who already have expertise and wisdom to offer on these topics. consider “SCI-PASS” to be short for “scientific passport,” a term that encapsulates the idea of portability and exchangeability of research outputs.

Adapting to Evolving Research Practices¶

To succeed, the REMARK/SCI-PASS project must embrace modern publishing technologies. As we recognized the limitations of $\LaTeX$, this grant helped us integrate modern scientific publishing tools like Jupyter Notebooks and MyST Markdown. This shift toward the Jupyter ecosystem is intended to enhance user-friendliness, interactivity, and accessibility for researchers. In the long-term, we aim to further integrate with open science infrastructure, such as established open science publishing platforms for peer-review and publishing like Open Journals and The Journal of Open Source Software.

While most existing REMARKs are built using python, we are committed to expanding language support to accommodate the diverse set of tools used in computational research. This expansion aims to broaden the project’s reach and applicability across disciplines. Moreover, the project’s long-term sustainability depends on active community engagement, which includes collaborating with journal and data editors, organizing workshops and tutorials, and establishing an independent board of advisors with expertise in computational science, library science, and relevant research domains. By embracing these adaptations, the REMARK/SCI-PASS project aims to evolve into a robust and sustainable standard for reproducible research, aligning with evolving practices and solidifying its place within the future of open science.

Taking REMARK/SCI-PASS on the road¶

This grant allowed us to host a workshop at the 30th International Conference for Computing in Economics and Finance in Singapore, where we discussed open science, reproducibility, and the REMARK standard. The experience and feedback we gathered were invaluable, and we are eager to take the project on the road to more conferences and events. Below, we describe a few of the ways in which we plan to advocate for open science and reproducibility:

1. Replication Competitions - Competitions where researchers are challenged to replicate the results of notable papers, which will provide a practical demonstration of the REMARK standard and incentivize adoption.
2. Tutorials and Workshops - Target students and researchers to provide hands-on experience using our tools, demonstrating how to create reproducible research outputs and highlighting the advantages of open science.
3. SCI-PASS Talks - Engage with journal editors and data editors to seek feedback and collaboration to understand better how our project can streamline the review process, ensure reproducibility, and enhance the credibility of published research.

Financial Information¶

Grant Expenditures¶

The majority of the funds were used to contract with Curvenote to integrate the MyST tools into the REMARK project. We also supported the work of three Johns Hopkins University PhD students over the summer of 2024 to produce REMARKs on their ongoing research projects. Finally, we hosted a workshop at the 30th International Conference for Computing in Economics and Finance in Singapore and used the remaining funds for travel and lodging.