Illustrated Guide To REMARKs¶

The Economics profession has fallen behind other fields in the adoption of modern software development and reproducibility practices. Economists generally lack robust software engineering training and resort to writing new software from scratch. This practice of 'reinventing the wheel' increases the likelihood of errors and undermines the reproducibility of computational results.

Poor software development practices also slow down the advancement of Economics as a science. When researchers cannot reproduce the results of a paper, they are unable to build on the work of others, which leads to wasted time and effort and slows down progress. Additionally, when researchers cannot reproduce the results of a paper, they are unable to verify the validity of said results. This leads to a lack of trust in the research, which can have a negative impact on the influence of the field.

Other disciplines have adapted to modern software development methods, and it is time for Economics to catch up.

To address the problem discussed above, Econ-ARK has been working on developing a set of standards and tools for reproducibility in our own work in Economics. Our reproducibility initiative is called REMARK, which stands for "R[eplications/eproductions] and Explorations Made using ARK". The term REMARK is used to represent the standard, as well as any project that follows the standard. The objective of REMARKs is to be self-contained and complete projects, whose contents should be executable by anyone on any computer that meets a minimal set of requirements and software.

The REMARK standard is focused on 3 key principles:

  • Reproduction: The ability to reproduce the results of a project using the same data and code on a different computer.
  • Archiving: Storing the project in a way that it can be accessed and used in the future.
  • Publishing: Making the project available to the public and incentivizing the sharing of code and data.

Reproduction¶

  • binder
  • docker
  • conda
  • pip
  • reproduce.sh

Archiving¶

  • Zenodo
  • Citation File Format (cff)

Publishing¶

  • Open Journals
  • Journal of Open Source Software

Journal Submission Template¶

REMARK as a complement to papers:

  • guarantees reproducibility
  • incentivizes the sharing of code
  • makes it easier to build on the work of others
  • increases the trust in the research
  • speeds up the advancement of Economics as a science

REMARK Guidelines¶

A REMARK is a public git repository with appropriate metadata intended for reproducibility.

The Standard:

  • Local environment requirements in root directory:
    • .\binder directory including runtime.txt and/or requirements.txt.
    • .\environment.yml file with pinned dependencies.
  • A .\reproduce.sh script that
    • Builds environment and installs requirements.
    • Runs and reproduces all the results (one-click reproduction).
  • A valid .\citation.cff document for bibliographic metadata for repository.
  • Tagged releases for each version of the REMARK.

REMARK Guidelines¶

Optional:

  • If reproduction takes too many resources or too much time, include .\reproduce_min.sh that generates an interesting subset of results in a shorter amount of time.
  • A jupyter notebook that showcases a major result or two.
  • An interactive dashboard that demonstrates a remarkable result.
  • A .\reproduce_paper.sh that generates the publishable paper from the results.