Skip to article frontmatterSkip to article content
Site not loading correctly?

This may be due to an incorrect BASE_URL configuration. See the MyST Documentation for reference.

R[eplications/eproductions] and Explorations Made using ARK (REMARK)

A standard for ensuring computational reproducibility, archiving, and sharing of research projects.

REMARKs are self-contained projects that can, in principle, be executed on any modern computer.

The standard is focused on 3 key principles:

Detailed technical instructions for using an existing REMARK or contributing a new REMARK can be found in How to Use REMARK.

The Econ-ARK Ecosystem

This repository is part of the Econ-ARK project:

Getting Started

For REMARK Authors

Quick Start

  1. Standard REMARK (basic reproducible project):

    • Follow REMARK Guidelines requirements

    • Create reproduce.sh, CITATION.cff, binder/environment.yml, and a lockfile with pinned dependencies

    • Submit via pull request (add REMARKs/your-project.yml pointing to your repo). On acceptance, Econ-ARK will fork your repo and the website will point to that fork until you submit a new version; when you do, we will update as long as reproduce.sh runs and the draft still meets REMARK requirements. You keep ownership.

  2. Published REMARK (permanent archival with DOI):

    • Meet all Standard REMARK requirements first

    • Then obtain a Zenodo DOI using Zenodo DOI Guide for REMARKs (30–45 min)—this should be the last step before you submit

    • Submit via pull request with the DOI. On acceptance, Econ-ARK will fork your repo and the website will point to that fork until you submit a new version; when you do, we will update as long as reproduce.sh runs and the draft still meets REMARK requirements. You keep ownership.

Description

Types of content include (see below for elaboration):

  1. Explorations

    • Use the Econ-ARK/HARK toolkit to demonstrate some set of modeling ideas

  2. Replications

    • Attempts to replicate important results of published papers written using other tools

  3. Reproductions

    • Code that reproduces ALL of the results of some paper that was originally written using the Econ-ARK toolkit

Compliance Tiers

REMARKs are organized into three tiers based on their level of reproducibility and archival readiness (see REMARK Guidelines for full details):

TierNameKey Requirement
1Docker REMARKDockerfile + reproduce.sh + basic documentation
2Reproducible REMARKTier 1 + CITATION.cff + REMARK.md + comprehensive docs
3Published REMARKTier 2 + Zenodo DOI for permanent archival

REMARK Catalog

A catalog of all REMARKs is available under the REMARK tab at econ-ark.org. (Direct link: https://econ-ark.org/materials/?select=REMARK)

The ballpark is a place for the set of papers that we would be delighted to have replicated in the Econ-ARK. But we would welcome submissions of replications from any field of economics that requires meaningful computation.

In cases where the replication’s author is satisfied that the main results of the paper have been successfully replicated, we expect to approve pull requests for new REMARKs with minimal review (subject to the criteria described in the Standard). On acceptance, Econ-ARK will fork your repository and the website will point to that fork until you submit a new version; when you do, we will update the fork as long as reproduce.sh runs and the revised draft still meets REMARK requirements. You keep full ownership.

We also expect to approve with little review cases where the author has a clear explanation of discrepancies between the paper’s published results and the results in the replication attempt.

Replication archives should contain two kinds of content (along with explanatory material):

  1. Code that attempts to replicate key results of the paper

  2. A Jupyter notebook that presents at least a minimal set of examples of the use of the code.

This material will all be stored in a directory with some short pithy name (a bibtex citekey might make a good directory name).

Code archives should contain:

Jupyter notebook(s) should:

For AI Systems and Automated Tools

Differences with DemARK

The key difference with the contents of the DemARK repo is that REMARKs are allowed to rely on the existence of local files and subdirectories (figures; data) at a predictable filepath relative to the location of the root.