Reproducibility Standards in Economics (and beyond)

REMARK Motivation

• Econ lags in reproducibility practices
• State of field: for published papers
  • .zip archive has some files
    • rarely works ‘out of the box’
  • Top-5 journals have a ‘data editor’
    • Very expensive
    • Excellent barrier to competition
    • Data \(\neq\) results

Econ-ARK REMARK solutions

• Standards for reproducibility of compute
  • self-contained and complete projects
  • executable on any modern computer
• Emphasis on:
  • minimally good code
    • enforced algorithmically
    • would be huge improvement
    • “ChatGPT, fix my documentation”
  • explicit claims (“Risk Aversion: [2.5,3.5]”)
  • standardized metadata

So Far: About 23+3 REMARKs

• Many replicating canonical papers

• Three new ones created for FossProF:

  • Imai_Keane (2004) by John Green
  • DeNardi (2004) by Ashish Kumar
  • Aiyagari (1994) by Adam Edwards

• Paul Romer:

  • This is the future of academic publishing

Jointly Reproducible: Code and Math and Text

• Solved: most computational aspects
  • Docker containers for universality
    • Linux, Windows, MacOS, AWS, …
  • version control (git/GitHub) = ownership
  • cff references make it findable/indexable
  • Zenodo: independent “release” archiving

Math and Text with Code?

• \(\LaTeX\) is standard for text, but…
  • definitely not lightweight
  • no dynamically executable content
• Now: tenuous connection between text and code
  • “where is equation 18 implemented”?
    • in a gestalt of interactions
• Future:
  • “this math: implemented by this code”

FOSSProF \(\rightarrow\) Curvenote

Funding let us hire open source contractor - To do things we could not do

At our direction:

• Integrated MyST Markdown into REMARK
• Filled gaps in tools to integrate text, math, and code
• Improved \(\LaTeX\) \(\rightarrow\) MyST engine
  • Tools required for some of our REMARKs

Link to FOSSProF Final Report

Curvenote: \(\LaTeX\)

Curvenote: Computation

Journal of Open Source Economics

Not Specific to Economics!

• conceived for projects using Econ-ARK
  • … but doesn’t require Econ-ARK
• Not bound to economics in any way

Rebranding: REMARK \(\rightarrow\) SCI-PASS

• Scholarly Communication Infrastructure for Publishing and Archiving Scientific Software
• Ambition:
  • universal standard for repr compute
    • nothing now exists (we’ve looked)
• Also short for “scientific passport”
  • portability/sharing of research outputs

Long-Term Goal: ISO standard

• Adoptable by journals, libraries, archives …
• Existing software standards got started this way
  • Somebody needed it for their own purposes
• Members of our team:
  • SB: “Information Science” PhD
    • connection to archivists, librarians
  • AS: Economics PhD
    • Vetted ISO software standards for Oz