Skip to article frontmatterSkip to article content
Site not loading correctly?

This may be due to an incorrect BASE_URL configuration. See the MyST Documentation for reference.

REMARK Ecosystem Workflow

This document provides a comprehensive overview of how the REMARK ecosystem works, including the interactions between the REMARK repository, individual research repositories, and the econ-ark.org website.

System Architecture Overview

CRITICAL DISTINCTION: The REMARK ecosystem has TWO SEPARATE SYSTEMS that serve different purposes:

  1. Website Generation System (populate_remarks.py) - Generates econ-ark.org content

  2. REMARK Validation System (cli.py) - Validates research reproducibility standards

These are INDEPENDENT systems with different requirements!

The REMARK ecosystem consists of three main components:

┌─────────────────────┐    ┌─────────────────────┐    ┌─────────────────────┐
│   REMARK Repo       │    │  Individual Repos   │    │  econ-ark.org       │
│  (Catalog/Standards) │    │  (Research Projects) │    │  (Public Website)   │
│                     │    │                     │    │                     │
│ • REMARKs/*.yml     │◄──►│ • CITATION.cff      │───►│ • _materials/*.md   │
│ • STANDARD.md       │    │ • REMARK.md         │    │ • Jekyll templates  │
│ • Validation tools  │    │ • reproduce.sh      │    │ • Search/filter UI  │
│ • CLI tools         │    │ • binder/env.yml    │    │ • Material pages    │
└─────────────────────┘    └─────────────────────┘    └─────────────────────┘

Detailed Workflow

1. REMARK Repository Structure

The REMARK repository serves as the catalog and standards authority:

REMARK/
├── REMARKs/                    # Catalog of all REMARKs
│   ├── BufferStockTheory.yml   # Minimal metadata per REMARK
│   ├── beyond-the-streetlight.yml
│   └── ...
├── STANDARD.md                 # Requirements for REMARK compliance
├── cli.py                      # Tools for validation and testing
├── .github/workflows/          # Automation workflows
│   └── transfer-remark-metadata.yml
└── Documentation files

REMARK Catalog Files (REMARKs/*.yml)

Each REMARK has a minimal YAML file containing:

name: project-name              # Short identifier
remote: https://github.com/...  # Repository URL  
title: Human Readable Title     # Display name
tag: v1.0.0                     # Optional: the release the entry pins
private: true                   # Optional: the repository is not public yet

An entry marked private: true records that a REMARK exists whose repository is not public yet. The entry itself is public, and the website shows nothing for it. When the repository becomes public, remove the line.

Critical Point: The REMARK repository does NOT contain the full metadata - only the minimal catalog entry pointing to the actual research repository.

2. Individual Research Repositories

Each research project is a self-contained repository that must meet REMARK standards:

Required Files

Optional Files

Example REMARK.md Structure

---
# Website-specific metadata (YAML frontmatter)
remark-name: beyond-the-streetlight
title-original-paper: "100 years of Economic Measurement..."
notebooks:
  - RS100_Discussion_Slides.ipynb
tags:
  - REMARK
  - Notebook
keywords:
  - forecast accuracy
  - Federal Reserve
---

# Abstract

This repository provides analysis of...

3. Website Generation Process

WEBSITE GENERATION SYSTEM (Primary: populate_remarks.py)

The econ-ark.org website is generated through an automated pipeline that is SEPARATE from the REMARK validation system:

Step 1: GitHub Workflows

Two workflows coordinate the integration:

A. REMARK Repo → Website Repo (.github/workflows/transfer-remark-metadata.yml)

B. Website Preprocessing (.github/workflows/site-preprocess.yml) - PRIMARY MECHANISM

Step 2: populate_remarks.py Script

This is the core integration script that:

  1. Clones REMARK catalog: Gets the current list of all REMARKs

  2. Reads catalog entries: Extracts repository URLs from REMARKs/*.yml files

  3. Clones individual repositories: Downloads each research project (using --sparse clone)

  4. Merges metadata: Combines data from two key source files:

    • CITATION.cff (bibliographic metadata)

    • REMARK.md (website-specific fields + abstract/body content)

  5. Generates material files: Creates _materials/{name}.md for Jekyll

IMPORTANT: This script only requires CITATION.cff to generate a basic webpage. For a rich, descriptive page, REMARK.md is essential. The script specifically looks for these two file names and ignores other markdown files (e.g., README.md or legacy {name}.md files) for website content generation.

Step 3: Jekyll Site Generation

Data Flow Diagram

┌─────────────────────┐
│ Author submits PR   │
│ to REMARK repo      │
│ (adds .yml file)    │
└──────────┬──────────┘
           │
           ▼
┌─────────────────────────────────────────────────────────────────────┐
│                    REMARK Repository                                │
│                                                                     │
│  REMARKs/new-project.yml ◄─── PR Review & Merge                   │
│  ┌─────────────────────┐                                           │
│  │ name: new-project   │                                           │
│  │ remote: github.com/ │                                           │
│  │ title: Project Name │                                           │
│  └─────────────────────┘                                           │
└─────────────────────────────────────────────────────────────────────┘
           │
           ▼ (Daily/Push triggers)
┌─────────────────────────────────────────────────────────────────────┐
│                populate_remarks.py Script                          │
│                                                                     │
│  1. Clone REMARK repo ──► Get catalog                              │
│  2. For each entry:                                                 │
│     ├─ Clone individual repo                                        │
│     ├─ Read CITATION.cff                                           │
│     ├─ Read REMARK.md                                              │
│     └─ Merge metadata                                               │
│  3. Generate _materials/{name}.md                                   │
└─────────────────────────────────────────────────────────────────────┘
           │
           ▼
┌─────────────────────────────────────────────────────────────────────┐
│                     econ-ark.org Website                           │
│                                                                     │
│  _materials/                                                        │
│  ├─ new-project.md  ◄─── Generated from merged metadata            │
│  │   ┌─────────────────────────────────────────────────────────┐   │
│  │   │ ---                                                     │   │
│  │   │ # From CITATION.cff                                     │   │
│  │   │ authors: [...]                                          │   │
│  │   │ title: Project Name                                     │   │
│  │   │ version: 1.0.0                                          │   │
│  │   │ # From REMARK.md frontmatter                            │   │
│  │   │ remark-name: new-project                                │   │
│  │   │ notebooks: [...]                                        │   │
│  │   │ tags: [REMARK, ...]                                     │   │
│  │   │ ---                                                     │   │
│  │   │                                                         │   │
│  │   │ # From REMARK.md body                                   │   │
│  │   │ # Abstract                                              │   │
│  │   │ This repository provides...                             │   │
│  │   └─────────────────────────────────────────────────────────┘   │
│  │                                                                 │
│  └─ Jekyll processes → /materials/new-project/ webpage             │
└─────────────────────────────────────────────────────────────────────┘

Key Integration Points

Metadata Merging Logic

The populate_remarks.py script combines metadata with this priority:

  1. Base data: CITATION.cff provides bibliographic information

  2. Website overlay: REMARK.md frontmatter adds website-specific fields

  3. Content: REMARK.md body becomes the webpage content

File Naming Convention

Error Handling

Submission Process

For New REMARKs

  1. Prepare repository meeting REMARK standards (your repo stays under your GitHub identity).

  2. Submit PR to REMARK repo adding REMARKs/{name}.yml that points to your repository.

  3. Editorial review checks compliance (e.g. cli.py lint, reproduce.sh runs).

  4. Merge PR adds the entry to the catalog.

  5. On acceptance, Econ-ARK will create a fork of your repository to preserve the state at which it was tested and verified to work; the catalog (and website) will then point to that fork until you submit a new version. You keep full ownership and can submit updated versions later.

  6. Automated integration (e.g. populate_remarks.py) generates website content from the catalog; your REMARK appears on econ-ark.org/materials (typically within 24 hours), serving content from the fork.

For Updates

  1. Update your repository (e.g. tag a new release).

  2. Submit a PR to REMARK that updates the catalog entry for the new version.

  3. Econ-ARK will update the fork (and the website will continue to show the fork) as long as reproduce.sh runs successfully and the revised draft still meets REMARK requirements.

Common Issues and Solutions

Issue: New REMARK not appearing on website

Diagnosis: Check if CITATION.cff exists and is valid YAML Solution: Ensure all required files are present and properly formatted

Issue: Metadata not updating

Diagnosis: Check GitHub Actions logs for populate_remarks.py Solution: Verify individual repository is publicly accessible

Issue: Website build failure

Diagnosis: YAML parsing error in metadata files Solution: Validate YAML syntax in CITATION.cff and REMARK.md

Issue: GitHub workflow appears “broken” (looking for .md files when only .yml exist)

Diagnosis: Misunderstanding the dual workflow system Solution: Remember that transfer-remark-metadata.yml is SECONDARY - the primary workflow is populate_remarks.py

Critical Understanding Points for AI Systems

Warning for AI Analysis: The REMARK ecosystem uses a dual workflow system:

  1. Primary: populate_remarks.py script that reads .yml catalog files and generates content

  2. Secondary: transfer-remark-metadata.yml workflow for edge cases with manual .md files

Do not assume the transfer workflow is misconfigured because it looks for .md files in a directory containing .yml files. This is by design - the two mechanisms serve different purposes.

Maintenance and Monitoring

Automated Processes

Manual Oversight


This workflow ensures that the REMARK ecosystem maintains high standards for reproducibility while providing a seamless integration between distributed research repositories and the centralized discovery platform at econ-ark.org.

REMARK Validation System vs Website Generation System

Critical Distinction

These are TWO COMPLETELY SEPARATE SYSTEMS with different purposes and requirements:

AspectWebsite Generation (populate_remarks.py)REMARK Validation (cli.py)
PurposeGenerate econ-ark.org website contentValidate research reproducibility
TriggerAutomatic (daily/push)Manual (editor workflow)
Required FilesCITATION.cff (required), REMARK.md (optional)reproduce.sh, CITATION.cff, binder/environment.yml
Clone Methodgit clone --sparse (metadata only)git clone --depth 1 (full repo)
Output_materials/*.md files for JekyllValidation reports and logs
Failure ImpactMissing materials on websiteCannot reproduce research

Website Generation Requirements

Minimum for website appearance:

Enhanced website features:

NOT required for website:

REMARK Validation Requirements

Required for REMARK compliance (reproducibility):

Optional for REMARK compliance:

Common Confusion Points

WRONG: “If cli.py lint shows missing files, the website won’t work” CORRECT: Website generation and REMARK validation are independent.

WRONG: “All materials on the website are REMARK compliant” CORRECT: Materials can appear on website without meeting full REMARK standards.

WRONG: “Website generation requires reproducibility files” CORRECT: Website only needs bibliographic metadata (CITATION.cff).

Workflow Integration

Research Repository
├── CITATION.cff           ────► Website Generation (populate_remarks.py)
├── REMARK.md (optional)   ────► Website Enhancement
├── reproduce.sh           ────► REMARK Validation (cli.py)
├── binder/environment.yml ────► REMARK Validation (cli.py)
└── other research files   ────► Research Content

Troubleshooting Guide

Issue: Material missing from website Check: Does repository have CITATION.cff? Is URL correct in REMARKs/*.yml?

Issue: REMARK validation fails Check: Does repository have reproduce.sh, binder/environment.yml? Does reproduce.sh run successfully?

Issue: Website shows incorrect content Check: Is CITATION.cff correct? Does REMARK.md have proper frontmatter?

AI System Warning

FOR AI SYSTEMS: Do not confuse these two systems. When analyzing REMARK compliance issues:

  1. First determine: Are you looking at website generation issues or reproducibility validation issues?

  2. Website problems: Check populate_remarks.py logs, CITATION.cff files, _materials/ generation

  3. Validation problems: Check cli.py output, reproduce.sh scripts, environment files

  4. Remember: A repository can appear on the website without being fully REMARK compliant for reproducibility