⚠️ IMPORTANT: The REMARK CLI tool (cli.py) is for REMARK VALIDATION AND TESTING, not website generation. The website (econ-ark.org) is generated by a separate system (populate_remarks.py).
🔧 CLI Tool vs 🌐 Website Generation¶
This CLI tool is used for:
✅ Validating REMARK compliance (reproducibility standards)
✅ Testing
reproduce.shscripts✅ Building and testing environments
✅ Editorial review workflow
This CLI tool is NOT used for:
❌ Generating website content (that’s
populate_remarks.py)❌ Creating
_materials/*.mdfiles❌ Website troubleshooting
The REMARK CLI tool (cli.py) provides automated operations for cloning, validating, building, and executing REMARKs. This tool is essential for maintainers and anyone working with multiple REMARKs.
Installation Requirements¶
# From REMARK repository root
pip install -r requirements.txtRequired dependencies:
Python 3.9+
PyYAML and jsonschema
Docker (for docker builds)
Conda (for conda builds)
repo2docker (for docker operations); version 2025 and later needs Docker’s buildx plugin (
docker-buildxon Ubuntu)
As a package¶
cli.py is a thin launcher for the remark package beside it, so python cli.py ... works from a checkout with nothing installed. Installing the package adds a remark command, which acts on the catalog in the current directory (or --root <checkout>), and makes each step importable:
pip install . # from a checkout
pip install "econ-ark-remark @ git+https://github.com/econ-ark/REMARK@<tag>" # pinned, elsewhere
remark --root /path/to/REMARK lint --tier 1 --allfrom remark.catalog import load_catalog # REMARKs/*.yml -> Metadata
from remark.git import pull_entry # clone or update at the catalogued tag
from remark.lint import lint_repo # (errors, warnings) for a tier
from remark.actions import build, execute # each step, with its record
from remark.records import validate # check a record against the schemaA project’s own runner can call execute for its jobs. Beyond the command line’s options it takes the script’s arguments, an environment directory, and the settings of a run that keeps its outputs apart: target (where the checkout is mounted), read_only, mounts (such as {'/out': job_folder}), tmpfs, variables and user ('host' for the invoking user).
The tests (pip install '.[test]', then pytest) run in CI on every pull request that touches the package; the container tests run wherever a docker daemon answers and are skipped elsewhere.
Core Commands¶
1. Pull/Clone REMARKs¶
Download REMARK repositories locally:
# Clone all REMARKs
python cli.py pull --all
# Clone specific REMARKs
python cli.py pull REMARKs/BufferStockTheory.yml REMARKs/beyond-the-streetlight.yml
# Pull updates for already cloned REMARKs
python cli.py pull --all # Updates existing reposWhat it does:
Creates
_REMARK/repos/directory structureClones repositories from GitHub URLs in catalog
Checks out the tag pinned in the catalog entry (the clone is made at that tag; pulling again re-fetches it)
Updates existing repositories if already cloned (a clone left detached at a tag whose entry no longer pins one needs
python cli.py clean repofirst)With
--all, leaves out the entries markedprivate: trueand names them;--privateincludes them, and an entry named on the command line is always attempted. This holds for every command that takes--all.Exits non-zero, naming the entry, when any git step fails
2. Lint/Validate REMARKs¶
Check REMARK compliance against standards:
# Check all REMARKs
python cli.py lint --all
# Check specific REMARKs
python cli.py lint REMARKs/BufferStockTheory.yml
# Include optional files in validation
python cli.py lint --all --include-optionalWhat it does:
Checks each REMARK for the files its tier requires (
--tier 1|2|3, default 2; the lists are inremark/lint.py, followingSTANDARD.md)Reports compliance issues with specific missing files, and README length
Optionally checks optional files (
reproduce_min.sh)Exits non-zero, naming the REMARKs with errors, when any has one; warnings alone do not fail it
🚨 CRITICAL: Lint results show REMARK COMPLIANCE (reproducibility), NOT website functionality:
❌ “missing reproduce.sh” = Cannot reproduce research (REMARK issue)
✅ Does NOT mean = Website won’t work (website only needs
CITATION.cff)❌ “missing binder/environment.yml” = Cannot build execution environment
✅ Does NOT mean = Website display problem
Example output:
------------------ REMARKs/BufferStockTheory.yml ------------------
/path/to/_REMARK/repos/BufferStockTheory
- missing reproduce_min.sh
- missing binder/environment.yml3. Build Environments¶
Create reproducible execution environments:
# Build conda environments for all REMARKs
python cli.py build conda --all
# Build docker images for specific REMARKs
python cli.py build docker REMARKs/BufferStockTheory.yml
# Use parallel processing (default: 4 jobs)
python cli.py build conda --all --jobs 8Conda builds:
Creates
./.condaenvin each repositoryUses
binder/environment.ymlfor dependenciesIsolated environment per REMARK
Docker builds:
Uses repo2docker (same as MyBinder)
Supports multi-language REMARKs
Names the image
econ-ark/<name>:<hash>, the hash being that of the environment’s files (binder/, or the repository’s root when there is none) as git records themAlways builds, even when an image of that name exists; docker’s layer cache keeps a build that changes nothing short
Labels the image with the commit it was built at, since repo2docker copies the whole checkout into it
Both kinds of build write an execution record.
An environment in a directory of its own (docker only). A repository can keep further environments beside its own, for instance one per solver, each in its own directory:
# Build, then run in, the environment kept in envs/solver of the REMARK
python cli.py build --environment envs/solver docker REMARKs/BufferStockTheory.yml
python cli.py execute --environment envs/solver docker REMARKs/BufferStockTheory.ymlThe directory is built alone: its
Dockerfilebydocker buildwhen it has one, else its binder files by repo2docker. Nothing else of the checkout enters the image.Without a
Dockerfile, the directory must hold a file repo2docker builds from, such asenvironment.ymlorrequirements.txt. It does not read.python-version, or apyproject.tomlthat has no build system; a directory with nothing it reads is refused, and no image or record is made.repo2docker installs from none of
uv.lock,poetry.lock,pdm.lockandpixi.lock. A directory that holds one of them and noDockerfileis refused before anything is built: its image would not be of the locked versions, though its record would carry the lock’s hash. To build from a lock file, give the directory aDockerfilethat installs from it. This applies to a directory named with--environmentonly; the REMARK’s own environment is built frombinder/as before, whatever lock the repository’s root holds.The image is named
econ-ark/<name>-<directory>:<hash>, hereecon-ark/bufferstocktheory-envs-solver:<hash>. Since the name says all the image depends on, a later commit that leaves the directory alone needs no rebuild.The checkout is still mounted at run time, so the script and the code it runs are those of the checkout.
4. Execute REMARKs¶
Run reproduction scripts in built environments:
# Execute all REMARKs using conda
python cli.py execute conda --all
# Execute specific REMARKs using docker
python cli.py execute docker REMARKs/BufferStockTheory.yml
# Force use of reproduce.sh (ignore reproduce_min.sh)
python cli.py execute conda --all --no-min
# Parallel execution
python cli.py execute conda --all --jobs 2
# Limit each container (docker only): no network, 2 CPUs, 8 GB, 2 BLAS/OpenMP threads, one hour
python cli.py execute --no-network --cpus 2 --memory 8g --threads 2 --timeout 3600 docker REMARKs/BufferStockTheory.ymlPut options before docker or conda: the parser does not accept an option between the type and the REMARK paths.
Script selection:
Uses
reproduce_min.shif available (faster)Falls back to
reproduce.sh--no-minforcesreproduce.sh
How a docker execution runs:
Without a terminal, so it works from CI, cron or a background job; the container is removed when it ends.
The checkout is mounted, writable, at the image’s working directory, where repo2docker copied the repository, and the script runs there.
In repo2docker’s
kernelornotebookconda environment when the image has one; otherwise the script runs directly.--no-network,--cpus,--memory,--threadsand--timeoutare off unless given. A REMARK stopped by--timeoutreturns 124.In the image built for the environment as the checkout has it now. If the environment’s files changed since the build there is no such image, and if the checkout moved to another commit the image is of the old one: either way nothing runs, and the message says to build first.
Output:
Logs stored in
_REMARK/logs/execute/Return codes tracked for success/failure
Stdout/stderr captured per REMARK
An execution record for every run
buildandexecuteexit non-zero, naming the REMARKs that failed or left no record, when any of them does
5. View Logs¶
Check results from build/execute operations:
# Show all recent results
python cli.py logs
# Show results for specific REMARKs
python cli.py logs REMARKs/BufferStockTheory.ymlExample output:
build
BufferStockTheory = 0
beyond-the-streetlight = 1
execute
BufferStockTheory = 0
beyond-the-streetlight = 0Return codes: 0 = success, non-zero = failure
6. Clean Environments¶
Remove built environments and data:
# Remove conda environments
python cli.py clean conda --all
# Remove docker images: every image built for each REMARK, whatever its tag
python cli.py clean docker --all
# Remove cloned repositories
python cli.py clean repo --allExecution Records¶
Every build and every execute writes a record of what it did, with its log beside it, in _REMARK/records/<name>/:
_REMARK/records/Aiyagari1994QJE/
├── 20261111T181502Z_build.json
├── 20261111T181502Z_build.log
├── 20261111T182240Z_execution.json
└── 20261111T182240Z_execution.logA record holds facts read from git, the machine and the process, and nothing inferred:
What ran: the remote, the commit, the catalogued tag when the checkout is at it, and whether the working tree was dirty (with its tree as git would write it, when it was); the script; the environment’s directory and the hash of its files; the image’s ID and the build record it came from.
Where and when: the CPU, cores, memory, operating system, kernel and docker version; the limits set on the run; the start, the end and the duration.
--no-hostleaves the machine’s name out.How it ended: the exit code, and whether the script exited by itself, was stopped by
--timeout, was killed by the memory limit, or by another signal. Docker does not report every memory-limit kill (it missed 1 of 40 in a test on docker 29.1.3); an unreported one is recorded as aSIGKILL.
The format is remark/schemas/execution-record.schema.json, and a record that does not match it is never written. A project’s own runner can write the same format and add fields of its own under extensions. Two limits to know:
A docker build that fails has no image, so it leaves a log and a return code but no record.
A record needs an
httpsremote. For an entry whose remote is a local path, the step runs and the command then exits non-zero, saying that no record was written.
A second execute starts from a checkout the first one wrote into, so its record says dirty; for a run of record, start from a fresh clone (clean repo, then pull).
Status Table¶
One command reads every entry and writes a table of what each repository holds. Nothing is run, and nothing in the table is kept by hand:
python cli.py status --all # every public entry
python cli.py status --all --private # private entries too, where this machine can reach them
python cli.py status REMARKs/BufferStockTheory.ymlWhat it does:
Pulls each entry at its pin, as
pulldoes, and records the commit read and the newest commit on the remote, marking a remote that has moved on as(newer).Writes
status.mdandstatus.jsoninto_REMARK/status/(or--output), with a row for every entry asked for. A private entry that was not asked for has a row that says only that it exists.Exits non-zero, naming the entries, when one could not be read: its pull failed, no reader is installed for its kind, or its reader failed. The row is written all the same.
What a row says comes from a reader chosen by the kind of unit, which a repository declares in the front matter of its REMARK.md:
---
kind: comparison
---A repository that declares no kind is a
remark. Its reader is built in and reports the highest tier whose required files the checkout holds.Any other kind is read by a function that an installed package offers under the entry-point group
remark.status, named for the kind. It receives the checkout and the entry, and returns a one-linesummary, and optionally its own Markdowntableanddata. Each kind’s table has the columns that suit it.
[project.entry-points."remark.status"]
comparison = "some_package.status:read"A run that read a private entry says so at the top of status.md, in status.json (private_entries_read) and on the terminal: those files hold what a private repository contains and are not for publication.
Advanced Usage¶
Working with Specific Versions¶
If a REMARK metadata file specifies a tag field:
name: BufferStockTheory
remote: https://github.com/econ-ark/BufferStockTheory
title: Buffer Stock Theory
tag: v1.0.0The CLI will automatically checkout that specific version.
Parallel Processing¶
Most operations support parallel execution:
# Build 8 REMARKs simultaneously
python cli.py build conda --all -J 8
# Execute with 2 parallel jobs (safer for memory-intensive REMARKs)
python cli.py execute docker --all --jobs 2Environment Management¶
The CLI creates isolated environments:
_REMARK/
├── repos/ # Cloned repositories
│ ├── BufferStockTheory/
│ │ ├── .condaenv/ # Conda environment (if built)
│ │ └── ...
│ └── beyond-the-streetlight/
├── status/ # status.md and status.json, from `status`
├── logs/ # The latest log and return code of each step
│ ├── build/
│ │ ├── BufferStockTheory_conda.log
│ │ └── BufferStockTheory_conda_rc.log
│ └── execute/
│ ├── BufferStockTheory_conda.log
│ └── BufferStockTheory_conda_rc.log
└── records/ # A record and a log for every build and execution
└── BufferStockTheory/Error Handling¶
Common Issues¶
Missing files: Use lint to identify missing required files
python cli.py lint --allBuild failures: Check build logs
python cli.py logs
cat _REMARK/logs/build/ProjectName_conda.logDocker issues: Ensure Docker daemon is running
docker --version
systemctl status docker # On LinuxConda issues: Verify conda installation
conda --versionIntegration with GitHub Workflows¶
The CLI is designed to integrate with GitHub Actions:
- name: Validate REMARKs
run: python cli.py lint --all
- name: Test REMARKs
run: |
python cli.py build conda --all
python cli.py execute conda --allMaintainer Workflows¶
Adding New REMARKs¶
Validate compliance:
python cli.py lint REMARKs/new-remark.ymlTest build:
python cli.py build conda REMARKs/new-remark.ymlTest execution:
python cli.py execute conda REMARKs/new-remark.ymlCheck logs:
python cli.py logs REMARKs/new-remark.yml
Bulk Operations¶
# Full validation pipeline
python cli.py pull --all
python cli.py lint --all
python cli.py build conda --all
python cli.py execute conda --all
python cli.py logsPerformance Considerations¶
Docker builds: Slower but more robust, good for multi-language REMARKs
Conda builds: Faster, good for Python-only REMARKs
Parallel jobs: More parallelism speeds builds but uses more memory
Disk space: Each REMARK requires ~100MB-1GB depending on dependencies
This CLI tool enables efficient management of the entire REMARK ecosystem, from validation through execution, with comprehensive logging and error handling.