Project Configuration

Each study has its own project config file: {project}_config.yaml, stored in config/project_config/. You specify which project to use with --project my_study, and the pipeline loads my_study_config.yaml automatically.

A complete annotated example follows.

Complete Example¶

# 1. branch_config.yaml — BRANCH study project configuration

prefix: "sub-"               # Subject directory prefix (always "sub-" for BIDS)
scripts_dir: "scripts/branch"  # Absolute, or relative to the directory that holds --config-dir

database:
  db_path: "$WORK_DIR/database/pipeline_jobs.db"

# 2. envir_dir paths become $UPPERCASE env vars in every wrapper script
envir_dir:
  container_dir: "/work/cglab/containers"
  virtual_envir: "/work/cglab/conda_env"
  template_dir: "/work/cglab/projects/BRANCH/all_data/for_AFNI/"
  atlas_dir: "/work/cglab/projects/BRANCH/all_data/for_AFNI/"
  freesurfer_dir: "/work/cglab/containers/.licenses/freesurfer"
  config_dir: "/work/cglab/conda_env/config_for_BIDS"
  stimulus_dir: "/work/cglab/projects/BRANCH/all_data/for_AFNI/processing_scripts"

# 3. Loaded on every compute node before the task runs (used for database logging)
global_python:
  - ml Python/3.11.3-GCCcore-12.3.0
  - . /home/$USER/virtual_environ/neuro_pipeline/bin/activate

# 4. Named module sets; tasks reference them by name via their `environ` field
modules:

  afni_24.3.06:
    - ml Flask/2.3.3-GCCcore-12.3.0
    - ml netpbm/10.73.43-GCC-12.3.0
    - ml AFNI/24.3.06-foss-2023a

  fsl_6.0.7.14:
    - ml FSL/6.0.7.14-foss-2023a
    - '[ -n "$FSLDIR" ] && source ${FSLDIR}/etc/fslconf/fsl.sh'

  freesurfer_7.4.1:
    - ml FreeSurfer/7.4.1-GCCcore-12.3.0

  data_manage_1:
    - ml p7zip/17.05-GCCcore-13.3.0
    - ml parallel/20240722-GCCcore-13.3.0

# 5. Keyed by task name (must match names in config.yaml)

tasks:

  unzip:
    environ: ["data_manage_1", "afni_24.3.06"]

  recon:
    container: "dcm2bids_3.2.0.sif"
    config: "branch_config.json"

  volume:
    environ: ["afni_24.3.06"]
    template: "HaskinsPeds_NL_template1.0_SSW.nii"

  rest_preprocess:
    remove_TRs: 6
    template: "MNI152NLin2009cAsym"
    container: "fmriprep_25.1.3.sif"
    license: "license.txt"

  rest_post:
    remove_TRs: 6
    template: "MNI152NLin2009cAsym"
    container: "xcp_d-0.11.0rc1.sif"
    rest_mode: "abcd"
    motion_filter_type: "notch"
    band_stop_min: "15"
    band_stop_max: "25"
    nuisance_regressors: "36P"
    license: "license.txt"

  cards_preprocess:
    remove_TRs: 2
    template: "HaskinsPeds_NL_template1.0_SSW.nii"
    blur_size: 4.0
    environ: ["afni_24.3.06"]
    censor_motion: "0.3"
    censor_outliers: "0.05"

  kidvid_preprocess:
    remove_TRs: 22
    template: "HaskinsPeds_NL_template1.0_SSW.nii"
    blur_size: 4.0
    environ: ["afni_24.3.06"]
    censor_motion: "0.3"
    censor_outliers: "0.05"

  cards_postprocess:
    environ: ["afni_24.3.06"]
    mask_frac: "1.0"
    conditions: ["BW", "BL", "LW", "LL", "NeutralW", "AntLongBW"]
    master_id: "027"
    jobs: "8"
    pthr: "0.001"
    athr: "0.05"
    nn: "1"

  kidvid_postprocess:
    environ: ["afni_24.3.06"]
    mask_frac: "1.0"
    contrasts: ["neg", "neut", "pos"]
    clustsim: "20"
    master_id: "027"
    pthr: "0.001"
    athr: "0.05"
    nn: "1"

  dwi_preprocess:
    container: "qsiprep-1.0.0.sif"
    license: "license.txt"
    template: "MNI152NLin2009cAsym"
    output_resolution: 2.0

  dwi_post:
    container: "qsirecon-1.2.0.sif"
    license: "license.txt"
    mode: "dsi_studio_autotrack"
    atlases: "Gordon333Ext"

  mriqc_preprocess:
    container: "mriqc_24.0.2.sif"

  mriqc_post:
    container: "mriqc_24.0.2.sif"

Field Reference¶

`prefix`¶

Subject directory prefix. Almost always "sub-" for BIDS datasets.

`scripts_dir`¶

Directory where your project’s shell scripts live. A relative path is resolved against the study directory, the parent of your --config-dir, so it does not depend on where you start neuromaestro. With the layout neuromaestro init creates (my_study/config/ next to my_study/scripts/), the template’s default finds the copied scripts:

scripts_dir: "scripts"   # → /scratch/my_study/scripts with --config-dir /scratch/my_study/config

An absolute path is used as given, e.g. for scripts on a shared lab directory:

scripts_dir: "/scratch/cglab/projects/my_study/scripts"

If a script is not found, the error message shows the resolved scripts_dir path and, when that directory exists, the files it contains.

`database`¶

db_path sets where the job database is written, and is required. The pipeline replaces the literal $WORK_DIR with the run --work directory without the project name, so the default "$WORK_DIR/database/pipeline_jobs.db" becomes e.g. /data/work/database/pipeline_jobs.db, with JSONL job logs in /data/work/database/json/. No other variable is expanded. Inside the job, $WORK_DIR is --work plus the project name (e.g. /data/work/my_study), whose log/ folder holds the job logs and wrappers.

`envir_dir`¶

All paths become $UPPERCASE_ENV_VARS inside wrapper scripts:

Field	Env var	Purpose
`container_dir`	`$CONTAINER_DIR`	Singularity `.sif` files (required)
`virtual_envir`	`$VIRTUAL_ENVIR`	Conda/venv base
`template_dir`	`$TEMPLATE_DIR`	MRI templates and atlases
`atlas_dir`	`$ATLAS_DIR`	Parcellation atlases
`freesurfer_dir`	`$FREESURFER_DIR`	FreeSurfer license
`config_dir`	`$CONFIG_DIR`	BIDS conversion configs
`stimulus_dir`	`$STIMULUS_DIR`	Task timing/stimulus files

`global_python`¶

Shell commands run on the compute node before every task, used to activate the Python environment for database logging. The environment needs typer. The logging script otherwise uses only the Python standard library.

`modules`¶

Named groups of module load commands. Reference them by name in tasks.<name>.environ. This way you define module versions once and reuse them across tasks.

`tasks`¶

Task-specific parameters keyed by task name. The key must exactly match a task name defined in config.yaml. An entry can also override any field of that task in config.yaml except stage and multi_stage, which pre-flight rejects.

Common per-task fields:

Field	Description
`environ`	List of module group names to load
`container`	Singularity `.sif` filename (looked up in `container_dir`)
`license`	License file (e.g., FreeSurfer) relative to `freesurfer_dir`
`template`	Template filename (looked up in `template_dir`)
`remove_TRs`	Number of dummy TRs to drop at the start
`blur_size`	Smoothing kernel FWHM in mm
`censor_motion`	Motion threshold for censoring (mm)
`censor_outliers`	Outlier voxel fraction threshold

How `tasks` and `modules` Relate¶

tasks.cards_preprocess.environ = ["afni_24.3.06"]
        ↓
modules.afni_24.3.06 = [ml Flask/..., ml netpbm/..., ml AFNI/...]
        ↓
Wrapper script: runs those module commands before executing the analysis script

This indirection lets you update the AFNI version in one place (modules) and have all tasks pick it up automatically.

End-to-End Example: `cards_preprocess`¶

This section traces exactly how config values flow from branch_config.yaml into your analysis script.

1. Config entries (branch_config.yaml)¶

envir_dir:
  container_dir: "/work/cglab/containers"
  template_dir:  "/work/cglab/projects/BRANCH/all_data/for_AFNI/"

modules:
  afni_24.3.06:
    - ml Flask/2.3.3-GCCcore-12.3.0
    - ml netpbm/10.73.43-GCC-12.3.0
    - ml AFNI/24.3.06-foss-2023a

tasks:
  cards_preprocess:
    remove_TRs: 2
    template:   "HaskinsPeds_NL_template1.0_SSW.nii"
    blur_size:  4.0
    environ:    ["afni_24.3.06"]
    censor_motion:    "0.3"
    censor_outliers:  "0.05"

2. Generated wrapper script (auto-created per submission)¶

The pipeline writes a wrapper to {--work}/{project}/log/wrapper/ before calling sbatch (--dry-run writes it too, without submitting). The file is named after the script, e.g. afni_cards_preprocessing_<timestamp>_wrapper.sh. You normally never need to read it. It is kept for debugging and reproducibility records. The example below assumes --work /work/cglab/BRANCH/work, --output /work/cglab/BRANCH/processed and --project branch.

#!/bin/bash

# Auto-generated wrapper script
# ...
# Submission Command:
# sbatch --partition=batch ... /work/cglab/BRANCH/work/branch/log/wrapper/afni_cards_preprocessing_1234567890_wrapper.sh

# Basic paths (always present)
export SUBJECTS='001 002 003 ...'
export INPUT_DIR='/work/cglab/BRANCH/processed/branch/BIDS'
export OUTPUT_DIR='/work/cglab/BRANCH/processed/branch/AFNI_derivatives'
export WORK_DIR='/work/cglab/BRANCH/work/branch'
export CONTAINER_DIR='/work/cglab/containers'
export LOG_DIR='/work/cglab/BRANCH/work/branch/log'
export DB_PATH='/work/cglab/BRANCH/work/database/pipeline_jobs.db'
export TASK_NAME='cards_preprocess'
export SCRIPT_DIR='/path/to/site-packages/neuromaestro/pipeline'
export EXECUTION_ID='12'

# Global Python (for database logging)
export GLOBAL_PYTHON_COMMANDS=$(cat << "PYTHON_EOF"
ml Python/3.11.3-GCCcore-12.3.0
. /home/$USER/virtual_environ/neuro_pipeline/bin/activate
PYTHON_EOF
)

# Module load commands (from modules.afni_24.3.06)
export ENV_COMMANDS=$(cat << "ENV_EOF"
ml Flask/2.3.3-GCCcore-12.3.0
ml netpbm/10.73.43-GCC-12.3.0
ml AFNI/24.3.06-foss-2023a
ENV_EOF
)

# envir_dir paths, then session, prefix and project from the run
export GLOBAL_ENV_VARS=$(cat << "GENV_EOF"
export CONTAINER_DIR='/work/cglab/containers'
export TEMPLATE_DIR='/work/cglab/projects/BRANCH/all_data/for_AFNI/'
...
export SESSION='01'
export PREFIX='sub-'
export PROJECT='branch'
GENV_EOF
)

# Task parameters (config.yaml and tasks.cards_preprocess keys, minus structural fields)
export TASK_PARAMS=$(cat << "TASK_EOF"
export STAGE='prep'
export MULTI_STAGE='True'
export REMOVE_TRS='2'
export TEMPLATE='HaskinsPeds_NL_template1.0_SSW.nii'
export BLUR_SIZE='4.0'
export CENSOR_MOTION='0.3'
export CENSOR_OUTLIERS='0.05'
TASK_EOF
)

# Execute
source "$SCRIPT_DIR/utils/wrapper_functions.sh"
execute_wrapper "/path/to/scripts/branch/afni_cards_preprocessing.sh"

Two rules for envir_dir and tasks values.

Values are passed through literally. They are written into the wrapper in single quotes, so a value containing $VAR, a backtick or $(...) reaches your script exactly as written rather than being expanded on the compute node. If you need something evaluated at run time, put it in global_python or modules instead: those are command lists and still expand normally.

A few key names are reserved. Every key is upper-cased, so a key named path would become export PATH=... and leave the job unable to find any command. Submission fails with an error if a key maps to PATH, HOME, SHELL, PWD, OLDPWD or IFS, or to a variable the wrapper sets itself: SUBJECTS, SUBJECT_ID, SUBJECTS_ARRAY, NUM_SUBJECTS, INPUT_DIR, OUTPUT_DIR, WORK_DIR, LOG_DIR, SUB_LOG_DIR, LOG_PATH, DB_PATH, TASK_NAME, SCRIPT_DIR, EXECUTION_ID or ENV_FILE. Rename it, for example to tool_path. A key that is not a valid shell variable name once upper-cased (anything but letters, digits and underscores, or a leading digit, as in my-param or voxel.size) is rejected the same way. Under envir_dir, keys named session, prefix or project are silently replaced by the values of the current run. Other environment variables are yours to set: a key named lc_all or pythonpath is accepted, since a task may legitimately need to override those.

3. Your analysis script uses those variables¶

#!/bin/bash
# scripts/branch/afni_cards_preprocessing.sh (simplified)
# The wrapper framework passes the current subject as $1.

subject=$1

afni_proc.py \
  -subj_id        sub-${subject} \
  -dsets          $INPUT_DIR/sub-${subject}/func/*.nii.gz \
  -tcat_remove_first_trs $REMOVE_TRS \
  -blur_size      $BLUR_SIZE \
  -tlrc_NL_warp \
  -tlrc_base      $TEMPLATE_DIR/$TEMPLATE \
  -regress_censor_motion   $CENSOR_MOTION \
  -regress_censor_outliers $CENSOR_OUTLIERS \
  -out_dir        $OUTPUT_DIR/sub-${subject}.results

4. Adding a custom parameter¶

Any key you add under tasks.<name> that is not a structural field (name, scripts, profile, array, input_from, output_pattern, environ) is automatically exported as $UPPERCASE with no code changes needed. So is container (as $CONTAINER), and so are the non-structural keys inherited from config.yaml (stage, multi_stage). For example:

tasks:
  cards_preprocess:
    remove_TRs:  2
    blur_size:   4.0
    censor_motion:   "0.3"
    censor_outliers: "0.05"
    environ: ["afni_24.3.06"]
    polort:    "A"           # → $POLORT  in your script
    bandpass:  "0.01 0.1"   # → $BANDPASS in your script

Then in your script:

3dDeconvolve \
  -polort $POLORT \
  ...

This makes every task fully configurable from the YAML without editing any pipeline code.

Complete Example¶

Field Reference¶

prefix¶

scripts_dir¶

database¶

envir_dir¶

global_python¶

modules¶

tasks¶

How tasks and modules Relate¶

End-to-End Example: cards_preprocess¶