Add a Custom Task

How To: Add a Custom Task¶

This walkthrough adds a new task called flanker_preprocess to the task fMRI pipeline. By the end you will have a fully integrated task that submits SLURM array jobs, receives parameters from config as environment variables, and supports --resume.

Overview¶

Adding a task involves four files, the last one optional:

Where task definitions live¶

Task types are defined in config.yaml inside your --config-dir. Add new pipeline sections directly to this file. Task-specific parameters (resources, paths, tool versions) go in your project config under tasks.

Step 1: Write the Analysis Script¶

Create the script in the directory set by scripts_dir in your project config:

{scripts_dir}/afni_flanker_preprocess.sh

The pipeline dispatches based on file extension: .py files are run with python, and every other file is run with bash. To use another language such as R, call it from a short bash script. The script receives one argument: the subject ID (e.g., 001). Everything else is available as environment variables that the pipeline exports automatically.

#!/bin/bash
set -euo pipefail

subject="$1"

# ── Environment variables always available ───────────────────────────────────
# These come from global project config and envir_dir:
#   $PREFIX          subject prefix, e.g. "sub-"
#   $PROJECT         project name, e.g. "my_study"
#   $SESSION         session label, e.g. "01"
#   $INPUT_DIR       --input, or the input_from task's output directory when
#                    that task runs in the same submission
#   $OUTPUT_DIR      this task's output directory (output_pattern resolved)
#   $WORK_DIR        working / log directory
#   $TEMPLATE_DIR    path to MNI templates  (from envir_dir.template_dir)
#   $CONTAINER_DIR   path to .sif containers (from envir_dir.container_dir)
#   $FREESURFER_DIR  FreeSurfer license dir  (from envir_dir.freesurfer_dir)
#
# These come from your task's entry in tasks.flanker_preprocess:
#   $REMOVE_TRS      remove_TRs value
#   $TEMPLATE        template filename
#   $BLUR_SIZE       blur_size value
#   $CENSOR_MOTION   censor_motion value
#   $CENSOR_OUTLIERS censor_outliers value
# ─────────────────────────────────────────────────────────────────────────────

echo "=== Flanker preprocessing: ${PREFIX}${subject} ==="
echo "Template:  ${TEMPLATE_DIR}/${TEMPLATE}"
echo "Blur:      ${BLUR_SIZE} mm"
echo "Remove TRs: ${REMOVE_TRS}"

afni_proc.py \
  -subj_id "${PREFIX}${subject}" \
  -script "proc.${subject}" \
  -dsets "${INPUT_DIR}/${PREFIX}${subject}/ses-${SESSION}/func/"*flanker*bold*.nii.gz \
  -tcat_remove_first_trs "$REMOVE_TRS" \
  -tlrc_base "${TEMPLATE_DIR}/${TEMPLATE}" \
  -blur_size "$BLUR_SIZE" \
  -regress_censor_motion "$CENSOR_MOTION" \
  -regress_censor_outliers "$CENSOR_OUTLIERS" \
  -execute

How parameters become environment variables¶

Your project config (step 3) lists task parameters like remove_TRs: 4. The pipeline upper-cases each key and exports it before calling your script:

Internal keys (name, environ, scripts, input_from, profile, array, output_pattern) are not exported — they are consumed by the pipeline itself.

Step 2: Register in config.yaml¶

Open config.yaml in your --config-dir and add your task as a new top-level section. For a staged task fMRI pipeline, add it like cards or kidvid:

# Add a new section for the flanker task
flanker:
  - name: flanker_preprocess
    stage: prep
    profile: standard          # resource profile (required, no default)
    array: true                # true = one array job per subject
    multi_stage: true          # staged pipeline: depends on intermed if requested
    scripts: [afni_flanker_preprocess.sh]
    input_from: recon          # $INPUT_DIR = recon's output when recon runs in the same submission
    output_pattern: "{base_output}/AFNI_derivatives"

Key fields:

For the full list of resource profiles and their CPU/memory/time limits, see HPC Configuration.

Step 3: Configure in Your Project Config¶

Add parameters to your project’s {project}_config.yaml under tasks, keyed by task name:

tasks:
  flanker_preprocess:        # must match exactly what you put in config.yaml
    remove_TRs: 4
    template: "HaskinsPeds_NL_template1.0_SSW.nii"
    blur_size: 4.0
    environ: ["afni_24.3.06"]       # module names to load (from your modules section)
    censor_motion: "0.3"
    censor_outliers: "0.05"

The environ list is resolved against your modules section. For example:

modules:
  afni_24.3.06:
    - ml AFNI/25.1.01-foss-2023a

At runtime the pipeline inserts those ml commands into the wrapper script’s environment block before calling your script.

Parameter naming rules¶

• Config key → environment variable: the key is upper-cased and nothing else changes (remove_TRs → REMOVE_TRS). Parameter names must therefore be valid shell identifiers: letters, digits and underscores, starting with a letter or underscore. A key containing - or ., or starting with a digit, stops run with an error naming the key when the wrapper is generated, which --dry-run also reports.

• All values are strings in the environment. Use "0.3" not 0.3 if your script does string comparison.

• Nested dicts are not supported — keep task parameters flat.

Step 4: Wire the CLI flag¶

The pipeline routes --staged-prep flanker to flanker_preprocess by looking up the flanker section in config.yaml and selecting tasks with stage: prep. Run neuromaestro list-tasks to verify.

Step 5: Run a Dry-Run Test¶

neuromaestro run \
  --subjects 001 \
  --input /data/BIDS \
  --output /data/processed \
  --work /data/work \
  --config-dir /data/config \
  --project my_study \
  --session 01 \
  --staged-prep flanker \
  --dry-run

The dry-run prints the DAG execution plan and a summary without submitting anything. It does not print sbatch commands. Instead, the generated wrapper in {--work}/{project}/log/wrapper/ records its full submission command in a # Submission Command: comment near the top. Check there:

• --job-name matches your script file name (afni_flanker_preprocess)

• --array=1-1%15 (1 subject, limit 15 concurrent)

• No --dependency= flag in this example: run on its own, flanker_preprocess waits for nothing. With --intermed volume in the same run it gets --dependency=afterany:... on the volume job

• The wrapper exports the right $REMOVE_TRS, $TEMPLATE, etc.

# Inspect the generated wrapper
cat /data/work/my_study/log/wrapper/afni_flanker_preprocess_*.sh

If the dry-run looks correct, remove --dry-run to submit.

Step 6: Add Output Checks for --resume¶

Without a checks entry, --resume will always resubmit this task with a warning. Add a rule to <config-dir>/results_check/{project}_checks.yaml:

flanker_preprocess:
  output_path: "{work_dir}/AFNI_derivatives/{prefix}{subject}/"
  required_files:
    - pattern: "proc.{subject}"
      min_size_kb: 100
    - pattern: "{prefix}{subject}.results/stats.{prefix}{subject}+tlrc.HEAD"
      min_size_kb: 1000
  count_check:
    bold_runs:
      pattern: "pb0*.r0*.scale+orig.HEAD"
      expected_count: 4        # number of runs expected
      tolerance: 0             # must match exactly

With this file in place, --resume will check each subject’s AFNI output directory and only resubmit subjects whose files are missing or too small.

For the full syntax — check types, placeholders, and a real-project example — see Output Checks Configuration.