HPC Configuration

hpc_config.yaml controls how the pipeline submits and monitors jobs on your HPC cluster: which scheduler to use, what resource profiles are available, and how submission flags are constructed.

Location: config/hpc_config.yaml

Scheduler¶

scheduler: slurm   # or: pbs

Switch to pbs and uncomment the PBS block at the bottom of the file to use PBS/Torque instead of SLURM.

Resource Profiles¶

Resource profiles are fully user-configurable. You can edit existing profiles and add new ones freely. Changing a profile only affects future job submissions; it has no impact on the database or any already-running jobs.

Each task in config.yaml references a profile by name via the profile: field. At submission time the pipeline looks up that profile, merges it with defaults, and translates the result into scheduler flags.

How merging works¶

defaults  +  profile  →  final resources

The defaults block provides a base for every profile. Any key present in the profile overrides the default for that submission only. This lets you set cluster-wide values (e.g. partition: batch) once and override them selectively per profile.

defaults:
  partition: "batch"      # used by all profiles unless overridden

resource_profiles:
  standard:
    memory: "32gb"
    time: "20:00:00"
    nodes: 1
    ntasks: 1
    cpus_per_task: 16
    # partition not listed → inherits "batch" from defaults

  heavy_long:
    memory: "64gb"
    time: "24:00:00"
    nodes: 1
    ntasks: 1
    cpus_per_task: 16
    partition: "bigmem"   # overrides defaults.partition for this profile only

Profile fields¶

Field	Required	Description
`memory`	Yes (or `memory_per_cpu`)	Total job memory. Becomes `--mem` in SLURM. Format: `"32gb"`, `"64gb"`.
`memory_per_cpu`	Alternative to `memory`	Per-CPU memory allocation. Becomes `--mem-per-cpu`. Use when your cluster bills by CPU-hour. If both are set, `memory_per_cpu` takes precedence.
`time`	Yes	Wall-clock time limit. Format: `"HH:MM:SS"` or `"D-HH:MM:SS"`. Jobs exceeding this are killed by the scheduler.
`nodes`	Yes	Number of compute nodes. Almost always `1` for neuroimaging jobs.
`ntasks`	Yes	Number of parallel tasks within a job. Almost always `1` (the wrapper launches one process per subject).
`cpus_per_task`	Yes	CPU cores allocated per task. Becomes `--cpus-per-task`. Set to match the parallelism of the underlying tool (e.g. 16 for AFNI, 16 for fMRIPrep).
`partition`	No	Queue/partition name. Overrides `defaults.partition` when set.
`additional_args`	No	List of raw scheduler flags appended verbatim to the submission command. Use for cluster-specific flags not covered above (e.g. `--constraint=avx512`, `--account=mylab`).

Adding a new profile¶

Add a named block under resource_profiles:. The name becomes the value you put in profile: in config.yaml.

Note: --job-name, --output, and --error are auto-generated by the pipeline from the task name and log paths; do not put these in a profile.

For example, to replicate this SLURM header:

#SBATCH --partition=gpu_p
#SBATCH --gres=gpu:A100:1
#SBATCH --ntasks=1
#SBATCH --cpus-per-task=2
#SBATCH --mem=40gb
#SBATCH --time=10:00:00
#SBATCH --mail-type=END,FAIL
#SBATCH --mail-user=yourID@xxx.edu

Write:

resource_profiles:
  gpu_a100:
    memory: "40gb"
    time: "10:00:00"
    nodes: 1
    ntasks: 1
    cpus_per_task: 2
    partition: "gpu_p"
    additional_args:
      - "--gres=gpu:A100:1"
      - "--mail-type=END,FAIL"
      - "--mail-user=yourID@xxx.edu"

If --mail-user should apply to every profile, put it in defaults instead so you don’t repeat it:

defaults:
  partition: "batch"
  additional_args:
    - "--mail-type=END,FAIL"
    - "--mail-user=yourID@xxx.edu"

additional_args from defaults and from the profile are not merged: the profile’s list replaces the default entirely if both are present. Put shared flags in defaults and profile-specific flags only in the profile.

Then reference the new profile in config.yaml:

intermed:
  - name: volume
    profile: gpu_a100   # ← new profile
    ...

Current profiles¶

Profile	Memory	Time	Use for
`data_manage`	2 GB	20 min	Unzip, file management
`light_short`	16 GB	4 h	MRIQC group, lightweight postprocessing
`standard_short`	32 GB	8 h	Intermed AFNI (`volume`), XCP-D
`standard`	32 GB	20 h	Task fMRI (AFNI), DWI
`heavy_long`	64 GB	24 h	fMRIPrep, MRIQC individual

SLURM Settings¶

slurm:
  submit_cmd: sbatch
  job_id_parse: last_word
  dependency_flag: "--dependency=afterany:{jobs}"
  array_flag: "--array={array}"
  resource_flags:
    partition:     "--partition={value}"
    nodes:         "--nodes={value}"
    ntasks:        "--ntasks={value}"
    cpus_per_task: "--cpus-per-task={value}"
    time:          "--time={value}"
    mem:           "--mem={value}"
    mem_per_cpu:   "--mem-per-cpu={value}"
    job_name:      "--job-name={value}"
    output:        "--output={value}"
    error:         "--error={value}"
  status_cmd: squeue
  status_args: ["--noheader", "--format=%i %T"]
  active_states: [PENDING, RUNNING]
  cancel_cmd: scancel

Submission¶

Key	Description
`submit_cmd`	The command used to submit jobs. `sbatch` for SLURM.
`job_id_parse`	How to extract the job ID from the scheduler’s stdout. `last_word` parses `"Submitted batch job 12345"` → `"12345"`. `first_word` is for PBS (`"12345.cluster"` → `"12345"`).

Dependencies and arrays¶

Key	Description
`dependency_flag`	Template for expressing upstream job dependencies. `{jobs}` is replaced with a colon-joined list of upstream job IDs: `--dependency=afterany:12345:12346`. Staged pipelines that wait for multiple intermed tasks will have all their job IDs joined here.
`array_flag`	Template for SLURM array submission. `{array}` is replaced with the range string from `array_config.pattern` in `config.yaml` (e.g. `1-50%15`).

Resource flags¶

resource_flags is a mapping from internal resource key → scheduler flag template. {value} is substituted with the resolved value from the profile. You can rename the flag if your cluster uses non-standard options, but do not rename the keys on the left; those are internal names the pipeline uses.

# Example: cluster uses --mem-gb instead of --mem
resource_flags:
  mem: "--mem-gb={value}"   # change the right side only

Job monitoring (`--wait`)¶

These fields are used when --wait is passed to neuropipe run:

Key	Description
`status_cmd`	Command to query job status. `squeue` for SLURM.
`status_args`	Arguments passed to `status_cmd`. The default `--noheader --format="%i %T"` produces `12345 RUNNING` lines, which the pipeline parses to extract job ID and state.
`active_states`	States considered “still running”. The pipeline keeps polling while any job has one of these states. SLURM: `PENDING`, `RUNNING`. PBS equivalent: `Q`, `R`, `H`.
`cancel_cmd`	Command used to cancel jobs (`scancel` for SLURM, `qdel` for PBS).

PBS Support¶

To use PBS/Torque, set scheduler: pbs and uncomment the pbs: block in hpc_config.yaml. The block maps the same resource flags to their PBS equivalents (e.g. partition → -q, --mem → -l mem=).

# pbs:
#   submit_cmd: qsub
#   job_id_parse: first_word        # qsub prints "12345.cluster.example.com"
#   dependency_flag: "-W depend=afterany:{jobs}"
#   array_flag: "-J 1-{array}"
#   resource_flags:
#     partition:     "-q {value}"
#     nodes:         "-l nodes={value}"
#     ntasks:        ""
#     cpus_per_task: "-l ncpus={value}"
#     time:          "-l walltime={value}"
#     mem:           "-l mem={value}"
#     mem_per_cpu:   ""
#     job_name:      "-N {value}"
#     output:        "-o {value}"
#     error:         "-e {value}"
#   status_cmd: qstat
#   active_states: [Q, R, H]
#   cancel_cmd: qdel