HPC Configuration

hpc_config.yaml controls how the pipeline submits and monitors jobs on your HPC cluster: which scheduler to use, what resource profiles are available, and how submission flags are constructed.

Location: config/hpc_config.yaml

Scheduler¶

scheduler: slurm   # or: pbs

Switch to pbs and uncomment the PBS block at the bottom of the file to use PBS/Torque instead of SLURM.

Resource Profiles¶

Resource profiles are fully user-configurable. You can edit existing profiles and add new ones freely. Changing a profile only affects future job submissions; it has no impact on the database or any already-running jobs.

Each task in config.yaml references a profile by name via the profile: field. At submission time the pipeline looks up that profile, merges it with defaults, and translates the result into scheduler flags.

How merging works¶

defaults  +  profile  →  final resources

The defaults block provides a base for every profile. Any key present in the profile overrides the default for that submission only. This lets you set cluster-wide values (e.g. partition: batch) once and override them selectively per profile.

defaults:
  partition: "batch"      # used by all profiles unless overridden

resource_profiles:
  standard:
    memory: "32gb"
    time: "20:00:00"
    nodes: 1
    ntasks: 1
    cpus_per_task: 16
    # partition not listed → inherits "batch" from defaults

  heavy_long:
    memory: "64gb"
    time: "24:00:00"
    nodes: 1
    ntasks: 1
    cpus_per_task: 16
    partition: "bigmem"   # overrides defaults.partition for this profile only

Profile fields¶

Field	Required	Description
`memory`	Yes, unless `memory_per_cpu` is set	Total job memory. Becomes `--mem` in SLURM. Format: `"32gb"`, `"64gb"`.
`memory_per_cpu`	Alternative to `memory`	Per-CPU memory allocation. Becomes `--mem-per-cpu` and replaces `--mem` when set. Use when your cluster bills by CPU-hour.
`time`	Yes	Wall-clock time limit. Format: `"HH:MM:SS"` or `"D-HH:MM:SS"`. Jobs exceeding this are killed by the scheduler.
`nodes`	Yes	Number of compute nodes. Almost always `1` for neuroimaging jobs.
`ntasks`	Yes	Number of parallel tasks within a job. Almost always `1` (the wrapper launches one process per subject).
`cpus_per_task`	Yes	CPU cores allocated per task. Becomes `--cpus-per-task`. Set to match the parallelism of the underlying tool (e.g. 16 for AFNI, 16 for fMRIPrep).
`partition`	Yes (here or in `defaults`)	Queue/partition name. Overrides `defaults.partition` when set. If neither the profile nor `defaults` sets it, submission (and `--dry-run`) stops with an error naming the profile. The same holds for the other required fields.
`gres`	No	Generic resources such as GPUs. Becomes `--gres` (e.g. `"gpu:1"` → `--gres=gpu:1`).
`array_limit`	No	Caps how many subjects of an array task run at once. Becomes the `%N` in `--array=1-50%15`. Only applies to tasks marked `array: true` in `config.yaml`. Omit it to leave the array unthrottled, letting the scheduler run as many subjects as the queue allows.
`additional_args`	No	List of raw scheduler flags appended verbatim to the submission command. Use for cluster-specific flags not covered above (e.g. `--constraint=avx512`, `--account=mylab`).

Adding a new profile¶

Add a named block under resource_profiles:. The name becomes the value you put in profile: in config.yaml.

Note: --job-name, --output, and --error are auto-generated by the pipeline. The job name is the script filename without its extension, and the two log paths point into the task’s log directory. Do not put these in a profile.

For example, to replicate this SLURM header:

#SBATCH --partition=gpu_p
#SBATCH --gres=gpu:A100:1
#SBATCH --ntasks=1
#SBATCH --cpus-per-task=2
#SBATCH --mem=40gb
#SBATCH --time=10:00:00
#SBATCH --mail-type=END,FAIL
#SBATCH --mail-user=yourID@xxx.edu

Write:

resource_profiles:
  gpu_a100:
    memory: "40gb"
    time: "10:00:00"
    nodes: 1
    ntasks: 1
    cpus_per_task: 2
    partition: "gpu_p"
    gres: "gpu:A100:1"
    additional_args:
      - "--mail-type=END,FAIL"
      - "--mail-user=yourID@xxx.edu"

If --mail-user should apply to every profile, put it in defaults instead so you don’t repeat it:

defaults:
  partition: "batch"
  additional_args:
    - "--mail-type=END,FAIL"
    - "--mail-user=yourID@xxx.edu"

additional_args from defaults and from the profile are not merged: the profile’s list replaces the default entirely if both are present. Put shared flags in defaults and profile-specific flags only in the profile.

Then reference the new profile in config.yaml:

intermed:
  - name: volume
    profile: gpu_a100   # ← new profile
    ...

Current profiles¶

Profile	Memory	Time	Array limit	Use for
`data_manage`	2 GB	20 min	15	Light file management (not used by any shipped task)
`light_short`	16 GB	4 h	15	`recon`, MRIQC group
`standard_short`	32 GB	8 h	15	Intermed (`volume`, `bfc`), XCP-D
`standard`	32 GB	20 h	15	Unzip, task fMRI (AFNI), DWI
`heavy_long`	64 GB	24 h	15	fMRIPrep, MRIQC individual

The shipped profiles all cap at 15, which is a conservative starting point rather than a tuned value. Since the limit is per profile, a light profile can usually be raised well above a heavy one without exceeding the same node budget. Adjust to match your cluster’s queue policy.

SLURM Settings¶

slurm:
  submit_cmd: sbatch
  job_id_parse: last_word
  dependency_flag: "--dependency=afterany:{jobs}"
  array_flag: "--array={array}"
  resource_flags:
    partition:     "--partition={value}"
    nodes:         "--nodes={value}"
    ntasks:        "--ntasks={value}"
    cpus_per_task: "--cpus-per-task={value}"
    time:          "--time={value}"
    mem:           "--mem={value}"
    mem_per_cpu:   "--mem-per-cpu={value}"
    gres:          "--gres={value}"
    job_name:      "--job-name={value}"
    output:        "--output={value}"
    error:         "--error={value}"
  status_cmd: squeue
  status_args: ["--noheader", "--format=%i %T"]
  active_states: [PENDING, RUNNING]

Submission¶

Key	Description
`submit_cmd`	The command used to submit jobs. `sbatch` for SLURM.
`job_id_parse`	How to extract the job ID from the scheduler’s stdout. `last_word` parses `"Submitted batch job 12345"` → `"12345"`. `first_word` takes the first token and is for PBS, where qsub prints the ID itself (`"12345.cluster"` → `"12345.cluster"`).

Dependencies and arrays¶

Key	Description
`dependency_flag`	Template for expressing upstream job dependencies. `{jobs}` is replaced with a colon-joined list of upstream job IDs: `--dependency=afterany:12345:12346`. Staged pipelines that wait for multiple intermed tasks will have all their job IDs joined here.
`array_flag`	Template for array submission. `{array}` is replaced with the complete range string the pipeline builds from the subject count and the profile’s `array_limit` (e.g. `1-50%15`), so the template supplies only the flag itself, not the range.

Resource flags¶

resource_flags is a mapping from internal resource key → scheduler flag template. {value} is substituted with the resolved value from the profile. You can rename the flag if your cluster uses non-standard options, but do not rename the keys on the left; those are internal names the pipeline uses.

# Example: cluster uses --mem-gb instead of --mem
resource_flags:
  mem: "--mem-gb={value}"   # change the right side only

Job monitoring (`--wait`)¶

These fields are used when --wait is passed to neuromaestro run:

Key	Description
`status_cmd`	Command to query job status. `squeue` for SLURM.
`status_args`	Arguments passed to `status_cmd`. The default `--noheader --format="%i %T"` produces `12345 RUNNING` lines, which the pipeline parses to extract job ID and state.
`active_states`	States considered “still running”. The pipeline keeps polling while any job has one of these states. SLURM: `PENDING`, `RUNNING`. PBS equivalent: `Q`, `R`, `H`.

PBS Support¶

To use PBS/Torque, set scheduler: pbs and uncomment the pbs: block in hpc_config.yaml. The block maps the same resource flags to their PBS equivalents (e.g. partition → -q, --mem → -l mem=).

# pbs:
#   submit_cmd: qsub
#   job_id_parse: first_word        # qsub prints "12345.cluster.example.com"
#   dependency_flag: "-W depend=afterany:{jobs}"
#   array_flag: "-J {array}"
#   resource_flags:
#     partition:     "-q {value}"
#     nodes:         "-l nodes={value}"
#     ntasks:        ""
#     cpus_per_task: "-l ncpus={value}"
#     time:          "-l walltime={value}"
#     mem:           "-l mem={value}"
#     mem_per_cpu:   ""
#     gres:          "-l ngpus={value}"
#     job_name:      "-N {value}"
#     output:        "-o {value}"
#     error:         "-e {value}"
#   status_cmd: qstat
#   active_states: [Q, R, H]