Skip to article frontmatterSkip to article content
Site not loading correctly?

This may be due to an incorrect BASE_URL configuration. See the MyST Documentation for reference.

Debug & Troubleshoot

Step 1: Identify Which Subjects Failed

Run Post-Run Verification first. Use check-outputs to see which subjects are missing expected output files, and the job database for exit codes and error messages. Once you know which subjects and tasks need attention, come back here to diagnose why.

Step 2: Read the SLURM Logs

SLURM output and error files are in {work_dir}/log/{task_name}/:

{work_dir}/log/{task_name}/{task_name}_{job_id}.out
{work_dir}/log/{task_name}/{task_name}_{job_id}.err

The detailed per-subject execution log (stdout + stderr combined) is written by the wrapper:

# Array jobs
{work_dir}/log/{task_name}/sub-{subject}/{task_name}_{job_id}_{array_task_id}_{timestamp}.log

# Non-array jobs
{work_dir}/log/{task_name}/{task_name}_{job_id}_{timestamp}.log

Step 3: Inspect the Wrapper Script

The generated wrapper script shows exactly what ran on the compute node:

cat /data/work/my_study/log/wrapper/afni_cards_preprocessing_*.sh

Verify:

Step 4: Test the Module Environment

SSH to a compute node (or use srun) and manually load the modules:

srun --pty bash
ml AFNI/24.3.06-foss-2023a
afni_proc.py --help     # should print help if loaded correctly

Step 5: Check Script Line Endings

Windows line endings cause $'\r': command not found errors:

file src/neuromaestro/pipeline/scripts/branch/afni_cards_preprocessing.sh
# Should say "POSIX shell script" or "Bourne-Again shell script", NOT "CRLF"

# Fix if needed:
dos2unix src/neuromaestro/pipeline/scripts/branch/*.sh

Database Shows Incomplete Records?

If the database is missing jobs (e.g., after a cluster crash), merge the raw JSONL logs manually:

neuromaestro merge-logs /data/work/my_study

JSONL event logs in {work_dir}/database/json/ accumulate independently of the SQLite database. Merging re-processes any unarchived files and fills in gaps.

If the JSONL files were already archived by a previous merge and the database is still incomplete (e.g., after restoring from backup), use force-rebuild to create a fresh database from all logs including archived ones:

neuromaestro force-rebuild /data/work/my_study
# → writes pipeline_jobs_rebuild_{timestamp}.db next to the original

Quick Diagnosis Checklist

Error Reference

Setup & Configuration

“No subjects found”

“Project configuration not found”

“Task not found” / task name mismatch

GUI won’t start

SLURM & Job Submission

module: command not found in job logs

SLURM Job ID: N/A in database

Jobs are queued but never start

Container Mounting (Apptainer / Singularity)

FATAL: container creation failed: ... squashfuse_ll failed to mount ... in 10s

Apptainer mounts a .sif image with squashfuse (a userspace FUSE mount) and gives up if the mount is not ready within a hard 10 second limit. The image lives on the shared network filesystem (container_dir, e.g. /work). When many jobs mount the same large image at the same time (a job array releasing a batch at once), they all read the image over the network and contend for storage bandwidth, so individual mounts exceed 10s and fail.

This is probabilistic, not deterministic. It is far more likely with:

The processing scripts avoid this by staging the image to node-local disk before running:

This is the block near the top of each processing script (before singularity run). Copy it into any new container script, then run singularity run ... ${local_container} ... instead of ${CONTAINER_DIR}/${CONTAINER}:

# Point Apptainer session/cache to node-local disk to avoid squashfuse mount timeout
apptainer_tmp="${SLURM_TMPDIR:-${TMPDIR:-/tmp}}/apptainer_${SLURM_JOB_ID:-$$}"
mkdir -p "${apptainer_tmp}"
export APPTAINER_TMPDIR="${apptainer_tmp}"
export APPTAINER_CACHEDIR="${apptainer_tmp}"
export SINGULARITY_TMPDIR="${apptainer_tmp}"
export SINGULARITY_CACHEDIR="${apptainer_tmp}"

# Stage container image to node-local disk to avoid squashfuse mount timeout.
# Per-node lock: only one job copies, others wait then reuse. Fall back to the
# network image if staging fails (e.g. local disk full).
src_container="${CONTAINER_DIR}/${CONTAINER}"
node_cache="/tmp/${USER}_sif"
staged="${node_cache}/${CONTAINER}"
local_container="${src_container}"
mkdir -p "${node_cache}" 2>/dev/null
(
    flock 9
    if [ ! -s "${staged}" ] || [ "$(stat -c%s "${staged}" 2>/dev/null)" != "$(stat -c%s "${src_container}")" ]; then
        tmp_copy="${staged}.tmp.$$"
        cp "${src_container}" "${tmp_copy}" 2>/dev/null && mv -f "${tmp_copy}" "${staged}" || rm -f "${tmp_copy}"
    fi
) 9>"${node_cache}/${CONTAINER}.lock"
if [ -s "${staged}" ] && [ "$(stat -c%s "${staged}" 2>/dev/null)" = "$(stat -c%s "${src_container}")" ]; then
    local_container="${staged}"
else
    echo "WARNING: staging to ${node_cache} failed (disk full?), using network image"
fi

If mount timeouts still appear (staging fell back to the network image), also:

cp: ... No space left on device when staging the image

Node-local /tmp filled up. On shared nodes /tmp is shared with other users’ jobs, so it can be nearly full before your copy starts. The scripts handle this gracefully (fall back to the network image), so the job does not hard-fail, but that job loses the staging benefit.

Python Environment

ModuleNotFoundError: No module named 'typer'

Resume & Output Checks

--resume does not skip any subjects

check-outputs reports unexpected failures

How-To Guides
Post-Run Verification and Reporting
Internals
DAG Execution & Preflight