Post-Run Verification and Reporting

Before starting, confirm your jobs are no longer running:

squeue -u $USER

If the queue is empty, proceed below. If jobs are still pending or running, wait for them to finish before verifying outputs.

After your SLURM jobs finish, three commands help you verify results and document what ran:

Steps 1 and 3 are the most common. Step 2 is only needed if the database looks incomplete.

Why you need both the database and check-outputs¶

After a run it is tempting to just query job_status and look for FAILED records. This catches some problems but misses others. Here is what each database status actually means and what it cannot tell you:

The critical gap is silent failure. An analysis tool can exit 0 while producing corrupt, empty, or partial outputs — fMRIPrep is a common example where certain failure modes produce an HTML report but no NIfTI files. The database records SUCCESS because the exit code was 0. Only check-outputs catches this by checking whether the expected files actually exist and meet minimum size requirements.

The practical rule:

• Database status tells you how the job ended — use it to find crashes and cancellations.

• check-outputs tells you whether the science succeeded — use it to confirm outputs are present and plausible.

• A subject needs to pass both: no FAILED/CANCELLED in the database and all check-outputs checks passing.

Step 1: Check Which Subjects Completed¶

Before generating any report, confirm which subjects actually have valid output files on disk. This is independent of SLURM job status — a job can exit successfully but still produce incomplete output.

neuropipe check-outputs \
  --project my_study \
  --work /data/work \
  --config-dir /data/config \
  --subjects 001,002,003,004,005 \
  --session 01

To check multiple sessions at once, pass them comma-separated:

neuropipe check-outputs \
  --project my_study \
  --work /data/work \
  --config-dir /data/config \
  --subjects 001,002,003,004,005 \
  --session 01,02

If your subject list is in a file:

neuropipe check-outputs \
  --project my_study \
  --work /data/work \
  --config-dir /data/config \
  --subjects subjects.txt \
  --session 01

To check only specific tasks rather than all configured tasks:

neuropipe check-outputs \
  --project my_study \
  --work /data/work \
  --config-dir /data/config \
  --subjects subjects.txt \
  --session 01 \
  --task rest_preprocess \
  --task volume

Reading the terminal output¶

The terminal shows only subjects with at least one failing check, grouped by task:

[check-outputs] Issues found:
  rest_preprocess: 002, 004
  volume: 004

This means subjects 002 and 004 are missing rest_preprocess outputs, and subject 004 is also missing volume outputs. Subjects not listed passed all checks.

A full CSV is saved automatically to {work_dir}/check_results_{timestamp}.csv. The CSV has one row per check item and tells you exactly which file was missing or too small — open it when the terminal summary is not enough to diagnose the problem.

What to do with the results¶

• Subjects that passed all checks → done, move to Step 3

• Subjects that failed → check the SLURM logs, fix the issue, then rerun those subjects (see Rerun Specific Subjects)

The check_results_*.csv file will also be used in Step 3 to overlay output validation status onto the HTML report.

Step 2: Sync the Database (if needed)¶

The pipeline writes JSONL logs during each job and you merge them manually afterwards with:

neuropipe merge-logs /data/work/my_study

This scans {work_dir}/database/json/ for unprocessed JSONL files, inserts them into pipeline_jobs.db, and moves processed files to archived/ subdirectories.

If the database path is not in the default location:

neuropipe merge-logs /data/work/my_study \
  --db-path /data/work/my_study/database/pipeline_jobs.db

After merging, check that the database looks right before generating the report:

sqlite3 /data/work/my_study/database/pipeline_jobs.db \
  "SELECT task_name, status, COUNT(*) FROM job_status GROUP BY task_name, status;"

To see all jobs from a specific pipeline run (using execution_id to link tables):

sqlite3 /data/work/my_study/database/pipeline_jobs.db \
  "SELECT j.subject, j.task_name, j.status, j.duration_hours
   FROM job_status j
   JOIN pipeline_executions p ON j.execution_id = p.execution_id
   WHERE p.id = 1
   ORDER BY j.task_name, j.subject;"

pipeline_executions.id is the SQLite row number shown by SELECT id, execution_time, command_line FROM pipeline_executions ORDER BY execution_time DESC LIMIT 5;.

If the database is missing records after a run¶

If jobs appear to have run but are not showing up in the database, the JSONL files are still in {work_dir}/database/json/. Run merge-logs to bring them in.

Rebuilding the database from scratch¶

If the database file is corrupted or you want a clean copy that includes all historical runs (including previously archived JSONL files), use force-rebuild:

neuropipe force-rebuild /data/work/my_study

This creates a new pipeline_jobs_rebuild_{timestamp}.db next to the original. The original database is never modified. The rebuild scans both active and archived JSONL files, so it recovers the full history.

Step 3: Generate the Report¶

Generate a standalone HTML report from the job database. --check-results is required — run check-outputs first (Step 1) to produce the CSV:

neuropipe generate-report \
  --db-path /data/work/my_study/database/pipeline_jobs.db \
  --project my_study \
  --session 01 \
  --check-results /data/work/my_study/check_results_20260401_120000.csv

The report is saved as pipeline_report_{project}_{timestamp}.html next to the database. To save it elsewhere:

neuropipe generate-report \
  --db-path /data/work/my_study/database/pipeline_jobs.db \
  --project my_study \
  --session 01 \
  --check-results /data/work/my_study/check_results_20260401_120000.csv \
  -o /data/reports/my_study_wave01.html

Including check-outputs results¶

The --check-results path must point to a check_results_*.csv produced by check-outputs. Pass the file saved in Step 1 directly:

What the report contains¶

The report is organised by session. Each session gets its own section in the navbar and contains all sub-sections below.

Putting It All Together¶

A typical post-run workflow for a completed session:

WORK=/data/work/my_study
DB=$WORK/database/pipeline_jobs.db
SUBJECTS=subjects.txt
SESSION=01
PROJECT=my_study

# 1. Verify outputs — saves check_results_<timestamp>.csv to $WORK
neuropipe check-outputs \
  --project $PROJECT \
  --work $WORK \
  --subjects $SUBJECTS \
  --session $SESSION

# 2. Sync the database (if needed)
neuropipe merge-logs $WORK

# 3. Generate the report — pass the CSV from Step 1 explicitly
CHECK_CSV=$(ls -t $WORK/check_results_*.csv | head -1)
neuropipe generate-report \
  --db-path $DB \
  --project $PROJECT \
  --session $SESSION \
  --check-results $CHECK_CSV \
  -o /data/reports/${PROJECT}_ses-${SESSION}_report.html

What the Database Looks Like¶

After a successful run and merge, the database contains records like these.

pipeline_executions — one row per neuropipe run call:

id | execution_id      | execution_time      | project   | session | status    | total_jobs
---|-------------------|---------------------|-----------|---------|-----------|----------
1  | 1746023412831     | 2026-04-30 09:03:32 | my_study  | 01      | COMPLETED | 150
2  | 1746109832145     | 2026-05-01 09:30:32 | my_study  | 01      | COMPLETED | 45

execution_id is a timestamp-based integer written into every JSONL file at submission time. It links this row to all jobs and wrappers from that run.

job_status — one row per subject per task:

execution_id   | subject | task_name        | status  | duration_hours | exit_code | node_name
---------------|---------|------------------|---------|----------------|-----------|----------
1746023412831  | 001     | rest_preprocess  | SUCCESS | 4.231          | 0         | node042
1746023412831  | 002     | rest_preprocess  | SUCCESS | 3.987          | 0         | node011
1746023412831  | 003     | rest_preprocess  | FAILED  | 0.041          | 1         | node017
1746023412831  | 001     | volume           | SUCCESS | 1.102          | 0         | node042

wrapper_scripts — one row per sbatch call (one per task per run, not per subject):

execution_id   | task_name       | job_id   | submission_time
---------------|-----------------|----------|---------------------
1746023412831  | rest_preprocess | 41693201 | 2026-04-30 09:03:35
1746023412831  | volume          | 41693202 | 2026-04-30 09:03:36

command_outputs — one row per subject per task, holds captured stdout/stderr (last 50 lines):

subject | task_name       | exit_code | job_id      | script_name
--------|-----------------|-----------|-------------|------------------------------
001     | rest_preprocess | 0         | 41693201_1  | afni_rest_preprocess.sh
002     | rest_preprocess | 0         | 41693201_2  | afni_rest_preprocess.sh
003     | rest_preprocess | 1         | 41693201_3  | afni_rest_preprocess.sh

Common Scenarios¶

“I just want to see which subjects have missing outputs”¶

check-outputs is the fastest way to confirm whether the expected files are on disk, without writing any SQL:

neuropipe check-outputs \
  --project my_study \
  --work /data/work \
  --config-dir /data/config \
  --subjects subjects.txt \
  --session 01

To narrow down to specific tasks:

neuropipe check-outputs \
  --project my_study \
  --work /data/work \
  --config-dir /data/config \
  --subjects subjects.txt \
  --task rest_preprocess \
  --task volume

“I want to see exit codes and error messages from the database”¶

Query the database directly for jobs that crashed (non-zero exit code):

sqlite3 /data/work/my_study/database/pipeline_jobs.db \
  "SELECT subject, task_name, status, error_msg
   FROM job_status
   WHERE status = 'FAILED'
   ORDER BY task_name, subject;"

To see all non-successful jobs at once:

sqlite3 /data/work/my_study/database/pipeline_jobs.db \
  "SELECT subject, task_name, status, error_msg
   FROM job_status
   WHERE status != 'SUCCESS'
   ORDER BY task_name, subject;"

“I want to share the report with someone”¶

Use -o to save to a path outside the work directory:

neuropipe generate-report \
  --db-path /data/work/my_study/database/pipeline_jobs.db \
  --project my_study \
  --session 01 \
  --check-results /data/work/my_study/check_results_20260401_120000.csv \
  -o ~/Desktop/my_study_report.html

The HTML file is fully standalone — pure HTML and inline CSS, no external dependencies. No server or extra files needed.

“My database seems incomplete after a run”¶

# Merge any unprocessed JSONL logs
neuropipe merge-logs /data/work/my_study

# Or rebuild from scratch (including archived logs) into a new file
neuropipe force-rebuild /data/work/my_study