← Lab Notes

SLURM patterns for molecular dynamics jobs

~6 min read · HPC, SLURM, job scripting

A LAMMPS run that dies at hour 47 of a 48-hour walltime limit, with no checkpoint, is the single most common way I've seen cluster time get wasted. None of what follows is exotic — it's just discipline that's easy to skip under deadline pressure.

1. Always request a checkpoint-friendly walltime, and use it

Write restart files on a schedule independent of when the job happens to end:

restart 50000 restart.*.data
# or, for a single rolling restart file:
restart 50000 restart.a.data restart.b.data

The alternating restart.a/restart.b pattern matters: if the job is killed mid-write to one file, the other still has a valid, complete restart point. A single restart file being overwritten when the job is killed is how you lose an entire run's progress in the last five minutes.

2. Chain long runs with job dependencies, not one giant walltime request

Clusters schedule short-to-medium jobs far more readily than one job asking for the maximum walltime. Instead of requesting 5 days upfront, chain segments:

#!/bin/bash
#SBATCH --job-name=lammps-md
#SBATCH --time=12:00:00
#SBATCH --nodes=1
#SBATCH --ntasks-per-node=32
#SBATCH --output=slurm-%j.out

module load lammps/2Aug2023

if [ -f restart.a.data ]; then
    srun lmp -in in.continue -var restart_file restart.a.data
else
    srun lmp -in in.equil
fi

# resubmit self if the simulation isn't finished yet
if [ ! -f production.done ]; then
    sbatch "$0"
fi

Each segment finishes cleanly within its walltime, writes a restart file, and resubmits itself. You get shorter queue waits per segment and a checkpoint between every one.

Detecting "finished" without babysitting the queue

Have the LAMMPS input script print a sentinel file (production.done) as its last command, and check for that file's existence in the submit script — not the job's exit code, which will read as "success" even for a run that finished early because it hit an unrelated error partway through.

3. Match the resource request to what the simulation actually uses

• MPI ranks vs. atoms: below roughly 2,000–5,000 atoms per MPI rank, LAMMPS spends more time on communication overhead than computation — requesting more cores doesn't linearly help below that floor. Profile with -in in.test -log none on a short run before committing to a large allocation.
• Memory: GCMC/hybrid runs with a growing number of guest molecules need more headroom than a fixed-composition MD run of the same atom count — pad the memory request, since an out-of-memory kill mid-run is much more disruptive than a wasted allocation.
• Don't request GPUs for GCMC-heavy hybrid runs unless your LAMMPS build's GPU package actually accelerates the fix you're using (fix gcmc is CPU-bound in most builds) — you'll just be paying queue-priority cost for hardware nothing is using.

4. Keep a job log outside the scheduler

SLURM's own accounting (sacct) disappears into the noise once you've run hundreds of jobs across a few projects. A one-line append per submission — job ID, input file, git commit hash of the input scripts, one-line intent — turns "which run was the humid MOF-177 case again?" from an afternoon of archaeology into a grep.