Slurm Cluster¶

The GPU cluster is the primary compute surface of PRESCIENT. You log in to the submission node, allocate GPUs on one of the GPU clusters through Slurm, and your workload runs there. You do not SSH directly to a GPU node — the scheduler places your job for you.

Clusters¶

There are two independent GPU clusters. Each runs its own scheduler but they share one accounting database, so your account and limits are identical on both.

Cluster (`-M`)	GPUs	Per-GPU memory	Good for
`core-hgx`	8 × NVIDIA H100 (SXM5, NVLink)	80 GB	large models, multi-GPU / tensor-parallel, FP8
`edge-lmd-1`	8 × NVIDIA L40S (PCIe)	46 GB	single-GPU jobs, inference, smaller training

/home and /shared are NFS-served from core-hgx, so they look identical on both clusters and on the submission node. Pick a cluster per job with -M core-hgx or -M edge-lmd-1 (the H100s on core-hgx are the default if you omit -M).

Logging in¶

ssh <name.#>@slurm-head.prescient.internal      # VPN required

slurm-head is the submission node — it has the Slurm client but no GPUs. You stage files, submit jobs, and monitor them from here.

You cannot SSH straight to core-hgx or edge-lmd-1 unless you already hold an active allocation on that node (enforced by pam_slurm_adopt — it guarantees GPU isolation between users). Submit your work from slurm-head and Slurm runs it on the GPU node. The one time you do connect to a compute node directly is to open a tunnel to a service running inside your own allocation.

On login you'll see the cluster MOTD — a short banner plus a system-information block.

Allocating GPUs¶

Interactive shell (`srun`)¶

The most common entry point — a shell with GPUs attached, running on the compute node:

srun -M core-hgx --gres=gpu:1 --time=2:00:00 --pty bash

Flag	Meaning
`-M core-hgx` / `-M edge-lmd-1`	Which cluster to run on.
`--gres=gpu:N`	Allocate N GPUs.
`--time=HH:MM:SS` or `--time=D-HH:MM:SS`	Wall-clock limit.
`--pty bash`	Returns an interactive shell.

The shell is adopted into the job's cgroup on the compute node and sees only the GPUs you were allocated. When you exit, the allocation ends. To survive a disconnect, start tmux before srun, or use salloc (below).

Detached allocation (`salloc`)¶

salloc reserves resources and returns a job ID without attaching a shell, so the allocation outlives your terminal:

salloc -M core-hgx --gres=gpu:1 --time=8:00:00      # prints a job ID
srun --jobid=<id> --pty bash                        # join it (repeatable)

Batch jobs (`sbatch`)¶

sbatch -M core-hgx my-job.sbatch                    # returns a job ID, then detaches

#!/bin/bash
#SBATCH --job-name=train
#SBATCH --gres=gpu:2
#SBATCH --time=12:00:00
#SBATCH --output=logs/%j.out

python train.py

sbatch is the right tool for anything long-running or detached — the job proceeds on the compute node independently of your session.

Containers run on a compute node — but you launch them from slurm-head

Pyxis is installed on slurm-head as well as the compute nodes, so an srun --container-image=… issued directly from the login node works. Slurm dispatches the container to a compute node and attaches your allocated GPUs. You can equally run container steps through sbatch (the script body executes on the node) or from inside an interactive allocation. See Containers.

Checking and cancelling jobs¶

myjobs                    # alias: squeue --me  (your queued / running jobs)
myhistory                 # your jobs over the last week
sacct -M core-hgx -X --starttime today
scancel <jobid>           # cancel a job and release its GPUs
gpu-status                # cluster-wide snapshot across both clusters

scancel is how you "kill a reservation" — it stops the job and frees the GPUs immediately. To cancel everything you have running on a cluster: scancel -M core-hgx -u <name.#>.

Surviving disconnects¶

Two reliable patterns:

tmux + srun. Start tmux on slurm-head first, then run srun --pty bash inside the tmux session. Detach with Ctrl-b then d. The allocation continues; reattach later with tmux attach.
salloc + reattach. salloc returns a job ID that survives your session; srun --jobid=<id> --pty bash rejoins it at any time.

Never wrap a long-running service (e.g. a vLLM server) in a bare interactive srun over SSH without tmux — a dropped connection kills the allocation. For services, prefer sbatch (see Common Workflows).

Your environment and helper commands¶

A handful of helpers are pre-installed (most are aliases from /etc/profile.d/); they work from slurm-head and the compute nodes alike:

Command	What it does
`gpu-status`	Cluster-wide snapshot — node state and queued jobs across both clusters (`sinfo`/`squeue -M all`). No SSH needed; works for everyone.
`myquota`	Your storage usage across `/home`, `/cache`, `/scratch`, plus the shared-dataset quota.
`myjobs`	Your current Slurm jobs (`squeue --me`).
`myhistory`	Your jobs over the last week.
`gpus-here`	`nvidia-smi` on the current host (the whole node, not just your allocation).

Environment defaults are set for you on login: HF_HOME/HF_HUB_CACHE for the HuggingFace cache, and pip/uv caches plus ~/.cache are redirected to local NVMe (see Storage).

Monitoring a running job¶

Scheduler view, from anywhere: gpu-status (both clusters), myjobs.
Live GPU + process view, inside an allocation: nvtop (installed on the compute nodes) gives a top-like view of per-GPU utilization, memory, and the processes using each GPU. nvidia-smi and gpus-here give one-shot snapshots.
Peek at a job's GPUs without taking new ones — attach a throwaway step to an existing allocation from slurm-head:

srun -M core-hgx --overlap --jobid=<id> nvidia-smi

--overlap shares the job's existing GPUs rather than requesting more.

You only see your own processes

/proc is mounted with hidepid=2, so ps, top, and nvtop show only your processes even when you share a node with other users. This is a privacy/security measure, not a malfunction.

Storage¶

PRESCIENT uses tiered storage. Pick the right tier for the data lifetime — putting hot data on /home is the most common cause of "why is my training slow."

Path	Backing	Shared across hosts	Lifetime	Use for
`/home/$USER`	NFS from `core-hgx` (RAIDZ2 vault)	✓	persistent, quota'd	code, configs, important results, papers
`/shared/hf-cache`	NFS from `core-hgx` (RAIDZ2 vault)	✓	persistent	community HuggingFace cache
`/cache/users/$USER`	local NVMe (stripe, no redundancy)	✗	persistent, no backup	Python venvs, working datasets, model outputs
`/cache/hf`	local NVMe	✗	persistent	per-host fast HF cache
`/scratch/jobs/$SLURM_JOB_ID`	local NVMe, set as `$TMPDIR`	✗	deleted at job end	intermediate files within a single job
`/scratch/users/$USER`	local NVMe	✗	persistent, self-managed	fast scratch you clean up yourself

Storage rules of thumb¶

Slow-but-redundant → fast-but-at-risk as you move down the list. /home and /shared live on a RAIDZ2 pool (survives a disk failure); /cache and /scratch are single-disk stripes with no redundancy — a disk failure loses that pool. Keep authoritative copies on /home or upstream.
/home is NFS over 1 GbE. Do not load multi-gigabyte model weights or run a Python venv from there hot-path — use /cache for working data.
Per-user directories auto-create on first login and at job-prolog time (/cache/users/$USER, /scratch/jobs/$JOBID). You don't need to ask an admin.
Quotas apply on /home by Slurm account; myquota shows your usage. Ask the operators if you need yours raised.
/scratch/jobs/$SLURM_JOB_ID is wiped when the job ends — copy anything you want to keep to /cache or /home before exit.

HuggingFace cache¶

Two environment variables are pre-set in your shell:

HF_HOME=/shared/hf-cache        # shared community cache (NFS, on core-hgx)
HF_HUB_CACHE=/cache/hf          # per-host local cache, checked first

The HuggingFace library checks the local /cache/hf first; on a hit it loads fast off local NVMe. On a miss the download lands in the shared /shared/hf-cache, so the next person who needs that model gets it without re-downloading. Writing into /shared/hf-cache requires membership in the shared-cache group — ask the operators if you need it.

For a gated or private model, point HF_HOME at your own home directory so your token and weights stay private:

HF_HOME=~/.cache/huggingface HF_TOKEN=<your-token> vllm serve <private-model>

Containers (Enroot + Pyxis)¶

Containers are the preferred way to run anything with a heavy dependency stack, and the only supported way to get the CUDA toolkit (nvcc) — the GPU nodes ship the NVIDIA driver but no host CUDA toolkit. Enroot is rootless (no Docker daemon, no setuid) and Pyxis hooks it into Slurm so containers run inside your allocation.

Run a container in an allocation¶

srun -M core-hgx --gres=gpu:1 --time=2:00:00 \
     --container-image=docker://vllm/vllm-openai:latest \
     nvidia-smi -L

--container-image accepts:

Docker Hub references (docker://vllm/vllm-openai:latest).
NVIDIA NGC or other OCI registries (nvcr.io/...).
A pre-imported squashfs file (faster repeat launches).

Pyxis is installed on slurm-head as well as the compute nodes, so you can issue srun --container-image=… directly from the login node — Slurm runs the container on a compute node with your allocated GPUs. sbatch and interactive allocations work too.

Host caches inside containers¶

A container inherits neither your login shell's environment nor the host's filesystem. Two consequences:

/cache and /shared are not visible inside the container unless you bind them in with --container-mounts. (Your home is unmounted too — ENROOT_MOUNT_HOME=no; add /home:/home if you need it.)
HF_HOME / HF_HUB_CACHE are not set inside the container. The login defaults (above) do not cross the boundary, so HuggingFace falls back to an empty in-container cache and re-downloads weights on every run.

To use the shared model cache from inside a container, mount the cache paths and re-export the HF variable in the container command:

srun -M core-hgx --gres=gpu:1 \
     --container-image=docker://vllm/vllm-openai:latest \
     --container-mounts=/cache:/cache,/home:/home \
     bash -c "export HF_HUB_CACHE=/cache/hf; vllm serve <model>"

With this, a model already in /cache/hf (or /shared/hf-cache) loads straight from local NVMe; without it, every job re-downloads into ephemeral container storage that vanishes when the job ends.

This is separate from Enroot's own state

Enroot's per-user runtime/cache/data under /cache/users/$USER/enroot/ are on local NVMe automatically (see below) — that's the container runtime state on the host. The mount + HF_HUB_CACHE export above are about what the code inside the container can see. You need both.

Pre-import once, reuse fast¶

Every srun --container-image=docker://… re-imports (re-unpacks ~10 GB) of the image. For repeated launches, import once to a squashfs on local /cache and point every run at that file:

enroot import -o /cache/users/$USER/enroot/vllm-openai.sqsh \
    docker://vllm/vllm-openai:latest

srun -M core-hgx --gres=gpu:1 \
     --container-image=/cache/users/$USER/enroot/vllm-openai.sqsh \
     --pty bash

Enroot's per-user runtime, cache, and data live under /cache/users/$USER/enroot/ — local fast NVMe, not NFS — by default fleet-wide. The .sqsh persists across jobs, so later runs skip the import entirely.

For a worked end-to-end example (serving an LLM and reaching it from your laptop), see Common Workflows.

What you cannot do¶

No sudo. Researchers cannot use sudo on any node (HBAC gates the sudo service itself). All elevated needs are met inside containers; if you genuinely need something at the host level, open an issue.
No SSH to a compute node without an active allocation.
No system-wide CUDA toolkit. There is no host nvcc. Get CUDA from your container image (the vllm/vllm-openai image bundles the full toolchain) or from pinned Python wheels.