Common Workflows¶

Practical, end-to-end recipes for things people actually do on PRESCIENT. These build on the basics in Slurm Cluster — allocation, storage, and containers — and assume you're logged in to slurm-head.

For now these cover the Slurm clusters. Kubernetes workflows will be added here once that cluster is open.

Serve an LLM with vLLM¶

Stand up an OpenAI-compatible inference server in a container and reach it from your laptop. vLLM runs in the official vllm/vllm-openai image, which bundles the full CUDA toolchain — so it works on the GPU nodes (driver-only) with no host setup.

Because a server is long-running and should outlive your session, use sbatch rather than an interactive srun. Create serve-vllm.sbatch:

#!/bin/bash
#SBATCH --job-name=vllm
#SBATCH --gres=gpu:1
#SBATCH --cpus-per-task=8
#SBATCH --mem=64G
#SBATCH --time=08:00:00
#SBATCH --output=%x-%j.log

MODEL="microsoft/phi-4"          # any ungated HF repo, or a local path
PORT=8000

# A container sees neither your login environment nor the host filesystem, so
# the HF cache must be made available explicitly: mount /cache into the container
# (--container-mounts below) AND set HF_HUB_CACHE *inside* it (the bash -c export).
# Without both, weights re-download into ephemeral container storage every run.
# See "Host caches inside containers" in the Slurm guide for the full rule.
export HF_HUB_CACHE=/cache/hf

srun --container-image=docker://vllm/vllm-openai:latest \
     --container-mounts=/cache:/cache,/home:/home \
     --no-container-entrypoint \
     bash -c "export HF_HUB_CACHE=$HF_HUB_CACHE; \
              exec vllm serve $MODEL --host 0.0.0.0 --port $PORT"

Submit it and watch startup:

sbatch -M core-hgx serve-vllm.sbatch     # returns a job ID, then detaches
myjobs                                    # wait for state R (running)
tail -f vllm-<jobid>.log                  # model download + load progress

The first run downloads the image (~10 GB) and the model weights — allow 15–30 minutes. Later runs reuse the cached .sqsh and weights and start in a few minutes. The server is ready when the log prints Application startup complete and http://localhost:8000/health returns 200 on the node.

Pre-import the image for faster repeat launches

Import the container once to a squashfs on /cache and point --container-image at that file instead of docker://… — see Pre-import once, reuse fast. Subsequent jobs skip the import entirely.

Reach a service from your laptop¶

The node firewall denies inbound connections except SSH, so the server's port isn't directly reachable. But because you hold an allocation on the node, pam_slurm_adopt lets you SSH in — and an SSH tunnel forwards the port over that connection. From your VPN-connected laptop:

ssh -N -L 8000:localhost:8000 <name.#>@core-hgx.prescient.internal

Leave that running and hit http://localhost:8000/v1/models (or point any OpenAI-compatible client at http://localhost:8000/v1). Closing the tunnel does not affect the job; reconnect any time.

When you're done, release the GPUs:

scancel <jobid>          # on slurm-head

Multi-model chat UI

For a richer setup — several models behind a single OpenWebUI chat interface, with health checks and a one-command status helper — see the demo/ directory in the infrastructure repo (serve.sh / status.sh / tunnel.sh). It's the same pattern as above, scaled to multiple GPU-pinned backends.

Sizing guidance¶

Each H100 provides 80 GB, each L40S 46 GB. Rough weight footprints: ~2 GB per billion parameters at FP16, ~1 GB at FP8, ~0.5 GB at 4-bit. Reserve roughly 20–40% of the GPU for the key-value cache; lower --max-model-len if a model approaches the memory limit. One GPU per model; add GPUs (and raise --gres) with --tensor-parallel-size N for models that don't fit on one.

Python without containers¶

Containers are the preferred path, but a pure-Python project (no CUDA compilation) can run from a virtual environment. uv is installed system-wide at /usr/local/bin/uv and resolves far faster than pip for ML stacks.

# Inside an allocation (or on slurm-head for CPU-only setup):
uv venv /cache/users/$USER/venvs/myenv --python 3.12
source /cache/users/$USER/venvs/myenv/bin/activate
uv pip install <packages>

Put the venv on /cache, never /home. NFS lookups for every import make a /home venv painfully slow to start. /cache/users/$USER is local NVMe.
No host nvcc. Anything that compiles CUDA kernels at install or run time (Triton's compiler, flashinfer, DeepGEMM) cannot build on the host — use a container for those. Pure-Python packages and prebuilt native wheels are fine.

Kubernetes workflows¶

Coming soon — workflows for the PRESCIENT Kubernetes cluster will be documented here once it is open to users. See Kubernetes Cluster Access.