Runtime Model
High-Level Model
slotd is a single-host scheduler. The runtime model is intentionally simple:
- one local daemon
- one local SQLite database
- one local execution host
- one local user workflow
There is no controller/worker split and no remote node launch protocol.
Core Resources
slotd schedules three resource types:
- CPU
- memory
- GPU
Current behavior:
- CPU reservation is
ntasks * cpus-per-task ntaskslaunches one local process per task rank for batch and foreground execution- total memory defaults to host-detected
MemTotalfrom/proc/meminfo, with a16384 MBfallback - memory is stored in MB
- GPUs are integer slots
- admission is reservation-based, not usage-based
- if
SLOTD_CGROUP_BASEis unset, CPU and memory remain reservation-only - if
SLOTD_CGROUP_BASEis set to a writable cgroup v2 subtree,slotdwritesmemory.maxandcpu.max - if cgroup setup fails after explicit configuration, launch fails instead of silently skipping enforcement
Partitions
Configured by environment:
SLOTD_CPU_PARTITIONSSLOTD_GPU_PARTITIONS
Rules:
- only configured partition names are accepted
- if there are no GPUs, no GPU partition is exposed
- if a GPU partition is selected and
--gpusis omitted, the default GPU request is1 - otherwise the default GPU request is
0 - CPU and GPU partitions are virtual views over one local host
- CPU and memory capacity stay shared across partitions; only GPU visibility/defaults differ by partition
GPU Detection
If SLOTD_GPU_COUNT is not set, slotd tries to detect GPUs from nvidia-smi.
The current implementation checks:
nvidia-smi/usr/bin/nvidia-smi/usr/lib/wsl/lib/nvidia-smi/bin/nvidia-smi
Job Types
Persisted records are one of:
- top-level batch jobs
- allocation-only jobs
- array tasks
- steps under allocations
Job States
Implemented states:
PENDINGRUNNINGCOMPLETINGCOMPLETEDFAILEDCANCELLEDTIMEOUTOUT_OF_MEMORY
Terminal states:
COMPLETEDFAILEDCANCELLEDTIMEOUTOUT_OF_MEMORY
Scheduling Rules
The daemon loop runs every 300ms.
Pending jobs are blocked by:
- dependencies
- array concurrency limits
- delayed start time
- exclusive host use
- insufficient reserved resources
- user hold state
Ordering:
- submission order is the base rule
- explicit job priority can override pure submission order
- array tasks are interleaved by array group
Runtime Files
Within SLOTD_ROOT:
run/slotd.sock: daemon socketlib/state.db: SQLite statelib/jobs/<job_id>/script.sh: batch scriptlib/jobs/<job_id>/runner.sh: daemon wrapperlib/jobs/<job_id>/exit_status: wrapper exit status
Notifications
If SLOTD_NOTIFY_CMD is set, slotd runs it for terminal top-level jobs.
Exported variables:
SLOTD_JOB_IDSLOTD_JOB_NAMESLOTD_JOB_STATESLOTD_JOB_PARTITIONSLOTD_JOB_REASON