Detect idle instances
Last updated
Automatically pause your bid when your instances become idle to prevent unnecessary spend.
Pausing your spot bid releases the instances and stops billing but keeps the cluster configuration and boot disk so it can resume later.
Add autostop to your task YAML:
resources:
# Pause bid after 10 minutes of inactivity.
autostop: 10mEnable autostop:
ml sky autostop my-cluster -i 10Disable autostop:
ml sky autostop my-cluster --cancelA cluster is considered idle when:
No jobs are pending or running (check ml queue)
No active SSH sessions exist
Once the cluster remains idle for the configured duration, a daemon inside the VM automatically pauses the bid.
You can change what counts as "activity" using wait_for.
jobs_and_ssh (default)
Wait for jobs in pending/running state and SSH sessions to finish
jobs
Ignore SSH sessions (useful if you keep long-running IDE/SSH connections open)
none
The bid will pause after the configured time regardless of cluster activity.
The none option is useful for enforcing a hard time limit, even if workloads like Jupyter notebooks are still running
You can specify commands to run before your bid is paused.
The hook runs on the cluster and has access to its filesystem and environment variables.
If the hook exits with a non-zero status, autostop will still proceed, but a warning will be logged.
By default, autostop hooks have a 1 hour timeout.
If the hook exceeds this limit, it will be terminated and autostop will continue.
Note that your bid will not be paused until the hook finishes or times out, so choose an appropriate hook_timeout.
The autostop functionality is inherited from the SkyPilot project. This page is largely adapted from the SkyPilot autostop documentation.
Last updated
resources:
autostop:
idle_minutes: 10
wait_for: jobs_and_sshresources:
autostop:
idle_minutes: 10
hook: |
cd /path/to/project
git add .
git commit -m "Commit my code"
git push
hook_timeout: 300