Handling spot interruptions

Working with a dynamic spot market means that your spot bids may get preempted if your bid is no longer winning. In order to work efficiently with changes in the spot market, we recommend implementing preemption and resumption handlers that will automatically take action during the 5-minute preemption and relocation windows.

Checking for a preemption

When a spot bid is preempted, the status of all instances within the bid changes to STATUS_PREEMPTING.

Previously, to check the instance status within the instance, the user had to generate an API key, embed it within an instance, and deploy a service within the instance to poll the instance status API endpoint.

As of February 2026, users can now simply check the /opt/mithril/MITHRIL_SIGNAL.yml file for the instance status. If the file is not present, the instance is not being preempted, and no instances in the order are being relocated 🎉

If the file is present, check the contents for:

• instance_status: The status of the particular instance

  • STATUS_PREEMPTING indicates that the instance is being preempted, and will be stopped at the end_time provided in the file

  • STATUS_RELOCATING indicates that the instance will be relocated at the end_time provided in the file. This means that the instance will stop and restart.

• bid_status: The status of the instance's bid

  • Relocating indicates that one or more of the instances in the order will be relocated at the end_time provided in the file.

• end_time: The time at which the instance will terminate, or another relocating instance will stop and restart.

Note: it is possible for the bid to be re-allocated during the 5-minute preemption window if the market price changes. In this case, the Mithril Signal file will be removed. We recommend saving your work when a preemption occurs without killing the workload, so that your workloads may resume immediately if the bid is reallocated during the preemption window.

Preemption

In the basic preemption case, the Mithril Signal file in each instance of the preempted bid will be as follows:

instance_status: STATUS_PREEMPTING
bid_status: Allocated
end_time: 2026-02-12 15:47:08.751581-08:00

Relocation

You have a 2-instance order called magnificent-leopard, of which magnificent-leopard-1 is detected to have a GPU error. Mithril support will relocate magnificent-leopard-1, so the Mithril Signal file on each instance will show:

## on magnificent-leopard-1
instance_status: STATUS_RELOCATING
bid_status: Relocating
end_time: 2026-02-12 15:47:08.751581-08:00

## on magnificent-leopard-2
instance_status: STATUS_RUNNING
bid_status: Relocating
end_time: 2026-02-12 15:47:08.751581-08:00

Note that the instance status for magnificent-leopard-2 is not relocating. The end_time in this case refers to the time at which magnificent-leopard-1 will be relocated.

However, you may still wish to take action on both instances in this case, for example by pausing your workload until the relocating instance resumes.

Handling preemptions

Here's a snippet of how to check the Mithril Signal file for preemption:

## Preemption check
def is_preempting() -> bool:
    status = ''
    try:
        with open('/opt/mithril/MITHRIL_SIGNAL.yml', 'r') as f:
            status = f.read()
    except FileNotFoundError:
        return False
    return "STATUS_PREEMPTING" in status

## Later, in your training loop, use preempting() to save a checkpoint on demand:
if is_preempting():
    save_checkpoint()

Resuming workloads

If market prices drop and your spot bid wins, instances will be reallocated. We recommend automating the resumption of jobs when instances restart in order to take full advantage of the time an instance is allocated.

To automatically start jobs, we recommend writing a systemd service that will execute on restart, and adding it to the startup script, which executes only once, when an instance is launched for the first time. The startup service will start each time an instance boots up.

Check the following documentation for more information: Running scripts on subsequent startups after relocation, preemption, or resuming a spot bid