# Handling spot interruptions

Working with a dynamic spot market means that your spot bids may get preempted if your bid is no longer winning. In order to work efficiently with changes in the spot market, we recommend implementing preemption and resumption handlers that will automatically take action during the 5-minute preemption and relocation windows.

### Checking for a preemption

When a spot bid is preempted, the status of all instances within the bid changes to `STATUS_PREEMPTING`. 

Previously, to check the instance status within the instance, the user had to generate an API key, embed it within an instance, and deploy a service within the instance to poll the instance status API endpoint.

**As of February 2026, users can now simply check the `/opt/mithril/MITHRIL_SIGNAL.yml` file for the instance status.** If the file is not present, the instance is not being preempted, and no instances in the order are being relocated :tada:

If the file is present, check the contents for:

* `instance_status`: The status of the particular instance
  * `STATUS_PREEMPTING` indicates that the instance is being preempted, and will be stopped at the `end_time` provided in the file
  * `STATUS_RELOCATING` indicates that the instance will be relocated at the `end_time` provided in the file. This means that the instance will stop and restart.
* `bid_status`: The status of the instance's bid
  * `Relocating` indicates that one or more of the instances in the order will be relocated at the `end_time` provided in the file.
* `end_time`: The time at which the instance will terminate, or another relocating instance will stop and restart.

{% hint style="info" icon="triangle-exclamation" %}
**Note:** it is possible for the bid to be re-allocated during the 5-minute preemption window if the market price changes. In this case, the Mithril Signal file will be removed.\
\
We recommend saving your work when a preemption occurs without killing the workload, so that your workloads may resume immediately if the bid is reallocated during the preemption window.
{% endhint %}

#### Preemption

In the basic preemption case, the Mithril Signal file in each instance of the preempted bid will be as follows:

```yml
instance_status: STATUS_PREEMPTING
bid_status: Allocated
end_time: 2026-02-12 15:47:08.751581-08:00
```

#### Relocation

You have a 2-instance order called `magnificent-leopard`, of which `magnificent-leopard-1` is detected to have a GPU error. Mithril support will relocate `magnificent-leopard-1`, so the Mithril Signal file on each instance will show:

```yml
## on magnificent-leopard-1
instance_status: STATUS_RELOCATING
bid_status: Relocating
end_time: 2026-02-12 15:47:08.751581-08:00
```

```yml
## on magnificent-leopard-2
instance_status: STATUS_RUNNING
bid_status: Relocating
end_time: 2026-02-12 15:47:08.751581-08:00
```

Note that the instance status for `magnificent-leopard-2` is *not* relocating. The `end_time` in this case refers to the time at which `magnificent-leopard-1` will be relocated.

However, you may still wish to take action on both instances in this case, for example by pausing your workload until the relocating instance resumes.

### Handling preemptions

Here's a snippet of how to check the Mithril Signal file for preemption:

```py
## Preemption check
def is_preempting() -> bool:
    status = ''
    try:
        with open('/opt/mithril/MITHRIL_SIGNAL.yml', 'r') as f:
            status = f.read()
    except FileNotFoundError:
        return False
    return "STATUS_PREEMPTING" in status

## Later, in your training loop, use preempting() to save a checkpoint on demand:
if is_preempting():
    save_checkpoint()
```

### Resuming workloads

If market prices drop and your spot bid wins, instances will be reallocated. We recommend automating the resumption of jobs when instances restart in order to take full advantage of the time an instance is allocated.

To automatically start jobs, we recommend writing a `systemd` service that will execute on restart, and adding it to the startup *script*, which executes only once, when an instance is launched for the first time. The startup *service* will start each time an instance boots up.

Check the following documentation for more information: [Startup scripts](/compute-and-storage/startup-scripts.md#running-scripts-on-subsequent-startups)


---

# Agent Instructions: Querying This Documentation

If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question.

Perform an HTTP GET request on the current page URL with the `ask` query parameter:

```
GET https://docs.mithril.ai/compute-and-storage/spot-bids/handling-spot-interruptions.md?ask=<question>
```

The question should be specific, self-contained, and written in natural language.
The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.

Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.