> For the complete documentation index, see [llms.txt](https://docs.mithril.ai/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.mithril.ai/compute-and-storage/persistent-storage.md). # Persistent storage A guide to where your data should live on Mithril, and why. If you only take away one thing: keep your source of truth in an object store, and use Mithril storage for the working set your GPUs touch during a run. {% hint style="info" %} Mithril does not charge for ingress, egress, or network bandwidth. You pay only for the storage you provision. Pricing is at [mithril.ai/pricing](https://mithril.ai/pricing). {% endhint %} While each instance comes with its own ephemeral storage (see [Instance types & specifications](/compute-and-storage/instance-types-and-specifications.md)), that storage is wiped on preemption or when the order is paused. Persistent storage gives you a fast, colocated working set that survives instance pauses and stops. It's the right place for data your GPUs actively use during a run and is especially valuable when: * You'd otherwise pay egress costs re-streaming data from a third-party provider on every run. * Your workloads would be accelerated by colocating data with your GPUs. * You're running on spot instances and need checkpoints that survive preemption within a run. Mithril offers two options for persistent storage:
File sharesPreformatted, shared storage that mounts automatically. Best for most use cases./pages/RdprnL0Yl6B4vLsfAyjc/pages/RdprnL0Yl6B4vLsfAyjc
Block storageA raw disk volume with full filesystem control. An advanced, single-instance option./pages/ARzw2YePZXNz7DMDVBC9/pages/ARzw2YePZXNz7DMDVBC9
### The two-tier model Most ML teams need two different kinds of storage and get the most out of Mithril when they keep them separate. **Tier 1: your source of truth.** Raw datasets and final checkpoints that need to live indefinitely. This belongs in an **object store** (Cloudflare R2, Amazon S3, Google Cloud Storage, Hugging Face Storage Buckets). It's cheap per gigabyte, durable, and you read from it once at the start of a run rather than continuously during it. **Tier 2: your working set.** The data your GPUs actually read and write during a run. This belongs on **Mithril persistent storage**; either a file share or block volume colocated with your instances on a fast network. It only needs to hold what one run or project needs, and only for as long as the run or project lasts. The job of your data pipeline is simple: **pull from the object store when a run starts; write results back when it finishes.** Everything in between happens on Mithril storage. {% hint style="danger" %} Mithril persistent storage is not a backup service. File shares and block volumes are not replicated across regions; if a region has an incident, your working set can be unavailable. Keep your durable source of truth in an object store. {% endhint %} ### Why an object store for source of truth Object stores are the right home for data that has to outlive individual runs. They're durable, cheap per gigabyte, and decouple your data's lifetime from any single instance or region so your source of truth persists whether or not you have compute running or a run is in flight. Mithril persistent storage is built for the opposite job: fast, colocated access during a run. The two are complementary, which is why the two-tier model pairs them. Once you've decided your source of truth lives in an object store, the next question is which one and here the economics matter. Mithril doesn't charge for moving data in or out, but your object store provider probably does. Egress from S3, GCS, or Azure Blob typically runs several cents per gigabyte. If you pull a multi-terabyte dataset onto a fresh working set at the start of every run (a natural pattern on ephemeral GPU compute) those charges add up fast. **Cloudflare R2 and Hugging Face Storage Buckets charge nothing for egress.** Storage cost is comparable to S3, but data leaves for free: to Mithril, to your laptop, to another cloud. If you're choosing an object store from scratch and expect to pull data more than once, R2 is usually the most economical option. If you're already established on S3 or GCS, they work fine; just be aware of the egress bill. To pull object-store data onto an instance using the Mithril CLI, use cloud bucket mounts (`file_mounts`) in your task YAML. See [Data & storage](/mithril-cli/task-yaml/data-and-storage.md). ### Choosing between file shares and block storage #### File shares: the default choice Use a **file share** when more than one instance needs to read or write the same files: multi-node training with shared checkpoints, a dataset several instances read from, or anything where you'd otherwise copy data to N machines and manage consistency yourself. * Readable and writable from **multiple instances concurrently** * **Read-optimized** (you'll typically see higher read than write speeds) * Auto-mounted at `/mnt/` on instances that select it * Maximum size 32TB; resize by contacting your account team This is the right choice for most workloads. See [File shares](/compute-and-storage/persistent-storage/file-shares.md). #### Block storage: the advanced choice Use **block storage** when you want a raw disk and full control over the filesystem on a single instance. * Readable and writable from **only one instance at a time**. Concurrent read-write mounts will corrupt the filesystem. * Requires manual setup: format, mount, and add to `fstab` yourself If you need shared storage, use a file share instead. See [Block storage](https://claude.ai/compute-and-storage/persistent-storage/block-storage.md). {% hint style="warning" %} Block volumes cannot be shrunk or expanded after creation. Size them correctly up front. File shares, by contrast, can be resized on request. {% endhint %} #### Ephemeral storage: free scratch space Every instance comes with NVMe SSD ephemeral storage mounted at `/mnt/local`, at no cost. Use it for scratch work, caches, and shuffle buffers. See [Ephemeral storage](/compute-and-storage/ephemeral-storage.md). {% hint style="warning" %} Ephemeral storage at `/mnt/local` is wiped on preemption, relocation, or when the order is paused. Don't keep anything there you need. It is reserved for ephemeral use, so don't mount persistent volumes to that path. {% endhint %} ### Quick decision guide | Use case | Recommended storage option | | --------------------------------------------------------- | ----------------------------- | | Raw datasets and final checkpoints that live indefinitely | Object store (R2 recommended) | | Shared files across multiple instances or nodes | File share | | Fast per-instance persistent disk, single writer | Block storage | | Temporary scratch, caches, shuffle buffers | Ephemeral (`/mnt/local`) | ### Region availability Storage is region-scoped: a volume can only be attached to instances in the same region.
RegionGPUFile sharesBlock storage
us-central2-aH100✓✓
us-central2-bH200✓✓
us-central3-aA100✗✓
us-central5-aB200✓✓
File shares are not currently available in the A100 region (us-central3-a). ### Attaching storage to existing instances Storage selected when you create a reservation or spot bid is mounted automatically. To add storage to an instance that's already running, the instance must be in a **Paused** state, and volumes attached this way won't auto-mount — you'll mount them manually.
Mounting a file share added to a running instance If the instance has already booted when you attach a new file share, mount it in the standard format with: ```bash FILESHARE_NAME= sudo mkdir /mnt/$FILESHARE_NAME sudo chmod 777 /mnt/$FILESHARE_NAME echo "$FILESHARE_NAME /mnt/$FILESHARE_NAME virtiofs defaults,nofail 0 1" | sudo tee -a /etc/fstab sudo mount -a ``` See [File shares](/compute-and-storage/persistent-storage/file-shares.md) for full details.
Mounting a block volume added to a running instance Block volumes require formatting on first use and manual mounting. The full sequence (finding the device name, formatting, mounting, and persisting across reboots) is documented on the [Block storage](/compute-and-storage/persistent-storage/block-storage.md) page.
### Next steps * [File shares](/compute-and-storage/persistent-storage/file-shares.md): provisioning, mounting, performance benchmarks * [Block storage](/compute-and-storage/persistent-storage/block-storage.md): raw disk setup and filesystem control * [Data & storage (task YAML)](/mithril-cli/task-yaml/data-and-storage.md): mounting object-store buckets and volumes in your runs * [Ephemeral storage](/compute-and-storage/ephemeral-storage.md): instance-local scratch specs --- # Agent Instructions This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com. ## Querying This Documentation If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question. Perform an HTTP GET request on the current page URL with the `ask` query parameter, and the optional `goal` query parameter: ``` GET https://docs.mithril.ai/compute-and-storage/persistent-storage.md?ask=&goal= ``` `ask` is the immediate question: it should be specific, self-contained, and written in natural language. `goal` is optional and describes the broader end goal you are ultimately trying to accomplish on behalf of the user. GitBook uses it to tailor the answer towards what is most useful for that goal. The response will contain a direct answer to the question and relevant excerpts and sources from the documentation. Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.