Persistent storage
A guide to where your data should live on Mithril, and why. If you only take away one thing: keep your source of truth in an object store, and use Mithril storage for the working set your GPUs touch during a run.
Mithril does not charge for ingress, egress, or network bandwidth. You pay only for the storage you provision. Pricing is at mithril.ai/pricing.
While each instance comes with its own ephemeral storage (see Instance types & specifications), that storage is wiped on preemption or reboot. Persistent storage gives you a fast, colocated working set that survives instance pauses and stops. It's the right place for data your GPUs actively use during a run and is especially valuable when:
You'd otherwise pay egress costs re-streaming data from a third-party provider on every run.
Your workloads would be accelerated by colocating data with your GPUs.
You're running on spot instances and need checkpoints that survive preemption within a run.
Mithril offers two options for persistent storage:
The two-tier model
Most ML teams need two different kinds of storage and get the most out of Mithril when they keep them separate.
Tier 1: your source of truth. Raw datasets and final checkpoints that need to live indefinitely. This belongs in an object store (Cloudflare R2, Amazon S3, Google Cloud Storage, Hugging Face Storage Buckets). It's cheap per gigabyte, durable, and you read from it once at the start of a run rather than continuously during it.
Tier 2: your working set. The data your GPUs actually read and write during a run. This belongs on Mithril persistent storage; either a file share or block volume colocated with your instances on a fast network. It only needs to hold what one run or project needs, and only for as long as the run or project lasts.
The job of your data pipeline is simple: pull from the object store when a run starts; write results back when it finishes. Everything in between happens on Mithril storage.
Mithril persistent storage is not a backup service. File shares and block volumes are not replicated across regions; if a region has an incident, your working set can be unavailable. Keep your durable source of truth in an object store.
Why an object store for source of truth
Object stores are the right home for data that has to outlive individual runs. They're durable, cheap per gigabyte, and decouple your data's lifetime from any single instance or region so your source of truth persists whether or not you have compute running or a run is in flight. Mithril persistent storage is built for the opposite job: fast, colocated access during a run. The two are complementary, which is why the two-tier model pairs them.
Once you've decided your source of truth lives in an object store, the next question is which one and here the economics matter. Mithril doesn't charge for moving data in or out, but your object store provider probably does. Egress from S3, GCS, or Azure Blob typically runs several cents per gigabyte. If you pull a multi-terabyte dataset onto a fresh working set at the start of every run (a natural pattern on ephemeral GPU compute) those charges add up fast.
Cloudflare R2 and Hugging Face Storage Buckets charge nothing for egress. Storage cost is comparable to S3, but data leaves for free: to Mithril, to your laptop, to another cloud. If you're choosing an object store from scratch and expect to pull data more than once, R2 is usually the most economical option. If you're already established on S3 or GCS, they work fine; just be aware of the egress bill.
To pull object-store data onto an instance using the Mithril CLI, use cloud bucket mounts (file_mounts) in your task YAML. See Data & storage.
Choosing between file shares and block storage
File shares: the default choice
Use a file share when more than one instance needs to read or write the same files: multi-node training with shared checkpoints, a dataset several instances read from, or anything where you'd otherwise copy data to N machines and manage consistency yourself.
Readable and writable from multiple instances concurrently
Read-optimized (you'll typically see higher read than write speeds)
Auto-mounted at
/mnt/<name>on instances that select itMaximum size 32TB; resize by contacting your account team
This is the right choice for most workloads. See File shares.
Block storage: the advanced choice
Use block storage when you want a raw disk and full control over the filesystem on a single instance.
Readable and writable from only one instance at a time. Concurrent read-write mounts will corrupt the filesystem.
Requires manual setup: format, mount, and add to
fstabyourself
If you need shared storage, use a file share instead. See Block storage.
Block volumes cannot be shrunk or expanded after creation. Size them correctly up front. File shares, by contrast, can be resized on request.
Ephemeral storage: free scratch space
Every instance comes with NVMe SSD ephemeral storage mounted at /mnt/local, at no cost. Use it for scratch work, caches, and shuffle buffers. See Ephemeral storage.
Ephemeral storage at /mnt/local is wiped on preemption, relocation, or reboot. Don't keep anything there you need. It is reserved for ephemeral use, so don't mount persistent volumes to that path.
Quick decision guide
Raw datasets and final checkpoints that live indefinitely
Object store (R2 recommended)
Shared files across multiple instances or nodes
File share
Fast per-instance persistent disk, single writer
Block storage
Temporary scratch, caches, shuffle buffers
Ephemeral (/mnt/local)
Region availability
Storage is region-scoped: a volume can only be attached to instances in the same region.
us-central2-a
H100
✓
✓
us-central2-b
H200
✓
✓
us-central3-a
A100
✗
✓
us-central5-a
B200
✓
✓
File shares are not currently available in the A100 region (us-central3-a).
Attaching storage to existing instances
Storage selected when you create a reservation or spot bid is mounted automatically. To add storage to an instance that's already running, the instance must be in a Paused state, and volumes attached this way won't auto-mount — you'll mount them manually.
Mounting a file share added to a running instance
If the instance has already booted when you attach a new file share, mount it in the standard format with:
See File shares for full details.
Mounting a block volume added to a running instance
Block volumes require formatting on first use and manual mounting. The full sequence (finding the device name, formatting, mounting, and persisting across reboots) is documented on the Block storage page.
Next steps
File shares: provisioning, mounting, performance benchmarks
Block storage: raw disk setup and filesystem control
Data & storage (task YAML): mounting object-store buckets and volumes in your runs
Ephemeral storage: instance-local scratch specs
Last updated