Mastodon

Importing to Storm Buckets

Import copies data into Storm and runs the transfer on Storm's
infrastructure. The source can be one of your own Storm buckets on another
endpoint, alpha to montreal-1 for example, or a bucket on another
S3-compatible provider. Either way the source is never changed or deleted.

This is a one-time copy. To point your own tools at a bucket you already
have, see Connecting S3 Tooling. To pull on
a schedule, see Backing up to Storm Buckets.

Import lives under Tools > Import in the dashboard, and
as a shortcut next to New bucket on your overview.

The two Import entry points: Tools > Import in the sidebar, and the Import shortcut next to New bucket on the overview

Move your buckets to another endpoint

The usual case is an alpha member who just bought space on montreal-1 and
wants their alpha data there: montreal-1 is the paid endpoint, sold in 1 TB
blocks, and alpha gets node updates first.

Once you hold montreal-1 capacity, your overview offers to bring your alpha
buckets over. Open Import starts with every alpha bucket ticked, each
landing in a new montreal-1 bucket of the same name. Untick what to skip and
add the pairs. The card stays until that import finishes, and Dismiss
hides it in that browser.

The overview card offering to copy three alpha buckets to Montreal 1, with Open Import and Dismiss

To start one yourself, pick Your Storm buckets at Connect. There is
nothing to paste. Storm makes a read-only key on each source bucket and
deletes it when the import ends, or a day after an import you never start.
The toggle shows once your account holds a bucket.

Step 1, Connection, with Your Storm buckets selected

Pick each end the same way, an endpoint and then one of its buckets. The
destination is a new bucket or one you already have, and a new one takes the
source's name unless you type another. Only endpoints where you hold capacity
are offered for a new bucket. A restricted bucket cannot be a source, and a
pair whose two ends are the same bucket is refused.

Step 2, Buckets: wiki-uploads on Alpha picked as the source, flowing to a new wiki-uploads bucket on Montreal 1

Review measures every pair against the free storage on its destination
endpoint before anything moves.

Step 3, Review: one pair measured at 19.3 GB against 2.2 TB free on Montreal 1

What carries over

  • Objects. All of them.

  • CORS rules. Once a bucket's objects are copied, Storm reads the
    source's CORS rules and writes them to the
    destination. A destination with rules of its own keeps them. If
    Change CORS rules is closed in Control Gates,
    none are copied. If the read or the write fails, the import still
    finishes and the step tells you to set them yourself.

  • Keys stay behind. Each endpoint has its own. When the import finishes,
    a card shows the new endpoint URL and its signing region, with
    Create a key.

From another provider

Any S3-compatible provider works: MinIO, Backblaze B2, Wasabi, DigitalOcean
Spaces, Cloudflare R2, another Storm account, or a self-hosted Garage or Ceph
cluster.

What you provide Source endpoint, access key, secret, and the bucket name(s) to pull
What Storm keeps Nothing from your source key past the moment it starts the job
What happens on failure Whatever already transferred stays, and the step reports why
  1. Connect. Pick Another provider and enter the endpoint URL and an
    access key for the bucket you're importing from, the same values you'd
    hand to any S3-compatible tool. When the endpoint contains the region
    (Backblaze, Wasabi, AWS and most others put it there), Storm fills the
    region field for you. It stays editable.

    Step 1, Connection: source endpoint, region, and access key fields, with the key-custody note visible

  2. Choose buckets. Name the source bucket, then decide where it lands:
    a brand-new Storm bucket, or one you already have. Storm warns you before
    pairing with a bucket that already holds data.

    Step 2, Buckets: a source bucket paired with a new Storm bucket, and a second pair pointed at an existing bucket showing the non-empty warning

  3. Review. Every pair is measured against your available storage before
    anything moves, with how much you're over by if it doesn't fit.

    Step 3, Review: each pair measured against available storage, with the measured size and the fits verdict per pair

  4. Import. Storm transfers each pair and reports bytes moved, objects
    moved, and a specific reason if a step fails.

    Step 4, Import: a transfer mid-flight showing bytes moved, objects moved, transfer rate, and ETA

Changing the source

Until you start the import, Start over at Connect cancels it and lets you
pick again. Storm revokes any key it made for it.

Importing into an existing bucket

If the destination already holds data, Storm asks you to prove you own it
first: pick the bucket's admin key, and Storm mints a fresh, single-use key
for the transfer, then deletes it once the job is done. Your admin key's
secret only proves ownership. The transfer never runs on it.

An empty destination needs no proof. Select it from your list and Storm sets
up the single-use transfer key itself, as it does for a brand-new bucket.

The ownership check for an existing destination: pick the bucket's admin key before the transfer key is minted

Verifying the import

Once a bucket's import finishes, run a restore test. Storm samples real
objects back out of the destination, the largest one, the smallest one, and
one from each top-level folder, and lists what it checked with each
object's size.

A passed restore test listing the exact objects verified: the largest, the smallest, and one per top-level folder, each with its size


Create a Storm account, then request space
on montreal-1 or alpha access. Import is turned on per account. Once it's on
for yours, start an import.