Migrate user guide

Overview

Migrate is The One Stack's cloud migration and on-prem archival product. The endpoint side of Migrate is the Migrate module of The One Agent — The One Agent is installed once per device and the Migrate module is enabled per device — and it streams files and mailboxes from a source system to a destination, covering on-prem to cloud and on-prem archival. Cloudflare R2 is used as zero-egress staging so staging adds no egress fees. Progress is tracked batch-by-batch, credentials stay encrypted at rest, and the volume transferred is measured so you can see it inside Migrate.

Key Features

• Clients, projects, and jobs with a full lifecycle
• Agents: registration, heartbeat, and command polling with per-agent keys
• Progress tracking that's idempotent by batch ID (so retries never double-count)
• Cloudflare R2 staging with pre-signed upload URLs and 7-day retention
• Credentials vault with AES-256-GCM encryption and an agent-fetch delivery flow
• Static connector catalog covering 15+ source types and 8+ destinations
• Reports with CSV export
• Settings per org
• Hub-connected team management with role hierarchy (owner, admin, technician, viewer)
• API keys and audit log
• Usage metering: a per-GB transfer summary computed hourly (a record of volume moved, not a charge)
• Agent key generation and revocation
• Background workers: stall detection, agent offline checker, staging cleanup, billing rollup — with leader-election locking

Supported Sources and Destinations

Sources: local drive, SMB share, NFS share, OneDrive, SharePoint, Google Drive, Google Workspace Drive, Dropbox, Box, Egnyte. Mailboxes: Gmail, Google Workspace Mail, Exchange Online, Exchange on-prem, PST files, local Outlook, IMAP.

Destinations: Cloudflare R2, Azure Blob, OneDrive, SharePoint, Exchange Online, SMB share. Google Workspace Mail and Google Drive are supported sources only — Migrate refuses either as a destination at job creation with 422 destination_engine_not_implemented (see the correction note at the top of this page).

How To: Plan a Cloud Migration

1. Open Migrate and go to Clients > New. Register the client whose data is moving.
2. Go to Projects > New. Tie the project to the client. A project groups related jobs (one per workload).
3. Go to Jobs > New for each workload: a file migration from an SMB share to OneDrive, a mailbox move from Exchange on-prem to Exchange Online, an archival run from a local drive to R2, and so on.
4. For each job, pick a job type: archive, migrate_files, migrate_mailbox, or sync. Two further types are catalogued — discovery_scan and validation — but nothing runs them today (see the correction note at the top of this page).
5. Pick source and destination connectors. Invalid source-destination-job-type combos are rejected — only realistic pairs are allowed.
6. Pick the source and destination credentials from the vault. If you don't have credentials yet, go to Credentials first and add them (encrypted).

How To: Install and Register an Agent

1. On the customer machine, install The One Agent and enable the Migrate module for that device. There is no separate Migrate installer — the endpoint side of Migrate is that module.
2. In Migrate, go to Agent Keys > Generate. Copy the key.
3. Paste the key into the agent. The agent hits POST /agents/register with the key, gets a hostname-bound record, and starts sending heartbeats.
4. The agent shows up under Agents with status Online, busy, or offline based on the last heartbeat. AGENT_OFFLINE_THRESHOLD_MS controls how long before offline.

How To: Track a Workload Migration

1. Open the job. Status moves through Queued, Running, to Completed or Failed.
2. Click Progress to see real-time counts: files scanned, uploaded, verified, errored, bytes transferred.
3. Progress ingestion is batch-id idempotent — if the agent retries a batch after a network blip, counts don't inflate.
4. If the job stalls (no progress for 20+ minutes while running for at least 5), Migrate raises a stall event and writes it to the job's audit trail — no message is pushed from it. The cron enforces a 2-hour cooldown per job to prevent spam.
5. Use Rerun, Pause, or Cancel to manage the job. Rollback and Resume are not implemented: the Rollback control is present but disabled and the API refuses rollback_job with 501 not_implemented, and a paused job is left Partial with no Resume control — Re-run continues from the saved checkpoint. See Rollback.

How To: Use R2 Staging

1. R2 is the default primary staging — zero-egress costs let you move data through without hidden fees.
2. The agent hits POST /staging/upload-url for each file and gets a pre-signed PUT URL (default 2-hour TTL). Larger batches use /staging/upload-urls-batch.
3. After upload, Hub (or a second agent) streams files from R2 into the actual destination.
4. 7 days after a job reaches a terminal state (completed, failed, cancelled), staging_cleanup_scheduled_at expires and the staging cleanup cron deletes the R2 prefix.
5. Check /staging/stats for how much is in staging right now; /staging/cleanup can be called manually if needed.

How To: Manage Credentials Safely

1. Go to Credentials > New.
2. Enter the provider (OneDrive, SharePoint, Exchange, Google Workspace, etc.) and the credentials required.
3. Migrate encrypts the credential blob with AES-256-GCM using a per-org 32-byte key and stores it.
4. When a job runs, the agent calls POST /credentials/agent-fetch with its job ID and the credential IDs it needs. Migrate decrypts and returns them (rate-limited to 10 fetches per hour per agent).
5. Every credential fetch is logged to the audit log.

How To: Track Archival Jobs

1. Archival is just a job type (archive) with a source and a destination (typically Azure Blob or R2).
2. Set up the job as normal. The agent reads from source, uploads to destination, and optionally deletes from source if delete_after_upload is enabled.
3. Archival is a one-time job like any other — no second pass is ever scheduled for any Migrate job. A Re-run resumes from the job's saved checkpoint: items already transferred are skipped and items that failed are retried.

Usage and billing

Migrate isn't sold at launch, so there's no Migrate pricing or billing yet. Migrate does measure the volume each job moves (you'll see it in job progress and reports). Treat that figure as a record of what moved, not as a charge.

How To: Manage Your Team

1. Go to Team. Owner and admin roles have full access; technician can create and run jobs and view reports but can't change team, billing, or settings; viewer is read-only.
2. Invite users by email; roles are set at invite time.

Frequently Asked Questions

Why R2 staging instead of direct source-to-dest? Cloudflare R2 has zero egress fees and is geographically agnostic. Staging lets Migrate buffer, verify, and retry without rebuffering from the source. Plus R2 lets any agent (not just the one that uploaded) pull down and write to destination.

What happens if an agent goes offline mid-job? Heartbeats stop and the offline checker flips the agent to offline after the threshold. Jobs with in-flight batches pause; when the agent comes back online, it picks up from the next batch by ID.

How do I prevent double-counting on retries? Every batch has a batch_id. If the agent retries a batch, Migrate sees the duplicate and returns 200 without re-applying counters. That's the idempotency story.

Can I delete data from the source after migration? Yes. Set delete_after_upload on the job, but that's destructive — test with a small batch first.

How are credentials protected? AES-256-GCM with a 32-byte key validated at startup. The key lives in environment secrets; the encrypted payload lives in the database. Decryption only happens on agent-fetch with job scope.

What if I need to move from one cloud tenant to another cloud tenant in the same provider? Not today. A job whose source and destination are both cloud connectors resolves to server-side execution and is refused at creation with 422 server_execution_coming_soon — so a cloud-to-cloud move such as OneDrive tenant A to OneDrive tenant B is not available yet. A migration runs when at least one endpoint is a local, network-share or IMAP endpoint.

Can I schedule a job to run regularly? No. No second pass is ever scheduled for any Migrate job — there is no recurring schedule. Job type sync runs the same enumerate-and-transfer as migrate_files. A Re-run resumes from the job's saved checkpoint: items already transferred are skipped and items that failed are retried. A file changed at the source after it was transferred is not re-transferred, because the checkpoint records it as done. A job can be scoped when it is created with a source filter (newer_than, older_than, max_age_days, file_types), which narrows what a run enumerates.

Troubleshooting

Agent won't register. The agent key may be wrong, the IP may be rate-limited (20 registrations per hour per IP), or the agent key was revoked. Generate a fresh key.

Job stuck in Queued. Check the agent is Online. If online, look at the job's config — invalid source/destination combos reject.

Credentials fetch returns 403. The agent is asking for credential IDs that aren't tied to its job, or the fetch rate limit is exceeded (10/hour/agent). Wait or check the job-credential link.

Staging URLs failing. R2 credentials (CLOUDFLARE_ACCOUNT_ID, R2_ACCESS_KEY_ID, R2_SECRET_ACCESS_KEY) aren't set. Ask an admin to configure them.

Staging cleanup didn't run. The cleanup cron runs daily at 03:00 UTC. Jobs are only cleaned up after staging_cleanup_scheduled_at expires (7 days after terminal state by default).

Stall event repeated within 2 hours. It shouldn't — the cron enforces a 2-hour cooldown per job. If you're seeing repeats, confirm the cron-leader lock is working.

Audit log missing entries. Audit writes are fire-and-forget — if the DB was briefly unavailable, an entry could be lost. The actual action still succeeded.