Skip to content
Back to

Backup and restore

Fontana separates the recovery questions that used to be answered by one mechanism, because they fail in different ways and on different timescales:

Question Path
“The new release is wrong.” Release rollback: pin the previous source.tag and run fontana apply
“We lost data, or a volume, or a cluster.” Nightly backups in your bucket, taken while everything keeps running, and fontana restore
“We want the workspace as plain files, or a copy of it on another install.” fontana export, and fontana restore --from to bring an export back
“Someone changed a file and we want the old one.” Restore Points and the version history of the namespace in Fontana Files

Keeping them separate means an upgrade never has to stop your workspace to be safe, and a backup window is not a maintenance window.

Backups run online

Section titled “Backups run online”

Every workspace runs a nightly backup job inside its own Kubernetes namespace. It exports the running workspace and uploads the set to an S3-compatible bucket you own. Nothing is stopped and no page is shown to your users.

Two backup modes
Section titled “Two backup modes”

Fontana Files holds every record your workspace authors: configuration, people, teams, and roles, agents, skills, projects, workflow runs, conversations, and Knowledge Graph settings. The backend’s data is built from those files. That is why two artifacts can restore a workspace, its people, teams, and roles included (see People, teams, and roles after a restore), and why the backup has two modes, chosen with remoteStore.backupComponents:

What is protected How full (default) files-and-vault
Fontana Files One versioned bundle per Fontana Files namespace, with its full history, Restore Points, and review sessions Yes Yes
Secrets A logical export of the workspace’s Vault Yes Yes
Relational data Logical dumps of every Postgres database in the workspace: the backend, identity, and lineage Yes Rebuilt from Fontana Files
Uploaded files and deployed functions Copied from the backend’s storage volume right after the database dump, so the two always agree Yes Rebuilt from Fontana Files
Knowledge Graph A point-in-time snapshot of the graph database Yes Each graph reloads from its namespace’s files when first opened
Run manifest Written last, with a checksum for every artifact and the backup mode. A run without it is incomplete and is never counted for retention Yes Yes

Stay on full until you have rehearsed a restore of your workspace the way a two-component run restores (see Restore). Switching is then one line in fontana.yaml and one fontana apply. A files-and-vault run restores onto the install it came from; to recover into a new install, restore a full run.

A run takes well under a minute on a typical workspace. Each run lands under its own timestamped prefix: <backupPrefix><workspace>/<timestamp>/.

Declaring the bucket and retention
Section titled “Declaring the bucket and retention”

Backups are configured in fontana.yaml and switched on by the presence of remoteStore. Without it, no backup job is created.

remoteStore:
  kind: s3
  endpoint: https://s3.example.com
  region: eu-central-1
  bucket: acme-fontana-backups
  backupPrefix: backups/ # optional
  backupComponents: full # optional: full (default) or files-and-vault


snapshots:
  schedule:
    daily: "02:00" # UTC
  retain:
    daily: 7 # nightly runs kept
    monthly: 12 # months whose first run is kept

The bucket credentials are environment variables in the shell that runs fontana apply, never values in the file. Apply fails rather than deploy a job that cannot upload, and it accepts only full or files-and-vault as the backup mode.

Retention runs inside the job, not as bucket lifecycle rules, so the policy lives in fontana.yaml with the same vocabulary on every install and works on any S3-compatible endpoint. After each run the job keeps the newest retain.daily runs plus the first run of each of the newest retain.monthly months, and removes the rest.

Restore

Section titled “Restore”

A restore is one command, run by your platform team from the workstation that runs fontana apply:

Terminal window
fontana restore <workspace> --run <timestamp> --yes

Leave out --run to take the newest complete backup. The command then does the whole recovery inside the cluster:

  1. The target’s secret store is checked first: it must be unsealed, and its unseal material must be in place. Otherwise the restore stops before anything changes.
  2. Every service in the workspace is paused so nothing writes while the data is replaced.
  3. The backup run is downloaded and every artifact is checked against the manifest’s checksums before any data store is touched. A run whose manifest is missing never finished and is refused.
  4. The data is put back in the run’s mode:
    • full: the databases are replayed from their dumps, the uploaded files and deployed functions are placed back on the backend’s storage volume, the secret store’s contents are written back, and the graph snapshot and the Fontana Files namespaces are placed on their volumes.
    • files-and-vault: the Fontana Files namespaces and your workspace’s secrets are put back, and the graph database is cleared so each Knowledge Graph reloads from its namespace’s files. The target install keeps its own install credentials and identity service.
  5. The services resume at their previous scale, and an apply rebuilds the backend’s data from Fontana Files, once per restore. It records a single restore entry in the Activity log rather than one entry per restored item. The same health check that follows fontana apply must pass.

The --yes flag is required because a restore replaces the workspace’s data with the chosen point in time.

To rehearse the two-component path before you switch the nightly job, restore any run, including a full one, the way a two-component run restores:

Terminal window
fontana restore <workspace> --run <timestamp> --files-and-vault --yes

Signing in afterwards. A full restore replays the identity database, so signed-in users sign in again. A files-and-vault restore keeps the install’s identity service, so nothing changes for your users.

Because a restore puts data back into the same layout that fontana apply creates, recovering into a rebuilt cluster is the same procedure: provision, apply, restore a full run.

Export a workspace as plain files

Section titled “Export a workspace as plain files”

fontana export writes a backup run to a folder on your workstation, so you can inspect, archive, or move a workspace with nothing but files:

Terminal window
fontana export <workspace> <dir> --run <timestamp>

Leave out --run to take the newest complete backup. The export reads a verified backup run, never the live workspace, so it is a consistent point in time and costs the running workspace nothing. Every artifact is checked against the manifest’s checksums first. The folder holds:

Path Content
ffs/ The complete Fontana Files data: every namespace with its full version history, Restore Points, and review sessions. fontana restore --from reads this folder
by-name/<namespace> A readable link into ffs/: the namespace’s current files, under its name. Large files hold their content, not a pointer
vault-kv.json The workspace’s secrets, readable only by your user
export.json The workspace, the backup run, the export time, and the namespaces it contains

Keep the folder together, and copy or archive it with symbolic links preserved: by-name only points into ffs/. The folder contains your secrets in usable form: store it like the backup bucket. The secret store’s unseal material is not part of an export.

Restore from an export

Section titled “Restore from an export”
Terminal window
fontana restore <workspace> --from <dir> --yes

The command packs the folder into a new backup run under the target workspace’s prefix in your bucket, then restores it like any other run, in files-and-vault mode, with the same checks and the same rebuild. Use it to bring an archived export back, or to copy a workspace to another install. People, teams, and roles come with the files: each person signs in on the target install and is matched to their own account (see People, teams, and roles after a restore).

Release rollback

Section titled “Release rollback”

Releases are immutable and content-addressed, and every installed release stays on disk beside the current one. To go back, point source.tag at the earlier release in fontana.yaml and run fontana apply. Workloads roll back the same way they rolled forward: a replacement starts, passes its readiness check, and only then does the current pod retire.

Schema changes are written to accept both the previous and the current shape for one release, and migrations run idempotently during apply. That is what makes a rollback safe to perform without restoring data.

A failed rollout does not need a manual rollback at all: if the new pods never become ready, the old pods keep serving. See Fontana CLI.

When to use which path

Section titled “When to use which path”
Scenario Recommended path
Undo a bad release Pin the previous source.tag, run fontana apply
A release that never became healthy Nothing to do: the rollout stalls and the old pods keep serving
Recover workflow run history, uploads, or a database Restore from the nightly backup
Recover an entire cluster Provision, fontana apply, restore from the nightly backup
Keep a workspace as plain files, or copy it to another install fontana export, then fontana restore --from
Recover secrets, workspace secrets, and BYOK The secret-store export in the nightly backup (both modes), restored into an install whose secret store is unsealed
Change control before a planned change Manual export, and an on-demand run of the backup job if you want a fresh point in time

Upgrades and change control

Section titled “Upgrades and change control”

Production deploys use immutable release tags, so every upgrade is reproducible and auditable, and the pinned tag in fontana.yaml is the record of what ran. Because upgrades do not stop the workspace and backups do not stop it either, your change window is about approval and verification rather than downtime: apply the release, watch fontana status, and validate in the workspace.

Section titled “Related documentation”