The deploy kit¶

For operators

This page is the deploy kit's own README (kit/ in the Cerea repository), included here at build time so the site and the kit never drift apart. It is the runbook: first run, presets, upgrades, backups, the identity provider, break-glass and the rest. Deploying is the map of how the three deployment paths relate; this page is the kit itself.

The README below is served from the same source the kit ships: https://github.com/paoloviviani/Cerea/blob/stable/kit/README.md. One Cerea release tag, vX.Y.Z, is the kit's version too.

The Cerea deploy kit¶

Run Cerea on your own server: the chat (Cerea), the model gateway and admin console (Pystino), a bundled identity provider (Authelia), the reverse proxy with automatic TLS (Caddy), and optional add-ons: PII redaction, a headless browser for web fetch, and agent machines (/chat/code).

Everything is in this directory (kit/ in the Cerea repository), and you can read all of it before running anything: one compose.yaml, one documented .env.example, the proxy's caddy/Caddyfile, the IdP's authelia/configuration.yml, and ./configure, a single standard-library Python script that writes .env for you.

Licence: Apache-2.0, Copyright 2026 Paolo Viviani (LICENSE, NOTICE). This kit contains no third-party code; see AI-DISCLOSURE.md for how it was written. The kit's version is Cerea's: one release tag vX.Y.Z of the repository is one tested kit.

Third-party software¶

This kit contains no third-party code. It references stock images you pull from their publishers under their own licences: Authelia (Apache-2.0), Caddy (Apache-2.0), PostgreSQL with pgvector (PostgreSQL Licence), Valkey (BSD-3-Clause) and MongoDB 8 or 4.4 (Server Side Public License v1, run here as Cerea's internal database, not offered as a service). The optional office profile runs Gotenberg (MIT) with LibreOffice (MPL-2.0 and others) inside it, unmodified. The optional backup sidecar's image, built on your machine from tools/backup/, copies in the official restic (BSD-2-Clause) and rclone (MIT) binaries and installs Alpine's PostgreSQL client (PostgreSQL Licence), MongoDB Database Tools (Apache-2.0) and supercronic (MIT). The Cerea and Pystino images carry their own NOTICE files.

First run¶

You need Docker Engine with Compose 2.24 or newer, Python 3.9 or newer (for ./configure only), and a DNS name pointing at the server with ports 80 and 443 open (or a TLS front forwarding to it: see TLS modes).

curl -fsSL https://raw.githubusercontent.com/paoloviviani/Cerea/stable/kit/get-kit.sh | sh
cd cerea/kit
./configure
docker compose up -d

get-kit.sh is a short POSIX shell script that needs only git. It fetches the current release of this directory (the one the stable branch points at), and nothing else from the repository, into ./cerea (a shallow, sparse git checkout of a few hundred kilobytes), and prints the commands to run next. You work in cerea/kit: that is where ./configure, .env and every docker compose command live. Prefer to read it first? Download it, read it, then run sh get-kit.sh.

sh get-kit.sh --help
sh get-kit.sh --version vX.Y.Z --dir /opt/cerea    # a release other than the newest, and another place

Follow a release tag or the stable branch, never main: stable moves to a release only after that release has been installed fresh and signed in to, and main may name images that are not published yet.

The images are public on the GitHub Container Registry (ghcr.io/paoloviviani/cerea, pystino-gateway, pystino-redaction); Compose pulls the versions this release pins, with no login.

./configure asks for your origin (https://cerea.example.org), the administrator's email, a preset, the TLS mode and the identity provider. It generates every secret and writes .env at mode 0600. The first sign-in for the bundled Authelia goes to first-sign-in.txt (mode 0600). Sign in at https://<your origin>/ (the console; the chat is at /chat/), then delete that file.

Prefer to edit by hand? Run cp .env.example .env && chmod 600 .env, then fill it in. Every option is explained there, and every secret names the command that generates it. Then run ./configure --check.

For scripted installs, every question has a flag:

./configure --non-interactive --origin https://cerea.example.org \
  --admin-email ops@example.org --preset team --tls acme --idp authelia

Presets¶

Preset	Runs	For
`homelab`	gateway, console, chat, local document reader; no ledger or quotas	one person or a family
`team`	homelab + ledger, quotas, pattern-based PII redaction	a team (the default)
`enterprise`	team + NER redaction (built locally) and headless-browser fetch	an organisation
`satellite`	the chat only, against a central Pystino and its IdP	a second site
`generic`	the chat only, against any OpenAI-compatible endpoint	no gateway at all

A preset only chooses compose profiles and feature switches in .env. You can change either afterwards.

Pystino mode or standalone mode, never both. Every preset but generic runs the chat in Pystino mode: each person's own token to the gateway, and no shared key (CHAT_OPENAI_API_KEY stays empty). generic is standalone mode: one shared key to one endpoint, with personal tokens off. ./configure writes only one of the two, ./configure --check reports a mix as an error, and ./configure --set refuses to create one. The chat itself refuses to start on a mix (Configuration). An .env from an older install that mixes them: run ./configure, which says so and offers the fix.

Upgrading¶

cd cerea/kit
sh get-kit.sh --upgrade            # the current release; --version vX.Y.Z for another
docker compose pull
docker compose up -d

--upgrade fetches the release and checks it out. It never touches a file git does not track: .env, compose.override.yaml, proxy.d/*.caddy, first-sign-in.txt and your backups stay as they are. (From elsewhere, name the checkout: sh get-kit.sh --upgrade --dir ~/cerea.) It stops, without changing anything, if you edited a file git tracks: put such changes in compose.override.yaml or proxy.d/ instead.

Without the script, the same thing by hand, from inside cerea/kit:

git fetch --depth 1 --filter=blob:none origin tag vX.Y.Z
git checkout vX.Y.Z

Each release pins the image versions it was tested with, inside compose.yaml, so you never type one. Back up first (see Backup and restore).

An upgrade lands on a tested pair of versions: check the CHANGELOG, then run the commands. The CHANGELOG also says when a release needs something more, such as re-running the install line on each agent machine.

Moving from a `cerea-deploy` install¶

The kit used to be its own repository, cerea-deploy. If you installed it by cloning that, get-kit.sh --from moves you to the new layout without losing anything:

curl -fsSL https://raw.githubusercontent.com/paoloviviani/Cerea/stable/kit/get-kit.sh |
  sh -s -- --from /path/to/cerea-deploy --dir /path/to/cerea     # installs; changes nothing in the old directory
cd /path/to/cerea-deploy && docker compose down                  # step 1, yours: stop the old stack, WITHOUT -v
cd /path/to/cerea/kit && docker compose up -d --wait             # step 2, yours: start it from the new place

--from copies the old install's .env (and its .env.bak-*), compose.override.yaml, proxy.d/*.caddy and first-sign-in.txt, and never runs docker. Docker names the volumes after the compose project, so the new stack finds the old data only if the project name is the same: it keeps the COMPOSE_PROJECT_NAME in your .env (./configure always writes one), and for an .env without one it writes the name the old compose.yaml defaulted to (cerea), or failing that the old directory's name. Check the line it prints before step 2. Backup directories next to the old kit are not copied.

Only files git tracks are replaced by the new release: a local edit to caddy/Caddyfile or another tracked file is not carried over (--from warns if there is one). Move such changes into compose.override.yaml or proxy.d/*.caddy first.

Prefer a clean start? Stop the old stack with docker compose down -v (this deletes its volumes: every conversation, account and key), install fresh as above, and run ./configure again. Keep the old directory until the new stack works.

What runs¶

Service	Profile	What it is
`proxy`	always	Caddy (stock image) with `caddy/Caddyfile`: TLS, and the routes `/chat`, `/authelia`, everything else to the gateway
`postgres`	always	PostgreSQL with pgvector: the gateway's data and the chat's knowledge store
`bootstrap`	always	a one-shot on every `up`: creates the chat's database role and extensions, and the bundled Authelia's signing key and first user if absent; it never overwrites anything
`gateway`, `migrate`, `valkey`	`gateway`	Pystino: the OpenAI-compatible API at `/v1` and the admin console
`chat`, `chat-mongo`	`chat`	Cerea at `/chat`
`documents`	`chat`	the document service (`cerea-documents`): creates and edits Word, Excel, PowerPoint and PDF files for the chat's `doc_*` tools. On with the chat; internal network only (see below)
`authelia`	`authelia`	the bundled identity provider at `/authelia` (stock image, `authelia/configuration.yml`)
`redaction`	`redaction`	PII redaction for the gateway
`extractor`	`documents`, `redaction`	the local document reader (Word, Excel, PowerPoint and old `.doc` files, scanned-PDF pages as images); `homelab` runs it through `documents`, without redaction
`playwright`	`fetch`	a headless browser for the chat's web fetch. On in the `enterprise` preset; on any other preset, add it later with `./configure --fetch && docker compose up -d`, and remove it with `./configure --no-fetch`. Boxed in: its own `fetch` network shared only with the chat, Chromium's sandbox on, read-only filesystem, no capabilities, limits (browser)
`gotenberg`	`office`	Office files as PDF previews in the chat, and the `convert_document` tool for models (Gotenberg: LibreOffice and Chromium). Off in every preset; add it with `./configure --office-preview && docker compose up -d`, remove it with `--no-office-preview`
`rustfs`	`rustfs`	RustFS, an S3-compatible object store that holds the chat's files (attachments, generated files, uploads, previews) instead of MongoDB. Off in every preset; add it with `./configure --object-store && docker compose up -d`. A fresh install uses it at once; an existing one keeps its files in MongoDB until they are copied (Storage). Your own S3 works without the profile

Every service names an image and nothing builds on up. COMPOSE_PROFILES in .env chooses the add-ons.

To add the browser to a running deployment, whatever its preset: ./configure --fetch && docker compose up -d. It adds the fetch profile, sets FETCH_BACKEND=playwright and points PLAYWRIGHT_WS_ENDPOINT at the bundled renderer, touching nothing else in .env. --no-fetch undoes it. A renderer you point PLAYWRIGHT_WS_ENDPOINT at yourself, elsewhere, is kept.

Office previews and document conversion (`office` profile)¶

./configure --office-preview && docker compose up -d adds Gotenberg, a document converter. With it, Word, Excel, PowerPoint and OpenDocument files preview as PDF in the chat, and models get a convert_document tool (a file in the conversation, or Markdown/HTML they write, to a PDF saved in the conversation). ./configure sets CHAT_GOTENBERG_URL=http://gotenberg:3000 while the profile is on and removes it when off (--no-office-preview); empty means the feature is off, and PowerPoint then previews in the browser, approximately. It is off in every preset. The image is large (about 1.5 GB) and a conversion takes a few hundred MB of memory.

Gotenberg runs LibreOffice and Chromium on documents users upload, so it is boxed in: it sits alone with the chat on an internal Docker network (no route to the internet, no published port), has no volumes (scratch space is tmpfs, the root filesystem read-only), no capabilities, CPU and memory limits (GOTENBERG_CPUS, GOTENBERG_MEM_LIMIT, defaults 2 and 2g), its own request timeout (GOTENBERG_API_TIMEOUT, 60s), short queues, no JavaScript in Chromium, and refuses non-public addresses and downloadFrom. Pin another image with GOTENBERG_IMAGE.

Object storage for files (`rustfs` profile)¶

By default the chat keeps files (attachments, generated files, knowledge-base uploads, Office previews) inside MongoDB. ./configure --object-store && docker compose up -d adds RustFS (Apache-2.0), an S3-compatible object store, and on a fresh install makes the chat use it. ./configure mints the access keys (CHAT_S3_ACCESS_KEY_ID, CHAT_S3_SECRET_ACCESS_KEY), sets CHAT_S3_ENDPOINT=http://rustfs:9000, the bucket cerea and path-style addressing, and has the chat create the bucket on first start.

On an existing install the profile starts RustFS but the chat keeps using MongoDB: switching the store under existing files would hide every one of them. Copy them first with scripts/blobs-migrate.mjs, then set CHAT_BLOB_STORE='s3'; the procedure, with a dry run and verification, is in Storage. ./configure --no-object-store refuses while the chat is using the bundled store, for the same reason.

RustFS sits alone with the chat on an internal Docker network (no route out, no published port, no web console), with its data in the rustfs-data volume, a read-only root filesystem, no capabilities, and memory and CPU limits (RUSTFS_MEM_LIMIT 1g, RUSTFS_CPUS 2). The image is pinned by version and digest in compose.yaml (RUSTFS_IMAGE replaces it). Back up the bucket with the rest (see Backup and restore; the restic sidecar does it for you): the Mongo backup holds only the index of the files, not their bytes.

Any other S3-compatible store (AWS S3, Cloudflare R2, Backblaze B2, MinIO, Ceph) works without the profile: set CHAT_BLOB_STORE='s3' and the CHAT_S3_* variables in .env (see .env.example and Storage).

Creating and editing Word, Excel, PowerPoint and PDF files (`documents` service)¶

With the chat, a small service (cerea-documents, the same version as the chat) gives models typed tools instead of code in the browser: doc_create (a new Word, Excel, PowerPoint or PDF file from a spec, with native or image charts and images) and doc_create_from_template (fill a template the person uploaded in the chat), doc_outline (a file's structure), doc_read (exact content) and doc_edit (typed operations: replace text, fill placeholders, set cells and formulas, edit tables, swap images, reorder slides, tracked changes in Word). An edit touches only the parts it names, the result is saved as a new version of the file next to the original, and the person gets an ordinary file card with the diff and a verification report in the conversation. There is no administrator template: the look is the built-in neutral style plus what the person asks for, or a file they upload. Without the service the tools are not offered and the skills fall back to the in-sandbox helpers. Decks can have slide backgrounds and up to 30 shapes per slide; a person who explicitly asks for a custom deck design gets one built in the browser sandbox instead (the PowerPoint skill's one exception). What users can ask for, and the limits, are on Word, Excel, PowerPoint and PDF files.

./configure mints CHAT_DOCUMENTS_SECRET, which the chat sends and the service requires; an install from before 0.30.0 needs ./configure run once more (then docker compose pull && docker compose up -d), or the documents container does not start and the tools stay off (the chat is unaffected). The chat finds the service at CHAT_DOCUMENTS_URL (default http://documents:8090; set it empty to turn the tools off). The service parses uploaded files, so it sits alone with the chat on an internal Docker network (no route to the internet, no published port), has no volumes, a read-only root filesystem with a small tmpfs, no capabilities, and CPU, memory and process limits (DOCUMENTS_CPUS, DOCUMENTS_MEM_LIMIT, defaults 2 and 2g); every request runs in a fresh, resource-limited subprocess (up to 30 s of work, 1.5 GiB of memory, 4 at a time; a request may be up to 64 MiB). Size the memory limit for your largest files. Pin another image with DOCUMENTS_IMAGE; its own settings are in documents/README.md.

./configure --office-preview (the office profile above) is separate: it adds Office previews and convert_document, not the document tools. Without it decks are previewed in the browser.

MongoDB: 8 by default, 4.4 where the host cannot run it¶

The chat's database is MongoDB 8 (mongo:8.0.x, an exact patch version) on a new install, and mongo:4.4.30 where the host cannot run 8. ./configure decides when it writes .env, says why, and writes the result as MONGO_IMAGE:

The host	`MONGO_IMAGE`
x86-64 with AVX, Linux kernel outside 6.19 to 7.0.13	`mongo:8.0.34`
no AVX (some small VPS hosts and low-power CPUs)	`mongo:4.4.30`
Linux 6.19 to 7.0.13	`mongo:4.4.30`
AVX or kernel version cannot be read	`mongo:4.4.30`

MongoDB 5 and later need AVX (grep -c -w avx /proc/cpuinfo is 0 without it), and MongoDB 8.0 does not start on Linux 6.19 to 7.0.13 (its memory allocator against a kernel change; kernel 7.0.14 and later are fixed). Only the upstream version counts, so 7.0.0-38-generic is 7.0.0. On Docker Desktop (macOS, Windows) the Docker VM's CPU and kernel are what matter, and ./configure cannot read them from the host, so it picks 4.4 and says so. Look from inside the VM, docker run --rm alpine sh -c 'grep -c -w avx /proc/cpuinfo; uname -r', and if the CPU has AVX and the kernel is fine, ./configure --set MONGO_IMAGE=mongo:8.0.34.

Your own MONGO_IMAGE is kept: ./configure never overwrites it, and ./configure --check reports one the host cannot run as an error. A .env with no MONGO_IMAGE runs mongo:8.0.34 (compose's default, kit 0.24 and later), and ./configure --check judges it as 8 (an error on a host without AVX, with the fix). Re-running ./configure on such a .env writes nothing but prints a note.

Upgrading from 0.23.x: earlier kits defaulted to 4.4, so an install whose .env has no MONGO_IMAGE has its data on 4.4, which MongoDB 8 cannot open. Before you pull the new kit, add MONGO_IMAGE='mongo:4.4.30' to .env (or ./configure --set MONGO_IMAGE=mongo:4.4.30), or move the data with tools/mongo-upgrade (below). A host without AVX stays on 4.4.

Upgrading an existing install to MongoDB 8¶

MongoDB cannot go from 4.4 to 8 in one move: it steps 4.4 to 5.0 to 6.0 to 7.0 to 8.0 and, after each, featureCompatibilityVersion must be raised before the next version will start. tools/mongo-upgrade does that, on hosts with AVX only (it refuses on others, changing nothing):

tools/mongo-upgrade --plan     # what it would do, nothing changed
tools/mongo-upgrade            # asks once, then upgrades (--yes skips the question)
tools/mongo-upgrade --target 7.0   # stop at 7.0, for a kernel MongoDB 8 cannot run on
docker compose up -d --wait

It stops the chat, takes a backup (a mongodump archive next to the script, mode 0600, and a backup-now too when the restic sidecar runs), then runs each version's image alone on the data volume (no network, no published port), sets and checks the compatibility version, stops it cleanly, and writes MONGO_IMAGE after every verified step so .env always names the version the data is at. The chat stays down until you start the stack. It takes minutes for a few gigabytes, mostly image pulls.

If a step fails it says which and why, and leaves the chat stopped: fix the cause and run it again. It resumes (the progress is in .mongo-upgrade.json and MONGO_IMAGE), skipping steps that are done. There is no downgrade: to go back, restore the backup into the older image (see Backup and restore). The archive is not deleted; remove it when you are satisfied.

Checking an install¶

./configure --check          # --offline skips DNS and port probes

This reports everything wrong at once: - keys missing for the profiles you run; - values that contradict each other, such as a TLS mode whose proxy settings don't match, or a client-secret digest that isn't the secret's; - leftovers from older installs; - images that aren't on the host, and whether the registry needs a login; - the Compose version; - a MONGO_IMAGE this host cannot run: MongoDB 5 and later without AVX, or MongoDB 8 on a Linux kernel from 6.19 to 7.0.13 (an error, with the image to use instead); - in acme mode, whether the name resolves to this host and ports 80 and 443 answer.

Among them, what identity configuration turns up: - an external IdP with no admin rule — set --admin-email, or --admin-claim with --admin-claim-value; a half-set claim pair, or an OIDC_ADMIN_EMAIL entry that is not a valid address, is reported the same way; - OIDC_LINK_BY_EMAIL on — the reminder to turn it off when the transition is done; - the team preset without SMTP — users cannot reset their own password; - pystino idp check --discovery-only, run in the gateway container against the configured issuer; - with an external IdP, any configured scope the issuer's scopes_supported does not list (--offline skips the lookup).

It changes nothing, so it is safe on a running install.

Reporting a problem¶

tools/diagnose               # writes ./diagnose-<date>.txt (mode 0600)
tools/diagnose --lines 500   # more log lines per service

One text file to attach to a bug report, collected by reading only: nothing is restarted or changed. It holds:

versions and pins: the kit's release (or commit), the Docker and Compose versions, the pins in compose.yaml and the image each service runs;
docker compose ps, and each service's state, health, restart count, OOM kill and exit code;
the last 200 log lines of chat, gateway and extractor;
the quota health: every rule's counter against the ledger, from pystino quota health in the gateway container. A gateway image too old to have it gets a SQL and valkey-cli listing instead;
disk usage (df and docker system df);
the keys of .env. A value is shown only for versions, presets, profiles and flags (COMPOSE_PROFILES, PYSTINO_PRESET, *_VERSION, CHAT_*_ENABLED, and a few more); every other key reads <redacted>, or <empty> if you left it unset.

It never includes a secret. Every line is masked by pattern: email addresses, JWTs and bearer tokens, API keys, password=/secret=/token= values, cookies, credentials inside URLs, private keys, password digests and long opaque strings. Any secret-named value in your .env is also removed wherever it appears, whatever shape it has. The patterns are the part that can miss something nobody has seen yet, so read the file before you send it. It still holds your hostnames, user ids and what your logs say.

Changing the configuration later¶

Re-run ./configure. It edits the existing .env and never overwrites it. Secrets are kept, your hand edits survive unless the answer they depend on changed, and the previous file is saved as .env.bak-<time>.
Set single values: ./configure --set KEY=VALUE … or ./configure --unset KEY ….
Apply with docker compose up -d. Compose recreates only the services whose settings changed.
Edited caddy/ or authelia/ yourself? Compose can't see a change inside a mounted file. Run docker compose up -d --force-recreate proxy authelia, or tools/pin to refresh the cerea.config-rev labels that make compose notice.
Own routes and headers go in proxy.d/*.caddy. Git ignores those files, and the proxy imports them (see proxy.d/README.md).
Own services go in compose.override.yaml: see Adding your own services.

Adding your own services¶

Four files, four owners:

File	Owner
`.env`	`./configure`. It keeps keys it doesn't know in a final "Not in .env.example" section, so your own variables (a setup key, say) survive a re-run
`compose.yaml`	this kit; an upgrade replaces it
`compose.override.yaml`	you. Compose merges it into `compose.yaml` by itself, and git ignores it
`proxy.d/*.caddy`	you: extra routes and headers

./configure, ./configure --check, tools/pin and dev/build.sh leave the override alone, and --check reads the merged result. Name no -f of your own on the command line: it turns the automatic merge off.

Example: a NetBird client in the stack¶

A NetBird client in the same Compose project gives the stack an address on your NetBird network without publishing a port on the host. The proxy joins the client's network namespace, so Caddy listens on the NetBird address:

# compose.override.yaml
services:
  netbird:
    image: netbirdio/netbird:latest
    restart: unless-stopped
    hostname: cerea                 # the peer's name on the NetBird network
    environment:
      NB_SETUP_KEY: ${NB_SETUP_KEY:?add NB_SETUP_KEY to .env}
      NB_HOSTNAME: cerea
    cap_add: [NET_ADMIN, SYS_ADMIN, SYS_RESOURCE]
    devices: ["/dev/net/tun"]
    volumes:
      - netbird-state:/var/lib/netbird

  proxy:
    network_mode: "service:netbird"  # Caddy shares the client's namespace
    ports: !reset []                 # the client owns the network now

  gateway:
    ports: !reset []                 # drop the loopback port 8000 as well

volumes:
  netbird-state:

Put the key in .env with ./configure --set NB_SETUP_KEY=…. The caps and the /dev/net/tun device are what NetBird's own Docker instructions list; check them there for your version. The client sits on the stack's network, so the proxy still reaches gateway, chat and authelia by name.

!reset needs Docker Compose 2.24, which ./configure already requires. On an older Compose, set PUBLIC_BIND=127.0.0.1 instead: the ports stay published, but only on the host's loopback.

TLS. Two setups work: - --tls upstream: NetBird's reverse proxy, or another edge, terminates TLS and forwards plain HTTP to the NetBird address on port 80. It must pass Host, forward WebSocket upgrades and not buffer (see TLS modes). Set TRUSTED_PROXIES to the range it connects from; for NetBird that is 100.64.0.0/10. - --tls internal: users reach the NetBird address or name directly, with Caddy's own certificate. Every browser and agent machine must trust that CA (Certificate trust).

PUBLIC_ORIGIN must be the address users type into the browser (with https://), and the agent machines have to reach it as well: they join the same NetBird network, or the origin is published some other way. The stack builds every sign-in redirect from it.

Example: backups with restic¶

A backup service takes the parts of Backup and restore on a schedule — .env, the Postgres dump, the Mongo archive, the Authelia volumes and, when the chat keeps its files in S3, the bucket's objects — and stores them in a restic repository: encrypted, deduplicated, on any backend restic speaks (S3, B2, Azure, GCS, SFTP, a REST server, a local path) and on any of rclone's (rclone:<remote>:<path>). It is configured by environment variables in .env.

cp tools/backup/compose.override.example.yaml compose.override.yaml   # or merge it into yours
./configure --set RESTIC_REPOSITORY=s3:s3.eu-central-1.amazonaws.com/my-bucket/cerea \
                  RESTIC_PASSWORD=… AWS_ACCESS_KEY_ID=… AWS_SECRET_ACCESS_KEY=…
docker compose up -d --build backup
docker compose logs backup               # "repository initialised", then "scheduler up"
docker compose exec backup backup-now    # a backup right now, in the foreground
docker compose exec backup restic snapshots

The image is built from tools/backup/ on your machine, because no published image carries restic, rclone, a Postgres 18 client and the Mongo tools together. It is Alpine, 250 MB, with pinned sources: the official restic/restic:0.18.1 and rclone/rclone:1.73.0 binaries, alpine:3.23.6 for the rest. After a upgrade that changes tools/backup/, run docker compose up -d --build backup. The service carries pull_policy: never, so the docker compose pull of the upgrade flow skips it — the image exists only here, and pulling it would fail the whole pull. (An override copied before this line needs the two words added by hand.)

Keep the repository password somewhere else. .env is in every snapshot, and it holds RESTIC_PASSWORD: the backup that explains how to open itself is no use on the day .env is gone. Put the password (and the backend's credentials) in a password manager as well.

Settings¶

All in .env (./configure --set KEY=VALUE); docker compose up -d applies a change.

Variable	Default
`RESTIC_REPOSITORY`	required	where the backups go; `rclone:remote:path` for rclone
`RESTIC_PASSWORD`	required	or `RESTIC_PASSWORD_FILE` (a mounted file) or `RESTIC_PASSWORD_COMMAND`
`AWS_`, `B2_`, `AZURE_`, `GOOGLE_`, `OS_`, `ST_`, …		whatever credentials restic reads for the backend
`RCLONE_CONFIG_<NAME>_TYPE`, `…_<OPTION>`		an rclone remote defined by variables, see below
`BACKUP_SCHEDULE`	`17 3 * * *`	cron, five fields; `TZ` sets the timezone (UTC by default)
`RESTIC_FORGET_ARGS`	`--keep-daily 7 --keep-weekly 4 --keep-monthly 6`	retention, applied with `forget --prune` after each backup; set it empty to keep everything
`BACKUP_CHECK_EVERY`	`7`	run `restic check` every N backups (`0`: never); `RESTIC_CHECK_ARGS=--read-data-subset=5%` also reads a share of the data
`BACKUP_MAX_AGE_HOURS`	`36`	the container turns `unhealthy` when no backup has succeeded for this long
`BACKUP_HOST`	the project name	the host name recorded on every snapshot, and the one retention groups by; keep it constant
`BACKUP_REPO_TIMEOUT`	`120`	seconds to wait for the repository to answer before a run fails as unreachable
`BACKUP_WORK_SIZE`	`2g`	size of the RAM disk the dumps are written to; at least the combined size of the dumps
`BACKUP_MONGO`	`auto`	`auto` backs up Mongo when the `chat` profile is on; `yes` or `no` forces it
`BACKUP_OBJECTS`	`auto`	`auto` backs up the chat's bucket when `CHAT_BLOB_STORE=s3`; `yes` or `no` forces it
`BACKUP_OBJECTS_TIMEOUT`	`60`	seconds to wait for the object store to answer before a run fails as unreachable
`BACKUP_OBJECTS_TRANSFERS`	`4`	files copied at once when syncing the bucket
`BACKUP_EXTRA_DIR`	`/extra`	where your own files are mounted in the service; copied into each snapshot as `extra/`

Repositories. - A local path. Mount it in the override (- /srv/backups/cerea:/repo) and set RESTIC_REPOSITORY=/repo. Keep it on another disk, or it protects you from nothing. - A restic backend with its own credentials, as above. Check the variable names in restic's documentation for your backend. - rclone, with the remote defined in .env: RESTIC_REPOSITORY=rclone:cloud:cerea and RCLONE_CONFIG_CLOUD_TYPE=webdav, RCLONE_CONFIG_CLOUD_URL=…, RCLONE_CONFIG_CLOUD_USER=…, RCLONE_CONFIG_CLOUD_PASS=… (obscured with rclone obscure). The remote's name is the CLOUD in the variable names, in lower case in the repository. - rclone, with an rclone.conf: uncomment the rclone.conf volume in the override and set RCLONE_CONFIG=/rclone.conf. Mounted read-only, so a remote whose token rclone must refresh (Google Drive, OneDrive) needs the file writable: drop the :ro. - rclone prints Config file "/root/.rclone.conf" not found - using defaults for every call when no file is mounted. It is harmless.

What a run does¶

Opens the repository, and runs restic init once, with a log line, if there is none. A wrong password or an unreachable backend stops here, with restic's own message.
Writes, into a RAM disk inside the container: .env, postgres.sql (pg_dumpall, over the stack's network, as POSTGRES_USER), chat-mongo.archive.gz (mongodump), authelia.tgz (both Authelia volumes, mounted read-only) and a small backup.info. They are the files Backup and restore names, taken one after the other in a few seconds, so they belong together. .env is never written to a disk; the RAM disk is deleted when the run ends, however it ends.
With CHAT_BLOB_STORE=s3 (BACKUP_OBJECTS=auto), after the Mongo dump, brings the persistent /objects volume (backup-objects) in line with the chat's bucket, through the S3 API with rclone, configured from CHAT_S3_* in .env and nothing else. Objects never change once written, so a run copies only the new ones and removes the ones the chat deleted; the first run copies everything. The bucket is not copied through the RAM disk, because it can be far larger than memory. A file added between the dump and the sync is an extra the index does not know, which is harmless; the other order could leave index entries without their bytes. If the bucket cannot be reached, or the keys are wrong, the run fails at step objects.
restic backups the dump directory and /objects as one snapshot, tagged cerea, host = the project name. A restic repository deduplicates, so the objects are stored once however many snapshots hold them.
restic forget --prune with RESTIC_FORGET_ARGS, and every BACKUP_CHECK_EVERY-th run a restic check.

The stack keeps running throughout and nothing is stopped. pg_dumpall and mongodump read consistent data while the services write. Authelia's SQLite store is copied with SQLite's own online backup, so it is consistent too; the rest of its volumes are a few small files that change rarely. A person who changes their password during the few seconds of a run is the only race, and the next run holds the result. The two stores are not one transaction: schedule the run for a quiet hour, and take a manual backup with the stack idle before an upgrade.

The container is not root on the host. It has no Docker socket, a read-only root filesystem, cap_drop: ALL (plus DAC_OVERRIDE, for Authelia's 0600 files and a repository directory that is not root's), and mounts .env and the Authelia volumes read-only.

Files of your own. Anything you mount under /extra in the backup service is copied into each snapshot as extra/, and restore returns it as restore/extra/. Use it for the files the kit does not create but you would want back after losing the machine, such as your compose.override.yaml (which holds the backup service itself), an rclone.conf, or a galopin mcp.json. Add one read-only line per file or directory to the override:

    volumes:
      - ./compose.override.yaml:/extra/compose.override.yaml:ro
      - ./rclone.conf:/extra/rclone.conf:ro

then docker compose up -d backup. They go through the same RAM disk as the dumps (BACKUP_WORK_SIZE), so keep it to configuration-sized files; bulk data belongs in its own backup. A different mount point can be set with BACKUP_EXTRA_DIR.

The disk cost of the objects. The backup-objects volume is a second copy of the whole bucket on this host, so it needs as much free space as the bucket holds (check where Docker keeps its volumes, docker info | grep 'Docker Root Dir', or point the volume at a bigger disk in the override), and that is on top of what the repository stores. What it buys is that a run is fast and the snapshot is complete. If the bucket is already replicated or versioned by its provider (S3 with versioning and cross-region replication, say) and you do not want a copy, set BACKUP_OBJECTS=no: the snapshot then holds the other parts, and the bucket's own protection must be restorable to the same moment as the Mongo archive. With the bundled RustFS the sidecar reaches the bucket over the objects network, which the override joins (drop that network from the override with your own S3).

Upgrading a sidecar set up before objects were covered: rebuild it and update the override (the backup-objects volume and the objects network, as in tools/backup/compose.override.example.yaml), then docker compose up -d --build backup. Without the volume a run with CHAT_BLOB_STORE=s3 fails at step objects and says so.

Failures¶

Every run ends in a log line you can search for: BACKUP OK or BACKUP FAILED at step '…', on stderr, with restic's message above it, and the run exits non-zero. docker compose logs backup | grep -E 'BACKUP|PREFLIGHT' is the whole history. The container's health turns unhealthy (docker compose ps) after a failed run, or when the last success is older than BACKUP_MAX_AGE_HOURS, which also catches a scheduler that never fired. At start the sidecar opens the repository and logs PREFLIGHT FAILED if it cannot, so a wrong password shows when you up it, not at 03:17. A run that finds another one still going stops with exit 75.

Restoring from the repository¶

On the machine you are rebuilding, with the kit cloned. All it needs are the repository's address, password and credentials, from your password manager:

docker build -t cerea-backup:1 tools/backup
cat > restore.env <<'EOT'        # no quotes: docker's --env-file keeps them
RESTIC_REPOSITORY=…
RESTIC_PASSWORD=…
EOT
mkdir -m 700 restore
docker run --rm -e BACKUP_RESTORE_CHOWN="$(id -u):$(id -g)" --env-file restore.env \
  -v "$PWD/restore:/restore" cerea-backup:1 restore          # or: restore <snapshot-id>
B=restore

Add what your backend needs: -v /srv/backups/cerea:/repo for a local repository, the RCLONE_CONFIG_* variables to restore.env, or -v "$PWD/rclone.conf:/rclone.conf:ro" -e RCLONE_CONFIG=/rclone.conf. On a box that still has its .env and override, docker compose run --rm --no-deps --cap-add CHOWN -e BACKUP_RESTORE_CHOWN="$(id -u):$(id -g)" -v "$PWD/restore:/restore" backup restore does the same. With restic installed on the host, restic restore latest:/work/cerea --tag cerea --target restore lands the same files.

restore prints the repository's snapshots and fills restore/ with the five files, newest snapshot by default (BACKUP_RESTORE_CHOWN hands them to you), plus restore/objects/, the bucket's files, when the snapshot has them; it checks their count against backup.info. Then follow the restore steps in Backup and restore from cp "$B/.env" .env, including the step that copies $B/objects back into the bucket. Delete restore/ afterwards: it holds every secret in the clear.

Server-side MCP servers¶

Simple stdio MCP servers (the kind a README tells you to start with uvx <package>) can run as containers beside the kit and appear in everyone's chat as built-in connectors, with no host port and no VPN. Each is on for everyone until an administrator sets otherwise under Admin → Connectors (off, on, or on and locked, and whether its tools are always loaded). tools/mcp/ has an example override with two of them, a web search (duckduckgo-mcp-server) and OpenStreetMap (osm-mcp, an npm package). Each service is one mcp-proxy wrapping one server: stdio in, Streamable HTTP out on the stack's own network, at http://mcp-search:8096/mcp. The image (tools/mcp/Dockerfile) is python:3.13-slim with mcp-proxy, uv and Node 22 with npm, all pinned: the upstream mcp-proxy image has neither, so uvx and npx servers need this one. Start npm servers with npx -y <package>@<version> and set NPM_CONFIG_CACHE=/cache/npm, as the example does.

The OpenStreetMap server calls free public services (Nominatim, Overpass, OSRM). Before you start it, set OSM_USER_AGENT in the override to a name with a contact address you answer at: those services answer 406 to anonymous clients, and Nominatim's usage policy asks for it. Walking and cycling routes use the FOSSGIS routed-foot / routed-bike servers; point OSRM_BASE_URL at your own if you need more than light use.

The public Overpass servers (the place search) are run by volunteers and are often busy. If they keep timing out for you, hide the tools that need them, so the assistant never waits on them: set OSM_DENY_TOOLS: "find_nearby_pois,poi_details,suggest_meeting_point" in the mcp-osm service's environment and run docker compose up -d mcp-osm. A denied tool is gone from the server's tool list; the chat picks the change up within 15 minutes, or at once with Check under Admin → Connectors. OVERPASS_BASE_URL takes a comma-separated list tried in order within one 50-second budget, so put the server that answers first and the ones that tend to hang last.

tools/mcp/skills/ has two example skills that tell the assistant which map tool answers which question. Both steer it away from the OpenStreetMap server's place search (find_nearby_pois), which depends on the public Overpass servers and in practice times out: places come from Geoapify or a web search instead. Add one of them, not both, for everyone in Admin → Skills (paste it, or import it as a zip of its folder):

maps-routing: the OpenStreetMap server alone;
maps-routing-geoapify: OpenStreetMap together with Geoapify (below), including how to pick a Geoapify place category.

They are not built-in skills because they name these servers' tools: edit them if you run other map servers.

Geoapify runs a hosted MCP server, so there is no container to add: it needs only a key from geoapify.com (a free tier exists; every call costs credits) and an entry in MCP_SERVERS with the key in a header, as in step 2. The key is shared by everyone who signs in. An administrator can instead add it under Admin → Connectors with a shared access key, which keeps it out of .env.

Add the example's services and its mcp-cache volume to compose.override.yaml: copy tools/mcp/compose.override.example.yaml if you have no override yet, otherwise paste the two services under your services: and the volume under volumes:, leaving the rest of your override as it is. The build: ./tools/mcp path is relative to the kit directory. (Compose's include: also accepts the file, from Compose 2.20, and the services join the stack's network. But it resolves build: relative to the included file, so the build path would need to be .; merging by hand keeps the example as it is.)
In .env, name the servers and the hosts they live on:

MCP_SERVERS='[{"name":"search","url":"http://mcp-search:8096/mcp"},{"name":"osm","url":"http://mcp-osm:8096/mcp"}]'
MCP_INTERNAL_HOSTS='mcp-search,mcp-osm'

With Geoapify as well (a hosted https server: nothing to add to MCP_INTERNAL_HOSTS), replace YOUR-GEOAPIFY-KEY with your key:

MCP_SERVERS='[{"name":"search","url":"http://mcp-search:8096/mcp"},{"name":"osm","url":"http://mcp-osm:8096/mcp"},{"name":"geoapify","url":"https://api.geoapify.com/v1/mcp","headers":{"x-api-key":"YOUR-GEOAPIFY-KEY"}}]'

docker compose up -d --build. The first start of each server downloads its package into the mcp-cache volume; the healthcheck allows 90 to 120 s for that.

MCP_INTERNAL_HOSTS is the only opening the chat needs. An MCP URL has to be https, and MCP_ALLOW_INSECURE_URLS only covers IP addresses; a Compose service name is a hostname that resolves to a private address over plain http. The listed hostnames, matched exactly, are allowed for MCP connections only (never for web_fetch or attachments), and the address they resolve to must still be a private one: link-local and cloud-metadata addresses stay refused. A name you do not list is refused as before. The same names are accepted as the address of a connector an administrator adds under Admin, Connectors (not of a user's own).

To add a server, copy a service, change its name and the package after --, and add the name to both .env lines. Containers you add by hand are not docker compose pulled (the example sets pull_policy: never); rebuild with docker compose build --pull mcp-search to refresh the base image.

A server here is shared by everyone who signs in, and so is any token it holds, so run only servers that need no personal credential. Personal accounts (mail, calendar, files) belong in local connectors, where each person's server runs on their own machine.

TLS modes¶

`--tls`	When	What you need
`acme` (default)	the server is reachable from the internet under its name	DNS pointing at the server; ports 80 and 443 open. Caddy gets and renews a Let's Encrypt certificate
`internal`	development, private networks	nothing; Caddy's own CA signs the certificate and browsers warn
`upstream`	something in front already terminates TLS (a load balancer, a tunnel, an edge proxy)	the front forwards plain HTTP to `--http-port` on this host

The origin is always https://…, including in upstream mode, because the applications build every URL from it.

In upstream mode, have the front: - pass the original Host; - forward WebSocket upgrades, at least for /chat/api/v2/code/machine, where each agent machine holds its link; - not buffer responses, because chat and completions stream.

The proxy trusts X-Forwarded-For only from TRUSTED_PROXIES (default private_ranges). If the front reaches this host from another range, such as a VPN's 100.64.0.0/10, set TRUSTED_PROXIES to that range. Otherwise the logs show the front's address instead of the client's.

Certificate trust with internal TLS¶

Caddy signs the certificate with its own CA, so every browser shows a warning until you trust that CA, or accept the warning once. The CA certificate is created on the first start; copy it out of the proxy:

docker compose cp proxy:/data/caddy/pki/authorities/local/root.crt cerea-root.crt

Agent machines (galopin) have no flag to skip certificate checks and no setting for an extra CA; they use the machine's own trust store. Install cerea-root.crt there: in the system store (macOS Keychain, Windows "Trusted Root Certification Authorities", or update-ca-certificates on Debian and Ubuntu), or, on Linux, point galopin at it with SSL_CERT_FILE=/path/to/cerea-root.crt (Go reads that variable). An agent machine on a server with a real certificate needs nothing.

Servers never call the public origin: the gateway and the chat reach the IdP on the internal network. The TLS mode therefore never means distributing a CA certificate to containers.

Identity¶

People sign in with OpenID Connect only: the console, the chat and galopin all sign in against one identity provider, named in .env by ./configure. The gateway takes its settings from .env on every start, so the provider is never edited anywhere else — the console shows it read-only.

Only one provider is supported at a time. To combine several sources of users (a corporate directory plus social logins, say), federate them in an IdP that brokers logins and point the stack at it. In Keycloak: create a realm, add the upstream providers under Identity providers (each an OIDC or SAML broker), and Keycloak signs everyone in itself — the stack sees a single issuer, Keycloak's, with a single set of clients. Use mappers to put what the stack reads into Keycloak's tokens: a groups claim, and email with email_verified.

The bundled Authelia (--idp authelia, the default) works with a dotted DNS name (cerea.example.org) or an IP address (https://192.168.1.10, with :port if you use one). It does not work with a single-word name such as myserver: Authelia rejects that as its cookie domain. If you would rather have a name than an IP on a private network, a wildcard DNS service gives you one: 192-168-1-10.sslip.io (or nip.io) resolves to 192.168.1.10. An IP or a private name cannot get a Let's Encrypt certificate, so use --tls internal there (see "Certificate trust" below). Its first account is created on the first up; the first sign-in with the administrator email you gave ./configure makes that account an administrator.

Its accounts are managed from the console's Users page:

Add user creates the Authelia login and the gateway account together and shows a one-time password once. Pass it on through a one-time channel (a password manager share, or in person); Authelia cannot force a change at first sign-in.
Create sign-in gives an existing account a bundled login — the case after a switch back from an external IdP or break-glass, for someone who only ever had an identity at the other provider.
Reset password mints a fresh one-time password for their login.
Disable and Enable cut a person's access everywhere without erasing anything (below).
Delete erases the person everywhere, chat included (below).

Groups and roles live in the console too (the Groups page): the bundled users file carries no groups of its own.

With SMTP set (./configure --smtp-host …), users reset their own password from the sign-in page. Without it, only an administrator can, by issuing a one-time password from the Users page.

Need MFA? The bundled Authelia is one-factor: its TOTP and WebAuthn are switched off. Deployments that need a second factor use an external IdP such as Keycloak — TOTP or WebAuthn is configured there — and point the stack at it through .env.

Your own IdP (--idp external): Keycloak, Entra ID, Google, Authentik or any OIDC issuer. Register three clients there, then pass their details:

Client	Kind	Redirect
console (`pystino-console`)	confidential	`https://<origin>/auth/callback/default`
chat (`cerea`)	confidential	`https://<origin>/chat/login/callback`
galopin (`opencode-enrollment`)	public	the device flow, and loopback `http://localhost/callback`

The chat client needs the refresh-token grant and offline_access among its allowed scopes too (the chat asks for it at sign-in when the issuer lists it, and keeps the refresh token for scheduled chats; the bundled Authelia client has both). The machine client needs the device-flow and refresh-token grants and offline_access in its scopes (galopin asks for it only when the issuer lists it; see below); the other two take the scopes openid profile email groups, less what the issuer does not offer. Map the groups claim the IdP sends (--oidc-groups-claim, groups by default), and make sure email arrives with email_verified as the JSON boolean true. If the IdP publishes no end_session_endpoint, set --oidc-logout-url 'https://id.example.org/logout?redirect={redirect}' so that signing out ends its session too.

Signing out of either the console or the chat signs you out of both, and out of the IdP.

export OIDC_ISSUER=https://id.example.org/realms/main   # your IdP's issuer
./configure --idp external --oidc-issuer "$OIDC_ISSUER" \
  --admin-email ops@example.org \
  --oidc-console-client-id pystino-console --oidc-console-client-secret … \
  --oidc-chat-client-id cerea --oidc-chat-client-secret …

Arguments on the command line are visible to other users of the machine; interactive ./configure asks for secrets without echoing them.

The chat client's secret (OIDC_CHAT_CLIENT_SECRET) also reaches the gateway, as GATEWAY_OIDC__CHAT_CLIENT_SECRET: an IdP whose access tokens are opaque rather than JWTs (GitLab) can only be asked about a token at its introspection endpoint, and the gateway does that authenticated as the chat client. ./configure already writes the value; there is nothing extra to set.

Agent machines with GitLab need their own application there: not confidential, scopes openid profile email, the device authorization grant enabled. Put its Application ID in OIDC_MACHINE_CLIENT_ID (GitLab cannot have the default opencode-enrollment). Its tokens cannot be introspected, since the application has no secret, so the gateway checks them at GitLab's /oauth/token/info with the token itself; see Agent machines.

Groups from the identity provider¶

A directory reports every group a person is in: one GitLab sign-in used to create 67 groups in the console, one per GitLab group. So, with an external IdP, ./configure asks:

Groups from the identity provider:
  manual      import by hand (recommended): the console lists them to import
  auto        import automatically: every group name becomes a group

or takes --group-import manual|auto (default manual), written as OIDC_GROUP_IMPORT. With manual, a sign-in creates no group: the names it carries are listed on the console's Groups page under Seen from your identity provider, with how many users carry each and when it was last seen. Import creates the group and gives it to those users at once (at their next sign-in if OIDC_GROUP_SYNC is not every_login); Dismiss stops listing it. A group that already exists is granted from the claim in both modes, as before. auto is the old behaviour: every name becomes a group at first sight. The bundled Authelia is unaffected: its groups are the console's.

OIDC_GROUP_ALLOWLIST (advanced, optional; comma-separated) limits which IdP group names are considered at all, in either mode.

Who may join¶

One answer governs what a stranger's first sign-in does, for any identity provider (ADR 0095). ./configure asks for it with an external IdP, or takes --join-mode open|approval|invite|group (a new install's default is approval; editing an existing .env keeps what it does today):

open — anyone who signs in gets an active account at once (the old OIDC_AUTO_PROVISION=true);
approval — the account is created disabled, in the console's Users page Awaiting approval queue; Approve enables it, Reject deletes it;
invite — only users you invited (or the directory pushed) can sign in; everyone else is refused with a text that says to ask for an invitation. Invitations expire after 14 days; re-inviting (the invite-text button) re-issues one;
group — only members of the required groups may sign in, checked at every sign-in and every API call. ./configure asks for the local group names (--required-groups, comma-separated; a directory name must be imported or mapped on Console > Groups first) and what a stranger without one gets (--group-fallback refuse|approval). group with no required groups refuses to start, by design.

The written keys are OIDC_JOIN_MODE, OIDC_REQUIRED_GROUPS and OIDC_GROUP_FALLBACK; the console's Identity page can set all of this instead, and a decision there wins. Administrators are never refused by the gate, so a wrong group cannot lock you out. The machine links survive a group loss: the gateway refuses with 403 outside_access_groups, Cerea closes the link (4404) without revoking the device, and it reconnects by itself once the person is back in a required group.

Everyone joins users. Each person is put in one default group at their first sign-in, with any IdP (OIDC_DEFAULT_GROUP, users; existing users at their next sign-in), so a new account has a group to bill and can use the public models before anything is imported. It grants no model by itself: restrict models by group on the Models screen, or keep them public. Someone an administrator removes from it stays removed. Set OIDC_DEFAULT_GROUP='' to turn it off. With the bundled Authelia it is the same users group its accounts always had.

Cleaning up groups created before (an upgrade from an auto install): in the console's Groups page, delete the ones you do not want. Under manual they are not recreated: their names come back to the Seen list at the next sign-in, to import again or dismiss.

Providers that don't offer a groups scope (e.g. Infomaniak)¶

Some providers publish a groups claim but no groups scope, and answer a sign-in that asks for it with invalid_scope. Infomaniak's discovery lists scopes_supported = openid, profile, email, phone, for one.

With --idp external, ./configure reads the issuer's /.well-known/openid-configuration and asks only for the scopes it lists (openid always stays), from openid profile email groups. It writes the result for both the console and the chat, and says so when groups is dropped:

OIDC_SCOPES='openid profile email'
OIDC_SCOPES_JSON='["openid","profile","email"]'

The two are one list: the chat reads the words, the gateway wants JSON. To change it later, ./configure --set OIDC_SCOPES='openid profile email' writes both, and --oidc-scopes does the same when configuring. A re-run keeps a list you edited unless the issuer changed. When the issuer's document lists no scopes_supported (or --offline is given), the default stays. ./configure --check warns about any configured scope the issuer does not list.

Without the scope, group-based features (group sync, quotas and billing by group, an --admin-claim groups rule) work only if the IdP puts a groups claim in its tokens regardless; check with docker compose run --rm --no-deps gateway pystino idp check --device, which prints the claim. With no groups claim, users sign in with no group memberships and an admin rule by --admin-email still works.

galopin follows the same rule: it asks for offline_access only when the issuer lists it. A provider that grants refresh_token without the scope (Infomaniak) is still asked, and if no refresh token comes back, galopin enroll stops with a message instead of enrolling a machine that would lose its sign-in within the hour; enable refresh tokens for the opencode-enrollment client at the provider.

Who is an administrator is decided in .env, and re-checked at every sign-in and at every gateway start:

--admin-email (OIDC_ADMIN_EMAIL): a comma-separated list of addresses. An address grants admin only when the IdP asserts email_verified: true for it. Removing an address takes its admin away at the next start; the last active administrator is never revoked.
--admin-claim and --admin-claim-value: the values of a claim (the groups claim, typically) that confer admin.

With an external IdP and neither set, the gateway refuses to start:

an external identity provider needs an admin rule: set OIDC_ADMIN_EMAIL, or
OIDC_ADMIN_CLAIM with OIDC_ADMIN_CLAIM_VALUE, in .env
(./configure --admin-email / --admin-claim --admin-claim-value)

./configure --check runs the same rule offline before you commit to it, and probes the configured issuer with pystino idp check --discovery-only in the gateway container. To test the admin rule with a real sign-in before switching, run the device-flow variant and sign in on any browser:

docker compose run --rm --no-deps gateway pystino idp check --device

It reports the token's sub, the email and email_verified with its JSON type (the string "true" is not accepted), the groups claim's value, the audience and client, whether each admin rule would grant, and — with link-by-email on — how many existing accounts a first sign-in would link to.

Switching the identity provider¶

Back up (see Backup and restore).
Ask everyone to sign in once since upgrading, before you switch. Accounts are keyed by their gateway id, which the chat learns at each person's first sign-in; for anyone who has not signed in yet, keeping their chat history falls to the merge in the next step.
Switch: ./configure --idp external … (or --idp authelia to switch back), then ./configure --check, then docker compose up -d.
Existing users. Their accounts and chat history stay. A person keeps theirs when one of these happens:
Link by email. With OIDC_LINK_BY_EMAIL=true (off by default; set it with --link-by-email), a first sign-in with a verified email attaches to the one existing account with that address. It never attaches to an administrator. Turn it off with --no-link-by-email once the transition is done; a warning prints at every start while it is on.
Merge. An administrator picks Merge into… on the Users page and moves the old account into the new one (or the other way round). The chat follows automatically: the person's conversations appear under the surviving account at their next sign-in, or within a minute of their next activity.
Each person with an agent machine re-enrolls it. Their /chat/code shows Re-enroll this machine: the identity provider changed with the command to run on that machine; tell them before you switch.
The old provider stays listed. The console's Settings screen keeps it as a disabled "previous" row, so users who signed in there can be linked back. If nobody ever signed in through it, Remove on that row deletes it; if users did, the row says how many and stays. Removing it loses nothing: pointing .env at that issuer again creates it afresh.

Merging accounts¶

The merge moves everything that belongs to the person: their identities at every issuer, group memberships, model access, API keys, spend history, and their quota and redaction rules. The target keeps their own profile and settings; an identity both accounts hold at one issuer is dropped, and named in the preview. The dialog asks you to type the source account's address to confirm, and takes an optional reason for the audit trail.

The merge is irreversible. There is no split-back: the undo is the backup taken before it.

Break-glass¶

When nobody can administer, or the identity provider itself is gone:

./configure --break-glass --admin-email ops@example.org --reason "IdP outage"

--reason is optional; when given, it is stored on the gateway's audit row.

It rewrites .env to the bundled Authelia — your external settings are kept in the .env.bak-<timestamp> file it makes and nowhere else — brings the stack up, and grants admin to the account with that email: creating it if it does not exist, re-enabling their bundled login if it was disabled, or creating one if they never had it. It prints the sign-in URL, the login and a one-time password, once. Everyone else's accounts, memberships and chat history are untouched. Each person with an agent machine has to re-enroll it, because its tokens name the old issuer; tell them before you switch.

To go back to your external IdP afterwards, switch again with its values from the backup file: ./configure --idp external … --oidc-issuer …, then --check, then docker compose up -d. The accounts and identities that provider held resolve again.

For someone who can already sign in, docker compose exec gateway pystino admin grant you@example.org grants or revokes admin without touching anything else.

Disabling and deleting a user¶

Disable (Users page) ends the person's console and chat sessions and closes their machine links within a minute, and refuses new sign-ins. Already-issued IdP tokens keep working at the IdP until they expire — the stack refuses them regardless. Enable brings the account back; sessions and machine links stay revoked, so they sign in again and re-enroll their machines.

Delete erases the person everywhere, chat included: conversations, files and attachments, projects, knowledge bases, memories, skills, MCP connectors, settings, sessions and machine links. Content they shared with other users disappears for them too; the delete confirmation names it and asks you to tick it before it proceeds. Their spend history stays in the ledger, anonymised. If the chat is down at delete time, the erasure is queued and retried until the chat confirms — docker compose exec gateway pystino erasure list shows the queue.

Agent machines¶

With --agents (CODE_AGENTS_ENABLED='true'), users can pair their own machines and drive coding agents on them from /chat/code. Each machine runs galopin, a small agent that dials out to your origin over WebSocket, so no inbound port is needed on the machine.

The pairing dialog prints one command to copy, already filled in with this deployment's issuer, gateway and client: install, enroll and run, one step per line, each continuing only if the one before succeeded.

curl -fsSL '<origin>/chat/galopin/install.sh' | sh &&
curl -fsSL https://opencode.ai/install | bash -s -- --version 1.18.34 &&
~/.local/bin/galopin enroll \
  --issuer '<issuer>' \
  --gateway '<origin>' \
  --cerea '<origin>/chat' &&
~/.local/bin/galopin run

The opencode line is there when the dialog's Install opencode checkbox is ticked. It installs the opencode release galopin is tested against (agent/packaging/opencode-version in the Cerea repository, 1.18.34 in this release) and not the newest one. A machine keeps whatever opencode it has until that line is run again. Checkboxes for the machine's policy add flags to enroll: terminals, slash commands that run shell and background subagents are on by default (--no-terminal, --no-command-shell and --no-background-subagents turn them off), and two plain checkboxes allow work outside the project folder and reading secret files without asking.

The browser terminal has two switches. The deployment's, --terminal (CODE_TERMINAL_ENABLED='true'), is off unless you turn it on; while it is off no terminal is offered anywhere, and the pairing dialog says so. Each machine's own is the dialog's Terminals checkbox, ticked by default; unticking it adds --no-terminal to the command. Together they grant interactive shell access on the machine to whoever controls the chat session — no model and no permission rule in the way once a terminal is open. The installer verifies the binary's checksum; enroll signs in through the browser. The machine then appears in /chat/code, where its owner confirms it. Revoking it there disconnects it for good. A machine whose identity provider changed, or whose access was revoked, shows Re-enroll this machine in the device list, with the same command. Everything in /chat/code needs a sign-in within the last 7 days; the machine's own link to the chat is not affected.

galopin run stops with its terminal. On Linux the machine's owner keeps it running as a systemd user unit, on macOS as a LaunchAgent (both files ship in agent/packaging/; the steps are in Keeping it running). After re-running the install line, systemctl --user restart galopin starts the new binary.

Local connectors¶

LOCAL_CONNECTORS_ENABLED='true' (the kit's default) lets users connect MCP servers that run on their own machine to their chat, without the Agents panel and without a coding agent on the machine. The machine runs galopin in connectors mode (galopin enroll --connectors-only, then galopin run), dials out to the same machine link as an agent machine, and is approved by its owner under Workspace → Local connectors, which also prints the whole how-to filled in with this deployment's address. The servers are listed in the machine's own mcp.json; nothing on the link starts a command that is not in that file. Set the variable to anything else to turn the feature off: connectors machines are then refused and the tab is hidden. See Local connectors.

Auto in the composer can have a decision-model reviewer behind it: an administrator chooses one under Administration → Auto-approval, and it stays off until then. CHAT_AUTO_PERMISSIONS='false' is the kill switch (on otherwise); then the third step reads Allow and no reviewer runs.

Scheduled actions come with --agents: users can have a prompt sent to a session on one of their machines on a timetable, and agents can make schedules on their own machine. CHAT_SCHEDULES_ENABLED='false' is the kill switch (on otherwise) and CHAT_SCHEDULES_MAX_PER_USER caps each person's schedules (default 20); see Scheduled actions.

opencode without Cerea¶

Anyone who uses opencode directly can point it at the gateway. The console's Overview shows the line, filled in with this deployment's address, beside the API keys:

curl -fsSL https://<your-host>/opencode/install.sh | bash

The gateway serves that script itself. It asks for an API key minted in the console (from the PYSTINO_API_KEY variable or a hidden prompt, never from the command line), adds one pystino provider to opencode's global config with the models that key may use, keeps everything else in that file and writes a backup first. --key-in-env keeps the key out of the file, and --install-opencode installs the opencode release the kit was tested with (1.18.34).

Backup and restore¶

What to keep: - .env. It holds every secret. GATEWAY_SECRET_KEY decrypts the provider keys stored in the database. AUTHELIA_STORAGE_KEY decrypts every user's identifier. Without them a database backup is only partly usable. - Postgres: everything the gateway (Pystino) keeps (users and groups, API keys, quotas and the ledger, providers and models, identity-provider settings) and the chat's knowledge store. pg_dumpall takes every database in the cluster at once. - Mongo: conversations. - The chat's bucket, only when the chat keeps its files in S3 (CHAT_BLOB_STORE=s3, with the bundled RustFS or your own S3): the files' bytes. The Mongo backup then holds just their index, so restore both from the same moment. The commands below copy it out through the S3 API. (Copying the rustfs-data volume of a running RustFS is not a safe backup.) - The Authelia volumes: the signing key, the users file, and the subject store. Only with the bundled Authelia; with an external identity provider the accounts live there, and these volumes hold nothing to keep.

Not kept, on purpose: Valkey (rate-limit counters and caches, rebuilt as requests arrive) and Caddy's volumes (certificates, fetched again).

B=backup-$(date +%Y%m%d-%H%M%S); mkdir -m 700 "$B"
cp .env "$B/"
docker compose exec -T postgres pg_dumpall -U gateway > "$B/postgres.sql"
docker compose exec -T chat-mongo mongodump --quiet --archive --gzip > "$B/chat-mongo.archive.gz"
P=cerea    # COMPOSE_PROJECT_NAME: the volumes are named after it
docker run --rm -v ${P}_authelia-config:/authelia-config:ro -v ${P}_authelia-data:/authelia-data:ro \
  -v "$PWD/$B:/out" alpine tar czf /out/authelia.tgz -C / authelia-config authelia-data

With the chat's files in S3, copy the bucket out after the Mongo dump (a file added in between is harmless; the other order is not). One rc helper serves the backup and the restore: rclone, pinned, run in a container and configured from .env through variables, so the keys are not on a command line. It joins the objects network, which is how it reaches the bundled RustFS; with your own S3, leave NET empty:

set -a; . ./.env; set +a
P=cerea    # COMPOSE_PROJECT_NAME
NET="--network ${P}_objects"      # the bundled RustFS; with your own S3: NET=
export RCLONE_CONFIG_CHAT_TYPE=s3 RCLONE_CONFIG_CHAT_PROVIDER=Other RCLONE_CONFIG_CHAT_NO_CHECK_BUCKET=true \
  RCLONE_CONFIG_CHAT_ENDPOINT="$CHAT_S3_ENDPOINT" RCLONE_CONFIG_CHAT_FORCE_PATH_STYLE="${CHAT_S3_FORCE_PATH_STYLE:-true}" \
  RCLONE_CONFIG_CHAT_REGION="${CHAT_S3_REGION:-us-east-1}" \
  RCLONE_CONFIG_CHAT_ACCESS_KEY_ID="$CHAT_S3_ACCESS_KEY_ID" RCLONE_CONFIG_CHAT_SECRET_ACCESS_KEY="$CHAT_S3_SECRET_ACCESS_KEY"
rc() { docker run --rm --user "$(id -u):$(id -g)" $NET $(env | sed -n 's/^\(RCLONE_CONFIG_CHAT_[A-Z_]*\)=.*/-e \1/p') \
  -v "$PWD/$B:/data" rclone/rclone:1.73.0 "$@"; }
rc sync "CHAT:$CHAT_S3_BUCKET" /data/objects      # after the mongodump above

(Your own S3 is the AWS S3 proper, with no CHAT_S3_ENDPOINT? Use RCLONE_CONFIG_CHAT_PROVIDER=AWS, leave the endpoint out, and set CHAT_S3_REGION. A CHAT_S3_PREFIX limits the chat to that part of the bucket: add --include "$CHAT_S3_PREFIX**" to copy only it. The Config file ... not found notice is harmless.)

Prefer it automatic, scheduled and off the box? The restic sidecar writes exactly these files, and the bucket's objects, encrypted, to a repository of your choice.

To restore into a fresh install with the same .env:

cp "$B/.env" .env && chmod 600 .env
docker compose up -d --wait postgres chat-mongo
docker compose exec -T postgres psql -U gateway -d postgres < "$B/postgres.sql"
docker compose exec -T chat-mongo mongorestore --quiet --archive --gzip --drop < "$B/chat-mongo.archive.gz"
# The archive stores `authelia-config/` and `authelia-data/` at its root;
# unpack, then move each into its volume.
P=cerea    # COMPOSE_PROJECT_NAME
docker run --rm -v ${P}_authelia-config:/c -v ${P}_authelia-data:/d -v "$PWD/$B:/in:ro" \
  alpine sh -c "tar xzf /in/authelia.tgz -C /tmp && cp -a /tmp/authelia-config/. /c/ && cp -a /tmp/authelia-data/. /d/"
docker compose up -d --wait

If the backup has an objects/ directory (the chat's files were in S3), put the files back into the bucket before the chat serves requests: start the object store alone first (with the bundled RustFS, docker compose up -d --wait rustfs; your own S3 needs nothing), copy, and only then bring the stack up. Replace the last line above with:

docker compose up -d --wait rustfs       # the bundled RustFS only; the bucket is empty
set -a; . ./.env; set +a                 # then define NET and the `rc` helper as in the backup above
rc mkdir "CHAT:$CHAT_S3_BUCKET"          # a new bucket; harmless when it exists
rc copy /data/objects "CHAT:$CHAT_S3_BUCKET"
rc check /data/objects "CHAT:$CHAT_S3_BUCKET"      # "0 differences found"
docker compose up -d --wait

Restoring from the restic sidecar's repository? $B is the directory its restore command fills; the steps are the same.

Take a backup before every upgrade: database migrations only go forward.

Restore all the parts from the same backup: Postgres (accounts, keys, the ledger), Mongo (conversations), the bucket's objects (the conversations' files; Mongo holds only their index), the authelia-config volume (the users file) and authelia-data (every person's subject identifier and sessions) reference one another, and restoring one without the others leaves accounts pointing at logins or subjects that no longer exist. The bootstrap creates the first account only when the users file is absent — it never recreates or overwrites an existing one — so a restored file stays exactly as backed up.

Moving from a `pystino init` install¶

An .env written by pystino init uses the same variable names. To move such an install:

Stop the old stack without -v.
Copy its .env here and keep its COMPOSE_PROJECT_NAME. The volumes are found by that name.
Run ./configure --check. It names the keys that no longer mean anything (COMPOSE_FILE, PYSTINO_DEPLOY_DIR, the version pins); remove them with ./configure --unset ….
Run docker compose up -d --wait.

The Caddy and Authelia configuration now comes from this kit instead of the pystino-proxy and pystino-authelia images, which are no longer needed.

Troubleshooting¶

chat-mongo restarts and its log says Illegal instruction: the CPU has no AVX but MONGO_IMAGE is MongoDB 5 or later. Set ./configure --set MONGO_IMAGE=mongo:4.4.30, and only if the data was never upgraded; otherwise restore the backup tools/mongo-upgrade took. On a MongoDB 8 that fails at start on a recent kernel, see MongoDB.
Asking for help: run tools/diagnose and attach the file it writes (see Reporting a problem).
docker compose logs <service> --tail 50 is the first stop. The bootstrap service explains any refusal in one line ("fix .env and run docker compose up -d again").
/chat redirects in a loop, or sign-in never sticks: the origin in .env doesn't match the one in the browser, or the host is a single-word name such as myserver (the bundled Authelia needs a dotted name or an IP address).
denied or manifest unknown on docker compose pull: .env names a version or registry that is not published (an old CEREA_VERSION/PYSTINO_VERSION override, or *_REGISTRY='local' left by dev/build.sh). Run dev/build.sh --reset, or remove those keys.
The console shows no models: add a provider and its models in the console. An upstream key in .env seeds a provider row, but no model is offered until one is configured.

Building the images yourself¶

Only for development, or to run commits that have no published image yet (both source repositories are public):

dev/build.sh           # clone Pystino and Cerea at the pinned commits, build, point .env at the result
docker compose up -d
dev/build.sh --reset   # back to the published images
dev/build.sh --pystino-ref main --cerea-ref main   # other commits
dev/build.sh --ner     # the NER redaction image, needed by the enterprise preset (never published)

An image is built only when local/<name>:sha-<commit> is missing. The first build of the chat image needs about 2 GB of memory. Maintainers re-pin a release with tools/pin --pystino <tag> --cerea <tag>, and CI checks the pins with tools/pin --check.

Changing the kit¶

Setup, tests, CI and the release procedure for the kit are in the "Deploy kit" and "Releasing" sections of Cerea's CONTRIBUTING.md.