Custody Tiers of Customer Data

This page is written for a security reviewer. It answers one question: where does customer data actually sit, who can reach it, and which of those places are protected by cryptography versus merely by access control.

"Custody" is used here in a precise sense: a tier is a physical location plus the conditions under which someone can read or delete the data in it. Four tiers exist today.

---

The four tiers at a glance

Tier	Where the data physically lives	Protected by	Deletion behaviour
Endpoint spool	~/.controlzero/spool/ on the operator's machine	encrypted at rest, key in the OS keystore by default	Auto-delete after ack + retention floor; bounded by a 256 MiB cap
Platform database	Platform Postgres (operational tables + secrets vault schema)	an encrypted envelope for stored secrets; RLS + app-layer org scoping	No retention auto-delete for operational rows; secrets deleted via the API
Audit store	audit_logs in a dedicated Postgres audit database	Append-only trigger; captured-I/O payloads encrypted per-record	No scheduled whole-row delete in production (audit_retention_days is stored but not yet enforced); deleting an organization removes that org's rows at the time of deletion, and if that removal fails the deletion still completes with the failure logged for manual follow-up; I/O payload crypto-shred past io_capture_retention_days
Archive	object storage	public-key-encrypted WAL (hosted platform); cold export carries ciphertext columns	WAL retained by object-store versioning; cold tier drops from Postgres only after dual-cloud verify

---

The boundary: what never leaves the endpoint

Three things stay on the endpoint and are never shipped to the platform:

1. The plaintext convenience log ~/.controlzero/audit.log (or ./controlzero.log in local-only mode). It is a human-readable JSON Lines copy and is never uploaded; the server-side store is a separate, independently-generated record.
2. The spool data-encryption key (DEK). When the OS keystore is used, the DEK never touches the disk on the endpoint, and it is never sent to the platform.
3. The spool's encrypted-at-rest form. The spool is decrypted locally and uploaded as plaintext audit entries over TLS; the server never receives the spool DEK or the spool ciphertext.

What does leave the endpoint is the audit entry itself. That includes id, project_id, user, hostname (plus user_email on enrolled devices), tool_name, decision, reason, policy_id, client name and version, an args_hash, and DLP outcome fields. The full args map is sent only when the integration supplies it, and the server then drops it, stores it encrypted, or stores it in plaintext according to the org's audit redaction mode. When I/O capture is enabled, the captured input/output payload is sent too. It lands in the audit store (tier 3).

---

Tier 1 — Local endpoint spool

For hosted (API-key) clients the SDK by default writes each audit event to a durable, encrypted, append-only spool before attempting any network send, then drains it to the backend in the background; CONTROLZERO_SPOOL=off disables it. If the spool cannot open, for example because a required dependency is missing or the disk is full, the SDK logs a warning, increments spool_init_degraded_total, and falls back to in-memory buffering, which is not durable.

Location and permissions. ~/.controlzero/spool/ (override CONTROLZERO_SPOOL_DIR). Directories are 0700, files 0600, and the implementation refuses to follow symlinks.

Cryptography. Each record is encrypted at rest under a per-device 32-byte DEK. The DEK is stored in the OS keystore by default (macOS Keychain / Linux Secret Service); the on-disk spool.key then holds only a sentinel. If no keystore is available the DEK falls back to an on-disk 0600 file, which co-locates the key with the ciphertext and is explicitly a weaker posture. This encryption protects against other local users, file exfiltration, and lost disks, but not root, same-UID malware, or the developer themselves before upload.

Retention and deletion.

• A sealed segment is deleted only when every record in it is acknowledged by the server and it is older than CONTROLZERO_SPOOL_RETENTION (default 24 hours). The floor exists for local forensics.
• A hard disk budget CONTROLZERO_SPOOL_MAX_BYTES (default 256 MiB) is enforced in this order: delete acked segments older than the floor, then acked segments younger than the floor, then evict the oldest unacked segments. Eviction is the documented data-loss path and is never silent: it writes an evicted control record and a synthetic spool_data_loss audit entry, and bumps the device epoch so the stream can continue.

What an administrator can and cannot do. The spool is controlled by environment variables (CONTROLZERO_SPOOL, CONTROLZERO_SPOOL_DIR, CONTROLZERO_SPOOL_RETENTION, CONTROLZERO_SPOOL_MAX_BYTES, CONTROLZERO_SPOOL_KEYCHAIN). An administrator can turn the spool off (CONTROLZERO_SPOOL=off), shorten the retention floor, raise or lower the disk cap, or relocate the spool. While the spool is on, they cannot stop the cap (default 256 MiB) from evicting the oldest unacked records under pressure, and they cannot remotely read or decrypt another machine's spool.

---

Tier 2 — Platform main database and the secrets vault

This is the platform's own Postgres: organizations, users, projects, policies, entitlements, and — separately — the secrets vault where customer credentials and API keys are stored.

Operational rows. These are access-controlled, not encrypted at rest: row-level security plus application-layer WHERE org_id = $1 scoping. RLS is defence-in-depth; the application's org predicate is the real boundary.

Secrets vault. Customer secrets live in cz_secrets and are sealed with encryption at rest. Envelope v1 seals everything under one master key derived from MASTER_KEY_PASSPHRASE with a memory-hard key-derivation function; envelope v2 wraps each secret under a per-org DEK that is itself wrapped by the master KEK. Postgres performs no crypto: sealing and unsealing happen in the application before the row is written and after it is read. Envelope v2 is additive and inert by default; re-sealing existing rows is a separate, human-gated step.

Retention. There is no age-based retention sweep for the operational database — the retention workers operate only on the audit store, described next. Secrets are removed through the API. A separate, dormant orphan janitor reaps vault ciphertext rows whose metadata pointer is gone (a deleted project), and only after a grace window (default 30 days, operator-tunable via CZ_VAULT_ORPHAN_GRACE) and only when CZ_VAULT_ORPHAN_JANITOR=true.

What an administrator can and cannot do. A platform operator can rotate the master key with czctl rotate-master-key, which requires the backend to be stopped first and re-encrypts every envelope-v1 secret in a single transaction. If any envelope-v2 row exists it refuses and commits nothing; v2 keys are rotated with the separate re-seal tool. Operators cannot read a stored secret in plaintext without going through the unseal path, which requires the master key.

---

Tier 3 — The audit store

Every uploaded decision lands in audit_logs, the authoritative record shown in the dashboard. It lives in a dedicated Postgres audit database, separate from the transactional database, and that database is the only audit read source in production.

Append-only by trigger. In the postgres-audit backend, a trigger rejects ordinary UPDATE and DELETE against audit_logs. There are exactly two code-mediated carve-outs. Each is gated by a session GUC that the backend arms only with SET LOCAL, inside its own erasure, purge or partition-maintenance transaction. Partition maintenance uses the delete carve-out only to move stranded rows between partitions of the same table, with every value unchanged:

• DELETE is permitted only when the session sets app.audit_allow_delete (the lawful-erasure carve-out).
• A narrow UPDATE is permitted only when the session sets app.audit_allow_io_shred and the statement nulls io_dek_wrapped, sets io_purged/io_purged_at, and changes nothing else.

Whole-row retention. Each org has an audit_retention_days setting, an org-tunable window of 1 to 2555 days (7 years), but no scheduled job deletes production audit-store rows past it today, so audit rows are kept until deleted. Production deletes whole audit rows in one case only: organization deletion. The owner can delete a single-owner, empty, free-tier organization, and that removes every audit row for the org through the trigger's lawful-delete carve-out. If that removal fails, the organization deletion still completes and the failure is logged for manual follow-up. When the removal succeeds it is a real DELETE, not a cryptographic erasure: the rows are gone.

Captured-I/O retention (crypto-shred). Captured input/output payloads are stored inline as input_ciphertext / output_ciphertext, sealed under a fresh per-record DEK that is wrapped by the org KEK and stored inline as io_dek_wrapped. A second worker runs every 6 hours and, past the org's io_capture_retention_days (default 30, org-tunable 7-365), crypto-shreds the payload by nulling io_dek_wrapped — the ciphertext becomes unrecoverable from the live database while the audit row's forensic fields stay intact. On the hosted platform, earlier copies of the row in the WAL archive and in the weekly encrypted audit-database dumps keep their wrapped key for as long as those archive objects are retained.

What an administrator can and cannot do.

• Reading either retention setting requires admin or higher; changing it requires the org owner.
• A manual I/O purge (one row or a range) is owner-only, requires a non-empty reason, enforces MFA step-up where configured, and is recorded to audit_purge_log before the key is destroyed.
• An administrator cannot edit or delete an individual audit row ad hoc: the append-only trigger rejects it. Through the gated paths above they can set both retention windows (only the captured-I/O window is currently enforced) and crypto-shred captured I/O. An owner can also delete a qualifying organization (single-owner, empty, free tier), which removes all of its audit rows; if that removal fails, the organization deletion still completes and the failure is logged for manual follow-up.

---

Tier 4 — Archive (WAL and cold tier)

The platform keeps three archive paths that copy customer data off the primary databases into object storage. The third is a weekly logical dump of the live tenant database (including the secrets schema) and the audit database. Each dump is encrypted to a recipient public key on the host before upload.

WAL archive to object storage (hosted platform). On the hosted platform, every closed Postgres WAL segment is encrypted to a recipient public key and uploaded to object storage, one object per segment, grouped by day. The Postgres container has no internet egress by design; a wal-archiver sidecar drains a local spool to object storage. The archive command is idempotent and object-store native versioning preserves prior uploads. The recipient public key is supplied through deployment configuration; the corresponding private key is not held on the archiver, so the archived segments are cryptographically withheld from anyone who can read the bucket but not the recipient key. The default self-managed and air-gapped stacks do not archive WAL; back those up with your own database backup tooling.

Audit cold tier. An operator tool moves audit_logs partitions older than the hot window out of postgres-audit into both object-store destinations as Parquet+zstd, and — only after a verified dual-cloud export matches the live partition on row count and checksum — drops them from Postgres. This tool is dormant by default (AUDIT_COLD_TIER_ENABLED unset) and the default hot window is 180 days. Cold objects are read back through DuckDB; encrypted bytea columns return as opaque BLOBs because the audit encryption key is not available to a raw SQL reader.

No S3 Object Lock. The archive path does not configure S3 Object Lock / WORM retention. Append-only-ness is enforced by the database trigger (tier 3), the WAL archive relies on object-store native versioning, and the cold tier relies on dual-cloud verification — none of these is an object lock. Do not describe the archive as immutable-by-lock; it is encrypted and dual-cloud-verified.

What an administrator can and cannot do. The archive paths are operator (platform) functions driven by environment and CLI flags, not tenant-visible settings. A tenant admin has no path to read or delete archived objects; an operator can arm/disarm the cold tier, set the hot window, and supply object store credentials. The dormant-by-default cold tier means nothing is exported until a human arms it.

---

Cryptographically withheld vs merely access-controlled

This is the distinction a reviewer should carry out of this page.

Data	Protection class	Where the key lives
Spooled audit records (endpoint)	Cryptographic — encryption at rest	Endpoint OS keystore (or 0600 file fallback)
Captured I/O payloads (audit store)	Cryptographic — per-record DEK, wrapped by org KEK	Wrapped key inline on the row; crypto-shred nulls it
Stored secrets (vault)	Cryptographic — encryption at rest	Master key (MASTER_KEY_PASSPHRASE), or per-org DEK under master KEK
WAL archive segments	Cryptographic — public-key encryption	Recipient private key, held off the archiver
Audit row metadata (tool, decision, timestamps, ids)	Access-controlled — append-only trigger + RLS + app-layer org scoping	n/a — plaintext in the database
Operational rows (orgs, users, policies)	Access-controlled — RLS + app-layer org scoping	n/a — plaintext in the database
Hosted policy bundles	Cryptographic: signed and encrypted; the SDK verifies the signature before decrypting	Per-project bundle key, delivered to the project's SDKs at bootstrap

The practical rule: captured I/O payloads are encrypted at rest, and so are audit arguments under the redact_with_admin_unmask mode; arguments under redact are dropped and arguments under store_full are kept as plaintext JSON; metadata is access-controlled and append-only; deletion of a payload is done by destroying its key (crypto-shred), deletion of metadata is done by a gated DELETE.