Skip to content

Indexes & retention

The Admin → Indexes panel shows what's stored and lets you manage retention. Indexes are created implicitly on first write — there's no index to declare — so this panel is about observing and aging out data, not provisioning it.

The Indexes panel: catalog summary and per-env index → day-partition rollup with doc/segment/size counts

The catalog

The panel lists the partition catalog as a rollup: environment → index → day-partition, with document, segment, and size counts. It refreshes periodically so counts tick up live during ingest. Use it to see which indexes exist, how big they are, and how data is distributed over time.

Retention

Retention drops whole day-partitions once they age past a limit. It's configured at two levels:

LevelWhereNotes
Global defaultretention.default_days (Admin → General) or HELIOS_RETENTION_DEFAULT_DAYSApplies to every environment without an override. The env var, when set, wins and locks the field.
Per-environmentAdmin → EnvironmentsOverrides the global default for one environment.

If neither is set, data is kept forever. A value of 0 (or blank) also means keep forever.

How the sweep works

A background sweeper checks for expired partitions on an interval (HELIOS_RETENTION_SWEEP_SECS, default hourly) and drops day-partitions older than the effective retention. In a cluster a single elected node performs the deletes; on a shared store, deletions leave a tombstone so replicas converge and never resurrect dropped data.

Retention deletes data permanently

Dropped day-partitions are gone. Set per-environment overrides deliberately, and keep backups (or a replicated bucket) if you need a longer recovery window than your retention.

Garbage collection

POST /api/admin/gc runs a retention sweep immediately and reports how many partitions and blocks it dropped — useful right after lowering a retention setting, instead of waiting for the next scheduled sweep.

Compaction is automatic

The block engine compacts small blocks into larger ones on its own (see Performance tuning). The older merge / commit maintenance actions are kept as no-ops for compatibility — there's nothing to trigger manually.