Compose a PRView SourceOpen Issue
> **Synced from Hive.** This page is pulled from [hivecommons/hive@v5](https://github.com/hivecommons/hive/blob/v5/src/docs/release-channels.md) during the docs build. Edit the canonical source in the Hive repository.

# Release Channels

Hive publishes three **release channels** — moving GHCR image tags an operator can point a hive at instead of a branch tag:

| Channel | Intended meaning |
|---|---|
| `stable` | Newest build promoted as generally safe to run. |
| `candidate` | A build believed good, awaiting soak before promotion to stable. |
| `edge` | The newest good build, with no soak period. |

> **Promotion policy:** the channels diverge by release line and maturity. Every green merge to **`v5`** retags **`candidate`** (and `:latest`); **`stable`** advances later by digest through the scheduled/manual stable-promotion workflow after the [stable soak and promotion policy](https://github.com/hivecommons/hive/blob/v5/src/docs/stable-soak-policy.md) passes. Merges to **`v6`** retag **`edge`**, so `edge` is an active-development v6 build, not a synonym for `stable`. **`v4`** is a maintenance line: its builds publish `v4-latest` and short-SHA tags, no channel (#7721 Phase 1).

The hub's release-channel block also shows the stable auto-promotion state.
Hub admins see a play/pause control on the `stable` row: play (the default)
lets the hourly promotion workflow catch `stable` up to `candidate` after the
24-hour soak and maintained-hive smoke evidence; pause records who paused and
when, and the workflow skips without moving tags until resumed. Non-admins see
a read-only badge.

## How channels are published

Channels are **retags, not rebuilds**. Each release line's `docker.yml` workflow adds fast-moving channels as extra tags in the same `docker buildx imagetools create` call that publishes the branch's `-latest` and immutable short-SHA tags, so a channel always points at an already-built, multi-arch digest. Builds of branch `v5` publish `candidate`; the separate stable-promotion workflow later retags `stable` by candidate digest after the soak gate passes. Builds of branch `v6` publish `edge`. All three images get their line's channels in both published orgs:

- `ghcr.io/hivecommons/hive` and `ghcr.io/hivecommons/hive`
- `ghcr.io/hivecommons/hive-contributor` and `ghcr.io/hivecommons/hive-contributor`
- `ghcr.io/hivecommons/hive-hub` and `ghcr.io/hivecommons/hive-hub`

The `hivecommons` packages are mirror tags of the same manifest digest as the native `hivecommons` packages during the org transfer, so operators can verify or pin the digest against either registry. builds of the release branches (`v5`, `v6`) publish channels — a feature-branch build can never move a production channel.

Publishing is monotonic by workflow run number. Every successful multi-arch build receives its immutable short-SHA tag even if a newer merge has already reached the branch. If that exact short-SHA tag already exists, a re-run leaves it untouched. Moving tags (the branch's `-latest` tag and fast channels such as `candidate`/`edge`) advance when that build is newer than the generation currently published; an older workflow that runs out of queue order publishes any missing immutable tag. `stable` uses the same generation guard during digest promotion, so a delayed promotion cannot move it backwards over a newer stable. Registry inspection failures fail the publish or promotion job instead of producing a silent green skip.

Short-SHA tags are retained as a bounded rollback/debug window, not forever. The scheduled GHCR pruning workflow deletes old package versions whose complete tag set is or more 7-hex short-SHA tags, after 90 days. Versions still carrying any moving tag (`v4-latest`, `latest`, `stable`, `candidate`, `edge`, or future channel names) are never deleted by that cleanup.

Pinning a hive back to of those short-SHA builds for all three images, and verifying by digest that the pin landed on the running spoke, is the [digest-verifiable rollback](https://github.com/hivecommons/hive/blob/v5/src/docs/release-rollback.md) runbook. Switching channels (below) is not a rollback: a channel is a moving tag, and the switch is judged complete when the heartbeat reports a matching *tag*, not a matching digest.

## Switching a hive to a channel

From the hub dashboard's **My Hives** list, click the blue version pill on a hive row. The menu lists branches first, then a **Channels** section with the three channels (most stable first). the hive's **owner** can switch.

Under the hood this is the same endpoint as a branch switch — the channel name goes in the `branch` field verbatim:

```text
POST /api/saas/hives/{id}/switch-branch
{"branch": "stable"}
```

The hive's image is set to `ghcr.io/hivecommons/hive:stable` (via kubectl for reachable hosted spokes, or delivered on the next heartbeat otherwise). The switch is considered complete when the spoke's heartbeat reports an image ref whose tag matches the channel ([#3761](https://github.com/hivecommons/hive/pull/3761)).

Because a channel tag is a moving (mutable) tag, a channel-tracking hive gets the same floating-tag auto-upgrade treatment as a `-latest` branch tag: upgrades roll the pod but keep the `:stable` image string rather than pinning a SHA ([#3757](https://github.com/hivecommons/hive/pull/3757)).

## The version pill: `stable (v4)`

A channel is a moving pointer, so the dashboard shows what it currently points at. The pill on a channel-tracking hive reads, for example:

- `stable (v4)` — the channel currently resolves to a build of branch `v4`;
- `stable (49e53e6)` — the channel resolves to a digest the hub could not attribute to a tracked branch (short digest shown);
- `stable (?)` — the channel tag could not be resolved on GHCR at all.

Resolution is live: the hub HEADs the GHCR manifests for each tracked branch's `-latest` tag and each channel tag, and matches digests ([#3742](https://github.com/hivecommons/hive/pull/3742)). Results are cached for 5 minutes; an unresolved refresh is never cached, so a transient GHCR blip retries on the next poll rather than latching `unknown` for the TTL ([#3721](https://github.com/hivecommons/hive/pull/3721)). If channel rows render as `unknown`, grep the hub log for `channel resolve:` — each failure path logs a WARN naming the cause (token failure, 401/403 package permission, 404 tag never published).

The **My Hives** page also shows a `Release channels:` block above the per-branch `Latest available images:` rows, mapping each channel to its currently resolved branch/digest. Each branch row also carries a compact per-line image-pulls bar chart (package pulls landing during each of that line's release windows; `—` when the line has no closed window yet), and the header's "Pulls per release" chart follows the **active** line — the branch `stable` currently resolves to — rather than any hard-coded branch.

## Channel distance: how much is queued for promotion

Each row in the `Release channels:` block also shows how far that channel has drifted from the stage **immediately upstream** of it in the promotion order ([#6418](https://github.com/hivecommons/hive/pull/6418)): `stable` is measured against `candidate`, `candidate` against `edge`. Edge has no upstream — it is where builds enter — so its row carries no distance. The upstream stage is derived from the promotion order rather than hardcoded, so a new track wires itself up. Measuring every channel against edge instead would roughly restate the sum of the hops and could not distinguish a starved soak from a stalled promotion; hop per row keeps each number actionable.

How to read it:

- `↓N vs candidate` (amber) — N commits are on `candidate` that `stable` does not have: the promotion backlog for that hop.
- `↑N` (blue) — N commits are on this channel that its upstream does not have. Both arrows can appear at: the channels follow different branches, and a branch synced from another both carries commits the other lacks and misses commits merged since the sync. GitHub's compare API reports this as *diverged*, and the UI shows **both** counts rather than collapsing them into a direction that does not exist. The tooltip spells out the full sentence.
- `in sync with candidate` — the two stages resolve to the same commit (or the compare returned no counts).
- `↓N 3d 5h vs candidate` — when a row is a plain ancestor of its upstream (compare status *behind*), the amber duration after the count is how much **older** the promoted build is than the upstream's: the difference between the two rows' commit timestamps. It appears for that pure-behind case. A *diverged* pair (`candidate` on v5 vs `edge` on v6) shares no single line of history, so a time gap there would compare two unrelated clocks — those rows show commit counts.
- **No distance shown at all** — the compare could not be resolved. This is deliberate: rendering `0` would read as "level with upstream", the answer that must never be guessed, since it turns a stalled promotion into a healthy-looking row.

Each row also carries the **commit timestamp** of the build it points at (`<sha> 2026-09-21 16:51`, viewer's local time, minute precision; the tooltip has the UTC RFC3339 value). It is the committer date of that SHA, fetched hub-side (`pkg/hub/channel_commit_date.go`) and cached permanently. A row whose date could not be fetched shows no stamp rather than a placeholder.

Distances are computed hub-side (`pkg/hub/channel_distance.go`) via GitHub's compare API and cached permanently: the distance between two fixed commits is immutable, and a moved channel is a new SHA pair, so entries become unreferenced rather than stale — there is no TTL after which a shown distance could be wrong.

Both reads, like every other hub-originated `api.github.com` call (branch tips, commit messages, workflow runs), are made anonymously unless **`HIVE_HUB_GITHUB_TOKEN`** is set on the hub. The anonymous budget is 60 requests/hour per source IP and the branch poller alone exhausts it the hub tracks more than a couple of branches; GitHub then answers `403`/`429`, the compares fail, and the distance column and timestamps silently disappear (the hub logs `channel distance: compare failed … HTTP 403`). Set the token (a fine-grained or classic token with public-repo read; 5000 requests/hour) and the rows come back on the next 5-minute channel refresh. See [`env-vars.md`](https://github.com/hivecommons/hive/blob/v5/src/docs/env-vars.md).

## Persistence: the tracked channel is durable

The hive's tracked channel is stored hub-side in the per-hive metadata record (`tracked_channel` in `/data/saas/hives/<hive-id>/meta.json`, on the hub PVC). It is set when you switch to a channel and cleared when you switch to a plain branch. Two failure modes are specifically handled:

- **Heartbeats do not erase it.** A channel image is a build of `v4`, so the spoke heartbeats `git_branch=v4`; the pill overlays the persisted channel at read time instead of trusting the reported branch ([#3750](https://github.com/hivecommons/hive/pull/3750)).
- **Hub restarts do not drop an in-flight switch.** The in-memory switch instruction is re-armed from the persisted record on the next heartbeat, as long as the spoke is not mid-upgrade and its reported image tag differs from the tracked channel ([#3771](https://github.com/hivecommons/hive/pull/3771)). This also self-heals a hive whose delivered upgrade drifted it off the channel tag.

## Spoke navbar badge

A spoke running a channel image shows both delivery dimensions in its own dashboard version badge: for example, `a1b2c3d · stable (v4) · floating`. The linked short SHA identifies the running build, `stable (v4)` identifies the release channel and built-from branch, and `floating` confirms that the Deployment follows a mutable tag. `candidate` and `edge` are displayed the same way when reported by the backend. A branch tag such as `v5-latest` renders as `a1b2c3d · v5 · floating`, while a SHA tag or digest pin renders as `a1b2c3d · v5 · pinned`.

The tooltip includes the running SHA, channel when present, built-from branch, authoritative Deployment image ref, and tracking mode. The spoke derives both channel and tracking mode server-side from its cached in-cluster Deployment image lookup; browser code does not infer mutability from tag strings. If the Deployment cannot be read (for example, during a plain `docker run`) or its image ref is malformed, the badge says `tracking unknown` and the API does not expose the untrusted ref. This preserves the distinction between unknown provenance and an intentional pin. The channel portion was introduced by [#3762](https://github.com/hivecommons/hive/pull/3762); the tracking detail is specified by [#6321](https://github.com/hivecommons/hive/issues/6321).

On a self-hosted Podman Quadlet spoke there is no Kubernetes Deployment to read. `bin/hive-podman-setup.sh` therefore writes the unit's own image metadata into `hive.env`, which `hive.container` already loads through `EnvironmentFile=`: `HIVE_SELF_IMAGE` carries the `Image=` reference and `HIVE_SELF_IMAGE_TRACKING` is `registry` when Quadlet registry auto-update is enabled for a non-digest image, otherwise `pinned`. The spoke dashboard uses those variables when the Deployment lookup is unavailable, so Kubernetes spokes keep the Deployment-derived behaviour above. A Quadlet spoke running `Image=ghcr.io/hivecommons/hive:candidate` with registry auto-update shows `candidate · tracking registry` in the header and `Channel: candidate` in the release-status panel; a digest-pinned Quadlet image shows pinned tracking and the digest while leaving the channel unresolved.

## Spoke self-service selector

Hosted spoke dashboards that are already following a release-channel tag show a **Follow channel** selector in the release-status panel. Choosing `stable`, `candidate`, or `edge` calls the spoke-local `POST /api/release-channel`, which relays the request to the hub's existing `POST /api/saas/hives/{id}/switch-branch` path using the spoke's dashboard-token proof. The hub still performs the authorization, channel/tag validation, GHCR publishability check, tracked-channel persistence, and kubectl-or-heartbeat delivery.

The spoke UI keeps reported state and intent separate: after a selection it continues to show the channel observed from the Deployment image, plus a pending "switch requested" note, until the rollout/heartbeat lands on the requested tag. Spokes that are self-hosted, pinned, branch-tracking, missing hub credentials, or otherwise unresolved show an honest "selection unavailable" explanation rather than a dead control.

For self-hosted Podman Quadlet spokes the selector is intentionally unavailable even when `HIVE_SELF_IMAGE` resolves a channel, because there is no hub-managed Deployment image to patch. The panel says to change `Image=` in `hive.container`; after editing, reload/restart through the Podman lifecycle so `hive.env` carries the updated `HIVE_SELF_IMAGE` and tracking mode.

## Known limitations

- **Bulk actions cannot set a channel.** The bulk *Switch branch* action validates against real branches and rejects channel names (`unknown branch`); it also never writes the tracked channel. Switching to a channel is per-hive.
- **A manual Upgrade on a channel-tracking hive resolves through the channel tag.** If `:stable`/`:candidate`/`:edge` already points at the commit the spoke is running, the hub refuses the click with a visible explanation instead of arming a no-op heartbeat upgrade. Operators who need a newer build must wait for the channel to advance or switch the hive to a newer channel/branch tag.
- **The spoke dashboard offer uses that same target.** `/api/version` keeps reporting the branch tip for provenance, but the `behind` flag, behind count, and Upgrade button are measured against the hub-delivered upgrade policy when present. A `:stable` spoke therefore does not offer "Upgrade available → <branch tip>" while `:stable` itself still points at the running commit.

## Channel-aware upgrade targeting

Automatic upgrade targeting resolves **through the tag the spoke's Deployment tracks**, not around it ([#5994](https://github.com/hivecommons/hive/issues/5994), landed in [#6005](https://github.com/hivecommons/hive/pull/6005), `pkg/hub/channel_targeting.go`).

The stable soak policy made this necessary: per-merge publishes move `candidate`, while `stable` advances later by digest. A spoke's Deployment tracks image tag, and rolling the pod re-pulls that tag — nothing the hub instructs can make a restart land on a digest the tag does not carry. A hub that targets branch HEAD is therefore asking a `:stable` spoke to reach a digest its own tag is deliberately withholding; before the fix this looped 41 spokes into permanent `UPGRADE FAILED` (hub instructs a SHA, spoke rolls, re-pulls `:stable`, reports the SHA it started on, hub re-sends the identical instruction).

How targets are now resolved (`reachableUpgradeTarget`, used by the manual upgrade handler, the auto-upgrade sweep, and the heartbeat spoke-managed path):

- **Which channel a spoke is on:** the spoke's *reported image ref* leads, because it is what the kubelet will pull; the hub-side `tracked_channel` record is intent, and the two disagree exactly while a channel switch is still on the wire. `tracked_channel` remains the fallback for spokes too old to report an image ref. Branch tags and SHA pins resolve to branch targeting, exactly as before.
- **Channel → commit:** the hub walks GHCR from the channel tag to the image index, picks the `linux/amd64` platform manifest (buildx attaches `unknown/unknown` provenance descriptors to the same index, so position is not enough), and reads the `org.opencontainers.image.revision` OCI label from the config blob — the commit identity that survives a retag. Answers are cached for 5 minutes (`channelDigestTTL`); when a refresh fails, the last good answer is served for up to 4× that (`channelRevisionStaleGrace`, 20 minutes) since a channel moves at most hourly, then resolution is treated as failed.
- **Unresolved channel = hold, loudly.** If the channel does not resolve to a commit, the hub instructs *nothing* for that spoke and logs a WARN — falling back to branch HEAD would be exactly the bug. Grep the hub log for `auto-upgrade held — the spoke's release channel did not resolve to a commit` (sweep) or `heartbeat: upgrade instruction withheld — the spoke's release channel did not resolve to a commit` (heartbeat path). The next cycle retries.
- **Downgrade guard.** A channel is a moving pointer, not a monotonic branch, so a spoke can legitimately sit *ahead* of the channel it tracks (rolled while the channel was further along, or switched from `candidate` moments ago). Such spokes are skipped rather than instructed to downgrade (`no upgrade — spoke is at or ahead of its release channel`).
- **"Up to date" is judged through the tag too.** The stale-latch recovery clears a floating-tag hive's upgrade latch it runs the newest build *its tag can deliver* (`clearing upgrade latch — floating-tag hive is at latest`). Judging a `:stable` spoke against branch HEAD instead kept it latched for the whole soak window — the failure mode measured above.

## Grouping hives by upgrade state

The My Hives **GROUP BY** selector includes an `Upgrade state` dimension ([#3805](https://github.com/hivecommons/hive/pull/3805)) with three buckets: `Queued (ready, not yet upgrading)` (auto-upgrade on and a target armed), `Upgrading`, and `Up to date`. Note that `Queued` requires auto-upgrade to be enabled — a hive that is behind latest with auto-upgrade off sorts under `Up to date`.