# Rapids automation and diagnostics

This page covers scripts, CI pipelines and AI agents that deploy and debug Rapids
containers without a human watching. Field names and value domains here change
additively, never in place.

> **Read [Platform automation](platform-automation) first.** The status model,
> polling rules, envelope, capabilities and diagnose contract described there are
> identical for every DanubeData product — cache, databases, queues, VPS, object
> storage, static sites, managed apps and more. This page restates them in Rapids
> terms and then covers what only Rapids has: image preflight, endpoint probing,
> revisions and deployment operations.
>
> Every failure code is catalogued at [Failure codes](failure-codes).

## Authentication and project scope

All requests use a bearer token against `https://danubedata.ro`:

```bash
curl -H "Authorization: Bearer $DANUBE_TOKEN" \
     -H "X-Team-Id: 42" \
     https://danubedata.ro/api/v1/serverless
```

`X-Team-Id` selects the project. If you omit it, the account default is used —
rarely what automation wants. With the CLI:

```bash
danube --project 42 rapids get my-api --json
danube project select --project 42          # persists the default, no prompt
```

`--team` is accepted as an alias. Supplying both with different values is an
error rather than a silent preference, so automation cannot end up pointed at
the wrong project without noticing.

Token abilities:

| Ability | Grants |
|---|---|
| `serverless:read` | Container metadata, deployment history, revisions |
| `serverless:write` | Create, update, deploy |
| `serverless:delete` | Delete |
| `serverless:diagnostics` | Logs and platform events |

Logs are gated separately from `serverless:read` on purpose: a token that may
list your containers should not automatically be able to read what those
containers print.

## The status model

Every container response carries `status_details`. Read this rather than the
legacy `status` string — it is the same object the dashboard and the websocket
broadcast use, so they cannot disagree with you.

```json
{
  "summary": "failed",
  "health": "unhealthy",
  "observed_at": "2026-08-03T18:59:00+00:00",
  "stale": false,
  "operation": { "state": "failed", "terminal": true },
  "error": {
    "code": "serverless.image_pull_auth",
    "source": "reconciler",
    "resource": { "kind": "Revision", "name": "my-api-00007" },
    "reason": "ContainerMissing",
    "message": "The registry rejected the configured pull credential.",
    "retryable": false,
    "observed_at": "2026-08-03T18:59:00+00:00"
  }
}
```

| Field | Meaning |
|---|---|
| `summary` | `pending`, `in_progress`, `ready`, `degraded`, `failed`, `stopped`, `deleting`, `unknown` |
| `health` | `healthy`, `degraded`, `unhealthy`, `unknown` — describes what is **currently serving** |
| `operation.state` | `queued`, `running`, `succeeded`, `failed`, `cancelled` |
| `operation.terminal` | **The stop condition for polling** |
| `observed_at` | When the platform last checked the cluster — not when the row was last written |
| `stale` | The platform could not reach a live source; the summary is the last known one |

### Polling correctly

Poll until `operation.terminal` is `true`. Do not infer terminality from the
status string, and do not treat `health: "unknown"` as a failure — during a
rollout the platform genuinely does not yet know whether the new revision is
healthy, and saying so is more honest than guessing.

```bash
until [ "$(danube rapids get my-api --json | jq -r '.status_details.operation.terminal')" = "true" ]; do
  sleep 5
done
```

`summary: "degraded"` is terminal but **not** an outage: a new revision failed
while an older one keeps serving traffic. Your site is up. Redeploying the same
configuration will fail the same way.

### Waiting for your own change

Every change that needs a rollout — a create, an update, a redeploy — is given a
number. The response to that request carries it as `spec_generation`, and the
container reports `observed_generation`: the number the latest rollout started
from. Your change is being rolled out once `observed_generation` has reached
your number, and it has finished once `operation.terminal` is `true` as well.

```bash
GEN=$(danube rapids update my-api --tag "$SHA" --json | jq -r '.spec_generation')
until danube rapids get my-api --json \
  | jq -e --argjson gen "$GEN" '.observed_generation >= $gen and .status_details.operation.terminal' >/dev/null; do
  sleep 5
done
```

`danube rapids update --wait` (CLI 1.3.0 and later) does exactly this.

While a change is waiting for its rollout, `operation.state` is `queued` and
`operation.terminal` is `false`, even if the container itself is `running` — the
settled status describes the previous rollout, not yours.

Rollouts of one container run one at a time, in order. A change sent while a
rollout is still in progress is queued behind it and rolled out next; several
changes sent before a queued rollout starts are rolled out together. Nothing is
dropped.

During a rollout the previous revision keeps serving until the new one is
ready, and for a few minutes after that — `max-scale` limits each revision, not
the container. If your container does one-off work when it starts (database
migrations, for example), expect it to run once per revision and guard it with a
lock.

## Failure codes

`status_details.error.code` is the automation key; `message` is for humans and
may be reworded. `retryable` says whether trying again can possibly help.

| Code | Meaning | Retryable |
|---|---|---|
| `serverless.image_pull_auth` | The registry rejected the pull credential | No — fix the credential |
| `serverless.image_not_found` | The image or tag does not exist | No — fix the reference |
| `serverless.image_arch_mismatch` | Image architecture is not amd64 | No — rebuild for amd64 |
| `serverless.invalid_image_ref` | The image reference is malformed | No |
| `serverless.image_pull_unknown` | The image could not be fetched, cause unclear | Yes |
| `serverless.config_error` | The container configuration was rejected | No |
| `serverless.oom_killed` | The container exceeded its memory limit | No — raise the profile |
| `serverless.crash_loop` | The container starts and exits repeatedly | No — fix the application |
| `serverless.progress_deadline` | The rollout did not complete in time | Yes |
| `serverless.revision_missing` | The expected revision is absent | Yes |
| `serverless.unknown` | Unclassified | Yes |

Retrying a non-retryable failure will not fix it and only consumes your build
and request quota.

## Diagnostic endpoints

```text
GET /api/v1/serverless/{id}/diagnose
GET /api/v1/serverless/{id}/logs
GET /api/v1/serverless/{id}/revisions
GET /api/v1/serverless/{id}/events
```

These return a `{success, data, error, meta}` envelope.

### Diagnose

`/diagnose` does the correlation for you and returns ranked findings. It reads
the status, the latest Revision and the Route together — which is what the three
manual steps below amount to — so start here and drop to the raw endpoints only
when you want to see what it was looking at.

Two findings are worth knowing by name, because nothing else reports them:

- **`serverless.no_pod_scheduled`** — the revision settled at `Ready=False`
  with zero replicas, so no pod ever ran. An empty log is the *expected*
  consequence, not a second problem to chase.
- **`serverless.ingress_not_ready`** — the revision is healthy and the public
  URL still does not serve. Every other signal says the deploy succeeded.

Revision checks are skipped while an operation is in flight and for a stopped
container: a healthy deploy is briefly `Ready=False` with zero replicas, and a
stopped container is meant to have no pods. Full list at
[Failure codes](failure-codes).

### Logs

Parameters: `since`, `until`, `cursor`, `limit`, `level`, `container`.

`container` is one of `user-container` (your process), `queue-proxy` (the
request sidecar) or `all`. `level` is one of `debug`, `info`, `warn`, `error`.

```bash
danube rapids logs my-api --since 1h --level error --json
```

**`data.available` is not the same as an empty `data.entries`.** An empty array
with `available: true` means your container printed nothing. `available: false`
with HTTP 503 means the log store did not answer — that says nothing about your
container, and is worth retrying.

**Always pass `since`.** Without it the query covers only the last 30 minutes,
so a container that failed an hour ago legitimately returns no entries.

Page with `meta.next_cursor`. When `meta.truncated` is `false` and
`next_cursor` is `null`, you have reached the end of the stream *within the
requested window* — widen `since` to look further back. Logs are retained for
10 days; a `since` older than that is rejected rather than silently clamped.

`level` matches the text of each line rather than a structured log level, so a
line that merely mentions "error" will match `level=error`. Treat it as a
filter, not a guarantee.

### Revisions

Returns **every** revision, not only those receiving traffic — the revision
that failed is precisely the one with no traffic.

Conditions are tri-state and the middle state carries meaning:

| `status` | Meaning |
|---|---|
| `True` | Satisfied |
| `False` | A settled verdict — this will not change on its own |
| `Unknown` | Still in progress |

Treating `Unknown` as a failure is the most common mistake here: it makes an
in-flight deploy look like an outage.

The response also includes Service and Route readiness. Route conditions
include `IngressReady`, which is what separates "the deployment succeeded" from
"the URL still returns 404".

### Events

Platform events for your container's service, revisions and pods. Requires
`serverless:diagnostics`.

Events are **ephemeral** — the platform garbage-collects them, so their absence
is not evidence that nothing happened. Revision conditions are the durable
signal; prefer them when the two disagree.

## Debugging a failed deploy

Ask the platform first — it correlates the three sources below in one call:

```bash
curl -sH "Authorization: Bearer $TOKEN" \
  https://danubedata.ro/api/v1/serverless/$ID/diagnose | jq '.data.findings'
```

To see the underlying signals yourself, or when a finding needs corroborating:

```bash
# 1. Is it terminal, and why?
danube rapids get my-api --json | jq '.status_details'

# 2. What did the platform observe?
danube rapids revisions my-api --json | jq '.data.revisions[0].conditions'

# 3. Did the container produce output?
danube rapids logs my-api --since 30m --json
```

If step 2 shows `Ready=False` with reason `ContainerMissing`, the image was
never fetched, so **no pod was ever created** and step 3 will legitimately
return nothing. That is not a logging problem — fix the image reference or the
registry credential and redeploy.

## Endpoints and DNS

Rapids containers are served from `*.danubedata.run`. Managed database, cache
and queue endpoints use `*.danubedata.ro`. These are different domains; do not
assume a Rapids URL follows managed-service DNS rules.

For custom domains, see [Rapids custom domains](serverless-domains).