# Troubleshooting CR + Rapids Deploys

When a push or deploy goes sideways, the failure is almost always at one of three points: the
**push** to the registry, the **pull** by Rapids, or the **container starting up**. This page
walks the failures you're most likely to hit, in the order they happen, with the fix for each.

> **First stop:** the Rapid's detail page surfaces the underlying Kubernetes event verbatim,
> and `danube rapids show hello` plus `danube rapids deployments hello` show status, the
> current image:tag, and per-revision traffic. Most diagnoses start there.

> **You'll hear about failures automatically.** If a deploy fails — even after the first
> rollout — Rapids flags the container **Degraded**, shows a plain-language reason (architecture
> mismatch, image or tag not found, out of memory, startup timed out) with a fix hint, and
> emails your team with a link to investigate. Your last healthy revision keeps serving the
> whole time, and the container clears back to healthy on its own once a deploy succeeds.

## Push failures (`docker push`)

### `denied: requested access to the resource is denied`

The first path segment of your image isn't your team slug. Images **must** be tagged
`cr.danubedata.ro/{your-team-slug}/...`.

```bash
# wrong — pushes to someone else's namespace
docker build -t cr.danubedata.ro/hello:1.0.0 .
# right — acme is your team slug
docker build -t cr.danubedata.ro/acme/hello:1.0.0 .
```

### `unauthorized: authentication required`

Your token is wrong, revoked, or expired. Re-issue a key (Container Registry → Access Keys)
and log in again. You can verify a token directly:

```bash
curl -i -u "you@example.com:$DD_REGISTRY_TOKEN" \
  "https://danubedata.ro/oci/token?service=cr.danubedata.ro&scope=repository:acme/hello:pull"
```

`200 OK` with a JSON `{"token":"..."}` body means the credentials are good; `401` means re-issue.

### `denied: Storage quota exceeded` / `Repository limit reached`

You've hit your registry plan's storage cap or repository count. Delete unused tags or
repositories (Container Registry → Repositories), or upgrade under **Plan & Usage**. Quota
checks use your live `bytes_used`, so a push succeeds immediately after you free space.

## Pull / deploy failures (`ImagePullBackOff`)

The image pushed fine but Rapids can't pull it into a pod. In order of likelihood:

1. **The tag doesn't exist** — a typo, or the push hadn't finished. Confirm the exact tag
   under Container Registry → Repositories → `hello` (expand the row).
2. **Team mismatch** — first-party pulls only work when the team that owns the Rapid matches
   the team slug in the image path. A Rapid in team `acme` cannot pull `cr.danubedata.ro/other/...`.
3. **Stale internal credential** — if you recently renamed the team, the auto-seeded pull
   secret can lag. Trigger a redeploy to re-materialise it: `danube rapids redeploy hello`.

The Rapid's status surface names which of these is in play.

## The container starts but never becomes Ready

The image pulled, but the revision won't take traffic. Rapids only routes to a revision once
it's listening, so a revision stuck "not ready" usually means a startup problem.

### `exec format error` in the logs

Architecture mismatch — an `arm64` image (common when building on Apple Silicon) on the
`linux/amd64` cluster. Rebuild for the right platform:

```bash
docker build --platform linux/amd64 -t cr.danubedata.ro/acme/hello:1.0.1 .
docker push cr.danubedata.ro/acme/hello:1.0.1
danube rapids update hello --image cr.danubedata.ro/acme/hello --tag 1.0.1
```

### The app isn't listening on `$PORT`

Rapids tells the container which port to use via the `PORT` env var and routes to that port.
If your app hardcodes a different port, or binds `127.0.0.1` instead of `0.0.0.0`, the
readiness check never passes. Read `PORT` (default `8080`) and bind all interfaces.

### `permission denied` binding the port

Containers run as a non-root user with Linux capabilities dropped, so binding a port below
`1024` fails even as `root`. Listen on `8080` (or any port ≥1024).

### The process crashes on startup

Check the revision's logs for the stack trace (Rapid → Logs, or `danube rapids show hello`).
A missing required env var is the usual culprit — set it with `danube rapids update hello --env KEY=VALUE`
and a new revision rolls automatically.

## "My deploy didn't change anything"

You pushed a new image but the running container is unchanged. Almost always this is the
`:latest` trap — Kubernetes won't re-pull a tag it has already cached. Either deploy by
immutable SHA tag (recommended), or force a re-roll of the current tag:

```bash
danube rapids redeploy hello
```

See [CI/CD Build → Push → Deploy](https://docs.danubedata.ro/tutorial-cr-rapids-cicd) for the
immutable-tag workflow that avoids this.

## Cold starts feel slow

With `--min-scale 0`, the first request after an idle period starts a fresh instance. To
reduce or remove that:

- Keep one instance warm: `danube rapids update hello --min-scale 1`.
- Shrink the image so it pulls and boots faster — a distroless or alpine base (see the Go
  example in the [first tutorial](https://docs.danubedata.ro/tutorial-cr-rapids-first-deploy))
  starts far quicker than a full OS image.

## Rolling back a bad deploy

```bash
danube rapids deployments hello     # find the last good <sha>
danube rapids update hello --image cr.danubedata.ro/acme/hello --tag <good-sha>
```

Traffic shifts to the older revision once it's ready — zero downtime.

## Still stuck?

- [Container Registry reference](https://docs.danubedata.ro/container-registry) — full
  registry error list and tag rules.
- [Serverless overview](https://docs.danubedata.ro/serverless-overview) — scaling, networking,
  and platform limits.
- Email **support@danubedata.ro** with your container name and the event/log text — that's
  enough for us to pinpoint it fast.

---

**Questions?** Contact support at support@danubedata.ro