{"slug":"failure-codes","title":"Failure codes","description":"Every failure the platform reports carries a stable code. Branch on the code;","section":"Automation","url":"https://docs.danubedata.ro/failure-codes","markdown_url":"https://docs.danubedata.ro/failure-codes.md","breadcrumbs":[{"title":"Automation","slug":null},{"title":"Failure Codes","slug":"failure-codes"}],"headings":[{"level":1,"title":"Failure codes","id":"failure-codes"},{"level":2,"title":"Universal codes","id":"universal-codes"},{"level":3,"title":"Event-derived codes","id":"event-derived-codes"},{"level":2,"title":"Object storage","id":"object-storage"},{"level":2,"title":"Static sites","id":"static-sites"},{"level":2,"title":"Managed apps","id":"managed-apps"},{"level":2,"title":"Serverless containers","id":"serverless-containers"},{"level":2,"title":"Uptime checks","id":"uptime-checks"},{"level":2,"title":"VPS block volumes","id":"vps-block-volumes"},{"level":2,"title":"Cache, databases, queues and VPS","id":"cache-databases-queues-and-vps"},{"level":2,"title":"Envelope errors","id":"envelope-errors"},{"level":3,"title":"Registry preflight","id":"registry-preflight"}],"format":"markdown","word_count":1688,"content":"# Failure codes\n\nEvery failure the platform reports carries a **stable code**. Branch on the code;\ndisplay the message. Messages are written for humans and may be reworded at any\ntime, codes are an interface and change additively.\n\nCodes appear in two places, with the same meaning in both:\n\n- `status_details.error.code` — why a resource is `failed` or `degraded`\n- `findings[].code` from `/diagnose` — everything diagnose correlated\n\nAll codes are namespaced by product: `cache.`, `database.`, `queue.`, `vps.`,\n`vps-volume.`, `storage.`, `static-site.`, `app.`, `uptime.`, `metric-alert.`,\n`serverless.`.\n\nSee [Platform automation](platform-automation) for the surrounding contract.\n\n## Universal codes\n\nEmitted for **every** product. `{product}` is the prefix from the list above.\n\n| Code | Severity | Retryable | Meaning | What to do |\n|---|---|---|---|---|\n| `{product}.healthy` | informational | — | Diagnose found nothing wrong | Nothing |\n| `{product}.operation_in_progress` | informational | yes | An operation is still running; diagnose short-circuits rather than reporting a half-built resource as broken | Keep polling `operation.terminal` |\n| `{product}.stale_observation` | action_required | yes | An in-flight operation has gone unconfirmed for 15 minutes | Retry the operation; if it repeats, open a ticket |\n| `{product}.warnings_recovered` | transient_recovered | — | Warning events occurred and the platform already recovered from them | **Nothing.** Not a current problem |\n| `{product}.unknown` | fatal | no | A failure was recorded but could not be classified | Read `message`; open a ticket if unclear |\n\n### Event-derived codes\n\nWhere a product exposes cluster events, warnings become findings coded\n`{product}.event_{reason}`, with the Kubernetes reason lowercased — so a\n`CrashLoopBackOff` on a cache instance is `cache.event_crashloopbackoff`.\n\nThese reasons are classified **fatal** — they do not resolve on their own:\n\n| Reason | Code suffix | Typical cause |\n|---|---|---|\n| `Failed` | `event_failed` | The container or controller reported a failure |\n| `FailedCreate` | `event_failedcreate` | The workload object could not be created |\n| `FailedScheduling` | `event_failedscheduling` | No node had room, or a volume was not placeable |\n| `ErrImagePull` | `event_errimagepull` | The image could not be fetched |\n| `ImagePullBackOff` | `event_imagepullbackoff` | Repeated pull failures; usually a bad tag or credential |\n| `CrashLoopBackOff` | `event_crashloopbackoff` | The process keeps exiting after start |\n| `BackOff` | `event_backoff` | Restarts are being throttled |\n| `FailedMount` | `event_failedmount` | A volume could not be attached or mounted |\n| `Evicted` | `event_evicted` | The node reclaimed resources |\n| `OOMKilling` | `event_oomkilling` | The container exceeded its memory limit |\n\nThese are classified **transient_recovered** and are *not* incidents —\nKubernetes emits them during ordinary reconciliation races:\n\n`InternalError` · `UpdateConflict` · `FailedUpdate`, plus any warning whose\nmessage matches a write-conflict pattern (`object has been modified`,\n`please apply your changes to the latest version`, `conflict`, `try again`).\n\nAutomation that treats every `Warning` event as an outage will page you for\nhealthy systems. This classification is why `/diagnose` exists rather than\nhanding you the raw event stream.\n\n## Object storage\n\n| Code | Severity | Retryable | Meaning | What to do |\n|---|---|---|---|---|\n| `storage.quota_exceeded` | fatal | no | The bucket is at its size limit. **Writes are rejected** while this holds | Delete objects, or raise the limit |\n| `storage.quota_nearly_exceeded` | action_required | no | Approaching the limit | Free space before writes start failing |\n| `storage.quota_not_applied` | action_required | yes | The bucket exists but the backend did not accept its quota, so **no limit is being enforced** | Retry the update; open a ticket if it persists |\n| `storage.usage_stale` | informational | yes | The usage figure is older than 30 minutes | Treat the number as approximate |\n\nUsage is sampled, not live. Quota findings state the age of the measurement they\nare based on, because acting on a stale number is how a \"you have room\" answer\nturns into a rejected write.\n\n## Static sites\n\n| Code | Severity | Retryable | Meaning | What to do |\n|---|---|---|---|---|\n| `static-site.build_failed` | fatal | no | The build command exited non-zero | Read the build logs |\n| `static-site.domain_unverified` | action_required | no | A custom domain has no valid DNS record | Add the record shown in the dashboard |\n| `static-site.pending_review` | action_required | no | The site is awaiting review before publishing | Wait, or contact support |\n| `static-site.suspended` | fatal | no | The site was suspended | Contact support |\n\n## Managed apps\n\n| Code | Severity | Retryable | Meaning | What to do |\n|---|---|---|---|---|\n| `app.upgrade_failed` | fatal | no | A version upgrade did not complete | Read the logs; the previous version may still be serving |\n| `app.upgrade_auto_rolled_back` | action_required | no | An upgrade failed and the platform restored the previous version | **Your app is running.** Investigate before retrying |\n\n`app.upgrade_auto_rolled_back` is deliberately not fatal: the rollback worked and\nthe app is serving. Reporting it as an outage would be false.\n\n## Serverless containers\n\nThese come from correlating the container's status with its live Knative\nRevision and Route — two sources that give the wrong answer when read alone.\n\n| Code | Severity | Retryable | Meaning | What to do |\n|---|---|---|---|---|\n| `serverless.no_pod_scheduled` | fatal | no | The latest revision settled at `Ready=False` with zero replicas: no pod ever started | Fix the image reference or its credential. **The empty log is expected** — nothing ran to write one |\n| `serverless.ingress_not_ready` | fatal | no | `IngressReady=False`: the revision is healthy but the public URL does not serve | Check the custom domain and its TLS certificate. Redeploying will not help |\n| `serverless.ingress_not_ready` | informational | yes | `IngressReady=Unknown`: the route is still converging | Wait and re-read |\n| `serverless.degraded_but_serving` | action_required | no | A new revision failed while an older one keeps serving. **The site is UP** | Do **not** roll back automatically — traffic is already on the last good revision |\n| `serverless.replica_count_lagging` | informational | no | `current_replicas` disagrees with the live revision count | Nothing. The container figure is a five-minute cache; the two legitimately differ after a scale-to-zero wake |\n| `serverless.revision_data_unavailable` | informational | yes | The Knative Service or Route could not be read, so some checks did not run | Retry. Do **not** read the missing findings as \"nothing wrong\" |\n\nTwo of these are the reason `/diagnose` exists rather than a status field:\n\n- **`no_pod_scheduled` explains an empty log.** Callers otherwise spend a long\n  time hunting for output that cannot exist, because the image was never pulled.\n- **`ingress_not_ready` separates \"deployed\" from \"reachable\".** Every other\n  signal — status, revision, events — says the deployment succeeded.\n\n`degraded_but_serving` arrives **alongside** the finding that explains why the\nnew revision failed — usually `no_pod_scheduled`. Read the whole array: the\nfirst says your site is still up, the second says what to fix.\n\nRevision checks are skipped, and no code is emitted, while an operation is still\nrunning (`operation.terminal` is `false`) and for a stopped container. A healthy\ndeploy is briefly `Ready=False` with zero replicas, and a stopped container is\n*meant* to have no pods; reporting either as a fault would be a false alarm.\n\n## Uptime checks\n\nAn uptime failure describes **your target**, not the monitor. A check reporting\n`down` is working correctly.\n\n| Code | Severity | Retryable | Meaning | What to do |\n|---|---|---|---|---|\n| `uptime.dns` | fatal | yes | The hostname did not resolve | Check DNS |\n| `uptime.timeout` | fatal | yes | The target did not respond in time | Check load and timeouts |\n| `uptime.refused` | fatal | yes | The connection was refused | Check the service is listening |\n| `uptime.http_error` | fatal | yes | An unexpected HTTP status | Check the endpoint |\n| `uptime.tls_expiring` | action_required | no | The certificate expires soon | Renew it |\n| `uptime.tls_expired` | fatal | no | The certificate has expired | Renew it now |\n| `uptime.flapping` | action_required | no | The target is alternating up and down | Investigate intermittent failures |\n| `uptime.not_running` | informational | no | The check is paused | Resume it |\n\n`retryable: true` here means the *target* may recover, not that re-reading the\ncheck will change anything.\n\n## VPS block volumes\n\n| Code | Severity | Retryable | Meaning | What to do |\n|---|---|---|---|---|\n| `vps-volume.instance_unavailable` | fatal | no | The parent VPS is stopped or gone, so the volume cannot be attached | Start the instance |\n| `vps-volume.awaiting_instance_boot` | informational | yes | The volume is attached and waiting for the guest to see it | Keep waiting |\n\nA volume carries its own `status_details`. Do not poll the parent VM — it stays\n`running` throughout an attach and would look like instant success.\n\n## Cache, databases, queues and VPS\n\nThese products report failures from two sources:\n\n1. **Recorded failures** — written by the reconciler that observed them, coded\n   `{product}.{category}`. The category depends on what the platform observed;\n   treat it as opaque and branch on the prefix plus `retryable`.\n2. **Event-derived findings** — the `{product}.event_*` codes above.\n\nIf you receive a code not listed on this page, apply the general rules: honour\n`retryable`, read `message` for the human explanation, and treat an unrecognised\ncode as a genuine failure rather than ignoring it.\n\n## Envelope errors\n\nThese describe the **request**, not the resource, and appear in `error.code`\nalongside an HTTP status.\n\n| Code | HTTP | Retryable | Meaning |\n|---|---|---|---|\n| `{product}.logs_not_supported` | 400 | no | This product has no log surface — check `capabilities.logs` first |\n| `{product}.logs_unavailable` | 503 | yes | The log backend did not respond |\n| `{product}.events_not_supported` | 400 | no | This resource has no cluster objects to read events from |\n| `{product}.events_unavailable` | 503 | yes | Cluster events could not be read |\n| `{product}.events_not_permitted` | 503 | no | The platform is not permitted to read events for this resource |\n| `operation.not_found` | 404 | no | No operation with that id |\n| `registry.no_project` | 400 | no | No project resolved; send `X-Team-Id` |\n\n`events_not_permitted` is separated from `events_unavailable` on purpose: one is\na transient upstream problem worth retrying, the other never resolves by\nretrying and needs an operator.\n\n### Registry preflight\n\n`GET /registry/preflight` returns findings rather than failing, so these arrive\nwith HTTP 200 in `findings[].code`:\n\n| Code | Retryable | Meaning |\n|---|---|---|\n| `registry.namespace_mismatch` | no | The push target is outside your project's namespace |\n| `registry.quota_exceeded` | yes | The push would exceed your storage or repository limit |\n| `registry.manifest_not_found` | no | No such tag in the repository — push it first |\n| `registry.architecture_mismatch` | no | The image was built for an architecture the platform does not run |\n| `registry.no_credential` | no | The image is private and no credential is configured |\n| `registry.external_image` | — | Informational: the image is outside your registry, so we cannot verify it |\n\nA 503 with `retryable: true` means an upstream we depend on is unreachable — back\noff and retry. A 403 will never succeed with the same token; mint one with the\nright ability instead.\n","prev":{"title":"Platform Automation & Diagnostics","slug":"platform-automation","url":"https://docs.danubedata.ro/platform-automation","markdown_url":"https://docs.danubedata.ro/platform-automation.md","json_url":"https://docs.danubedata.ro/platform-automation.json"},"next":{"title":"Overview","slug":"terraform-overview","url":"https://docs.danubedata.ro/terraform-overview","markdown_url":"https://docs.danubedata.ro/terraform-overview.md","json_url":"https://docs.danubedata.ro/terraform-overview.json"},"index_url":"https://docs.danubedata.ro/index.json"}