Skip to content

Failure codes

Every failure the platform reports carries a stable code. Branch on the code; display the message. Messages are written for humans and may be reworded at any time, codes are an interface and change additively.

Codes appear in two places, with the same meaning in both:

  • status_details.error.code — why a resource is failed or degraded
  • findings[].code from /diagnose — everything diagnose correlated

All codes are namespaced by product: cache., database., queue., vps., vps-volume., storage., static-site., app., uptime., metric-alert., serverless..

See Platform automation for the surrounding contract.

Universal codes

Emitted for every product. {product} is the prefix from the list above.

CodeSeverityRetryableMeaningWhat to do
{product}.healthyinformational—Diagnose found nothing wrongNothing
{product}.operation_in_progressinformationalyesAn operation is still running; diagnose short-circuits rather than reporting a half-built resource as brokenKeep polling operation.terminal
{product}.stale_observationaction_requiredyesAn in-flight operation has gone unconfirmed for 15 minutesRetry the operation; if it repeats, open a ticket
{product}.warnings_recoveredtransient_recovered—Warning events occurred and the platform already recovered from themNothing. Not a current problem
{product}.unknownfatalnoA failure was recorded but could not be classifiedRead message; open a ticket if unclear

Event-derived codes

Where a product exposes cluster events, warnings become findings coded {product}.event_{reason}, with the Kubernetes reason lowercased — so a CrashLoopBackOff on a cache instance is cache.event_crashloopbackoff.

These reasons are classified fatal — they do not resolve on their own:

ReasonCode suffixTypical cause
Failedevent_failedThe container or controller reported a failure
FailedCreateevent_failedcreateThe workload object could not be created
FailedSchedulingevent_failedschedulingNo node had room, or a volume was not placeable
ErrImagePullevent_errimagepullThe image could not be fetched
ImagePullBackOffevent_imagepullbackoffRepeated pull failures; usually a bad tag or credential
CrashLoopBackOffevent_crashloopbackoffThe process keeps exiting after start
BackOffevent_backoffRestarts are being throttled
FailedMountevent_failedmountA volume could not be attached or mounted
Evictedevent_evictedThe node reclaimed resources
OOMKillingevent_oomkillingThe container exceeded its memory limit

These are classified transient_recovered and are not incidents — Kubernetes emits them during ordinary reconciliation races:

InternalError · UpdateConflict · FailedUpdate, plus any warning whose message matches a write-conflict pattern (object has been modified, please apply your changes to the latest version, conflict, try again).

Automation that treats every Warning event as an outage will page you for healthy systems. This classification is why /diagnose exists rather than handing you the raw event stream.

Object storage

CodeSeverityRetryableMeaningWhat to do
storage.quota_exceededfatalnoThe bucket is at its size limit. Writes are rejected while this holdsDelete objects, or raise the limit
storage.quota_nearly_exceededaction_requirednoApproaching the limitFree space before writes start failing
storage.quota_not_appliedaction_requiredyesThe bucket exists but the backend did not accept its quota, so no limit is being enforcedRetry the update; open a ticket if it persists
storage.usage_staleinformationalyesThe usage figure is older than 30 minutesTreat the number as approximate

Usage is sampled, not live. Quota findings state the age of the measurement they are based on, because acting on a stale number is how a "you have room" answer turns into a rejected write.

Static sites

CodeSeverityRetryableMeaningWhat to do
static-site.build_failedfatalnoThe build command exited non-zeroRead the build logs
static-site.domain_unverifiedaction_requirednoA custom domain has no valid DNS recordAdd the record shown in the dashboard
static-site.pending_reviewaction_requirednoThe site is awaiting review before publishingWait, or contact support
static-site.suspendedfatalnoThe site was suspendedContact support

Managed apps

CodeSeverityRetryableMeaningWhat to do
app.upgrade_failedfatalnoA version upgrade did not completeRead the logs; the previous version may still be serving
app.upgrade_auto_rolled_backaction_requirednoAn upgrade failed and the platform restored the previous versionYour app is running. Investigate before retrying

app.upgrade_auto_rolled_back is deliberately not fatal: the rollback worked and the app is serving. Reporting it as an outage would be false.

Serverless containers

These come from correlating the container's status with its live Knative Revision and Route — two sources that give the wrong answer when read alone.

CodeSeverityRetryableMeaningWhat to do
serverless.no_pod_scheduledfatalnoThe latest revision settled at Ready=False with zero replicas: no pod ever startedFix the image reference or its credential. The empty log is expected — nothing ran to write one
serverless.ingress_not_readyfatalnoIngressReady=False: the revision is healthy but the public URL does not serveCheck the custom domain and its TLS certificate. Redeploying will not help
serverless.ingress_not_readyinformationalyesIngressReady=Unknown: the route is still convergingWait and re-read
serverless.degraded_but_servingaction_requirednoA new revision failed while an older one keeps serving. The site is UPDo not roll back automatically — traffic is already on the last good revision
serverless.replica_count_lagginginformationalnocurrent_replicas disagrees with the live revision countNothing. The container figure is a five-minute cache; the two legitimately differ after a scale-to-zero wake
serverless.revision_data_unavailableinformationalyesThe Knative Service or Route could not be read, so some checks did not runRetry. Do not read the missing findings as "nothing wrong"

Two of these are the reason /diagnose exists rather than a status field:

  • no_pod_scheduled explains an empty log. Callers otherwise spend a long time hunting for output that cannot exist, because the image was never pulled.
  • ingress_not_ready separates "deployed" from "reachable". Every other signal — status, revision, events — says the deployment succeeded.

degraded_but_serving arrives alongside the finding that explains why the new revision failed — usually no_pod_scheduled. Read the whole array: the first says your site is still up, the second says what to fix.

Revision checks are skipped, and no code is emitted, while an operation is still running (operation.terminal is false) and for a stopped container. A healthy deploy is briefly Ready=False with zero replicas, and a stopped container is meant to have no pods; reporting either as a fault would be a false alarm.

Rapids volumes, backups and restores

The Rapids Volumes API refuses a request with success: false and an error whose code is one of the codes below, with the HTTP status in the table. retryable says whether the same request can succeed later with nothing changed on your side. Branch on the code, never on the message.

Refused because of the account

CodeHTTPRetryableMeaningWhat to do
serverless.volume_not_available403noVolumes, or backups of volumes, are not offered to this accountAsk support to enable them. Reading and deleting still work
serverless.volume_manual_approval_required403noThis project is under review and creates no new resources until support has cleared itContact support
serverless.volume_budget_exceeded403noThis month's or today's spend has reached the project's budget, which is set to refuse new resourcesRaise the budget under Billing > Budget, or wait for the next billing period
serverless.volume_team_limit_reached403noThe project holds all the volume storage it may (100 GB unless raised)Delete a volume, or ask support to raise the limit

Not found

CodeHTTPRetryableMeaningWhat to do
serverless.volume_not_found404noThe container has no such volume. A volume that is being deleted stays at its address, as deleting, until it is goneCheck the id
serverless.volume_backup_not_found404noThe container has no such backupCheck the id
serverless.volume_restore_not_found404noThe container has no such restoreCheck the id
serverless.volume_container_gone404noThe container was deleted while the request was handledNothing to do

The request is wrong

CodeHTTPRetryableMeaningWhat to do
serverless.volume_cannot_shrink422noA volume can be made larger, never smaller. error.field is size_gbSend a larger size
serverless.volume_size_out_of_range422noThe size is outside what a volume may be (1 to 50 GB). A request with such a size is normally answered by the validation answer below, with errors.size_gb, before the platform's own check raises this code. error.field is size_gbSend a size from 1 to 50 GB
serverless.volume_invalid_mount_path422noThe container cannot mount a volume at that path. A mount_path in your request that is not valid is normally answered by the validation answer below, with errors.mount_path. This code comes from a restore into a container without a volume, when you send no mount_path and the path the backup was taken at can no longer be mounted. error.field is mount_pathSend a mount_path of your own. See Mount path

A body that is not valid at all (a field missing, a size that is not a number) is answered 422 with Laravel's {"message": "...", "errors": {...}} instead.

The state does not allow it now

CodeHTTPRetryableMeaningWhat to do
serverless.volume_already_exists409noThe container has a volume already (one per container)Use the volume it has
serverless.volume_container_not_deployed409yesThe container has not been deployed yet, so there is nothing to roll a volume out withWait for its first deployment
serverless.volume_container_busy409yesAnother change to the container is under way or queuedWait for it to finish, or cancel it
serverless.volume_tier_unsupported409noThe container is on the 10 Gb network tier, which cannot have a volume yetUse a container on the standard tier
serverless.volume_not_mounted409noThe volume is being deleted, so it cannot be moved or made largerWait until it is gone, then add a new one
serverless.volume_cannot_resize_now409yesThe volume is still being made or grownWait until it is active. A volume that is still pending after 10 minutes, with failure.operation provision, will not come up by waiting: delete it and add it again
serverless.volume_being_restored409yesA backup is being restored into the volumeWait for the restore to finish
serverless.volume_changed_meanwhile409yesThe volume changed while the request was handledRead the volume and send the request again
serverless.volume_backup_no_volume409noThe container has no volume to back upAdd a volume, or restore a backup into the container
serverless.volume_backup_volume_not_ready409yesThe volume is being created, grown or restoredWait until it is active. A volume that is still pending after 10 minutes, with failure.operation provision, will not come up by waiting: delete it and add it again
serverless.volume_backup_in_progress409yesA backup of this container is being made (one at a time)Wait until it is terminal
serverless.volume_backup_limit_reached409noThe container keeps as many backups as it may (10)Delete a backup
serverless.volume_backup_deletion_pending409yesA backup is being removed from the backup store, which takes about 5 minutes plus half a minute for every GB the backup held (an hour at most). No backup can be taken or restored meanwhileWait. When you are at the limit of 10 backups, delete one ahead of time
serverless.volume_backup_too_soon409yesThe last backup was made less than 10 minutes agoWait
serverless.volume_backup_daily_limit_reached409yes24 backups were requested in the last 24 hours, failed and deleted ones includedTry again when the oldest of them is more than 24 hours old
serverless.volume_backup_being_restored409yesA backup of this volume is being restored, and a deletion would pull the data from under itWait for the restore to finish
serverless.volume_restore_backup_not_ready409noThe backup is still being made, failed, or is being deletedWait for the backup to be ready (terminal) before restoring it
serverless.volume_restore_in_progress409noA restore of this container is under way. Not retryable: asked again once it has finished, it would restore a second time over what was written since. meta.restore_id and meta.poll_url name it. Adding a volume to a container that has none answers this too while a restore, or the clean-up after a failed one, holds the container's placeFollow that restore
serverless.volume_restore_cleaning_up409yesThe last restore failed and the platform is still cleaning up after it. Nothing was restored. meta.restore_id and meta.poll_url name it. Only a restore request answers thisTry again in a few minutes
serverless.volume_restore_backup_too_large409noThe backup is of a larger volume than the container has nowMake the volume at least that large, then restore
serverless.volume_restore_daily_limit_reached409yes12 restores were requested in the last 24 hours, failed ones includedTry again when the oldest of them is more than 24 hours old

The platform cannot do it now

CodeHTTPRetryableMeaningWhat to do
serverless.volume_capacity_unavailable503yesThere is no room to provision the volume, or the larger sizeTry a smaller size, or later
serverless.volume_placement_unknown503yesWe could not work out where the volume would be placed just nowTry again in a minute

When a container with a volume does not start

These two arrive in status_details.error.code (and in findings[].code), like the other runtime failures, when the platform cannot start an instance of a container that has a volume.

CodeRetryableMeaningWhat to do
serverless.volume_node_fullyesAll instances of a container with a volume run on the server that holds the volume, and it cannot take this one right nowLower Maximum replicas or the resource profile, or try again later
serverless.volume_not_readynoThe volume's claim is missing, not bound yet, or being removedLook at the volume on the Storage tab. Delete it and add it again only if it shows Not provisioned yet: it never held your files. If the volume was working before, or you did not delete it yourself, delete nothing and contact support

Uptime checks

An uptime failure describes your target, not the monitor. A check reporting down is working correctly.

CodeSeverityRetryableMeaningWhat to do
uptime.dnsfatalyesThe hostname did not resolveCheck DNS
uptime.timeoutfatalyesThe target did not respond in timeCheck load and timeouts
uptime.refusedfatalyesThe connection was refusedCheck the service is listening
uptime.http_errorfatalyesAn unexpected HTTP statusCheck the endpoint
uptime.tls_expiringaction_requirednoThe certificate expires soonRenew it
uptime.tls_expiredfatalnoThe certificate has expiredRenew it now
uptime.flappingaction_requirednoThe target is alternating up and downInvestigate intermittent failures
uptime.not_runninginformationalnoThe check is pausedResume it

retryable: true here means the target may recover, not that re-reading the check will change anything.

VPS block volumes

CodeSeverityRetryableMeaningWhat to do
vps-volume.instance_unavailablefatalnoThe parent VPS is stopped or gone, so the volume cannot be attachedStart the instance
vps-volume.awaiting_instance_bootinformationalyesThe volume is attached and waiting for the guest to see itKeep waiting

A volume carries its own status_details. Do not poll the parent VM — it stays running throughout an attach and would look like instant success.

Cache, databases, queues and VPS

These products report failures from two sources:

  1. Recorded failures — written by the reconciler that observed them, coded {product}.{category}. The category depends on what the platform observed; treat it as opaque and branch on the prefix plus retryable.
  2. Event-derived findings — the {product}.event_* codes above.

If you receive a code not listed on this page, apply the general rules: honour retryable, read message for the human explanation, and treat an unrecognised code as a genuine failure rather than ignoring it.

Envelope errors

These describe the request, not the resource, and appear in error.code alongside an HTTP status.

CodeHTTPRetryableMeaning
{product}.logs_not_supported400noThis product has no log surface — check capabilities.logs first
{product}.logs_unavailable503yesThe log backend did not respond
{product}.events_not_supported400noThis resource has no cluster objects to read events from
{product}.events_unavailable503yesCluster events could not be read
{product}.events_not_permitted503noThe platform is not permitted to read events for this resource
operation.not_found404noNo operation with that id
registry.no_project400noNo project resolved; send X-Team-Id
approval.already_pending409noAn approval for this action is already waiting for the project owner; approval_id names it
account_limits.request_pending409noA limit increase request is already open for the project
approval.not_found404noNo approval with that id in this project

events_not_permitted is separated from events_unavailable on purpose: one is a transient upstream problem worth retrying, the other never resolves by retrying and needs an operator.

Registry preflight

GET /registry/preflight returns findings rather than failing, so these arrive with HTTP 200 in findings[].code:

CodeRetryableMeaning
registry.namespace_mismatchnoThe push target is outside your project's namespace
registry.quota_exceededyesThe push would exceed your storage or repository limit
registry.manifest_not_foundnoNo such tag in the repository — push it first
registry.architecture_mismatchnoThe image was built for an architecture the platform does not run
registry.no_credentialnoThe image is private and no credential is configured
registry.external_image—Informational: the image is outside your registry, so we cannot verify it

A 503 with retryable: true means an upstream we depend on is unreachable — back off and retry. A 403 will never succeed with the same token; mint one with the right ability instead.