Deploy Succeeded But the Site Is Down

There is a specific kind of bad morning that starts with a green checkmark. The deploy succeeded but the site is down, the dashboard says running, and someone is messaging you a screenshot of a 502.

This is not a rare edge case. It is the predictable result of a platform reporting one thing and measuring another, and it is worth understanding precisely, because the fix is not "check harder" — it is changing what the word running is allowed to mean.

Three different questions, one status light

When a platform says a service is running, it could be answering any of these:

1. Did the container start? The process exists and has not exited.
2. Is the port open? Something is listening where the platform expects.
3. Does the application answer correctly? A request gets a response that means the app is ready to work.

These are wildly different guarantees, and most incidents in this shape come from a dashboard answering question 1 while you assumed question 3.

A Node process that boots, fails to connect to its database, and sits in a retry loop satisfies question 1 forever. It has not crashed. It will never serve a request. The container is "running" in every sense the orchestrator cares about.

The gap where the outage lives

The dangerous window is between "the new version started" and "the new version can work." In that window a naive platform has already moved traffic, because starting was the only thing it measured.

What makes this worse than a plain crash is the rollback story. A crash loop is loud: the container exits, restarts, exits, and the platform eventually notices. A boot-and-hang is silent. Nothing restarts, nothing alarms, and the previous working version has usually already been torn down.

That last part is the real damage. The old version was fine. It was removed because a new one started, and starting was mistaken for working.

What a real health gate does

The fix is structural, not procedural. Traffic should not move until the new version has answered a request.

On Dockup a release runs like this: the new version is built in isolation, started alongside the version currently serving, and then asked a question. Only when it answers does the domain point at it. If it never answers, the release stops there and the previous version keeps serving — nobody outside your dashboard ever knows a deploy was attempted.

That is why a failed deploy on Dockup is not an outage. The old container was never removed on the assumption that the new one would be fine.

# The health gate is per-service configuration, not a platform default you inherit
dockup info my-project/my-api --json

The healthCheck block in that output is the whole contract: which path is requested, how long to wait for an answer, how many times to try, and how long between attempts.

Configure the check to answer question 3

A health endpoint that returns 200 unconditionally is worse than none, because it converts a real gate into a rubber stamp. The point of the check is to fail when the application cannot do its job.

A useful readiness endpoint verifies the things the app cannot work without:

// Not this — it proves only that the process is alive
app.get('/healthz', (req, res) => res.send('ok'))

// This — it proves the app can actually serve a request
app.get('/healthz', async (req, res) => {
  try {
    await db.query('select 1')       // the dependency that is usually the problem
    if (!cacheReady) throw new Error('cache warming')
    res.status(200).json({ ok: true })
  } catch (err) {
    res.status(503).json({ ok: false, reason: err.message })
  }
})

Two rules make this work in practice:

Check dependencies you cannot serve without, and nothing else. If your app can degrade gracefully when the search index is down, do not fail readiness on the search index — you will block deploys for something that is not an outage.

Keep it cheap. The endpoint is called repeatedly during every release. A readiness check that runs an expensive query is a self-inflicted load problem.

Give it enough time, but not unlimited time

Two settings decide whether the gate helps or hurts:

• Timeout per attempt should exceed your slowest legitimate cold start. An app that connects to a database and warms a cache in eight seconds will fail a three-second check every time, and you will "fix" it by disabling the gate — which puts you back where you started.
• Retries should cover the total start time, not one attempt. Interval × retries is the real budget.

On Dockup those are healthCheckInterval, healthCheckTimeout and healthCheckRetries, and they are per service because a Rails monolith and a Go sidecar do not start on the same schedule.

When it is already down

If you are reading this mid-incident, the order that resolves it fastest:

1. Check whether the app answers directly, bypassing the domain. If it answers on its port but not through the domain, this is a routing problem, not an application problem, and you should stop debugging your code.
2. Read the runtime logs, not the build logs. The build succeeded — that is the premise. What you want is what the process did after it started.
3. Roll back before you diagnose. Diagnosis is cheaper when nobody is watching.

dockup logs my-project/my-api --follow            # what the running process is saying
dockup deployments my-project/my-api              # what was live before this
dockup rollback <deployment-id> my-project/my-api # put that back

On Dockup a rollback is a switch rather than a rebuild, because the previous version is still on disk. That matters at 3am: the fastest recovery is the one that does not have to compile anything.

The question to ask a platform

When you are choosing where to run production, this is a good thing to test deliberately: deploy an application that starts successfully and then fails to reach its database. Watch what the dashboard says.

If it says running, you now know exactly what that word will be worth during your next incident.

Frequently asked questions

Why does my dashboard say running when the site is down? Because "running" usually means the container process exists, not that the application can serve a request. A process stuck retrying a database connection satisfies that definition indefinitely.

Should a health check hit the database? Yes, if your application cannot serve requests without it. Check the dependencies you genuinely need and skip the ones you can degrade without.

What is the difference between liveness and readiness? Liveness asks whether the process should be restarted. Readiness asks whether it should receive traffic. The gate that prevents this failure is readiness, and it has to run before traffic moves.

How do I stop a bad deploy from taking the site down at all? Only switch traffic after the new version answers a real request, and keep the previous version until the switch is confirmed. Then a failed release is a release that never happened rather than an outage.