> ## Documentation Index
> Fetch the complete documentation index at: https://docs.fjall.io/llms.txt
> Use this file to discover all available pages before exploring further.

# Boot Gates & Deployment Safety

> The layers Fjall puts between a bad task and a broken production: schema-version boot gates, the deployment circuit breaker, the task-stop watchdog, and health guidance for worker services.

## What can go wrong, and what catches it

A deploy that ships a task which cannot boot — wrong schema, missing secret, broken image — fails in one of two very different ways:

* **Loudly**, when the service sits behind a load balancer: the new task fails its health checks, the deployment circuit breaker trips, and ECS rolls back to the previous revision automatically.
* **Silently**, when the service has no health check — a queue worker, a poller, a scheduled task. ECS counts a worker task healthy the moment it reaches `RUNNING`; a container that starts and then exits is invisible to the circuit breaker's failure arithmetic. The deployment can report **successful** while the service crash-loops, and once ECS expunges the stopped tasks (about an hour) there is almost no trace left.

Fjall layers four defences so both failure shapes are caught:

| Layer                      | What it does                                                           | Where it acts            |
| -------------------------- | ---------------------------------------------------------------------- | ------------------------ |
| Schema-version boot gate   | A task refuses to serve against a schema it was not built for          | Inside the container     |
| Deployment circuit breaker | Failed deployments roll back automatically                             | ECS control plane        |
| Task-stop watchdog         | Every abnormal task stop is recorded durably and alarmed on            | EventBridge + CloudWatch |
| Deploy-time ECS event tail | `fjall deploy` surfaces task stops and rollbacks live in deploy output | The deploy engine        |

## The schema-version boot gate

Declare `migrations:` once on a database and every service whose `connections:` include that database receives an `EXPECTED_SCHEMA_VERSION` environment variable at synthesis — the name of the latest migration the deployed code was built against. (`EXPECTED_CH_SCHEMA_VERSION` is the ClickHouse counterpart.)

```typescript theme={null}
const database = app.addDatabase(
  DatabaseFactory.build("AppDatabase", {
    type: "Instance",
    databaseName: "appDb",
    migrations: {
      tool: "prisma",
      path: "./prisma/migrations",
    },
  }),
);
```

Your application checks the variable at boot — the `verifyExpectedSchemaVersion` helper from `@fjall/util/migration` queries the live database and compares — and exits before serving traffic if the schema is behind the code. That closes the "new code, old schema" window: the failure is one structured log line at boot, not scattered runtime errors on unlucky requests.

The comparison is deliberately tolerant in the safe direction:

* **Rollbacks pass.** An older image booting against a database that has already applied a *newer* expand-only migration starts cleanly (`actual >= expected` for orderable migration names). Expand/contract discipline guarantees the old code's columns still exist, so a rollback must never be refused at boot.
* **Forward skew refuses.** New code on an older schema (`actual < expected`) exits — the code may read columns the database lacks.

Per-service control:

* `schemaGate: false` on a service opts it out of auto-injection entirely; inject the variable yourself if you still want the gate.
* An author-set `EXPECTED_SCHEMA_VERSION` in a service's `environment:` block wins over the injected value (with a synth-time warning).
* A service connected to two migrated databases of the same kind is rejected at synth — the gate cannot inject one expectation unambiguously. Split the service or set `schemaGate: false`.

<Note>
  Web services get deployment-level enforcement for free: a boot-refusing task
  fails its health checks and the circuit breaker rolls the deployment back.
  Workers get the correctness half (no wrong-schema processing) but not the
  rollback half — see the worker guidance below.
</Note>

## The deployment circuit breaker

Every Fjall ECS service ships with the [deployment circuit breaker](https://docs.aws.amazon.com/AmazonECS/latest/developerguide/deployment-circuit-breaker.html) enabled and set to roll back:

```typescript theme={null}
{
  name: "api",
  capacityProvider: "FARGATE",
  containers: [{ name: "app", port: 3000 }],
  // circuitBreaker defaults to { rollback: true } — nothing to write
}
```

When a deployment's new tasks repeatedly fail to reach a healthy state, ECS stops launching replacements and restores the previous service revision without operator involvement.

### Hardening knobs

Two properties tighten the breaker beyond AWS defaults (available from `@fjall/components-infrastructure` 9.1):

| Property             | Type      | Default      | Meaning                                                                                                                                                                                                                       |
| -------------------- | --------- | ------------ | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `resetOnHealthyTask` | `boolean` | `false`      | Whether a single healthy task resets the breaker's failure count. Fjall defaults this **off** (AWS defaults it on) so an intermittently-booting revision cannot keep resetting the count and stall the rollback indefinitely. |
| `threshold`          | `number`  | ECS-computed | Absolute number of failed task launches before the breaker trips (`COUNT` semantics). Without it, ECS derives a percentage-based threshold that can be surprisingly high for low-`desiredCount` services.                     |

```typescript theme={null}
{
  name: "worker",
  capacityProvider: "FARGATE",
  desiredCount: 2,
  containers: [{ name: "app" }],
  circuitBreaker: { threshold: 5 },   // trip after 5 failed launches
}
```

`circuitBreaker: false` disables the breaker entirely — only do this if you run your own deployment verification.

<Warning>
  The circuit breaker only counts tasks that fail **to become healthy**: failed
  load-balancer health checks, or tasks that never reach `RUNNING`. A worker
  container that starts successfully and then exits — a crash loop after boot —
  is invisible to it. Do not rely on the breaker as your only safety net for
  services without health checks; that is what the task-stop watchdog below is
  for.
</Warning>

## The task-stop watchdog

From `@fjall/components-infrastructure` 9.1, every ECS cluster gets a **task-stop watchdog** by default: a per-cluster EventBridge rule that captures abnormal task stops into a durable log, derives a per-service metric from them, and alarms on churn.

What it watches: task-state-change events with `lastStatus: STOPPED` and a stop code of `EssentialContainerExited` (any crash after start, including crash loops) or `TaskFailedToStart` (image pull, secret resolution, dependency-condition failures). Routine stops — scale-in, deploy draining, operator `StopTask`, Spot interruption — are deliberately excluded so the signal stays clean.

What it creates:

| Piece        | Detail                                                                                                                                                                |
| ------------ | --------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| Forensic log | `/fjall/<cluster>/task-stops`, 30-day retention — the **full stop event**, including container exit codes and stop reasons, survives long after ECS expunges the task |
| Metric       | `Fjall/ECS · StoppedTaskCount`, dimensioned by cluster and service                                                                                                    |
| Alarm        | One per service: **3 or more abnormal stops in 5 minutes** → your cluster's `alertsTopic`. Notification-only — it never takes action on the service                   |

If the cluster has no `alertsTopic`, the forensic capture and metric still run; only the alarms are skipped. A service opted out of alarms with `alarms: false` is likewise excluded from the churn alarms — its stops still land in the forensic log and the metric.

When the alarm fires, the log group answers *why*:

```bash theme={null}
fjall aws exec --target production -- aws logs filter-log-events \
  --log-group-name /fjall/AppCluster/task-stops \
  --query 'events[].message' --output text
```

Each entry is the raw ECS event — look at `detail.stoppedReason`, `detail.stopCode`, and `detail.containers[].exitCode`.

Opt out per cluster with `taskStopWatchdog: false` (on `EcsCluster` or the ECS compute pattern). Cost is modest — roughly \$0.50–0.70 per service per month in CloudWatch alarm and log charges.

<Note>
  The watchdog grants EventBridge write access to its log group via a native
  CloudWatch Logs **resource policy**. AWS caps these at 10 per account per
  region, and each cluster's watchdog uses one. If you run many clusters in a
  single account and region and approach the quota, opt out of the watchdog on
  the clusters you care least about — or ask AWS Support to raise the limit.
</Note>

## Deploy-time visibility

During `fjall deploy`, the engine tails ECS service events for the services in the stack while CloudFormation converges. Task stops, failed placements, and circuit-breaker rollbacks appear in the deploy output as they happen, rather than the deploy reporting only CloudFormation's final verdict. A deployment that CloudFormation calls successful but whose service is churning tasks is surfaced in the same terminal that ran the deploy.

## Guidance for worker services

Services without a load balancer are where deployment failures hide. Three practices close the gap:

1. **Keep the boot gate on.** The auto-injected `EXPECTED_SCHEMA_VERSION` gate means a mis-ordered deploy refuses at boot with one structured log line instead of corrupting work mid-message. Wire the check into your worker's entrypoint with `verifyExpectedSchemaVersion` from `@fjall/util/migration` — the same helper and tolerance semantics web services use.
2. **Let the watchdog alarm reach a human.** Set `alertsTopic` on the cluster. A crash-looping worker then pages within minutes, with a 30-day forensic record of every stop, instead of failing silently until queue-depth symptoms surface.
3. **Give the breaker something to count where you can.** A container `healthCheck` (even a trivial process liveness probe) moves failures from "invisible post-`RUNNING` exit" towards territory the circuit breaker can act on, restoring automatic rollback for boot-time failures.

```typescript theme={null}
{
  name: "worker",
  capacityProvider: "FARGATE",
  containers: [
    {
      name: "app",
      healthCheck: {
        command: ["CMD-SHELL", "test -f /tmp/healthy || exit 1"],
        startPeriod: 30,
      },
    },
  ],
  circuitBreaker: { threshold: 3 },
}
```