Boot Gates & Deployment Safety

​

What can go wrong, and what catches it

• Loudly, when the service sits behind a load balancer: the new task fails its health checks, the deployment circuit breaker trips, and ECS rolls back to the previous revision automatically.
• Silently, when the service has no health check — a queue worker, a poller, a scheduled task. ECS counts a worker task healthy the moment it reaches RUNNING; a container that starts and then exits is invisible to the circuit breaker’s failure arithmetic. The deployment can report successful while the service crash-loops, and once ECS expunges the stopped tasks (about an hour) there is almost no trace left.

​

The schema-version boot gate

const database = app.addDatabase(
  DatabaseFactory.build("AppDatabase", {
    type: "Instance",
    databaseName: "appDb",
    migrations: {
      tool: "prisma",
      path: "./prisma/migrations",
    },
  }),
);

• Rollbacks pass. An older image booting against a database that has already applied a newer expand-only migration starts cleanly (actual >= expected for orderable migration names). Expand/contract discipline guarantees the old code’s columns still exist, so a rollback must never be refused at boot.
• Forward skew refuses. New code on an older schema (actual < expected) exits — the code may read columns the database lacks.

• schemaGate: false on a service opts it out of auto-injection entirely; inject the variable yourself if you still want the gate.
• An author-set EXPECTED_SCHEMA_VERSION in a service’s environment: block wins over the injected value (with a synth-time warning).
• A service connected to two migrated databases of the same kind is rejected at synth — the gate cannot inject one expectation unambiguously. Split the service or set schemaGate: false.

​

The deployment circuit breaker

{
  name: "api",
  capacityProvider: "FARGATE",
  containers: [{ name: "app", port: 3000 }],
  // circuitBreaker defaults to { rollback: true } — nothing to write
}

​

Hardening knobs

{
  name: "worker",
  capacityProvider: "FARGATE",
  desiredCount: 2,
  containers: [{ name: "app" }],
  circuitBreaker: { threshold: 5 },   // trip after 5 failed launches
}

​

The task-stop watchdog

fjall aws exec --target production -- aws logs filter-log-events \
  --log-group-name /fjall/AppCluster/task-stops \
  --query 'events[].message' --output text

​

Deploy-time visibility

​

Guidance for worker services

1. Keep the boot gate on. The auto-injected EXPECTED_SCHEMA_VERSION gate means a mis-ordered deploy refuses at boot with one structured log line instead of corrupting work mid-message. Wire the check into your worker’s entrypoint with verifyExpectedSchemaVersion from @fjall/util/migration — the same helper and tolerance semantics web services use.
2. Let the watchdog alarm reach a human. Set alertsTopic on the cluster. A crash-looping worker then pages within minutes, with a 30-day forensic record of every stop, instead of failing silently until queue-depth symptoms surface.
3. Give the breaker something to count where you can. A container healthCheck (even a trivial process liveness probe) moves failures from “invisible post-RUNNING exit” towards territory the circuit breaker can act on, restoring automatic rollback for boot-time failures.

{
  name: "worker",
  capacityProvider: "FARGATE",
  containers: [
    {
      name: "app",
      healthCheck: {
        command: ["CMD-SHELL", "test -f /tmp/healthy || exit 1"],
        startPeriod: 30,
      },
    },
  ],
  circuitBreaker: { threshold: 3 },
}