Fjall Docs — deploy AWS infrastructure into your own account

What can go wrong, and what catches it

A deploy that ships a task which cannot boot (wrong schema, missing secret, broken image) fails in one of two very different ways.

Loudly, when the service sits behind a load balancer. The new task fails its health checks, the deployment circuit breaker trips, and ECS rolls back to the previous revision automatically.
Silently, when the service has no health check. Queue workers, pollers and scheduled tasks are counted healthy by ECS the moment they reach RUNNING. A container that starts and then exits is invisible to the circuit breaker’s failure arithmetic. The deployment can report successful while the service crash-loops, and once ECS expunges the stopped tasks (about an hour) almost no trace is left.

Fjall layers four defences so both failure shapes are caught.

Layer	What it does	Where it acts
Schema-version boot gate	A task refuses to start against a schema it was not built for	Inside the task
Deployment circuit breaker	Failed deployments roll back automatically	ECS control plane
Task-stop watchdog	Every abnormal task stop is recorded durably and alarmed on	EventBridge + CloudWatch
Deploy-time ECS event tail	`fjall deploy` surfaces task stops and rollbacks live in deploy output	The deploy engine

The schema-version boot gate

Declare migrations: once on a database. Every service whose connections: include that database then gets two things at synthesis: the expected schema version injected as environment variables, and a synthetic gate container that verifies the live database before any application container starts.

Declare the migrations directory

const database = app.addDatabase(
  DatabaseFactory.build("ApiDatabase", {
    type: "Instance",
    databaseName: "apiDb",
    migrations: {
      tool: "prisma",
      dir: "./prisma/migrations",
    },
  }),
);

The key is dir, not path. tool: "prisma" uses the built-in ^\d{14}_/ directory-name resolver. For Drizzle, Flyway, Rails or any other tool, supply your own resolver:

migrations: {
  tool: "custom",
  dir: "./db/migrations",
  versionResolver: (dir) => latestMigrationName(dir),
}

ClickHouse databases take a migrations: block of their own, with dir and an optional versionResolver. It carries no tool field, because the default resolver picks the lexicographically latest *.sql file under dir.

Injected environment variables

Variable	Injected when	Meaning
`EXPECTED_SCHEMA_VERSION`	A connected relational database declares `migrations:`	Latest migration name the image was built against
`EXPECTED_SCHEMA_VERSION_TOOL`	Always alongside `EXPECTED_SCHEMA_VERSION`	`"prisma"` or `"custom"`, so the gate picks a resolver
`EXPECTED_CH_SCHEMA_VERSION`	A connected ClickHouse database declares `migrations:`	ClickHouse counterpart of the version variable

EXPECTED_SCHEMA_VERSION and EXPECTED_SCHEMA_VERSION_TOOL travel as a pair. The runtime gate refuses a version supplied without a tool. If you inject either yourself, inject both.

The materialised `fjall-schema-gate` container

By default Fjall prepends a synthetic container named fjall-schema-gate to every eligible service, and gives every other container in the task a dependsOn: [{ container: "fjall-schema-gate", condition: "SUCCESS" }] edge. The gate connects to the live database as a read-only identity, compares the applied schema against the expected version, and exits. A non-zero exit stops the task before it reaches RUNNING. That counts toward the deployment circuit breaker, so a schema refusal rolls the deployment back rather than crash-looping invisibly. Enforcement does not depend on your application implementing anything. In init-container migration mode the ordering is migrate, then gate, then app. The migrate container never waits on the gate, because it runs against the pre-migration schema by definition.

Gate exit codes

Code	Kind	What it means
0	`pass`	Every configured target verified. Application containers start.
1	`refused`	The live schema does not satisfy the expected version. Land the migration or roll back.
2	`configError`	The gate could not attempt verification. Missing or invalid `FJALL_SCHEMA_GATE_*` env.
3	`connectionError`	The database was unreachable within the gate’s retry budget. Indeterminate.
4	`busy`	The database answered but refused a query slot for the whole budget (concurrency cap or quota).
5	`denied`	The database rejected the gate’s identity, credentials or grants. Re-run the migration task to reprovision.

When targets disagree, configError wins first, then refused, then denied, then busy, then connectionError.

Reading the exit code during a deploy

The gate’s exit hint rides the ECS tail’s stopped-task progress line, not a standalone callout:

[ecs] api: 2 task(s) from the new revision stopped since the update began — latest: Essential container in task exited (fjall-schema-gate exit 1 — schema gate refused: live schema does not satisfy the expected version — land the migration or roll back)

The [ecs] prefix appears under --non-interactive. In an interactive run the same line renders in the ECS operation box, truncated to the box width, and lands in full in the deploy log file. The final error block repeats the hint only when the ECS completion is the classifier’s source. A captured CloudFormation failure takes precedence and produces a cfn.* record with no gate hint, so read the ECS tail rather than waiting for the summary.

When the gate container does not materialise

Materialisation turns itself off, with a synth warning, in these cases. Environment injection stays on, so your own boot check still works.

Condition	Why
`migrations.mode: "post-deploy"`	Expected is deliberately ahead of the database at scale-up, so the gate would refuse every task
`migrations.mode: "lifecycle-hook"` without `separateTaskDef`	The migration RunTask reuses the service task definition, so the gate would refuse the migration itself. Add `separateTaskDef` to restore full gating
`tool` is not `"prisma"`	The relational half of the gate supports Prisma only today
The connected database is not postgres-family	The relational half supports postgres-family engines only today
The constructs package version cannot be resolved	The default gate image tag is underivable. Set `schemaGate.image` explicitly

The two relational half rows are not all-or-nothing. The relational and ClickHouse halves are resolved independently, so a service whose ClickHouse half is eligible still materialises a gate container even when the relational half is skipped for a non-Prisma tool or a non-postgres engine. The other three rows disable materialisation outright. Scheduled tasks never materialise a gate container. They receive the environment variables, and their own entrypoint decides whether to act on them.

Per-service control

{
  name: "worker",
  capacityProvider: "FARGATE",
  containers: [{ name: "app" }],
  schemaGate: { materialise: false },   // env injection only
}

Value	Effect
omitted (default)	Environment injection on, gate container materialised when eligible
`false`	Everything off. No injection, no gate container. Auditable and grep-recoverable
`{ materialise: false }`	Environment injection only. Your application’s own boot gate is the sole enforcement
`{ image: "..." }`	Bring your own gate image URI, for VPCs that cannot reach the mirrored default

An author-set EXPECTED_SCHEMA_VERSION in a service’s environment: block wins inside that container, with a synth warning. The materialised gate container still enforces the resolved value. Use schemaGate: { materialise: false } when the author value must be authoritative. A service connected to two migrated databases of the same kind is rejected at synth, because the gate cannot inject one expectation unambiguously. Split the service or set schemaGate: false.

Adding an application-side check

An application-side gate is optional now, and still useful for local runs, integration tests and CLI checks. verifyExpectedSchemaVersion from @fjall/util/migration queries the live database and returns the comparison. It never exits the process, so your boot code owns the exit code and the log shape.

import { verifyExpectedSchemaVersion } from "@fjall/util/migration";

const { matches, expected, actual } = await verifyExpectedSchemaVersion({
  tool: "prisma",
  expected: process.env.EXPECTED_SCHEMA_VERSION ?? "",
  client: pgClient,
});

if (!matches) {
  logger.error("Schema gate refused", { expected, actual });
  process.exit(1);
}

The comparison is deliberately tolerant in the safe direction.

Rollbacks pass. An older image booting against a database that has already applied a newer expand-only migration starts cleanly (actual >= expected). Expand and contract discipline guarantees the old code’s columns still exist.
Forward skew refuses. New code on an older schema (actual < expected) fails, because the code may read columns the database lacks.
A never-migrated database refuses. actual is null when no migration has been applied, and that never matches.

The tolerance applies only when both versions are orderable migration identifiers, a Prisma migration name or a ClickHouse .sql filename, whose leading numeric prefixes are the same width. Prisma’s fixed 14-digit timestamps always qualify. Anything else falls back to strict identity, which refuses the rollback: a tool: "custom" version that is a content hash rather than a name, and ClickHouse filenames whose NNN- prefixes differ in width or are absent. Zero-pad ClickHouse migration filenames to a fixed width so rollbacks stay tolerant there too.

The deployment circuit breaker

Every Fjall ECS service ships with the deployment circuit breaker enabled and set to roll back:

{
  name: "api",
  capacityProvider: "FARGATE",
  containers: [{ name: "app", port: 3000 }],
  // circuitBreaker defaults to { rollback: true }, nothing to write
}

When a deployment’s new tasks repeatedly fail to reach a healthy state, ECS stops launching replacements and restores the previous service revision without operator involvement.

Options

Property	Type	Default	Meaning
`rollback`	`boolean`	`true`	Restore the previous revision when the breaker trips. `false` detects a failed deployment without reverting it
`resetOnHealthyTask`	`boolean`	`false`	Whether one healthy task resets the breaker’s failure count. Fjall defaults this off (AWS defaults it on) so an intermittently-booting revision cannot keep resetting the count and stall the rollback
`threshold`	`number`	ECS-computed	Absolute number of failed task launches before the breaker trips (`COUNT` semantics). Without it, ECS derives a percentage-based threshold that can be surprisingly high for low-`desiredCount` services

{
  name: "worker",
  capacityProvider: "FARGATE",
  desiredCount: 2,
  containers: [{ name: "app" }],
  circuitBreaker: { threshold: 5 },   // trip after 5 failed launches
}

circuitBreaker: false disables the breaker entirely. Only do this if you run your own deployment verification.

The circuit breaker counts tasks that fail to become healthy: failed load-balancer health checks, tasks that never reach RUNNING, and gate refusals. A container that starts successfully and then exits later, a crash loop after boot, is invisible to it. That gap is what the task-stop watchdog below covers.

The task-stop watchdog

Every ECS cluster gets a task-stop watchdog by default: a per-cluster EventBridge rule that captures abnormal task stops into a durable log, derives a per-service metric from them, and alarms on churn. It watches task-state-change events with lastStatus: STOPPED and a stop code of EssentialContainerExited (any crash after start, including crash loops) or TaskFailedToStart (image pull, secret resolution, dependency-condition failures). Routine stops are deliberately excluded so the signal stays clean: scale-in and deploy draining (ServiceSchedulerInitiated), operator StopTask (UserInitiated), SpotInterruption and TerminationNotice.

Piece	Detail
Forensic log	`/fjall/<cluster>/task-stops`, 30-day retention. The full stop event, including container exit codes and stop reasons, survives long after ECS expunges the task
Metric	`Fjall/ECS · StoppedTaskCount`, dimensioned by `ClusterName` (the cluster ARN) and `ServiceName` (`service:<name>`)
Alarm	One per service. 3 or more abnormal stops in 5 minutes notifies your cluster’s `alertsTopic`. Notification-only, it never acts on the service

Without an alertsTopic on the cluster, the forensic capture and the metric still run and only the alarms are skipped. A service carrying alarms: false is excluded from the churn alarms too, while its stops still land in the log and the metric. Standalone and scheduled tasks (family:* groups) are logged for forensics but raise no alarm, because they have no service to dimension on. When the alarm fires, the log group answers why:

fjall aws exec --target production-euw1 -- aws logs filter-log-events \
  --log-group-name /fjall/ApiCluster/task-stops \
  --query 'events[].message' --output text

Run fjall target list to see the target names available to you. They carry a region suffix, such as production-euw1 or production-use2. Each entry is the raw ECS event. Read detail.stoppedReason, detail.stopCode and detail.containers[].exitCode. An exit code from the gate taxonomy above on the fjall-schema-gate container tells you the deployment was stopped on purpose. Opt out per cluster with taskStopWatchdog: false on EcsCluster or the ECS compute pattern. Cost is modest, roughly

0.50 to

0.70 per service per month in CloudWatch alarm and log charges.

The watchdog grants EventBridge write access to its log group with a native AWS::Logs::ResourcePolicy, one policy per log group. AWS caps account-level CloudWatch Logs resource policies at 10 per region. If you run many clusters in one account and region and approach the quota, opt out of the watchdog on the clusters you care least about, or ask AWS Support to raise the limit.

Deploy-time visibility

During fjall deploy, the engine tails ECS service events for the services in the stack while CloudFormation converges. Task stops, failed placements and circuit-breaker rollbacks appear in the deploy output as they happen, rather than the deploy reporting only CloudFormation’s final verdict. A deployment that CloudFormation calls successful but whose service is churning tasks is surfaced in the same terminal that ran the deploy.

Guidance for worker services

Services without a load balancer are where deployment failures used to hide. Three practices close the gap.

Leave the gate materialised. With the default gate container, a worker that would boot against the wrong schema is stopped pre-RUNNING, which the circuit breaker counts and rolls back. If you set schemaGate: { materialise: false } or schemaGate: false, you give that rollback up and your own boot check becomes the only enforcement.
Let the watchdog alarm reach a human. Set alertsTopic on the cluster. A crash-looping worker then pages within minutes, with a 30-day forensic record of every stop, instead of failing silently until queue-depth symptoms surface.
Give the breaker something to count for post-boot crashes. The gate covers schema refusals. A container healthCheck, even a trivial process liveness probe, moves later crashes from an invisible post-RUNNING exit into territory the circuit breaker can act on.

{
  name: "worker",
  capacityProvider: "FARGATE",
  containers: [
    {
      name: "app",
      healthCheck: {
        command: ["CMD-SHELL", "test -f /tmp/healthy || exit 1"],
        startPeriod: 30,
      },
    },
  ],
  circuitBreaker: { threshold: 3 },
}

Next Steps

Destructive Changes

Consent gates for replacements, deletions and remediation verbs.

Circuit Breaker Errors

Diagnose a deployment that ECS rolled back.

ECS Cluster

Service, container, scaling and health-check options in full.

Engine Compatibility

Keep the CLI and your infrastructure code on compatible versions.

Install & Setup

Application Deploy

Deployment Errors

Organisation Setup

Organisation Deploy

Boot Gates & Deployment Safety

What can go wrong, and what catches it

The schema-version boot gate

Declare the migrations directory

Injected environment variables

The materialised `fjall-schema-gate` container

Gate exit codes

Reading the exit code during a deploy

When the gate container does not materialise

Per-service control

Adding an application-side check

The deployment circuit breaker

Options

The task-stop watchdog

Deploy-time visibility

Guidance for worker services

Next Steps

Destructive Changes

Circuit Breaker Errors

ECS Cluster

Engine Compatibility

​What can go wrong, and what catches it

​The schema-version boot gate

​Declare the migrations directory

​Injected environment variables

​The materialised fjall-schema-gate container

​Gate exit codes

​Reading the exit code during a deploy

​When the gate container does not materialise

​Per-service control

​Adding an application-side check

​The deployment circuit breaker

​Options

​The task-stop watchdog

​Deploy-time visibility

​Guidance for worker services

​Next Steps

Destructive Changes

Circuit Breaker Errors

ECS Cluster

Engine Compatibility

What can go wrong, and what catches it

The schema-version boot gate

Declare the migrations directory

Injected environment variables

The materialised `fjall-schema-gate` container

Gate exit codes

Reading the exit code during a deploy

When the gate container does not materialise

Per-service control

Adding an application-side check

The deployment circuit breaker

Options

The task-stop watchdog

Deploy-time visibility

Guidance for worker services

Next Steps