Skip to main content

Overview

fjall drift detects and remediates out-of-band changes to CloudFormation-managed resources. Its focus is the deploy-wedging class of drift: a resource (a KMS key, an S3 bucket, a secret) that CloudFormation still tracks but that was deleted directly in AWS. The next deploy of that stack fails, rolls back, and fails again on every retry until the drift is remediated. The family has three subcommands:

Usage

For detect and repair, target is an application name or path. Without one, a single local application is picked up automatically; with several you must name one. Both probe the application’s CloudFormation stacks (Network, Database, Storage, Compute, Cdn, Messaging), or exactly one stack with --stack. resume is different: it takes the operation id of a journalled repair (printed when the repair starts) and continues that one operation. It probes no target and takes no --stack.

Detecting Drift

Detection is read-only against your AWS resources. For each stack it scans the last operation’s failure events, merges in suspects journalled from earlier failed deploys, and then verifies every suspect against the service that owns the resource. Only an authoritative not-found from the owning service (for example kms:DescribeKey returning NotFoundException) produces a confirmed finding. Anything less (throttling, access denied, a network failure) stays unconfirmed, and unconfirmed findings are never offered a destructive remediation.
A KMS key scheduled for deletion, or a secret inside its recovery window, counts as confirmed deleted: CloudFormation already treats it as gone, even though the deletion is still cancellable on the AWS side. The finding detail names the reversal API while the window lasts.

Verdicts

Example output with one confirmed finding:
And a clean run:

Detection is wired into deploys

You rarely need to run fjall drift detect on a schedule. After a failed deploy, suspects are journalled locally (under ~/.fjall/drift), and every fjall deploy runs a drift pre-flight before touching AWS: stacks in a failed or rolled-back state, and stacks carrying journalled suspects, are re-probed. A confirmed deletion blocks the deploy with the finding and its cure rather than burning a doomed CloudFormation run:
Unconfirmed findings never block a deploy; they surface as warnings and are re-probed next time.

drift detect options

Repairing Drift

fjall drift repair remediates confirmed deletions with the forget verb: CloudFormation stops tracking the deleted resource, and nothing else is touched (the physical resource is already gone). Unconfirmed findings are never repairable here. Repair is consented, not automatic. Run without consent it stops with a ticket naming each confirmed resource, its exact physical name, and the verbs you may choose, then exits with code 4 (nothing was changed):
Consent by naming the resource exactly (case-sensitive) with the verb:
Misspelt names, surplus consents, and illegal verbs all fail closed with a per-name verdict table; nothing runs until every confirmed resource is covered by a matching consent. The consent semantics are shared with the deploy-time destruction gate. See Destructive Changes for the full ceremony.
recreate (replace the deleted resource with a new one under the same name) is consented on the deploy, not through fjall drift repair. The repair surgery’s verb is forget.

The forget surgery takes two deploys

A consented forget is deliberately split so that each step is provable and interruptible:
  1. fjall drift repair captures a forensics bundle (to ~/.fjall/forensics/remediation/<opId>/), then applies a metadata-only change to the stack that marks the deleted resources as retained. The change is proven metadata-only via a CloudFormation change set before it is executed. The operation is journalled locally with an operation id.
  2. The next fjall deploy (the converge) removes the resources from CloudFormation’s view and closes the operation. Remove the resource from your fjall config first (for example with fjall remove); if the synthesised template still declares it, the deploy pre-flight names the resource and asks you to remove it before converging.
After step 1 the CLI reports the surgery, for example:
The operation is kill-safe: state is journalled on disk and also marked in the stack template itself, so a killed terminal, a lost laptop session, or a machine swap degrades to correct re-detection rather than a half-applied surgery.

Deferred findings

Some findings cannot be repaired in the current run, for example when the same stack already carries an in-flight remediation that must converge first. Those findings are deferred: the run reports each deferred row with its reason and a cure specific to that reason. For an in-flight operation on this machine the cure is sequencing (resume it, run the converge deploy, then re-run fjall drift repair). Some deferrals have different cures: an operation whose journal lives on another machine completes there (or via a deploy that converges it), and a marked resource the repair cannot safely adopt is cured by finishing the original operation or updating the template by hand. Always follow the cure printed on the row rather than assuming resume. Deferral exits 0. It is a safe, complete outcome, not a failure and not a missing consent.

Deleted resources that reference each other

CloudFormation resolves a resource’s Ref/Fn::GetAtt references against the live service during any update that modifies it — including references to and from resources that were deleted out-of-band. When a deleted resource still needs its retention policy changed (step 1 above) but it references another deleted resource (the common CDK pairing of a KMS key and its alias), that policy change can never execute: CloudFormation rolls the update back at execution time. fjall completes this shape without the policy change. The policy exists only to keep the converge’s removal non-destructive, and removing an already-deleted resource deletes nothing — so when the affected resource is verified deleted against its owning service, the repair proceeds directly (the CLI reports it as “flipless by necessity”). The verification does not stop there: the absence is re-proven against the live service immediately before every converge deploy, so if the resource has been recreated under the same name in the meantime, the deploy halts instead of deleting it.

Recorded limitation: when the policy change is blocked and absence cannot be verified

fjall drift repair and fjall drift resume stop before mutating anything — with the two workable cures on the error itself — when:
  • the affected resource’s absence cannot be verified against its owning service (the probe fails, the resource is in fact present, or the operation record carries no probeable physical id), or
  • other resources in the stack still reference a deleted resource that needs its policy changed — that update genuinely cannot execute, regardless of verification.
  1. Restore the referenced resource out-of-band (for example, cancel a scheduled KMS key deletion), then resume the operation. The repair verifies the restore against the live service and proceeds.
  2. Remove the affected resource and every reference to it (including stack Outputs) in one manual template update. Removal succeeds where modification cannot: CloudFormation deletes template-removed resources from its stored state, and deleting an already-deleted resource is a safe no-op.
Resources whose live policies already satisfy retention (the CDK default for stateful types such as keys and buckets) are unaffected: their step 1 is journal-only, with no CloudFormation call at all.

drift repair options

Resuming a Remediation

Resumes an in-flight forget remediation by operation id. The id comes from the fjall drift repair output or the deploy pre-flight block. No re-consent is needed: the journalled operation already carries the consent you gave, and every step re-verifies live state before acting. A recreate operation cannot be resumed here; it resumes through its deploy (fjall deploy <app>). The CLI names the correct command when this applies.

drift resume options

Exit Codes

Treat exit 4 as “a human must decide”, not as a failure. See Destructive Changes for CI guidance.
fjall drift detect exits 0 whether or not it finds confirmed deletions: detection is a report, and the exit code reflects only whether the probe run itself succeeded. Do not gate a pipeline on detect’s exit status to alarm on drift; parse the result block (drift-findings vs no-drift-detected) with --agent, or rely on the deploy pre-flight, which blocks a wedged stack loudly. The 4 row above applies to fjall drift repair.

Agent Mode

With --agent, each subcommand emits a structured result block instead of prose. The result field carries the outcome: drift-findings or no-drift-detected for detect; drift-repair-applied, drift-repair-deferred, drift-repair-partial, no-confirmed-deletions, or destruction-consent-required for repair; drift-resume-applied for resume. Agents are instructed to relay findings and never to self-select a destructive verb. See Agent Mode.

Next Steps

Destructive Changes

The consent ceremony shared by deploys and drift repair.

fjall deploy

The converge deploy that completes a forget.

fjall remove

Remove a forgotten resource from infrastructure.ts before the converge.

CI/CD Integration

How pipelines should handle exit code 4.