Overview
fjall drift detects and remediates out-of-band changes to CloudFormation-managed resources. Its focus is the deploy-wedging class of drift: a resource (a KMS key, an S3 bucket, a secret) that CloudFormation still tracks but that was deleted directly in AWS. The next deploy of that stack fails, rolls back, and fails again on every retry until the drift is remediated.
The family has three subcommands:
Usage
detect and repair, target is an application name or path. Without one, a single local application is picked up automatically; with several you must name one. Both probe the application’s CloudFormation stacks (Network, Database, Storage, Compute, Cdn, Messaging), or exactly one stack with --stack. resume is different: it takes the operation id of a journalled repair (printed when the repair starts) and continues that one operation. It probes no target and takes no --stack.
Detecting Drift
kms:DescribeKey returning NotFoundException) produces a confirmed finding. Anything less (throttling, access denied, a network failure) stays unconfirmed, and unconfirmed findings are never offered a destructive remediation.
A KMS key scheduled for deletion, or a secret inside its recovery window,
counts as confirmed deleted: CloudFormation already treats it as gone,
even though the deletion is still cancellable on the AWS side. The finding
detail names the reversal API while the window lasts.
Verdicts
Example output with one confirmed finding:
Detection is wired into deploys
You rarely need to runfjall drift detect on a schedule. After a failed deploy, suspects are journalled locally (under ~/.fjall/drift), and every fjall deploy runs a drift pre-flight before touching AWS: stacks in a failed or rolled-back state, and stacks carrying journalled suspects, are re-probed. A confirmed deletion blocks the deploy with the finding and its cure rather than burning a doomed CloudFormation run:
drift detect options
Repairing Drift
fjall drift repair remediates confirmed deletions with the forget verb: CloudFormation stops tracking the deleted resource, and nothing else is touched (the physical resource is already gone). Unconfirmed findings are never repairable here.
Repair is consented, not automatic. Run without consent it stops with a ticket naming each confirmed resource, its exact physical name, and the verbs you may choose, then exits with code 4 (nothing was changed):
recreate (replace the deleted resource with a new one under the same name)
is consented on the deploy, not through fjall drift repair. The repair
surgery’s verb is forget.The forget surgery takes two deploys
A consented forget is deliberately split so that each step is provable and interruptible:fjall drift repaircaptures a forensics bundle (to~/.fjall/forensics/remediation/<opId>/), then applies a metadata-only change to the stack that marks the deleted resources as retained. The change is proven metadata-only via a CloudFormation change set before it is executed. The operation is journalled locally with an operation id.- The next
fjall deploy(the converge) removes the resources from CloudFormation’s view and closes the operation. Remove the resource from your fjall config first (for example withfjall remove); if the synthesised template still declares it, the deploy pre-flight names the resource and asks you to remove it before converging.
Deferred findings
Some findings cannot be repaired in the current run, for example when the same stack already carries an in-flight remediation that must converge first. Those findings are deferred: the run reports each deferred row with its reason and a cure specific to that reason. For an in-flight operation on this machine the cure is sequencing (resume it, run the converge deploy, then re-runfjall drift repair). Some deferrals have different cures: an operation whose journal lives on another machine completes there (or via a deploy that converges it), and a marked resource the repair cannot safely adopt is cured by finishing the original operation or updating the template by hand. Always follow the cure printed on the row rather than assuming resume. Deferral exits 0. It is a safe, complete outcome, not a failure and not a missing consent.
Deleted resources that reference each other
CloudFormation resolves a resource’sRef/Fn::GetAtt references against the live service during any update that modifies it — including references to and from resources that were deleted out-of-band. When a deleted resource still needs its retention policy changed (step 1 above) but it references another deleted resource (the common CDK pairing of a KMS key and its alias), that policy change can never execute: CloudFormation rolls the update back at execution time.
fjall completes this shape without the policy change. The policy exists only to keep the converge’s removal non-destructive, and removing an already-deleted resource deletes nothing — so when the affected resource is verified deleted against its owning service, the repair proceeds directly (the CLI reports it as “flipless by necessity”). The verification does not stop there: the absence is re-proven against the live service immediately before every converge deploy, so if the resource has been recreated under the same name in the meantime, the deploy halts instead of deleting it.
Recorded limitation: when the policy change is blocked and absence cannot be verified
fjall drift repair and fjall drift resume stop before mutating anything — with the two workable cures on the error itself — when:
- the affected resource’s absence cannot be verified against its owning service (the probe fails, the resource is in fact present, or the operation record carries no probeable physical id), or
- other resources in the stack still reference a deleted resource that needs its policy changed — that update genuinely cannot execute, regardless of verification.
- Restore the referenced resource out-of-band (for example, cancel a scheduled KMS key deletion), then resume the operation. The repair verifies the restore against the live service and proceeds.
- Remove the affected resource and every reference to it (including stack Outputs) in one manual template update. Removal succeeds where modification cannot: CloudFormation deletes template-removed resources from its stored state, and deleting an already-deleted resource is a safe no-op.
drift repair options
Resuming a Remediation
fjall drift repair output or the deploy pre-flight block. No re-consent is needed: the journalled operation already carries the consent you gave, and every step re-verifies live state before acting.
A recreate operation cannot be resumed here; it resumes through its deploy (fjall deploy <app>). The CLI names the correct command when this applies.
drift resume options
Exit Codes
Treat exit
4 as “a human must decide”, not as a failure. See Destructive Changes for CI guidance.
fjall drift detect exits 0 whether or not it finds confirmed deletions:
detection is a report, and the exit code reflects only whether the probe run
itself succeeded. Do not gate a pipeline on detect’s exit status to alarm on
drift; parse the result block (drift-findings vs no-drift-detected) with
--agent, or rely on the deploy pre-flight, which blocks a wedged stack
loudly. The 4 row above applies to fjall drift repair.Agent Mode
With--agent, each subcommand emits a structured result block instead of prose. The result field carries the outcome: drift-findings or no-drift-detected for detect; drift-repair-applied, drift-repair-deferred, drift-repair-partial, no-confirmed-deletions, or destruction-consent-required for repair; drift-resume-applied for resume. Agents are instructed to relay findings and never to self-select a destructive verb. See Agent Mode.
Next Steps
Destructive Changes
The consent ceremony shared by deploys and drift repair.
fjall deploy
The converge deploy that completes a forget.
fjall remove
Remove a forgotten resource from infrastructure.ts before the converge.
CI/CD Integration
How pipelines should handle exit code 4.