> ## Documentation Index > Fetch the complete documentation index at: https://docs.fjall.io/llms.txt > Use this file to discover all available pages before exploring further. # fjall drift > Detect resources deleted outside CloudFormation and repair the wedged stacks they leave behind. ## Overview `fjall drift` detects and remediates out-of-band changes to CloudFormation-managed resources. Its focus is the deploy-wedging class of drift: a resource (a KMS key, an S3 bucket, a secret) that CloudFormation still tracks but that was deleted directly in AWS. The next deploy of that stack fails, rolls back, and fails again on every retry until the drift is remediated. The family has three subcommands: | Subcommand | Purpose | | --------------------- | ------------------------------------------------------------------------ | | `drift detect` | Probe stacks for resources deleted outside CloudFormation (read-only) | | `drift repair` | Remediate confirmed deletions via the consented forget surgery | | `drift resume ` | Resume an interrupted remediation by operation id (no re-consent needed) | ## Usage ```bash theme={null} fjall drift detect [target] [options] fjall drift repair [target] [options] fjall drift resume [options] ``` For `detect` and `repair`, `target` is an application name or path. Without one, a single local application is picked up automatically; with several you must name one. Both probe the application's CloudFormation stacks (Network, Database, Storage, Compute, Cdn, Messaging), or exactly one stack with `--stack`. `resume` is different: it takes the operation id of a journalled repair (printed when the repair starts) and continues that one operation. It probes no target and takes no `--stack`. ## Detecting Drift ```bash theme={null} fjall drift detect api ``` Detection is read-only against your AWS resources. For each stack it scans the last operation's failure events, merges in suspects journalled from earlier failed deploys, and then verifies every suspect against the service that owns the resource. Only an authoritative not-found from the owning service (for example `kms:DescribeKey` returning `NotFoundException`) produces a confirmed finding. Anything less (throttling, access denied, a network failure) stays unconfirmed, and unconfirmed findings are never offered a destructive remediation. A KMS key scheduled for deletion, or a secret inside its recovery window, counts as **confirmed deleted**: CloudFormation already treats it as gone, even though the deletion is still cancellable on the AWS side. The finding detail names the reversal API while the window lasts. ### Verdicts | Verdict | Meaning | What to do | | ----------------- | --------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------ | | **CONFIRMED** | The owning service reports the resource gone while CloudFormation still tracks it | Remediate with `fjall drift repair`; the next deploy of the stack fails until you do | | **UNCONFIRMED** | The probe could not verify the deletion (throttle, access, timeout) | Repair-only by design: re-run `fjall drift detect` once access recovers | | **STACK MISSING** | The stack itself no longer exists | Nothing to remediate; redeploy the application if the deletion was unexpected | | **RECOVERED** | A previously suspected resource was verified present | No action needed; the journal entry is cleared | Example output with one confirmed finding: ``` CONFIRMED deleted outside CloudFormation (1): ApiDatabase › EncryptionKey (AWS::KMS::Key) [1a2b3c4d-5e6f-7890-abcd-ef1234567890] — kms:DescribeKey returned NotFoundException The next deploy of these stacks will fail until remediated. Remediation verbs: forget (stop managing the resource — the physical resource is never touched). Remediate with 'fjall drift repair' — the forget surgery stops CloudFormation tracking the deleted resources (nothing else is touched) so the next deploy converges cleanly. ``` And a clean run: ``` No out-of-band deletions detected across 6 stack(s) in 123456789012/eu-west-2. ``` ### Detection is wired into deploys You rarely need to run `fjall drift detect` on a schedule. After a failed deploy, suspects are journalled locally (under `~/.fjall/drift`), and every `fjall deploy` runs a drift pre-flight before touching AWS: stacks in a failed or rolled-back state, and stacks carrying journalled suspects, are re-probed. A confirmed deletion blocks the deploy with the finding and its cure rather than burning a doomed CloudFormation run: ``` Deploy blocked: CloudFormation still tracks 1 resource(s) deleted outside CloudFormation — deploying would fail and re-wedge the stack: ApiDatabase › EncryptionKey (AWS::KMS::Key) [1a2b3c4d-5e6f-7890-abcd-ef1234567890] — kms:DescribeKey returned NotFoundException Run 'fjall drift detect' for the full report, then 'fjall drift repair --remediate =' to remediate confirmed deletions. ``` Unconfirmed findings never block a deploy; they surface as warnings and are re-probed next time. ### drift detect options | Option | Description | | ----------------------- | -------------------------------------------------------------------------------- | | `--stack ` | Probe a single CloudFormation stack by exact name instead of the target's stacks | | `-r, --region ` | AWS region override | | `-v, --verbose` | Enable verbose logging | | `--non-interactive` | Force plain CLI output (no UI) | ## Repairing Drift ```bash theme={null} fjall drift repair api ``` `fjall drift repair` remediates **confirmed** deletions with the `forget` verb: CloudFormation stops tracking the deleted resource, and nothing else is touched (the physical resource is already gone). Unconfirmed findings are never repairable here. Repair is consented, not automatic. Run without consent it stops with a ticket naming each confirmed resource, its exact physical name, and the verbs you may choose, then exits with code `4` (nothing was changed): ``` ! resource-deleted AWS::KMS::Key ApiDatabase/EncryptionKey physical name: 1a2b3c4d-5e6f-7890-abcd-ef1234567890 cause: deleted outside CloudFormation — kms:DescribeKey returned NotFoundException legal verbs: abort | forget awaiting consent: 1a2b3c4d-5e6f-7890-abcd-ef1234567890 Deployment withheld — destructive changes require named consent; nothing was applied. Consent to each destructive change by exact physical name, then re-run: fjall drift repair api --remediate 1a2b3c4d-5e6f-7890-abcd-ef1234567890= ``` Consent by naming the resource exactly (case-sensitive) with the verb: ```bash theme={null} fjall drift repair api --remediate 1a2b3c4d-5e6f-7890-abcd-ef1234567890=forget ``` Misspelt names, surplus consents, and illegal verbs all fail closed with a per-name verdict table; nothing runs until every confirmed resource is covered by a matching consent. The consent semantics are shared with the deploy-time destruction gate. See [Destructive Changes](/deployment/destructive-changes) for the full ceremony. `recreate` (replace the deleted resource with a new one under the same name) is consented on the deploy, not through `fjall drift repair`. The repair surgery's verb is `forget`. ### The forget surgery takes two deploys A consented forget is deliberately split so that each step is provable and interruptible: 1. **`fjall drift repair`** captures a forensics bundle (to `~/.fjall/forensics/remediation//`), then applies a metadata-only change to the stack that marks the deleted resources as retained. The change is proven metadata-only via a CloudFormation change set before it is executed. The operation is journalled locally with an operation id. 2. **The next `fjall deploy`** (the converge) removes the resources from CloudFormation's view and closes the operation. Remove the resource from your fjall config first (for example with [`fjall remove`](/cli/remove)); if the synthesised template still declares it, the deploy pre-flight names the resource and asks you to remove it before converging. After step 1 the CLI reports the surgery, for example: ``` Forget surgery on ApiDatabase — operation 3f9c2a1b (phase: flip-applied) Forensics captured at /Users/you/.fjall/forensics/remediation/3f9c2a1b/2026-07-12T101500.000Z The Retain flip is applied; CloudFormation still tracks the resources until the converge deploy. Run 'fjall deploy api' to complete the forget. If interrupted, resume with 'fjall drift resume ' — the journal already carries the consent. ``` The operation is kill-safe: state is journalled on disk and also marked in the stack template itself, so a killed terminal, a lost laptop session, or a machine swap degrades to correct re-detection rather than a half-applied surgery. ### Deferred findings Some findings cannot be repaired in the current run, for example when the same stack already carries an in-flight remediation that must converge first. Those findings are **deferred**: the run reports each deferred row with its reason and a cure specific to that reason. For an in-flight operation on this machine the cure is sequencing (resume it, run the converge deploy, then re-run `fjall drift repair`). Some deferrals have different cures: an operation whose journal lives on another machine completes there (or via a deploy that converges it), and a marked resource the repair cannot safely adopt is cured by finishing the original operation or updating the template by hand. Always follow the cure printed on the row rather than assuming resume. Deferral exits `0`. It is a safe, complete outcome, not a failure and not a missing consent. ### Deleted resources that reference each other CloudFormation resolves a resource's `Ref`/`Fn::GetAtt` references against the live service during any update that modifies it — including references to and from resources that were deleted out-of-band. When a deleted resource still needs its retention policy changed (step 1 above) but it references another deleted resource (the common CDK pairing of a KMS key and its alias), that policy change can never execute: CloudFormation rolls the update back at execution time. fjall completes this shape without the policy change. The policy exists only to keep the converge's removal non-destructive, and removing an already-deleted resource deletes nothing — so when the affected resource is verified deleted against its owning service, the repair proceeds directly (the CLI reports it as "flipless by necessity"). The verification does not stop there: the absence is re-proven against the live service immediately before every converge deploy, so if the resource has been recreated under the same name in the meantime, the deploy halts instead of deleting it. ### Recorded limitation: when the policy change is blocked and absence cannot be verified `fjall drift repair` and `fjall drift resume` stop before mutating anything — with the two workable cures on the error itself — when: * the affected resource's absence cannot be verified against its owning service (the probe fails, the resource is in fact present, or the operation record carries no probeable physical id), or * other resources in the stack still reference a deleted resource that needs its policy changed — that update genuinely cannot execute, regardless of verification. 1. **Restore the referenced resource out-of-band** (for example, cancel a scheduled KMS key deletion), then resume the operation. The repair verifies the restore against the live service and proceeds. 2. **Remove the affected resource and every reference to it (including stack Outputs) in one manual template update.** Removal succeeds where modification cannot: CloudFormation deletes template-removed resources from its stored state, and deleting an already-deleted resource is a safe no-op. Resources whose live policies already satisfy retention (the CDK default for stateful types such as keys and buckets) are unaffected: their step 1 is journal-only, with no CloudFormation call at all. ### drift repair options | Option | Description | | --------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------- | | `--stack ` | Repair a single CloudFormation stack by exact name instead of the target's stacks | | `-r, --region ` | AWS region override | | `--remediate ` | Consent to remediate the named resource (repeatable; exact physical name, case-sensitive) | | `--destruction-consent-digest ` | Bind consents to the ticket digest they were minted against, so a changed ticket withholds instead of silently carrying consents over | | `-v, --verbose` | Enable verbose logging | | `--non-interactive` | Force plain CLI output (no UI) | ## Resuming a Remediation ```bash theme={null} fjall drift resume ``` Resumes an in-flight forget remediation by operation id. The id comes from the `fjall drift repair` output or the deploy pre-flight block. No re-consent is needed: the journalled operation already carries the consent you gave, and every step re-verifies live state before acting. A `recreate` operation cannot be resumed here; it resumes through its deploy (`fjall deploy `). The CLI names the correct command when this applies. ### drift resume options | Option | Description | | ----------------------- | ------------------------------ | | `-r, --region ` | AWS region override | | `-v, --verbose` | Enable verbose logging | | `--non-interactive` | Force plain CLI output (no UI) | ## Exit Codes | Code | Meaning | | ---- | --------------------------------------------------------------------------------------------- | | `0` | Success: repairs applied, nothing to repair, or every finding safely deferred | | `4` | Consent withheld: confirmed findings need a named `--remediate` consent (nothing was changed) | | `1` | A real error | Treat exit `4` as "a human must decide", not as a failure. See [Destructive Changes](/deployment/destructive-changes#exit-code-4-in-ci) for CI guidance. `fjall drift detect` exits `0` whether or not it finds confirmed deletions: detection is a report, and the exit code reflects only whether the probe run itself succeeded. Do not gate a pipeline on detect's exit status to alarm on drift; parse the result block (`drift-findings` vs `no-drift-detected`) with `--agent`, or rely on the deploy pre-flight, which blocks a wedged stack loudly. The `4` row above applies to `fjall drift repair`. ## Agent Mode With `--agent`, each subcommand emits a structured result block instead of prose. The `result` field carries the outcome: `drift-findings` or `no-drift-detected` for detect; `drift-repair-applied`, `drift-repair-deferred`, `drift-repair-partial`, `no-confirmed-deletions`, or `destruction-consent-required` for repair; `drift-resume-applied` for resume. Agents are instructed to relay findings and never to self-select a destructive verb. See [Agent Mode](/cli/agent-mode/index). ## Next Steps The consent ceremony shared by deploys and drift repair. The converge deploy that completes a forget. Remove a forgotten resource from infrastructure.ts before the converge. How pipelines should handle exit code 4.