> ## Documentation Index
> Fetch the complete documentation index at: https://docs.fjall.io/llms.txt
> Use this file to discover all available pages before exploring further.

# ClickHouse backups can't start on a 16 GiB host

> Known issue in fjall 40.0.1 and 39.2.4. A ClickHouse host with 16 GiB of memory has no room for the nightly backup task, so no backup is taken and no alarm says so.

In fjall 40.0.1 and 39.2.4, a ClickHouse database on a host with 16 GiB of memory never takes its nightly backup. The server is given almost all the memory ECS can place on the host, and the backup task needs 256 MiB of what's left. No alarm fires, so the backups stop without a word. Earlier releases weren't checked, so run the check below whatever version you're on.

A database on a 4 GiB, 8 GiB or 32 GiB host takes its backup, on every type measured. The exception is one on a 4 GiB or 8 GiB host that also sets `optimiseTables`, because the backup and the optimise task can't run at the same time there.

## Which hosts are short of memory

ECS registers a little less memory than an instance type's nominal size. The server's memory limit allows for neither that gap nor the backup and optimise tasks, which need 256 MiB each. These figures were measured on 18 November 2026 with fjall 40.0.1 on arm64 hosts.

| Host memory | Types measured | Left beside the server | Backup alone | Backup and optimise together |
| - | - | - | - | - |
| 32 GiB | `r7g.xlarge`, `r8g.xlarge` | 522 MiB | Starts | Both run |
| 16 GiB | `m7g.xlarge`, `m8g.xlarge`, `r7g.large`, `r8g.large` | 61 MiB | Can't start | Can't start |
| 8 GiB | `m7g.large`, `m8g.large`, `r7g.medium`, `r8g.medium` | 343 MiB | Starts | Can't both run |
| 4 GiB | `t4g.medium` | 507 MiB | Starts | Can't both run |
| 4 GiB | `m7g.medium`, `m8g.medium` | 483 MiB | Starts | Can't both run |

`m7g.xlarge` is the enterprise tier's default, `m7g.large` the resilient tier's and `t4g.medium` the standard tier's. Hosts with 32 GiB of memory have room for both tasks.

With `optimiseTables` set on a 4 GiB or 8 GiB host, a backup or optimise run that starts while the other is running can't start.

## Nothing alarms when a run can't start

When ECS can't place the task, EventBridge still counts the scheduled run as delivered. The schedule's failed invocation alarm stays quiet, and nothing reaches its dead letter queue. The task never starts, so it writes no log line, and the backup status alarms have nothing to read. Fjall measured this on fjall 40.0.1 on 18 November 2026.

In fjall 40.0.1 the backup heartbeat alarm waits for a first successful backup before it pages, so a database that has never had room for a backup never pages. A database deployed without an alarm topic has no ClickHouse alarms at all.

## Check whether your database is affected

Run these from your Fjall project, with the target that holds the database.

1. Find the ClickHouse host and its instance type.

   ```bash theme={"theme":{"light":"github-light","dark":"github-dark-high-contrast"}}
   fjall aws exec --target <target> -- aws ec2 describe-instances --filters Name=tag:ClickHouseRole,Values=server Name=instance-state-name,Values=running --query "Reservations[].Instances[].[InstanceId,InstanceType]" --output text
   ```

2. Read how much memory ECS has left on the host. The ClickHouse cluster is named `Compute`. Read it outside the backup window, because a running backup or optimise task holds its 256 MiB.

   ```bash theme={"theme":{"light":"github-light","dark":"github-dark-high-contrast"}}
   fjall aws exec --target <target> -- aws ecs list-container-instances --cluster Compute --query "containerInstanceArns" --output text
   fjall aws exec --target <target> -- aws ecs describe-container-instances --cluster Compute --container-instances <arn> --query "containerInstances[].remainingResources[?name=='MEMORY'].integerValue" --output text
   ```

A figure under 256 means the nightly backup can't start. A figure under 512 on a database that sets `optimiseTables` means the backup and the optimise task can't run at the same time.

## Keep a copy of the data until a fixed release ships

No setting in a released version lets the backup fit on a 16 GiB host. These steps keep a copy of the data until one does.

1. Find the ClickHouse data volume. It's the volume whose owner ends in `PersistentDataVolume` under your database's name.

   ```bash theme={"theme":{"light":"github-light","dark":"github-dark-high-contrast"}}
   fjall aws exec --target <target> -- aws ec2 describe-volumes --filters Name=tag:fjall:Lifecycle,Values=data-volume --query "Volumes[].[VolumeId,Tags[?Key=='fjall:OwnerLogicalId']|[0].Value]" --output text
   ```

   If that lists nothing, list the volumes attached to the host from the first check. The data volume is the one that isn't the root volume.

   ```bash theme={"theme":{"light":"github-light","dark":"github-dark-high-contrast"}}
   fjall aws exec --target <target> -- aws ec2 describe-volumes --filters Name=attachment.instance-id,Values=<instance id> --query "Volumes[].[VolumeId,Attachments[0].Device,Size]" --output text
   ```

2. See whether AWS Backup already covers it. Recent recovery points mean the volume is already in a backup plan.

   ```bash theme={"theme":{"light":"github-light","dark":"github-dark-high-contrast"}}
   fjall aws exec --target <target> -- aws backup list-recovery-points-by-resource --resource-arn arn:aws:ec2:<region>:<account id>:volume/<volume id>
   ```

3. If it isn't covered, take a snapshot of the volume each day until you've upgraded.

   ```bash theme={"theme":{"light":"github-light","dark":"github-dark-high-contrast"}}
   fjall aws exec --target <target> -- aws ec2 create-snapshot --volume-id <volume id> --description "ClickHouse data until the backup fix"
   ```

A snapshot copies the whole disk at one moment, while ClickHouse is still writing to it. It isn't the ClickHouse backup the nightly task makes, and nothing checks it. Restoring one means creating a new volume from it, and Fjall hasn't yet tested a ClickHouse restore from a snapshot. Snapshots are billed as EBS snapshot storage, and each one after the first stores only the blocks that changed.

## When the fix ships

The fix sizes the server so the backup and optimise tasks fit beside it, and adds an alarm for a scheduled run that doesn't start. It's planned for the next fjall release and for a 39.x release after 39.2.5, and this page will name both versions when they ship. Upgrading restarts ClickHouse once, because the server's memory limit changes.


This documentation is built and hosted on [Mintlify](https://mintlify.com), a developer documentation platform.