# Linux kernel crash report Report generated from syzbot email [69eb21b8.a00a0220.17a17.0056.GAE@google.com](https://lore.kernel.org/r/69eb21b8.a00a0220.17a17.0056.GAE@google.com) — `[syzbot] [btrfs?] WARNING in __btrfs_update_delayed_inode (4)` ## Key elements | Field | Value | Implication | | ----- | ----- | ----------- | | UNAME | `syzkaller #0 PREEMPT(full)` | syzbot upstream build; no distro | | PROCESS | `kworker/u4:5` | Kernel worker thread | | PID | 70 | | | CPU | 0 | | | HARDWARE | QEMU Standard PC (Q35 + ICH9, 2009) | QEMU VM; syzbot fuzzer environment | | BIOS | 1.16.3-debian-1.16.3-2 04/01/2014 | | | HEAD_COMMIT | [`bea8d77e45a8`](https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=bea8d77e45a8) | Merge tag 'staging-7.1-rc1'; linux-7.1-rc1 merge window | | TAINT | (none) | Not tainted | | DISTRO | syzbot/upstream | | | VMLINUX | `oops-workdir/syzbot/vmlinux-bea8d77e` | Downloaded from syzbot assets | | SOURCEDIR | `oops-workdir/linux` @ [`bea8d77e45a8`](https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=bea8d77e45a8) | | | MSGID | `<69eb21b8.a00a0220.17a17.0056.GAE@google.com>` | | | MSGID_URL | [69eb21b8.a00a0220.17a17.0056.GAE@google.com](https://lore.kernel.org/r/69eb21b8.a00a0220.17a17.0056.GAE@google.com) | | | CRASH_TYPE | WARNING | | | WARNING_TEXT | `BTRFS: Transaction aborted (error -28)` | -28 = -ENOSPC | | WARNING_SOURCE | `fs/btrfs/delayed-inode.c:1027` | Inside `__btrfs_update_delayed_inode` | | ERROR_CODE | -28 = `-ENOSPC` | No space left on device (metadata space exhausted) | | WORKQUEUE | `events_unbound btrfs_async_reclaim_data_space` | Async data space reclaim path | | CONFIG_REQUIRED | `CONFIG_BUG` | Always enabled in production builds; governs `WARN()` macro | ## Kernel modules | Module | Flags | Backtrace | Location | Flag Implication | | ------ | ----- | --------- | -------- | ---------------- | | *(no modules — fully built-in syzbot kernel)* | | | | | ## Backtrace | Address | Function | Offset | Size | Context | Module | Source location | | ------- | -------- | ------ | ---- | ------- | ------ | --------------- | | `0xffffffff84082230 (0xffffffff840813f0 + 0xe40)` | `__btrfs_update_delayed_inode` | `0xe40` | `0x1070` | Task | — | [fs/btrfs/delayed-inode.c:1027](#0-__btrfs_update_delayed_inode--crash-site-fs-btrfs-delayed-inodec1027) | | | `btrfs_update_delayed_inode` (inlined) | | | Task | — | [fs/btrfs/delayed-inode.c:1103](#1-btrfs_update_delayed_inode-inlined--fsbtrfsdelayed-inodec1103) | | `0xffffffff84080fca (0xffffffff8407f1d0 + 0x1dfa)` | `__btrfs_commit_inode_delayed_items` | `0x1dfa` | `0x1f50` | Task | — | [fs/btrfs/delayed-inode.c:1127](#2-__btrfs_commit_inode_delayed_items--fsbtrfsdelayed-inodec1127) | | `0xffffffff8407e7a6 (0xffffffff8407e5b0 + 0x1f6)` | `__btrfs_run_delayed_items` | `0x1f6` | `0x510` | Task | — | [fs/btrfs/delayed-inode.c:1158](#3-__btrfs_run_delayed_items--fsbtrfsdelayed-inodec1158) | | `0xffffffff83ee8d56 (0xffffffff83ee8520 + 0x836)` | `btrfs_commit_transaction` | `0x836` | `0x30e0` | Task | — | `fs/btrfs/transaction.c:2376` | | `0xffffffff84153a23 (0xffffffff84153780 + 0x2a3)` | `flush_space` | `0x2a3` | `0xe20` | Task | — | `fs/btrfs/space-info.c` | | `0xffffffff841552fa (0xffffffff84155060 + 0x29a)` | `do_async_reclaim_data_space` | `0x29a` | `0x520` | Task | — | `fs/btrfs/space-info.c:1464` | | `0xffffffff8414e611 (0xffffffff8414e5d0 + 0x41)` | `btrfs_async_reclaim_data_space` | `0x41` | `0x90` | Task | — | `fs/btrfs/space-info.c:1512` | | | `process_one_work` (inlined) | | | Task | — | `kernel/workqueue.c:3302` | | `0xffffffff818eb2ed (0xffffffff818ea790 + 0xb5d)` | `process_scheduled_works` | `0xb5d` | `0x1860` | Task | — | `kernel/workqueue.c:3385` | | `0xffffffff818f3353 (0xffffffff818f2900 + 0xa53)` | `worker_thread` | `0xa53` | `0xfc0` | Task | — | `kernel/workqueue.c:3466` | | `0xffffffff8190ae58 (0xffffffff8190aad0 + 0x388)` | `kthread` | `0x388` | `0x470` | Task | — | `kernel/kthread.c:436` | | `0xffffffff816bfae4 (0xffffffff816bf5d0 + 0x514)` | `ret_from_fork` | `0x514` | `0xb70` | Task | — | `arch/x86/kernel/process.c:158` | | `0xffffffff813370aa (0xffffffff81337090 + 0x1a)` | `ret_from_fork_asm` | `0x1a` | `0x30` | Task | — | `arch/x86/entry/entry_64.S:245` | ## Backtrace source code ### 0. `__btrfs_update_delayed_inode` — crash site (`fs/btrfs/delayed-inode.c:1027`) [https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/fs/btrfs/delayed-inode.c?id=bea8d77e45a8#n996](https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/fs/btrfs/delayed-inode.c?id=bea8d77e45a8#n996) ```c 996 static int __btrfs_update_delayed_inode(struct btrfs_trans_handle *trans, 997 struct btrfs_root *root, 998 struct btrfs_path *path, 999 struct btrfs_delayed_node *node) 1000 { 1001 struct btrfs_fs_info *fs_info = root->fs_info; 1002 struct btrfs_key key; 1003 struct btrfs_inode_item *inode_item; 1004 struct extent_buffer *leaf; 1005 int mod; 1006 int ret; ... 1012 if (test_bit(BTRFS_DELAYED_NODE_DEL_IREF, &node->flags)) 1013 mod = -1; 1014 else 1015 mod = 1; 1016 1017 ret = btrfs_lookup_inode(trans, root, path, &key, mod); // RBX/RSI/R13 = 0xffffffe4 = -28 (-ENOSPC) 1018 if (ret > 0) 1019 ret = -ENOENT; 1020 if (ret < 0) { 1021 /* 1022 * If we fail to update the delayed inode we need to abort the 1023 * transaction, because we could leave the inode with the 1024 * improper counts behind. 1025 */ 1026 if (unlikely(ret != -ENOENT)) 1027 btrfs_abort_transaction(trans, ret); // <- crash here (-ENOSPC triggers WARN) 1028 goto out; 1029 } ... 1088 } ``` The `btrfs_abort_transaction` macro (in `fs/btrfs/transaction.h:249`) expands to call `WARN(btrfs_abort_should_print_stack(error), "BTRFS: Transaction aborted (error %d)\n", error)`. `btrfs_abort_should_print_stack(-28)` returns `true` because `-ENOSPC` is not in its exception list (`-EIO`, `-EROFS`, `-ENOMEM`), so `WARN()` fires and prints the stack trace. **Disasm window (from vmlinux):** ```asm ffffffff84082200: test %ebx,%ebx ffffffff84082202: jne ffffffff8408220e <__btrfs_update_delayed_inode+0xe1e> ... ffffffff84082226: lea 0xc34e863(%rip),%rdi # bug_table entry (WARN string) ffffffff8408222d: mov %r13d,%esi # error code = -28 (ENOSPC) → R13 = 0xffffffe4 ffffffff84082230: call ffffffff8bbc4e98 <__SCT__WARN_trap> <<< crash ``` ### 1. `btrfs_update_delayed_inode` (inlined) — `fs/btrfs/delayed-inode.c:1103` [https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/fs/btrfs/delayed-inode.c?id=bea8d77e45a8#n1090](https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/fs/btrfs/delayed-inode.c?id=bea8d77e45a8#n1090) ```c 1090 static inline int btrfs_update_delayed_inode(struct btrfs_trans_handle *trans, 1091 struct btrfs_root *root, 1092 struct btrfs_path *path, 1093 struct btrfs_delayed_node *node) 1094 { 1095 int ret; 1096 1097 mutex_lock(&node->mutex); 1098 if (!test_bit(BTRFS_DELAYED_NODE_INODE_DIRTY, &node->flags)) { 1099 mutex_unlock(&node->mutex); 1100 return 0; 1101 } 1102 1103 ret = __btrfs_update_delayed_inode(trans, root, path, node); // <- call here 1104 mutex_unlock(&node->mutex); 1105 return ret; 1106 } ``` ### 2. `__btrfs_commit_inode_delayed_items` — `fs/btrfs/delayed-inode.c:1127` [https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/fs/btrfs/delayed-inode.c?id=bea8d77e45a8#n1108](https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/fs/btrfs/delayed-inode.c?id=bea8d77e45a8#n1108) ```c 1108 static inline int 1109 __btrfs_commit_inode_delayed_items(struct btrfs_trans_handle *trans, 1110 struct btrfs_path *path, 1111 struct btrfs_delayed_node *node) 1112 { 1113 int ret; 1114 1115 ret = btrfs_insert_delayed_items(trans, path, node->root, node); 1116 if (ret) 1117 return ret; 1118 1119 ret = btrfs_delete_delayed_items(trans, path, node->root, node); 1120 if (ret) 1121 return ret; 1122 1123 ret = btrfs_record_root_in_trans(trans, node->root); 1124 if (ret) 1125 return ret; 1126 1127 return btrfs_update_delayed_inode(trans, node->root, path, node); // <- call here 1128 } ``` ### 3. `__btrfs_run_delayed_items` — `fs/btrfs/delayed-inode.c:1158` [https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/fs/btrfs/delayed-inode.c?id=bea8d77e45a8#n1136](https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/tree/fs/btrfs/delayed-inode.c?id=bea8d77e45a8#n1136) ```c 1136 static int __btrfs_run_delayed_items(struct btrfs_trans_handle *trans, int nr) 1137 { 1138 struct btrfs_fs_info *fs_info = trans->fs_info; ... 1141 struct btrfs_path *path; 1142 struct btrfs_block_rsv *block_rsv; ... 1149 path = btrfs_alloc_path(); ... 1153 block_rsv = trans->block_rsv; 1154 trans->block_rsv = &fs_info->delayed_block_rsv; // ← RSV switched to delayed_block_rsv 1155 1156 curr_node = btrfs_first_delayed_node(fs_info, &curr_delayed_node_tracker); 1157 while (curr_node && (!count || nr--)) { 1158 ret = __btrfs_commit_inode_delayed_items(trans, path, // <- call here 1159 curr_node); 1160 if (unlikely(ret)) { 1161 btrfs_abort_transaction(trans, ret); 1162 break; 1163 } ... ``` ## What-how-where analysis ### What The `WARN()` macro fires inside the `btrfs_abort_transaction` macro expansion at `fs/btrfs/delayed-inode.c:1027` inside `__btrfs_update_delayed_inode`. `btrfs_lookup_inode()` (line 1017) returned `-ENOSPC` (-28). Because this error is not `-ENOENT`, the guard at line 1026 (`if (unlikely(ret != -ENOENT))`) passes and `btrfs_abort_transaction(trans, -ENOSPC)` is called. Within the macro, `btrfs_abort_should_print_stack(-28)` returns `true` because `-ENOSPC` is not in its short exception list (`-EIO`, `-EROFS`, `-ENOMEM`). This causes `WARN("BTRFS: Transaction aborted (error %d)\n", -28)` to fire, printing the stack trace that syzbot captured. The register dump confirms: `RBX = RSI = R13 = 0x00000000ffffffe4` = -28 = `-ENOSPC`. These are the error value as it is passed through the macro. The `btrfs_abort_transaction` macro is a **developer-inserted diagnostic assertion** that fires only on the first transaction abort since mount (guarded by `test_and_set_bit(BTRFS_FS_STATE_TRANS_ABORTED, ...)`). Its purpose is to capture the exact call stack at the moment of the first abort. ### How The call chain leading to the WARN is: 1. **`btrfs_async_reclaim_data_space`** (work item on `events_unbound`) — an asynchronous worker attempting to reclaim data space on a nearly-full filesystem. 2. **`do_async_reclaim_data_space`** → **`flush_space(..., COMMIT_TRANS, ...)`** — after exhausting cheaper flush states, the reclaim loop reaches `COMMIT_TRANS` and calls `btrfs_commit_current_transaction()`. 3. **`btrfs_commit_transaction`** → **`__btrfs_run_delayed_items`** — during transaction commit, all pending delayed-inode updates are flushed. At line 1154, the active block reserve is replaced with `delayed_block_rsv`: ```c trans->block_rsv = &fs_info->delayed_block_rsv; ``` 4. **`__btrfs_update_delayed_inode`** → **`btrfs_lookup_inode`** → **`btrfs_search_slot(trans, root, &key, path, ins_len=-1, cow=1)`** — With `BTRFS_DELAYED_NODE_DEL_IREF` set, `mod = -1`, so `btrfs_lookup_inode` requests a slot with COW enabled. `btrfs_search_slot` must perform copy-on-write on internal B-tree nodes. Even a deletion path requires allocating new tree blocks for the COW copies. 5. **`btrfs_search_slot` returns `-ENOSPC`** — `delayed_block_rsv` does not have enough metadata space for the required B-tree node COW operations. Btrfs pre-reserves metadata space in `delayed_block_rsv` for delayed items, but this reservation is insufficient when the filesystem is under extreme metadata pressure (nearly full, or the reservation did not account for enough tree depth / COW overhead). 6. The `-ENOSPC` propagates back to `__btrfs_update_delayed_inode`. Since it is not `-ENOENT`, `btrfs_abort_transaction` is called. Because `-ENOSPC` is not in `btrfs_abort_should_print_stack()`'s exception list, `WARN()` fires. **Root cause:** This is a circular ENOSPC problem: the async data-space reclaim path commits a transaction to free data space, but committing that transaction requires metadata space for B-tree COW operations (via `delayed_block_rsv`). When both data and metadata space are critically low, the transaction commit itself fails with `-ENOSPC`, triggering a transaction abort and an unexpected `WARN()`. The WARN is arguably a **false positive** for this scenario. An ENOSPC-triggered transaction abort during space reclaim is a known btrfs edge case, not a programming error — `-ENOSPC` should arguably be added to the `btrfs_abort_should_print_stack()` exception list alongside `-EIO`, `-EROFS`, and `-ENOMEM`. ### Where Two levels of fix are possible: **Level 1 — Suppress the spurious WARN (targeted fix):** Add `-ENOSPC` to the exception list in `btrfs_abort_should_print_stack()` at `fs/btrfs/transaction.h:234`: ```diff --- a/fs/btrfs/transaction.h +++ b/fs/btrfs/transaction.h @@ -236,6 +236,7 @@ static inline bool btrfs_abort_should_print_stack(int error) switch (error) { case -EIO: case -EROFS: + case -ENOSPC: case -ENOMEM: return false; } return true; ``` This suppresses the `WARN()` for ENOSPC-triggered aborts, which is a known-valid condition when the filesystem is nearly full. The transaction is still aborted correctly via `__btrfs_abort_transaction()`; only the spurious stack trace is removed. **Level 2 — Prevent the abort entirely (proper fix):** The underlying issue is that `delayed_block_rsv` under-reserves metadata when the filesystem is in extreme space pressure. A more robust fix would ensure the delayed block reservation accounts for the worst-case COW depth, or gracefully degrade when ENOSPC is encountered in the deletion path (since a deletion that fails with ENOSPC in COW may be retried after the current flush cycle frees some blocks). This is a harder problem that touches the btrfs space-reservation accounting. The Level 1 fix is appropriate as an immediate mitigation. ## Bug introduction The `btrfs_abort_should_print_stack()` function (with its current exception list of `-EIO/-EROFS/-ENOMEM`) was moved from `fs/btrfs/ctree.c` to its current location in `fs/btrfs/transaction.h` by: - [`b815a78e17b9`](https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=b815a78e17b9dd90398561ec7d91891d95f25301) — `btrfs: move abort_should_print_stack() to transaction.h` (Filipe Manana, Dec 2024 — content unchanged, only a code reorganisation) The omission of `-ENOSPC` from the exception list is a pre-existing condition that predates this move; it is not attributable to a single "introducing" commit. The issue manifests only in the specific scenario of async data-space reclaim + transaction commit + metadata pressure (all simultaneously), which explains its infrequent appearance (syzbot reports it as occurrence "(4)"). Git blame for the code at lines 1017–1027 of `fs/btrfs/delayed-inode.c` traces to foundational commits: - [`0e8c36a9fd81`](https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=0e8c36a9fd8169a8b96c2ddc8446894bcd07b6b1) — `Btrfs: fix lots of orphan inodes when the space is not enough` (Miao Xie, Dec 2012) — introduced `__btrfs_update_delayed_inode` - [`16cdcec736cd`](https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=16cdcec736cd214350cdb591bf1091f8beedefa0) — `btrfs: implement delayed inode items operation` — foundational Bug introduction commit: not identified within search budget — this is a long-standing omission in the exception list rather than a regression introduced by a specific recent commit. ## Analysis, conclusions and recommendations **Summary:** A `WARN()` fires in `__btrfs_update_delayed_inode` via the `btrfs_abort_transaction` macro when `btrfs_lookup_inode()` returns `-ENOSPC` (-28) under severe metadata space pressure. This occurs during the async data-space reclaim path, which commits a transaction to free data space — but the transaction commit itself needs metadata space for B-tree COW operations, creating a circular failure. **Classification:** The WARNING is a **developer diagnostic false positive**. The transaction abort is legitimate (the inode update genuinely failed), but the `WARN()` stack trace was not intended for this common filesystem-full scenario. The syzbot title "(4)" indicates this reproduces reliably under stress conditions. **Recommended fix:** 1. **Immediate:** Add `-ENOSPC` to `btrfs_abort_should_print_stack()` in `fs/btrfs/transaction.h` so that ENOSPC-triggered transaction aborts do not fire `WARN()`. This matches the intent of the exception list (which already excludes other "expected-but-unrecoverable" errors like `-EIO` and `-EROFS`) and eliminates the syzbot report. 2. **Follow-up:** Investigate whether `delayed_block_rsv` can be improved to avoid running out of metadata during transaction commits triggered by space reclaim. This is the deeper root cause that causes the abort in the first place. **Confidence:** High — exact source match, register values confirmed, call chain fully traced. **Patch recipients:** `linux-btrfs@vger.kernel.org`, `dsterba@suse.com`, `clm@fb.com`, `linux-kernel@vger.kernel.org`