Linux kernel crash report

Crash reported by Anderson Nascimento via LKML: CAPhRvkyZGKHRTBhV3P2PCCRxmRKGEvJQ0W5a9SMW3qwS2hp2Qw@mail.gmail.com

Key elements

Kernel modules

Backtrace

CPU registers

Code bytes: … 0f 85 49 dd 01 00 <0f> 0b … The <0f> 0b sequence is the UD2 instruction — the x86 encoding of BUG(). The preceding 0f 85 (JNZ) is the if (!idr_is_empty(...)) branch that was taken.

Backtrace source code

1. rxrpc_destroy_client_conn_ids (inlined) — crash site (conn_client.c:64)

addr2line resolves rxrpc_purge_client_connections+0x58 to rxrpc_destroy_client_conn_ids inlined into rxrpc_purge_client_connections at conn_client.c:145. The crash fires inside the inlined callee:

net/rxrpc/conn_client.c — kernel-6.18.13-0

 54 static void rxrpc_destroy_client_conn_ids(struct rxrpc_local *local)
 55 {
 56     struct rxrpc_connection *conn;
 57     int id;
 58
 59     if (!idr_is_empty(&local->conn_ids)) {    // ← branch taken: conn_ids is NOT empty
 60         idr_for_each_entry(&local->conn_ids, conn, id) {
 61             pr_err("AF_RXRPC: Leaked client conn %p {%d}\n",
 62                    conn, refcount_read(&conn->ref));
 63         }
 64         BUG();    // ← crash here — refcount={1}, one connection leaked
 65     }
 66
 67     idr_destroy(&local->conn_ids);
 68 }

One connection was logged: 00000000bf02a6a7 {1} — refcount=1 means one reference holder has not released it.

2. rxrpc_destroy_local — call site (local_object.c:451)

net/rxrpc/local_object.c — kernel-6.18.13-0

 420 void rxrpc_destroy_local(struct rxrpc_local *local)
 421 {
      ...
 432     rxrpc_clean_up_local_conns(local);         // ← only cleans idle_client_conns list
 433     rxrpc_service_connection_reaper(&rxnet->service_conn_reaper);
 434     ASSERT(!local->service);
      ...
 451     rxrpc_purge_client_connections(local);     // ← call site → crash
 452     page_frag_cache_drain(&local->tx_alloc);
 453 }

rxrpc_clean_up_local_conns() (called at line 432) only drains the local->idle_client_conns list. A connection that has been allocated and added to conn_ids but has not yet reached the idle state is invisible to it.

3. rxrpc_clean_up_local_conns — cleanup gap (conn_client.c:813)

net/rxrpc/conn_client.c — kernel-6.18.13-0

 813 void rxrpc_clean_up_local_conns(struct rxrpc_local *local)
 814 {
 815     struct rxrpc_connection *conn;
 816
 817     local->kill_all_client_conns = true;
 818     timer_delete_sync(&local->client_conn_reap_timer);
 819
 820     while ((conn = list_first_entry_or_null(&local->idle_client_conns,  // ← only idle!
 821                                             struct rxrpc_connection, cache_link))) {
 822         list_del_init(&conn->cache_link);
 823         atomic_dec(&conn->active);
 824         trace_rxrpc_client(conn, -1, rxrpc_client_discard);
 825         rxrpc_unbundle_conn(conn);
 826         rxrpc_put_connection(conn, rxrpc_conn_put_local_dead);
 827     }
 828 }

Connections move to idle_client_conns only in rxrpc_disconnect_client_call(), after all channels on the connection go idle. An in-flight connection never reaches this list before cleanup runs.

What-how-where analysis

What

The BUG() at net/rxrpc/conn_client.c:64 fires because local->conn_ids IDR is non-empty when the local rxrpc endpoint is destroyed. The kernel logged one leaked connection (00000000bf02a6a7, refcount=1), meaning exactly one reference holder has not released the connection before destruction.

The IDR conn_ids is the definitive registry of all live client connections on a local endpoint. Its non-emptiness at destruction time means a connection was allocated (idr_alloc_cyclic) but never freed (idr_remove via rxrpc_put_client_connection_id). That release only happens inside rxrpc_unbundle_conn(), which is only reachable via the idle-connection path.

How

The race has three actors: the application thread, the rxrpc I/O thread (krxrpcio/7001), and the socket-close thread (spawned by pthread_create in the reproducer’s server).

1. Application thread calls sendmsg() with RXRPC_CHARGE_ACCEPT, queuing a call onto local->new_client_calls.

2. I/O thread runs rxrpc_connect_client_calls(): it allocates a rxrpc_bundle, moves the call to bundle->waiting_calls (under client_call_lock), and calls rxrpc_activate_channels() → rxrpc_alloc_client_connection() → idr_alloc_cyclic() adds the new connection to conn_ids.

3. Close thread calls close(sk) concurrently. This wakes the I/O thread’s exit path: rxrpc_io_thread() returns → rxrpc_destroy_local():

   • rxrpc_clean_up_local_conns() drains idle_client_conns — the just-allocated connection is not there yet (it is on bundle->conns[], not idle).
   • rxrpc_purge_client_connections() → rxrpc_destroy_client_conn_ids() → conn_ids is non-empty → BUG().

The missing lock identified by commit b1fdb0bb3b65 widens this window: without client_call_lock protecting list_del_init(&call->wait_link) in the conn == NULL abort path of rxrpc_disconnect_client_call(), a concurrent abort can corrupt the new_client_calls list. This leaves a stale call visible to the I/O thread, which then allocates a connection for it — a connection that will never be disconnected or freed before the local is destroyed.

Root cause: The connection lifecycle has a gap: connections that are allocated (in conn_ids) but have not yet been moved to idle_client_conns are invisible to rxrpc_clean_up_local_conns(). The missing client_call_lock in the abort-before-connect path is the proximate trigger that makes this race reproducible.

Where

Immediate fix: Commit b1fdb0bb3b65 (upstream: fc9de52de38f) closes the missing-lock race that makes the bug reliably triggerable. It is present in fedora/master but not in 6.18.13-200.fc43.x86_64.

Deeper fix: rxrpc_clean_up_local_conns() should also sweep connections that are still bundled (active state, in bundle->conns[]) and not yet idle. The safest approach would be to walk conn_ids directly during cleanup — the same IDR that rxrpc_destroy_client_conn_ids() iterates — and force-unbundle any remaining connections before calling rxrpc_purge_client_connections(). This would close the window regardless of lock correctness.

Bug introduction

Commit 9d35d880e0e4 — “rxrpc: Move client call connection to the I/O thread” (David Howells, Oct 2022) — restructured connection setup so that the actual idr_alloc_cyclic into conn_ids happens on the I/O thread rather than the application thread. This made the connection-allocation window overlap with the socket-close path, introducing the race.

The commit’s own message notes: “This also completes the fix for a race that exists between call connection and call disconnection” — acknowledging that races in this area were a known concern. The missing client_call_lock in the abort path (b1fdb0bb3b65, which itself carries Fixes: 9d35d880e0e4) is a direct consequence of the same restructuring.

Analysis, conclusions and recommendations

Summary: A client rxrpc connection is allocated on the I/O thread and registered in conn_ids, but the socket is closed concurrently before the connection can be moved to the idle list. rxrpc_clean_up_local_conns() only drains the idle list, so the active connection survives into rxrpc_destroy_client_conn_ids() which hits BUG().

Recommendations:

1. Apply b1fdb0bb3b65 to 6.18.13-200.fc43. This is a stable-tagged fix already present in fedora/master; it should land in the next Fedora 43 kernel update.

2. Upstream investigation: The cleanup gap in rxrpc_clean_up_local_conns() (only idle connections) should be reported to David Howells. A robust fix would have the cleanup path iterate conn_ids directly and force-disconnect any non-idle connections, removing the dependency on the connection having reached the idle state before destruction.

3. Reproducer confirmed: Anderson Nascimento’s server+client reproducer reliably triggers this. It should be forwarded to the rxrpc maintainer alongside the above analysis.