SIGABRT: owned tokio Runtime dropped from its own worker thread during chain sync teardown

[jvsena42]

Summary

On Android production (arm64-v8a, ldk-node-android 0.7.0-rc.52) we see a recurring SIGABRT caused by a tokio panic:

Cannot drop a runtime in a context where blocking is not allowed.
This happens when a runtime is dropped from within an asynchronous context.

Symbolication shows the owned tokio Runtime is dropped by an ldk-node chain-sync background task while that task is running on a worker thread of the very runtime being dropped. Release builds are panic = abort, so this terminates the app process with no catchable error.

Tracked downstream as synonymdev/bitkit-android#982 (top production crash cluster, mainnet 2.2.0 → 2.4.0).

Symbolicated stack

Symbolicated with the unstripped arm64-v8a/libldk_node.so from the native-debug-symbols artifact of com.synonym:ldk-node-android:0.7.0-rc.52 (.text vaddr 0x551980, size 0x7d16ec, matching the packaged library byte-for-byte).

#35 std::thread lifecycle → tokio-1.52.3 runtime/blocking/pool.rs:161      (tokio worker thread)
#28 tokio runtime/scheduler/multi_thread/worker.rs:503
#27 tokio runtime/scheduler/multi_thread/worker.rs:537  run
#23 tokio runtime/task/raw.rs:267  poll
#19 ldk_node::runtime::Runtime::spawn_background_task
      <Node::spawn_chain_sync_task_with_receiver>                          src/runtime.rs:73
#18 async block                                                            src/lib.rs
#17 async fn                                                               src/chain/mod.rs:766
#16 async fn                                                               src/chain/mod.rs:913
#15 async fn                                                               src/chain/electrum.rs:307 (from :121)
#14 drop_slow<ldk_node::chain::electrum::ElectrumRuntimeClient>            ← last Arc released here
#13 drop_in_place<ldk_node::runtime::Runtime / ldk_node::runtime::RuntimeMode>
#12 drop_in_place<tokio::runtime::Runtime → tokio::runtime::blocking::pool::BlockingPool>
#11 tokio runtime/blocking/shutdown.rs:51  wait → panic
#10 core::panicking::panic_fmt
#04 __rust_start_panic  (panic_abort)
#03 __rust_abort → #02 std::process::abort → #01 abort_internal → SIGABRT

Line numbers are from the rc.52 build; the structure is unchanged on main.

Analysis

ldk_node::runtime::Runtime can own the tokio runtime outright (src/runtime.rs):

let mode = match tokio::runtime::Handle::try_current() {
    Ok(handle) => RuntimeMode::Handle(handle),
    Err(_) => {
        let rt = tokio::runtime::Builder::new_multi_thread().enable_all().build()?;
        RuntimeMode::Owned(rt)          // <-- owned runtime
    },
};

On Android the Node is built from a plain JNI/UniFFI thread with no ambient tokio context, so RuntimeMode::Owned is always the branch taken.

ElectrumRuntimeClient holds a strong reference to it (src/chain/electrum.rs:601):

struct ElectrumRuntimeClient {
    electrum_client: Arc<ElectrumClient>,
    bdk_electrum_client: Arc<BdkElectrumClient<Arc<ElectrumClient>>>,
    tx_sync: Arc<ElectrumSyncClient<Arc<Logger>>>,
    runtime: Arc<Runtime>,              // <-- keeps the owned runtime alive
    config: Arc<Config>,
    logger: Arc<Logger>,
}

and sync paths take their own clone out of the status before awaiting (src/chain/electrum.rs:575):

fn client(&self) -> Option<Arc<ElectrumRuntimeClient>> {
    match self {
        Self::Started(client) => Some(Arc::clone(&client)),
        Self::Stopped { .. } => None,
    }
}

while teardown simply releases the status's own reference (src/chain/electrum.rs:571):

pub(super) fn stop(&mut self) {
    *self = Self::new()
}

That produces the race:

1. The chain-sync background task, spawned via Runtime::spawn_background_task, is executing on a worker thread of the owned runtime and holds an Arc<ElectrumRuntimeClient> clone.
2. Teardown (ElectrumRuntimeStatus::stop, or any other path releasing the status) drops the status's Arc, so the running task now holds the last reference.
3. The task finishes; its clone drops → ElectrumRuntimeClient drops → its Arc<Runtime> is the last one → RuntimeMode::Owned(tokio::runtime::Runtime) drops.
4. tokio::runtime::Runtime::drop → BlockingPool::shutdown needs to block, detects it is inside a runtime context, and panics.
5. panic = abort → SIGABRT, whole process gone.

Nothing on the consumer side can prevent or catch this: the drop happens entirely inside ldk-node's own task on ldk-node's own runtime.

Impact

• Top production crash cluster for Bitkit Android mainnet across 2.2.0 → 2.4.0 (SIGABRT, arm64-v8a, foreground, Android 13–16, multiple OEMs).
• No Java exception and no recovery path — the process dies outright.
• Almost certainly the same mechanism behind two sibling downstream reports: a free_node TimeoutException on the FinalizerDaemon during node teardown ([Bug]: $Proxy4.uniffi_ldk_node_fn_free_node — TimeoutException on Node destroy (Play 2.2.0) bitkit-android#984), and process death during a stop→rebuild cycle triggered by a stale network-graph reset ([Bug]: Mainnet wallet restore crashes during LDK network-graph reset restart bitkit-android#1096).

Suggested directions

1. Never let an owned tokio Runtime be dropped by a task running on it. E.g. have RuntimeMode::Owned drop go through Runtime::shutdown_background() / shutdown_timeout(), or hand the runtime off to a dedicated non-runtime thread for teardown.
2. Keep runtime ownership out of ElectrumRuntimeClient — hold a tokio::runtime::Handle (or a Weak<Runtime>) there instead of Arc<Runtime>, so a sync task can never own the last reference.
3. Make teardown deterministic: have stop() await outstanding chain-sync tasks (background_tasks / cancellable_background_tasks are already tracked in JoinSets) before releasing the client, so the final drop happens on the caller's thread rather than a worker.

Environment

• ldk-node-android 0.7.0-rc.52 (also present on main / v0.7.0 by inspection), tokio 1.52.3
• Android arm64-v8a, Android 13–16, release profile (panic = abort)
• Consumer: Bitkit Android 2.4.0 (mainnet, to.bitkit)

Note on symbolication

Play cannot symbolicate any of this itself: the published libldk_node.so carries no GNU build-id, which is the key Play uses to pair an uploaded native debug symbol file with the crashing library. That is why the stack above had to be resolved by hand. Tracked separately in #95.