Debugging SQLite-backed Durable Objects: Resolving Concurrency & Alarm Desync in Cloudflare workerd

Distributed state is notoriously hard to get right. When you combine an in-memory actor runtime, transactional SQLite persistence, and an asynchronous event loop managing timers, edge cases don't just lurk—they compound.

Recently, I dove into Cloudflare's open-source runtime, workerd (the engine powering Cloudflare Workers and Durable Objects), to track down and eliminate a critical concurrency desynchronization bug in SQLite-backed Durable Objects.

This is the technical breakdown of Issue #7330 and the fix landed in Pull Request #7464.

The Architecture: Durable Objects, SQLite, and Alarms

Cloudflare Durable Objects (DOs) provide strongly consistent, distributed in-memory state with coordinated persistence. In the modern workerd architecture, Durable Object storage can be backed by embedded SQLite.

Beyond standard key-value and tabular storage, Durable Objects support Alarms—guaranteed, scheduled wake-up timers that trigger an alarm() handler on the actor instance at a designated timestamp in the future:

// Typical Durable Object alarm invocation
await ctx.storage.setAlarm(Date.now() + 1000 * 60 * 5); // 5 minutes

Under the hood in C++, workerd coordinates alarms through two distinct layers:

1. The SQLite Metadata Table (ActorSqlite): The ground truth on disk. The alarm time is transactionally committed alongside application data.
2. The Host AlarmManager (alarmScheduledNoLaterThan): The runtime scheduler responsible for waking the isolate when the timer fires.

To minimize redundant cross-boundary IPC calls to the host scheduler, ActorSqlite caches the earliest scheduled time in alarmScheduledNoLaterThan. When an actor moves an alarm to a later time, workerd attempts to coalesce these "move-later" requests so it doesn't repeatedly reschedule the underlying timer until commits finish.

The Anomaly: Issue #7330

Developers began noticing intermittent anomalies in production:

"Our Durable Object scheduled an alarm for 10 seconds in the future, but it didn't trigger until 45 minutes later—or failed to trigger at all until the next cold start."

When inspecting the SQLite database directly, the table recorded the correct, earlier alarm timestamp. Yet the runtime timer failed to fire. The AlarmManager had become desynchronized from the database.

The Root Cause: Anatomy of an Asynchronous Interleaving

Let's trace how asynchronous execution in C++ KJ event loops created the stranded state.

1. In-Flight Move-Later Requests

When a transaction requests moving an alarm to a later time while another move-later request is already in-flight, workerd parked the target time into a scratch variable: pendingLaterAlarmTime.

2. The Overlapping Earlier Commit

Before that in-flight request returned, a concurrent operation executed a commit that set the alarm to an earlier time.

Critically:

// Previous behavior in startPrecommitAlarmScheduling
// If the new time was not strictly earlier than alarmScheduledNoLaterThan,
// pendingLaterAlarmTime was NOT invalidated!

3. The Window of Inconsistency

While this subsequent transaction was awaiting its database commitCallback:

1. The first in-flight request resolved.
2. The completion handler ran and found the parked pendingLaterAlarmTime.
3. It dutifully applied the parked timestamp, overwriting alarmScheduledNoLaterThan with the stale later time.

4. The Skipped Resync

When the database commit for the earlier alarm finally completed, it executed:

if (willFireEarlier(alarmScheduledNoLaterThan, alarmStateForCommit)) {
    // skipped! alarmScheduledNoLaterThan was already later!
}

Because alarmScheduledNoLaterThan had been overwritten with the stale future timestamp, the logic concluded no synchronization was needed.

The result?

• SQLite DB state: Wake up at T = 10s.
• AlarmManager state: Wake up at T = 3600s.
• The alarm was stranded.

Time ──►
[Tx 1] Request move-later (T=3600s) ──► in-flight...
[Tx 2] Commit earlier alarm (T=10s) ──────────────► writes to SQLite
[Tx 1 Completion Handler]           ──► Applies parked T=3600s to AlarmManager
[Tx 2 Post-Commit Check]            ──► Skips sync (thinks AlarmManager is fine)
                                        💥 DEADLOCK / DESYNC

Engineering the Solution

To make the alarm scheduler bulletproof against arbitrary re-entrancy and commit ordering, we implemented a multi-layered guard in workerd/src/workerd/api/actor-sqlite.c++:

1. Clearing Parked Timers on Earlier Commits

In startPrecommitAlarmScheduling, whenever an alarm is being committed that fires earlier than the parked pendingLaterAlarmTime, we actively purge the parked state:

if (pendingLaterAlarmTime.isSet() && willFireEarlier(alarmTime, *pendingLaterAlarmTime)) {
  pendingLaterAlarmTime = nullptr;
}

2. Validating Against DB Truth in Completion Handlers

In the scheduleLaterAlarm completion handler, before blindly applying the drained timestamp, we verify whether metadata.getAlarm() has since committed an earlier time:

if (metadata.getAlarm().isSet() && willFireEarlier(*metadata.getAlarm(), nextTime)) {
  // Database has committed an earlier alarm; do not apply the stale later time.
  return;
}

3. Post-Commit Resynchronization

In commitImpl, we added an explicit resync branch:

if (willFireEarlier(alarmStateForCommit, alarmScheduledNoLaterThan)) {
  // A concurrent operation pushed alarmScheduledNoLaterThan later
  // while our commit was persisting. Resynchronize immediately.
  scheduleEarlierAlarm(alarmStateForCommit);
}

Verifying with Deterministic Concurrency Testing

Testing asynchronous race conditions requires determinism. In workerd/src/workerd/api/actor-sqlite-test.c++, we constructed a targeted unit test:

"parked move-later alarm is not applied when subsequent commit moved alarm earlier"

The test explicitly interleaves:

1. Triggering an asynchronous setAlarm() into the distant future.
2. Committing a near-term alarm prior to the promise resolution of the first.
3. Yielding the event loop to force completion handler drainage.
4. Asserting that the host AlarmManager receives the exact near-term timestamp registered in SQLite.

With all test suites passing under TSAN (ThreadSanitizer) and ASAN (AddressSanitizer), the PR was approved and merged into cloudflare/workerd.

Takeaways for Runtime & Systems Engineers

1. State caches must be treated as optimistic leases: Whenever an in-memory value (alarmScheduledNoLaterThan) mirrors durable storage (SQLite), any asynchronous boundary can invalidate the cache.
2. Coalescing optimizations require explicit cancellation: Parked/coalesced operations (pendingLaterAlarmTime) must be cleared the instant a conflicting invariant is established.
3. Open-source runtimes empower end-to-end reliability: Being able to inspect the exact C++ implementation running your serverless code is why open infrastructure wins.

Special thanks to the Cloudflare Workers runtime engineering team for the swift reviews and collaborative feedback on PR #7464.