Recommending READ COMMITTED as Mojaloop's Default MySQL Isolation Level

Recommending READ COMMITTED as Mojaloop's Default MySQL Isolation Level
August 2026 - a recommendation from the Mojaloop core team to Mojaloop adopters and the wider community.
TL;DR
While investigating deadlocks reported against a Mojaloop deployment, we traced the cause to MySQL's default transaction isolation level, REPEATABLE READ, and found that switching to READ COMMITTED fixed two problems in the same deployment:

1. Incoming transfers were being blocked by the close settlement window process.
2. A long-standing deadlock in central-ledger's timeout handler (transfer timeout sweeps colliding with live transfer inserts) disappeared. We've already updated the reference example-mojaloop-backend chart in mojaloop/helm to set this correctly (PR merged post v17.2.0). In addition, one pre-requisite for this to work is the central-settlement version v17.4.0 .

This post explains the root cause, why we think it should be your production default too, and exactly how to make the same change on your own deployment - safely, reversibly, and without waiting for a future Mojaloop release.

The problem
Mojaloop's core write path (central-ledger) and its settlement path both hit the same InnoDB tables - transferStateChange, transferTimeout, and participantPosition - under MySQL's default isolation level, REPEATABLE READ.
Two symptoms showed up together:
● Settlement blocking live traffic. The close settlement window process - which scans and aggregates over ranges of transfers to close out a window - was taking shared locks that live incoming transfers then queued behind. Under load, transfer processing latency would spike every time a settlement window was closed.
● A timeout-handler deadlock. The cron-based timeout sweep runs an INSERT … SELECT to pull expired transfers. Under REPEATABLE READ, that statement takes shared next-key locks on the index range it scans - a normal part of how InnoDB prevents phantom reads at that isolation level. Those locks collided with live, in-flight transfer commits landing in the same index range at the same moment.
Both symptoms have the same root cause: REPEATABLE READ's phantom-read protection is implemented with locking (gap locks / next-key locks on range scans), and Mojaloop's write path is
dense enough, on the same narrow set of hot tables, that those locks become contention rather than protection.

Why REPEATABLE READ is the wrong default here
REPEATABLE READ exists to give a transaction a consistent snapshot across multiple statements - so if you SELECT the same range twice in one transaction, you see the same rows both times, and to guarantee that, InnoDB locks the gaps between index records to stop other transactions inserting into them.
Mojaloop's hot-path transactions don't need that guarantee:
● Transfer processing is short-lived: read position, apply a delta, write, commit. There's no multi-statement read-then-read-again pattern in the transfer path that depends on a frozen snapshot.
● Consistency where it actually matters: participant position balances - is enforced by explicit row locking (SELECT … FOR UPDATE) and unique constraints, not by gap locks preventing phantom rows.
● The thing that was relying on wide range scans (settlement aggregation, timeout sweeps) doesn't need repeatable-read semantics either - it needs to not block the write path while it scans.
READ COMMITTED removes the gap-lock behavior on non-matching rows for these scans (InnoDB still takes locks on rows it actually examines and modifies, just not the gaps around them). That's the entire fix: less locking on ranges, same row-level consistency guarantees where they're enforced explicitly.

Is this safe? What to check before you flip the switch
READ COMMITTED is not a free lunch - it changes semantics, and you should confirm these are true for your deployment before rolling it out:
● No code relies on repeatable snapshots within a transaction. Audit anything that runs more than one SELECT against the same range inside a single DB transaction expecting the second read to match the first (some reporting/reconciliation jobs are the usual suspects - the live transfer/settlement path in central-ledger is not one of them).
● binlog_format is ROW or MIXED. Statement-based replication is unsafe under READ COMMITTED because the same statement can legitimately produce different results depending on timing. ROW/MIXED (MySQL's default since 5.7.7) sidesteps this entirely - if you've explicitly forced STATEMENT anywhere, stop and fix that first.
● Nothing pins the isolation level per-connection. We confirmed database-lib (central-ledger's DB layer) has no afterCreate hook or per-connection isolation override - it inherits whatever the server default is. If your fork or a service you run does pin isolation level in application code, that code wins over the server default and won't be affected by this change either way - check for it so you know what you're actually changing.

Results
After the change: no recurrence of the timeout-handler deadlock, and closing a settlement window no longer visibly stalls incoming transfer throughput. This isn't a partial mitigation - the locking mode that caused both symptoms is gone.

What we've already changed upstream
The example-mojaloop-backend chart in mojaloop/helm already reflects this recommendation on its MySQL primary (which is provided for reference).
This alone doesn't change your production database. example-mojaloop-backend is a demo composition, not a values file any real deployment inherits automatically - if you're running Mojaloop in production, you almost certainly maintain your own values.yaml, and it needs the same change applied deliberately. That's what the rest of this post walks through.

How to change it on your own deployment
Below are the detailed steps for the configuration changes, once you’ve confirmed using central-settlement version v17.4.0 . This change (below) is server-wide, not central-ledger-only.

1. Test it live, no persistence, reversible in one command
   SET GLOBAL transaction_isolation = 'READ-COMMITTED';
   This affects new connections only - sessions already open keep their old isolation level until they reconnect. It does not survive a MySQL restart. This is the right first step: flip it, let connection pools cycle, watch your deadlock/lock-wait metrics, and revert with the same command if anything looks wrong.
   Check current state at any time:
   SELECT @@GLOBAL.transaction_isolation, @@SESSION.transaction_isolation;

2. Make it durable without a restart (MySQL 8.0+)
   SET PERSIST transaction_isolation = 'READ-COMMITTED';
   SET PERSIST writes the value to mysqld-auto.cnf in the data directory and applies it immediately and globally - no pod restart, no downtime. It survives a MySQL process restart too, as long as the data directory itself persists.
   That last clause matters more than it looks. If your MySQL data directory doesn't persist across pod restarts - an ephemeral volume, common in dev/test deployments - mysqld-auto.cnf disappears the moment the pod is recreated, and you silently fall back to REPEATABLE READ. SET PERSIST alone is not enough on ephemeral storage - you need step 3 as well.

3. Bake it into your persistent server configuration (durable across restarts)
   SET PERSIST only helps if the data directory it writes to survives. Either way, the isolation level should end up in whatever configuration your MySQL server loads on startup, so a redeploy, a restore, or a fresh replica doesn't quietly revert to REPEATABLE READ.
   If you're running the Bitnami MySQL Helm chart (used by the Mojaloop reference deployment):
   mysql:
   primary:
   extraFlags: "--transaction-isolation=READ-COMMITTED"
   (An alternative, primary.configuration, fully overrides the default my.cnf - only worth it if you're already overriding the full config for other reasons, since you'd need to carry forward every default setting alongside the isolation line.) Apply with helm upgrade --install -f values.yaml.
   If you're running MySQL another way - a self-managed server, a different Helm chart, or a managed service like RDS/Aurora/Cloud SQL - the mechanism differs but the target is the same: set transaction-isolation=READ-COMMITTED under [mysqld] in my.cnf, or the equivalent parameter in your provider's DB parameter group, and apply it through whatever config-management path you already use for that server.
   Either way, this restarts the MySQL primary. On a single-primary/standalone topology that's a brief write-unavailability window, so schedule it like any other maintenance-window change: apply SET PERSIST first for immediate effect, then land the durable config change, so you get zero-downtime and durability without the two steps needing to be simultaneous.

4. Verify and monitor
   SHOW VARIABLES LIKE 'transaction_isolation';
   SELECT * FROM performance_schema.data_locks; -- confirm gap-lock behavior is gone on hot ranges
   SELECT * FROM information_schema.INNODB_TRX; -- watch for long-running transactions post-change
   Watch deadlock counters (SHOW ENGINE INNODB STATUS, or whatever you already scrape into Prometheus) before and after - you're looking for the timeout-handler-vs-live-insert deadlock signature to stop appearing entirely, not just get rarer.

Rollback
SET GLOBAL / SET PERSIST back to REPEATABLE-READ (or RESET PERSIST transaction_isolation to drop the override and fall back to compiled default), and revert whatever persistent configuration you changed in step 3. Nothing about this change is one-way.

A note to the community
This is a recommendation, not a mandate - you don't need our sign-off to try it, and every deployment is free to make this change on its own timeline. That said, if you're running Mojaloop in production and know of a reason REPEATABLE READ semantics are load-bearing somewhere we haven't audited - a reporting job, a reconciliation flow, a fork with different transaction boundaries - we'd like to hear about it, so we can document the exception for others weighing the same change.