Mojaloop Community Central

Recommending READ COMMITTED as Mojaloop's Default MySQL Isolation Level

Megan Cannon — Thu, 03 Sep 2026 19:04:07 +0000

Recommending READ COMMITTED as Mojaloop's Default MySQL Isolation Level
August 2026 - a recommendation from the Mojaloop core team to Mojaloop adopters and the wider community.
TL;DR
While investigating deadlocks reported against a Mojaloop deployment, we traced the cause to MySQL's default transaction isolation level, REPEATABLE READ, and found that switching to READ COMMITTED fixed two problems in the same deployment:

Incoming transfers were being blocked by the close settlement window process.
A long-standing deadlock in central-ledger's timeout handler (transfer timeout sweeps colliding with live transfer inserts) disappeared. We've already updated the reference example-mojaloop-backend chart in mojaloop/helm to set this correctly (PR merged post v17.2.0). In addition, one pre-requisite for this to work is the central-settlement version v17.4.0 .

This post explains the root cause, why we think it should be your production default too, and exactly how to make the same change on your own deployment - safely, reversibly, and without waiting for a future Mojaloop release.

The problem
Mojaloop's core write path (central-ledger) and its settlement path both hit the same InnoDB tables - transferStateChange, transferTimeout, and participantPosition - under MySQL's default isolation level, REPEATABLE READ.
Two symptoms showed up together:
● Settlement blocking live traffic. The close settlement window process - which scans and aggregates over ranges of transfers to close out a window - was taking shared locks that live incoming transfers then queued behind. Under load, transfer processing latency would spike every time a settlement window was closed.
● A timeout-handler deadlock. The cron-based timeout sweep runs an INSERT … SELECT to pull expired transfers. Under REPEATABLE READ, that statement takes shared next-key locks on the index range it scans - a normal part of how InnoDB prevents phantom reads at that isolation level. Those locks collided with live, in-flight transfer commits landing in the same index range at the same moment.
Both symptoms have the same root cause: REPEATABLE READ's phantom-read protection is implemented with locking (gap locks / next-key locks on range scans), and Mojaloop's write path is
dense enough, on the same narrow set of hot tables, that those locks become contention rather than protection.

Why REPEATABLE READ is the wrong default here
REPEATABLE READ exists to give a transaction a consistent snapshot across multiple statements - so if you SELECT the same range twice in one transaction, you see the same rows both times, and to guarantee that, InnoDB locks the gaps between index records to stop other transactions inserting into them.
Mojaloop's hot-path transactions don't need that guarantee:
● Transfer processing is short-lived: read position, apply a delta, write, commit. There's no multi-statement read-then-read-again pattern in the transfer path that depends on a frozen snapshot.
● Consistency where it actually matters: participant position balances - is enforced by explicit row locking (SELECT … FOR UPDATE) and unique constraints, not by gap locks preventing phantom rows.
● The thing that was relying on wide range scans (settlement aggregation, timeout sweeps) doesn't need repeatable-read semantics either - it needs to not block the write path while it scans.
READ COMMITTED removes the gap-lock behavior on non-matching rows for these scans (InnoDB still takes locks on rows it actually examines and modifies, just not the gaps around them). That's the entire fix: less locking on ranges, same row-level consistency guarantees where they're enforced explicitly.

Is this safe? What to check before you flip the switch
READ COMMITTED is not a free lunch - it changes semantics, and you should confirm these are true for your deployment before rolling it out:
● No code relies on repeatable snapshots within a transaction. Audit anything that runs more than one SELECT against the same range inside a single DB transaction expecting the second read to match the first (some reporting/reconciliation jobs are the usual suspects - the live transfer/settlement path in central-ledger is not one of them).
● binlog_format is ROW or MIXED. Statement-based replication is unsafe under READ COMMITTED because the same statement can legitimately produce different results depending on timing. ROW/MIXED (MySQL's default since 5.7.7) sidesteps this entirely - if you've explicitly forced STATEMENT anywhere, stop and fix that first.
● Nothing pins the isolation level per-connection. We confirmed database-lib (central-ledger's DB layer) has no afterCreate hook or per-connection isolation override - it inherits whatever the server default is. If your fork or a service you run does pin isolation level in application code, that code wins over the server default and won't be affected by this change either way - check for it so you know what you're actually changing.

Results
After the change: no recurrence of the timeout-handler deadlock, and closing a settlement window no longer visibly stalls incoming transfer throughput. This isn't a partial mitigation - the locking mode that caused both symptoms is gone.

What we've already changed upstream
The example-mojaloop-backend chart in mojaloop/helm already reflects this recommendation on its MySQL primary (which is provided for reference).
This alone doesn't change your production database. example-mojaloop-backend is a demo composition, not a values file any real deployment inherits automatically - if you're running Mojaloop in production, you almost certainly maintain your own values.yaml, and it needs the same change applied deliberately. That's what the rest of this post walks through.

How to change it on your own deployment
Below are the detailed steps for the configuration changes, once you’ve confirmed using central-settlement version v17.4.0 . This change (below) is server-wide, not central-ledger-only.

Test it live, no persistence, reversible in one command
SET GLOBAL transaction_isolation = 'READ-COMMITTED';
This affects new connections only - sessions already open keep their old isolation level until they reconnect. It does not survive a MySQL restart. This is the right first step: flip it, let connection pools cycle, watch your deadlock/lock-wait metrics, and revert with the same command if anything looks wrong.
Check current state at any time:
SELECT @@GLOBAL.transaction_isolation, @@SESSION.transaction_isolation;
Make it durable without a restart (MySQL 8.0+)
SET PERSIST transaction_isolation = 'READ-COMMITTED';
SET PERSIST writes the value to mysqld-auto.cnf in the data directory and applies it immediately and globally - no pod restart, no downtime. It survives a MySQL process restart too, as long as the data directory itself persists.
That last clause matters more than it looks. If your MySQL data directory doesn't persist across pod restarts - an ephemeral volume, common in dev/test deployments - mysqld-auto.cnf disappears the moment the pod is recreated, and you silently fall back to REPEATABLE READ. SET PERSIST alone is not enough on ephemeral storage - you need step 3 as well.
Bake it into your persistent server configuration (durable across restarts)
SET PERSIST only helps if the data directory it writes to survives. Either way, the isolation level should end up in whatever configuration your MySQL server loads on startup, so a redeploy, a restore, or a fresh replica doesn't quietly revert to REPEATABLE READ.
If you're running the Bitnami MySQL Helm chart (used by the Mojaloop reference deployment):
mysql:
primary:
extraFlags: "--transaction-isolation=READ-COMMITTED"
(An alternative, primary.configuration, fully overrides the default my.cnf - only worth it if you're already overriding the full config for other reasons, since you'd need to carry forward every default setting alongside the isolation line.) Apply with helm upgrade --install -f values.yaml.
If you're running MySQL another way - a self-managed server, a different Helm chart, or a managed service like RDS/Aurora/Cloud SQL - the mechanism differs but the target is the same: set transaction-isolation=READ-COMMITTED under [mysqld] in my.cnf, or the equivalent parameter in your provider's DB parameter group, and apply it through whatever config-management path you already use for that server.
Either way, this restarts the MySQL primary. On a single-primary/standalone topology that's a brief write-unavailability window, so schedule it like any other maintenance-window change: apply SET PERSIST first for immediate effect, then land the durable config change, so you get zero-downtime and durability without the two steps needing to be simultaneous.
Verify and monitor
SHOW VARIABLES LIKE 'transaction_isolation';
SELECT * FROM performance_schema.data_locks; -- confirm gap-lock behavior is gone on hot ranges
SELECT * FROM information_schema.INNODB_TRX; -- watch for long-running transactions post-change
Watch deadlock counters (SHOW ENGINE INNODB STATUS, or whatever you already scrape into Prometheus) before and after - you're looking for the timeout-handler-vs-live-insert deadlock signature to stop appearing entirely, not just get rarer.

Rollback
SET GLOBAL / SET PERSIST back to REPEATABLE-READ (or RESET PERSIST transaction_isolation to drop the override and fall back to compiled default), and revert whatever persistent configuration you changed in step 3. Nothing about this change is one-way.

A note to the community
This is a recommendation, not a mandate - you don't need our sign-off to try it, and every deployment is free to make this change on its own timeline. That said, if you're running Mojaloop in production and know of a reason REPEATABLE READ semantics are load-bearing somewhere we haven't audited - a reporting job, a reconciliation flow, a fork with different transaction boundaries - we'd like to hear about it, so we can document the exception for others weighing the same change.

Central Bank of Sudan

Steve Haley — Thu, 07 May 2026 13:57:11 +0000

RFP out for the Central Bank of Sudan. Please comment here with your information if you are looking for collaborators and partners and what you're looking for!

https://cbos.gov.sd/en/content/tender-announcement-national-instant-payment-system-project-nips

The Mojaloop Foundation Publishes Its Policy on the Responsible Use of AI Tools in the Community

James Bush — Mon, 13 Apr 2026 13:33:14 +0000

The Mojaloop Foundation is pleased to announce the publication of its policy on the responsible use of AI tools within the Mojaloop community. As AI technologies become increasingly embedded in how we design, build, and operate software, it is important that their use aligns with the Foundation’s principles of openness, transparency, and trust.

The policy provides clear guidance on how community members may use AI tools in ways that support collaboration and maintain the integrity of the Mojaloop open source ecosystem. It covers areas such as the use of AI for documentation, code development, and participation in community discussions, with an emphasis on human accountability and transparency.

This policy has been reviewed and endorsed by the Community Council, reflecting a shared commitment across the community to adopt AI responsibly while continuing to encourage innovation and contribution.

Given the rapid pace of change in AI technologies and their applications, this policy will be reviewed and updated regularly to ensure it remains relevant and effective.

You can read the full policy here:
https://docs.mojaloop.io/community/standards/ai_policy.html

We welcome feedback from the community as we continue to evolve our approach in this important area.

Replica Configuration, Scheduling, and Repeatability (Part 5 of 6)

Chris Law — Mon, 23 Mar 2026 12:19:56 +0000

In Part 5 of our Mojaloop v17 series, we focus on repeatability, ensuring performance remains stable and predictable under sustained load.
We refined deployment topology using topology-aware scheduling and resource isolation, reducing run-to-run variance and stabilising behaviour across replicas.

At a national scale, inconsistent pod placement and resource contention can quickly undermine throughput gains. These changes ensure the platform delivers consistent transaction processing, even during peak periods.

Replica Configuration, Scheduling, and Repeatability (Part 5 of 6)

Delivered in support of programmes led by the COMESA Clearing House, COMESA Business Council, GamSwitch Company Ltd, Institute for Inclusive Digital Africa, and the Central Bank of The Gambia.

Kafka Design for Real Throughput: Partitioning, Ordering, and Concurrency (Part 4 of 6)

Chris Law — Thu, 19 Mar 2026 11:26:52 +0000

In Part 4 of our v17 series, we focus on the event backbone — how Kafka design underpins sustained, high-throughput processing across the switch.

We revisited message flow design to enable true parallel processing, aligning partitioning with business domains and tuning concurrency to maintain correctness without limiting scale.

At a national level, small inefficiencies in partitioning, ordering, or consumer stability can quickly become bottlenecks. These changes ensure the platform delivers stable, predictable performance under sustained load.

Kafka Design for Real Throughput: Partitioning, Ordering, and Concurrency (Part 4 of 6)

Delivered in support of programmes led by the COMESA Clearing House, COMESA Business Council, GamSwitch Company Ltd, Institute for Inclusive Digital Africa, and the Central Bank of The Gambia.

Zero-Touch Kubernetes Networking with NetBird

Chris Law — Wed, 11 Mar 2026 13:10:30 +0000

A recent engineering write-up on netbird.io highlighted how the INFITX Africa platform team built a zero-touch private networking for Kubernetes environments using NetBird.

As our infrastructure spans on-premise and AWS environments, we needed a way for clusters to automatically join a secure private network without manual configuration or VPN management.

Using NetBird, Kubernetes Operators, and Crossplane, networking is now fully declarative and automatically provisioned as new clusters are created.

A great example of how cloud-native tooling can simplify secure infrastructure at scale.

INFITX Africa - Zero-Touch Kubernetes Networking

netbird.io - INFITX Africa Engineering in Focus: Zero-Touch Kubernetes Networking

Delivered in support of programmes led by the COMESA Clearing House, COMESA Business Council, GamSwitch Company Ltd, Institute for Inclusive Digital Africa, and the Central Bank of The Gambia.

Removing Bottlenecks at the “Front Door” (Ingress and Gateway Path) (Part 3 of 6)

Chris Law — Tue, 10 Mar 2026 13:00:10 +0000

In Part 3 of our v17 series, we look at the ingress and gateway layer — the critical “front door” of the switch that determines how efficiently transaction traffic enters the platform.

We modernised the gateway path to unlock the full performance potential of the core services, introducing high-performance ingress components and optimising identifier handling for high-concurrency processing.

At a national scale, even small architectural constraints can become throughput bottlenecks. These changes ensure the platform can sustain high transaction volumes while maintaining security, efficiency, and global interoperability.

Removing Bottlenecks at the “Front Door” (Ingress and Gateway Path) (Part 3 of 6)

Delivered in support of programmes led by the COMESA Clearing House, COMESA Business Council, GamSwitch Company Ltd, Institute for Inclusive Digital Africa, and the Central Bank of The Gambia.

Batching, Caching, and Cutting “Chatty” Work (Part 2 of 6)

Chris Law — Mon, 02 Mar 2026 15:30:04 +0000

In Part 2 of our v17 series, we highlight the architectural refinements designed to unlock sustained, high-volume throughput.

We have optimized the critical path by introducing high-efficiency batching for transfer workflows and streamlining processing steps to minimize latency.

At a national scale, every millisecond counts. These structural enhancements ensure the platform remains lean and responsive as transaction volumes grow, providing a future-proof foundation for global financial inclusion

If you operate or govern a national payment infrastructure, this series is written for you.

Batching, Caching, and Cutting “Chatty” Work (Part 2 of 6)

Delivered in support of programmes led by the COMESA Clearing House, COMESA Business Council, GamSwitch Company Ltd, Institute for Inclusive Digital Africa, and the Central Bank of The Gambia.

Building World-Class Mojaloop Platforms Together (Part 1 of 6)

Chris Law — Mon, 23 Feb 2026 11:56:40 +0000

Mojaloop is increasingly being relied upon as a national and regional payment infrastructure. In that role, performance is not a vanity metric; it is a trust requirement.

In Part 1 of our six-part series, we explore why sustained throughput, predictable latency, operational stability, and security-enabled performance are foundational to production-grade payment switches

Drawing on INFITX’s performance engineering work in Mojaloop v17, delivered in collaboration with the Mojaloop Foundation and adoption partners including COMESA DRPP and GISP Bantaba 2.0, we outline the core optimisations and reproducible deployment approach that strengthen real-world platform readiness

If you operate or govern a national payment infrastructure, this series is written for you.

Building World-Class Mojaloop Platforms Together (Part 1 of 6)

Delivered in support of programmes led by the COMESA Clearing House, COMESA Business Council, GamSwitch Company Ltd, Institute for Inclusive Digital Africa, and the Central Bank of The Gambia.

Attention developers: Nathan Delma is here to work with you!

Paula Hunter — Fri, 07 Nov 2025 15:00:00 +0000

https://mojaloop.io/nathan-delma-mojaloop-foundation-community-engineering-lead/

Workstream Status Updates

Paul Makin — Mon, 06 Oct 2025 14:00:52 +0000

Following in-depth updates at the Workstream Leads call last Thursday, 2nd October, the latest, detailed statuses of all of the workstreams can be reviewed at https://community.mojaloop.io/active-workstreams

You can also view a brief, summary version of these statuses by clicking here.

Community Council Minutes 2025-09-08

Juliana Gordon — Thu, 11 Sep 2025 16:29:14 +0000

1.0 WELCOME, ROLL CALL, AGENDA REVIEW, MINUTES APPROVAL

Mr. Simeon Oriko welcomed the group, called the meeting to order at 9:04 AM ET and reviewed the agenda. Quorum was not reached.

2.0 MOJALOOP CONVENING UPDATE

Mr. Oriko updated the Council on activities surrounding the upcoming Convening. A request was previously sent to Council members for recommendations of those who could sit on the Program Advisory Committee. None have been submitted. The goal of the PAC is to create a draft agenda, champion the event within participant organizations, and bring in more speakers.
Mr. Oriko asked the group if we should move forward with the PAC, which the Council affirmed. Mr. Oriko then requested names of proposed participants by end of day Tuesday, 9 September.
It was proposed that the PAC consist of three MLF staff members and three external volunteers, which the group agreed with. Ms. Reica Rampersadh volunteered to participate in the PAC as a volunteer.

ACTION: Mr. Oriko will bring the proposal of the PAC including three MLF staff members and three volunteers back to leadership, and requests that Council members check in with their proposed volunteers, submitting them by EOD Tuesday, 9 September.

MLF has been pursuing Central Bank of Kenya in cohost role of event. Innocent Ephraim, Steve Haley, and Victor Malu have spoken with Kenya Bank Association and Central Bank of Kenya which have tentatively agreed to participate as cohost. Mr. Haley will be making the final decision of which/both will serve in the role. If Council members have any thoughts, they should contact Mr. Haley.

ACTION: Council members should consider submitting sessions to the Convening and registering for the event.

4.0 ADJOURN: Meeting adjourned at 9:23 AM ET.