Quick Recap
The team reviewed the vNext Beta, highlighting several implementation issues. James
emphasized the importance of understanding the code related to the transfer process and the potential issues with having
only one instance of a service running concurrently. The team discussed the scalability of their system, the advantages
of using a distributed database, and the challenges in implementing the TigerBeetle system. They also discussed the
potential performance issues with the Central Ledger Service and the need to balance performance and reliability.
Summary
vNext Beta Review and Implementation Issues
James initiated the meeting and noted that Michael Richards was absent. The main focus of the meeting was a review of
the vNext Beta presented to the Mojaloop Foundation. James, Sam, and Paul had summarized their findings in a document on
the vNext Beta Review Slack channel (#vnext-beta-review), which had garnered significant comments from community
members. James highlighted several issues with the current ledger implementation in the vNext code base and planned to
bring these issues to the design authority for discussion. Karim inquired about the differences between the vNow and
vNext implementations, which James confirmed and planned to discuss further.
System Architecture and Implementation Issues
James provided an overview of the system architecture, focusing on the issues related to the transfers and accounts and
balances bounded contexts. He explained that incoming requests are processed in a specific way, with the transfers
context handling the business logic and the accounts and balances context updating the ledger and account balances.
However, James highlighted several problems with the implementation, such as single threading, lack of synchronized
access to account balances, and lack of atomicity in writing entries to the persistent store and flushing account
balances. He was explaining the code related to the transfer process, noting the absence of a duplicate check function,
and
emphasized the importance of understanding this code for system improvement.
Concurrency Issues in Service Instances Discussed
James highlighted the potential issues with having only one instance of a service running concurrently in the system.
He explained the problem with the account balance synchronization and the possibility of overspending if two instances
assume the available liquidity balance is higher than it actually is. James suggested that the system should have
multiple instances of each service that can handle any request, ensuring redundancy, scalability, and reliability. Aung
proposed a possible solution using Kafka partitioning, but James reiterated his concern about the affinity between DFSPs
and service instances, suggesting that multiple instances of each service able to deal with requests from any DFSP would
be a more desirable solution.
Instant Payment System Implementation
Karim and James discussed the implementation of an instant payment system. Karim shared that the system can process 2
million transactions, with 1 million going through their data center. He emphasized the importance of ensuring data
persistence before committing operations. The system relies on Oracle as a database, and any issues with data
persistence could lead to problems if services are switched to another data center. James highlighted that the system
always writes to persistent storage, but reads only happen from a local cache at runtime and how the prevents multiple
processes from acting concurrently. They agreed on the need to balance system
scalability with zero tolerance on potential data loss.
Scalability and Distributed DBMS Discussion
James and Karim expressed concerns about the scalability of their system, specifically the issue of affinity. James
suggested that having only one instance of a service for any particular DFSP was a
problem and should be discussed further. Paul agreed, highlighting the complexity of effectively creating a distributed
DBMS.
However, James proposed leveraging an existing distributed DBMS, such as TigerBeetle, to avoid developing their own.
The team decided to kick off a discussion to gather different views on this matter.
Distributed Databases for Data Safety and Redundancy
James discussed the advantages of using a distributed database. He explained that distributed databases allow for
increased processing throughput by distributing the work across multiple machines, which avoids reaching the processing
throughput limit of a single hardware node. He also emphasized the importance of replicating data across multiple
physical storage devices, especially for mission-critical data. This approach ensures data redundancy and minimizes the
risk of data loss due to storage device failures. He noted that distributed databases allow for data to be stored in
different physical modes, different failure boundaries, and even different data centers, providing an added layer of
safety. He concluded that distributed databases are highly desirable for Mojaloop as they offer data redundancy and can
ensure the
safety of data in multiple physical locations.
Non-Functional Requirements and Solutions for Financial Systems
James proposed potential solutions for future clusters and emphasized the significance of non-functional requirements
for financial systems, including regulatory approval. He mentioned an open PR discussing operational characteristics and
non-functional requirements for Mojaloop (Mojaloop Invariants: https://github.com/mojaloop/documentation/pull/428),
urging the team to review and decide on possible changes to the ledger
implementation. James also noted that Pedro was working on a TigerBeetle integration, but it was still in its early
stages. James stressed the importance of reaching consensus on the correct approach for
Mojaloop and repeated his requested for all DA members to review the invariants, particularly the section on operational
characteristics. He
encouraged team members to comment on any disagreements in the PR and aimed to resolve these issues within the next
week.
TigerBeetle System Implementation Challenges
James and Paul discussed the challenges in implementing the tiger beetle system, with a focus on synchronization issues
between processes and the need for locking mechanisms to prevent dirty reads and writes. James
suggested the use of thread synchronization code across processes and expressed his hope that Pedro would deliver a
solution that utilizes tiger beetle as the single store for ledges and account balances, with multiple instances running
due to its distributed cluster nature. Paul proposed having a MySql version as an interim solution until the tiger
beetle tool matures. Karim added that they currently have multiple instances due to the absence of a context boundary.
Performance vs Reliability: Central Ledger Service Challenges
The team discussed the potential performance issues with a shared atomic account balance, particularly in handling bulk
transfers and high-volume transactions. Karim emphasized the need for reliability
over performance, while James highlighted the limitations of a single instance of a process writing to a single instance
of a database. The discussion concluded without a definitive solution, acknowledging the need to balance performance and
reliability. James also initiated a discussion about the invariants topic in the DA channel on Slack, encouraging team
members to contribute their thoughts. The meeting ran over time, prompting
James to suggest extending the discussion in the next meeting, which the team agreed to.
Next steps
• James will share the code for the critical path for transfer prepare and transfer commit on DA core slack channel.
• Discuss the need for multiple instances of services in Mojaloop to ensure high availability and scalability.
• All DA members to review the open PR on operational characteristics and non-functional requirements for Mojaloop (
Mojaloop Invariants)
• Discuss the implications of the Open PR on the vNext ledger design
• Consider the use of TigerBeetle as a single store for ledgers and account balances
• Consider the use of MySQL as a stopgap until TigerBeetle becomes production-ready
Please note that this content was originally generated by AI but has been reviewed for accuracy by James Bush
Top comments (0)