Minutes of DA Meeting 2023-10-04

Attendees:

• Paul Baker paul.baker@infitx.com (PB)
• Pedro Barreto pedrob@crosslaketech.com (PSB)
• James Bush jbush@mojaloop.io (JB)
• Tom Daly tdaly61@gmail.com (TD)
• Johann Foley johannes.foley@sybrin.com (JF)
• Sam Kummary skummary@mojaloop.io (SK)
• Simeon Oriko simeonoriko@gmail.com (SO)
• Michael Richards Michael.Richards@infitx.com (MR) (Chair)

Apologies:

• Miguel de Barros miguel.debarros@infitx.com (MdB)
• Jason Bruwer jason.bruwer@coil.com (JB)
• Godfrey Kutumela godfreyk@crosslaketech.com (GK)
• Paul Makin pmakin@mojaloop.io (PM)
• Jane Stroucken Jane.Stroucken@infitx.com (JS)

Absent:

• Aung Thaw Aye AungThaw.Aye@thitsaworks.com AT
• Greg McCormick greg@frms.io (GM)
• Justus Ortlepp justus.ortlepp@gmail.com (JO)

Agenda:

• Report on actions from last meeting
  • Where are we on currency conversion?
  • Post the DA charter somewhere public
• Position handler batching (VK)
• Adding security to the DA charter (TD)
• Issue 107: Mojaloop security review (TD)
• Issue 108: Design and strategy questions (TD)
• DA involvement in API changes (PSB)
• AOB

Minutes

• Actions from previous meeting
  • Where are we on currency conversion?
  • OK to proceed with PoC.
  • We will revisit improvements starting at the convening
  • Post the DA charter somewhere public
  • Still open
• Position handler batching
  • Link to documentation
  • PB: there was batching, but it didn't work.
  • VK picked this up and designed it.
  • Some initial test results
  • VK: existing batching only for prepare handler, this change applies to all messages
  • Kafka consumes a fixed number of messages, selected by participant currency ID
  • Processing speed depends on batch size: 1 op/sec start position; 100 ops/sec with batches of 10; 1000 ops/sec with batches of 100.
  • Performance on a single machine with position handler only.
  • One handler per account ID.
  • PSB: can we get a deadlock? VK: no. Each handler will deal with a separate account, there should be no competitive locks.
  • JB: if lock at the point of reading the balance then you could have multiple processors processing the same account.
  • SK: we are using Kafka to partition.
  • PSB: how do you assign where the fulfil has two accounts? SK: we don't have to do that. In fulfil, we only update the creditor account. In step 1, payer account subtracts from the position. In fulfil, payee account gets credited.
  • PSB: how do we assign messages to bins? PB: all actions are assigned to bins automatically
  • PSB: how do we manage technical failures?
  • JB: this doesn't work around the presence of a single account balance. PB: the results are so good that it's not a problem.
  • PSB: this is scale up, not scale out. PB: yes, we need to do a full test. VK: we did try scaling out to several instances on one node. Improvement was preserved.
  • PSB: how do you commit to Kafka when it fails? VK: another handler picks it up. PSB: that's not a good strategy, because it gets the batch again. You need something like a dead letter queue. PB: it processes the failure, then commits the message. VK: if there is batch level failure, then we don't commit the messages. JB: you can't choose which messages to process. PSB: you have to assume that you can only process the whole batch.
  • JB: also a resilience thing - you need a backup strategy if a process dies. VK: Kafka will manage that itself.
  • PSB: this is what we've done in vNext. If done well, it's great.
  • TD: how do we do full-scale performance tests? Why do we need to improve performance on vNow? JB: INFITX have a business case that they need to meet. TD: seems to trivialise the TGB statement that performance was sufficient.
  • Foley: in principle it all makes sense. How do we handle errors? We think that's OK, but we need to be sure. Opinion: would be great to see an E2E performance test with real circumstances and make a decision based on that.
  • PSB: So, we make PoC decisions based on the agreement that the performance is good enough, then we have a business need to improve performance for the PoCs? Were Infitx not part of the ones that said vNow's performance is enough?
  • SK: “We” who? Who said performance is good enough of which version ? Sorry it isn’t clear
• Adding security to the DA charter (TD)
• Issue 107: Mojaloop security review (TD)
• Issue 108: Design and strategy questions (TD)
• DA involvement in API changes (PSB)
• AOB

Actions:

• SK: liaise with performance workstream to review tasks for donation.