An order/order line update process needs to atomically update the database and send messages/events to track audit log to avoid data inconsistencies and bugs.
There’s no guarantee that the transaction will commit. Similarly, if a service sends a message after committing the transaction there’s no guarantee that it won’t crash before sending the message. This problem can be solved using transactional outbox pattern
Need to perform following steps:
- create new table that will store events
- modify source code of order/order line create/edits to store events into new separate table
- implement separate process that will fetch data from new table and send messages to kafka
Flow described in this diagram:
More details can be found here: https://microservices.io/patterns/data/transactional-outbox.html
There is potential issue with this apporach when there is more than 1 instance of mod-orders-srotage deployed. It can lead to situation that these instances simultaniously reading messages from this new 'Messages' table and we will have message's duplicated.
Approach to implement this:
- Add Timer Interface to mod-orders-storage https://github.com/folio-org/okapi/blob/master/doc/guide.md#timer-interface . Okapi will call necessary logic in mod-orders-storage to reade outbox table and send messages to kafka. Examples can be found in mod-users, mod-licenses, mod-data-export, mod-circulation-storage
- Create new table outbox_table_bloker with one line
- Implement listener in mod-orders-storage for this call from Okapi. This listener should call SELECT * FOR UPDATE FROM OUTBOX_TABLE_BLOCKER. If without success - nothing need to do, otherwise - continue the main logic to read messages from outbox_table and send them to kafka. After finish need to do commit/rollback for outbox_table_blocker
This approach well documented in this document: https://wiki.folio.org/display/DD/Implementing+the+Transactional+Outbox+pattern