Uploaded image for project: 'UX Product'
  1. UX Product
  2. UXPROD-3210

NFR: R1 2022 Lotus Data import Stability/Reliability work



    • New Feature
    • Status: Closed (View Workflow)
    • P2
    • Resolution: Done
    • None
    • Lotus (R1 2022)
    • None
    • Lotus R1 2022
    • Out of scope
    • Low
    • Jumbo: > 45 days
    • Folijet
    • 125
    • R1
    • R1


      BE est is Jumbo; rough estimate is 120 days

      Current situation or problem:

      1. High CPU/Memory consumption on modules
      2. Duplicates may created upon import for holdings and items (instances were fixed)
      3. Confirm that SRS does fail when processing during import
        # If we have infrastructure issue (like DB not available, module being restarted or network failure), we are sending DI_ERROR instead of retrying

      Investigation required for:

      • Race condition on start (Kafka consumers start working before DB is configured) OR Periodical DB shutdown after SRS restart. Jobs get stuck if not able to update status in DB (messages ACKed even if we could not process them)
      • Kafka consumers stop reading messages eventually, breaking job progress until module restart.
      • mod-data-import stores input file in memory, limiting size of uploaded file and possibly having oom
      • Consumer gets disconnected from Kafka cluster

      Proposed solution/stories

      1. Make consumers behave idempotent. Add pass-through identifier to de-duplicate messages.
      2. Generate "INSTANCE CREATED" from mod-inventory. Consume in SRS to update HRID in BIB and in INVENTORY to continue processing.
      3. Do not ACK messages in Kafka if there's not a logic, but infrastructure error/exception. Split failed processing results into 2 categories:
        • IO errors - do not ack. retry until fixed
        • Business logic - DI_ERROR and Ack current message
      4. Remove unnecessary topics (* ready for post processing and hrid set)
      5. De-duplicate status messages per-record while tracking progress

      One possible solution: Split to chunks, put to database, work with database/temp storage. Partially done (to be investigated)

      Links to additional info:
      Update to wherever the plan is now stored
      Data Import Stabilization plan - Vladimir Shalaev - FOLIO Wiki


      TestRail: Results


          Issue Links



                abreaux Ann-Marie Breaux
                Taisiya Trunova Taisiya Trunova (Inactive)
                0 Vote for this issue
                2 Start watching this issue



                  TestRail: Runs

                    TestRail: Cases