Uploaded image for project: 'UX Product'
  1. UX Product
  2. UXPROD-3210

NFR: R1 2022 Lotus Data import Stability/Reliability work

    XMLWordPrintable

Details

    • New Feature
    • Status: Closed (View Workflow)
    • P2
    • Resolution: Done
    • None
    • Lotus (R1 2022)
    • None
    • Lotus R1 2022
    • Out of scope
    • Low
    • Jumbo: > 45 days
    • Folijet
    • 125
    • R1
    • R1

    Description

      BE est is Jumbo; rough estimate is 120 days

      Current situation or problem:

      1. High CPU/Memory consumption on modules
      2. Duplicates may created upon import for holdings and items (instances were fixed)
      3. Confirm that SRS does fail when processing during import
        # If we have infrastructure issue (like DB not available, module being restarted or network failure), we are sending DI_ERROR instead of retrying

      Investigation required for:

      • Race condition on start (Kafka consumers start working before DB is configured) OR Periodical DB shutdown after SRS restart. Jobs get stuck if not able to update status in DB (messages ACKed even if we could not process them)
      • Kafka consumers stop reading messages eventually, breaking job progress until module restart.
      • mod-data-import stores input file in memory, limiting size of uploaded file and possibly having oom
      • Consumer gets disconnected from Kafka cluster

      Proposed solution/stories

      1. Make consumers behave idempotent. Add pass-through identifier to de-duplicate messages.
      2. Generate "INSTANCE CREATED" from mod-inventory. Consume in SRS to update HRID in BIB and in INVENTORY to continue processing.
      3. Do not ACK messages in Kafka if there's not a logic, but infrastructure error/exception. Split failed processing results into 2 categories:
        • IO errors - do not ack. retry until fixed
        • Business logic - DI_ERROR and Ack current message
      4. Remove unnecessary topics (* ready for post processing and hrid set)
      5. De-duplicate status messages per-record while tracking progress

      One possible solution: Split to chunks, put to database, work with database/temp storage. Partially done (to be investigated)

      Links to additional info:
      Update to wherever the plan is now stored
      Data Import Stabilization plan - Vladimir Shalaev - FOLIO Wiki

      Questions

      TestRail: Results

        Attachments

          Issue Links

            continues

            New Feature - New feature requests UXPROD-3193 NFR: R3 2021 Kiwi Data import Stability/Reliability work

            • P2 - normal priority level, must be included in the current development cycle
            • Closed
            is continued by

            New Feature - New feature requests UXPROD-3429 NFR: R2 2022 Morning Glory Data import Stability/Reliability/Performance work

            • P2 - normal priority level, must be included in the current development cycle
            • Closed
            is defined by

            Task - A task that needs to be done. KAFKAWRAP-3 Implement error handler contract for KafkaConsumerWrapper

            • P3 - low priority level, item will be considered for inclusion in the next dev cycle
            • Closed

            Story - Created by Jira Software - do not edit or delete. Issue type for a user story. KAFKAWRAP-7 SPIKE: Prevent losing Kafka messages due to infrastructure failure

            • P2 - normal priority level, must be included in the current development cycle
            • Closed

            Story - Created by Jira Software - do not edit or delete. Issue type for a user story. MODDATAIMP-473 SPIKE: Review PTF reports and create tickets for improvements

            • P2 - normal priority level, must be included in the current development cycle
            • Closed

            Task - A task that needs to be done. MODDATAIMP-491 Improve logging to be able to trace the path of each record and file_chunks

            • P3 - low priority level, item will be considered for inclusion in the next dev cycle
            • Closed

            Task - A task that needs to be done. MODDATAIMP-495 SPIKE: Analysis of the possibilities of implementing idempotence through monitoring processed records with the participation of the persistence level

            • P2 - normal priority level, must be included in the current development cycle
            • Closed

            Story - Created by Jira Software - do not edit or delete. Issue type for a user story. MODDATAIMP-500 SPIKE: Design approach for assigning UUIDs for entities created by DI

            • P2 - normal priority level, must be included in the current development cycle
            • Closed

            Story - Created by Jira Software - do not edit or delete. Issue type for a user story. MODDATAIMP-566 SPIKE: Investigate unreachable (garbage) objects

            • P3 - low priority level, item will be considered for inclusion in the next dev cycle
            • Closed

            Task - A task that needs to be done. MODINV-408 Implement ProcessRecordErrorHandler for Kafka Consumers

            • P2 - normal priority level, must be included in the current development cycle
            • Closed

            Task - A task that needs to be done. MODINV-460 SPIKE: Analyze possibilities to implement some constraints on the persistence level for blocking duplicate "holdings" and "items".

            • P3 - low priority level, item will be considered for inclusion in the next dev cycle
            • Closed

            Task - A task that needs to be done. MODINV-547 Idempotence for createInstanceEventHandler

            • P3 - low priority level, item will be considered for inclusion in the next dev cycle
            • Closed

            Story - Created by Jira Software - do not edit or delete. Issue type for a user story. MODINV-548 SPIKE: Investigate approach for events deduplication during holdings and item creation

            • P3 - low priority level, item will be considered for inclusion in the next dev cycle
            • Closed

            Task - A task that needs to be done. MODINV-584 Improve logging to be able to trace the path of each record and file_chunks mod-inventory

            • P3 - low priority level, item will be considered for inclusion in the next dev cycle
            • Closed

            Story - Created by Jira Software - do not edit or delete. Issue type for a user story. MODINV-588 Implement deduplication for Instances

            • P3 - low priority level, item will be considered for inclusion in the next dev cycle
            • Closed

            Story - Created by Jira Software - do not edit or delete. Issue type for a user story. MODINV-589 Implement deduplication for Holdings

            • P3 - low priority level, item will be considered for inclusion in the next dev cycle
            • Closed

            Story - Created by Jira Software - do not edit or delete. Issue type for a user story. MODINV-590 Implement deduplication for Items

            • P3 - low priority level, item will be considered for inclusion in the next dev cycle
            • Closed

            Story - Created by Jira Software - do not edit or delete. Issue type for a user story. MODINV-591 Implement deduplication for Authorities

            • P3 - low priority level, item will be considered for inclusion in the next dev cycle
            • Closed

            Task - A task that needs to be done. MODINVOICE-252 Implement ProcessRecordErrorHandler for Kafka Consumers

            • P3 - low priority level, item will be considered for inclusion in the next dev cycle
            • Closed

            Task - A task that needs to be done. MODSOURCE-290 Implement ProcessRecordErrorHandler for Kafka Consumers

            • P3 - low priority level, item will be considered for inclusion in the next dev cycle
            • Closed

            Task - A task that needs to be done. MODSOURCE-402 Properly handle DB failures during events processing

            • P2 - normal priority level, must be included in the current development cycle
            • Closed

            Task - A task that needs to be done. MODSOURMAN-474 Implement ProcessRecordErrorHandler for Kafka Consumers

            • P3 - low priority level, item will be considered for inclusion in the next dev cycle
            • Closed

            Task - A task that needs to be done. MODSOURMAN-598 Properly handle DB failures during events processing

            • P2 - normal priority level, must be included in the current development cycle
            • Closed
            relates to

            New Feature - New feature requests UXPROD-3471 NFR: R2 2022 Morning Glory: Implement flow control for Data Import

            • P2 - normal priority level, must be included in the current development cycle
            • Closed

            Activity

              People

                abreaux Ann-Marie Breaux
                Taisiya Trunova Taisiya Trunova (Inactive)
                Votes:
                0 Vote for this issue
                Watchers:
                2 Start watching this issue

                Dates

                  Created:
                  Updated:
                  Resolved:

                  TestRail: Runs

                    TestRail: Cases