Uploaded image for project: 'UX Product'
  1. UX Product
  2. UXPROD-3191

NFR: R3 2021 Kiwi Data import performance work

    XMLWordPrintable

Details

    • Medium
    • XXXL: 30-45 days
    • Folijet
    • 116

    Description

      Team estimation - 45 days

      UXPROD-3135 was split into UXPROD-3193 for stability and reliability and UXPROD-3191 for performance; abreaux to close UXPROD-3135 once all issues moved from it to the new features

      Current situation or problem:

      1.Kafka producer closed after sending

      2.WARN message when no handler found

      3. Kafka cache resource consumption
      4. Data import impacts other processes

      5. High resource consumption to get job(s) status/progress

       

      In scope

      Out of scope

      Use case(s)

      Proposed solution/stories
      *1.*Create pool of active producers. Start pool on module launch, close on shutdown. Reuse connections. Add max/min pool sizes.
      2. Do not subscribe to messages you're not going to process OR Lower log lever for this type of messages.

      *3.*Remove Kafka cache. Modules that do not do persistent changes will sometimes (on duplicates read) do unnecessary calls. Can be optimized further upon adding distributed in-memory cache (ex hazelcast) (blocked by 6 PUT LINK TO FEATURE in p.6)
      4. SPIKE REQ.: Need investigation (possible solution - configure rate limiter). Relates to High CPU/Memory consumption on modules
      5. Add some kind of caching for progress tracking (database or in-memory)

      Links to additional information:

      Data Import Stabilization plan - Vladimir Shalaev - FOLIO Wiki

      Questions

      TestRail: Results

        Attachments

          Issue Links

            defines

            Epic - Created by Jira Software - do not edit or delete. Issue type for a big user story that needs to be broken down. UXPROD-47 Batch Importer (Bib/Acq)

            • P3 - low priority level, item will be considered for inclusion in the next dev cycle
            • Analysis Complete
            is continued by

            New Feature - New feature requests UXPROD-3261 NFR: R1 2022 Lotus Data import performance work

            • P2 - normal priority level, must be included in the current development cycle
            • Closed
            is defined by

            Story - Created by Jira Software - do not edit or delete. Issue type for a user story. MODDATAIMP-499 SPIKE: Use active Kafka producer from the pool for sending messages

            • P2 - normal priority level, must be included in the current development cycle
            • Closed

            Task - A task that needs to be done. MODINV-427 Testing of loading 500K MARC records on Folijet Rancher

            • P2 - normal priority level, must be included in the current development cycle
            • Closed

            Task - A task that needs to be done. MODINV-508 Block sending a requests with an empty barcode - KIWI

            • P3 - low priority level, item will be considered for inclusion in the next dev cycle
            • Closed

            Task - A task that needs to be done. MODINV-572 Block sending a requests with an empty barcode - JUNIPER HF4

            • P3 - low priority level, item will be considered for inclusion in the next dev cycle
            • Closed

            Bug - A problem which impairs or prevents the functions of the product. MODINVSTOR-792 GET item-storage/items?query=barcode=="" is very slow - takes 58 seconds

            • P2 - normal priority level, must be included in the current development cycle
            • Closed

            Bug - A problem which impairs or prevents the functions of the product. MODSOURCE-307 Updates becoming increasingly slow

            • P2 - normal priority level, must be included in the current development cycle
            • Closed

            Task - A task that needs to be done. MODSOURCE-340 Lower log level for messages when no handler found

            • P3 - low priority level, item will be considered for inclusion in the next dev cycle
            • Closed

            Task - A task that needs to be done. MODSOURCE-388 SPIKE: Slow query from mod-oai-pmh

            • P2 - normal priority level, must be included in the current development cycle
            • Closed

            Bug - A problem which impairs or prevents the functions of the product. MODSOURMAN-537 SPIKE: Slow Queries observed from jobExecutions lookup from DI Landing Page

            • P2 - normal priority level, must be included in the current development cycle
            • Closed

            Bug - A problem which impairs or prevents the functions of the product. MODSOURMAN-548 Sorting causes module to crash

            • P2 - normal priority level, must be included in the current development cycle
            • Closed

            Bug - A problem which impairs or prevents the functions of the product. MODSOURMAN-550 Reduce BE response payload for DI Landing Page to increase performance

            • P2 - normal priority level, must be included in the current development cycle
            • Closed
            relates to

            New Feature - New feature requests UXPROD-3023 NFR: R2 2021 Juniper Data Import Stabilization and Reliability work

            • P2 - normal priority level, must be included in the current development cycle
            • Closed

            New Feature - New feature requests UXPROD-3135 NFR: R3 2021 Kiwi Data Import Stabilization and Reliability work

            • P2 - normal priority level, must be included in the current development cycle
            • Closed
            mentioned in

            Page Loading...

            Activity

              People

                abreaux Ann-Marie Breaux
                Taisiya Trunova Taisiya Trunova (Inactive)
                Votes:
                0 Vote for this issue
                Watchers:
                4 Start watching this issue

                Dates

                  Created:
                  Updated:
                  Resolved:

                  TestRail: Runs

                    TestRail: Cases