Uploaded image for project: 'UX Product'
  1. UX Product
  2. UXPROD-3261

NFR: R1 2022 Lotus Data import performance work

    XMLWordPrintable

Details

    • Lotus R1 2022
    • Out of scope
    • Medium
    • Jumbo: > 45 days
    • Folijet
    • 124

    Description

      Team estimation - 45 days

      UXPROD-3135 was split into UXPROD-3193 for stability and reliability and UXPROD-3191 for performance; abreaux to close UXPROD-3135 once all issues moved from it to the new features

      Current situation or problem:

      1.Kafka producer closed after sending

      2.WARN message when no handler found

      3. Kafka cache resource consumption
      4. Data import impacts other processes

      5. High resource consumption to get job(s) status/progress

      NOTE: Several stories in this feature are related to Spring; those may move to Morning Glory; need input from architects on whether Spring Kafka Template can be use in DI modules. Probably will need an architecture spike in Lotus, and then the Spring stories may move to Morning Glory

      Proposed solution/stories
      *1.*Create pool of active producers. Start pool on module launch, close on shutdown. Reuse connections. Add max/min pool sizes.
      2. Do not subscribe to messages you're not going to process OR Lower log lever for this type of messages.

      *3.*Remove Kafka cache. Modules that do not do persistent changes will sometimes (on duplicates read) do unnecessary calls. Can be optimized further upon adding distributed in-memory cache (ex hazelcast) (blocked by 6 PUT LINK TO FEATURE in p.6)
      4. SPIKE REQ.: Need investigation (possible solution - configure rate limiter). Relates to High CPU/Memory consumption on modules
      5. Add some kind of caching for progress tracking (database or in-memory)

      Links to additional information:

      Data Import Stabilization plan - Vladimir Shalaev - FOLIO Wiki

      Questions

      TestRail: Results

        Attachments

          Issue Links

            continues

            New Feature - New feature requests UXPROD-3191 NFR: R3 2021 Kiwi Data import performance work

            • P2 - normal priority level, must be included in the current development cycle
            • Closed
            defines

            Epic - Created by Jira Software - do not edit or delete. Issue type for a big user story that needs to be broken down. UXPROD-47 Batch Importer (Bib/Acq)

            • P3 - low priority level, item will be considered for inclusion in the next dev cycle
            • Analysis Complete
            is continued by

            New Feature - New feature requests UXPROD-3429 NFR: R2 2022 Morning Glory Data import Stability/Reliability/Performance work

            • P2 - normal priority level, must be included in the current development cycle
            • Closed
            is defined by

            Story - Created by Jira Software - do not edit or delete. Issue type for a user story. MODDATAIMP-557 Close Vertx Kafka producers that have not been closed

            • P2 - normal priority level, must be included in the current development cycle
            • Closed

            Task - A task that needs to be done. MODDATAIMP-580 SPIKE: Investigate Connection Pool Waiters behaviour on create and update

            • P3 - low priority level, item will be considered for inclusion in the next dev cycle
            • Closed

            Story - Created by Jira Software - do not edit or delete. Issue type for a user story. MODDATAIMP-588 Spike: Investigate possibility of removing Kafka cache

            • P2 - normal priority level, must be included in the current development cycle
            • Closed

            Sub-task - The sub-task of the issue MODDATAIMP-602 Investigate possibilities to remove Kafka cache from mod-inventory

            • P3 - low priority level, item will be considered for inclusion in the next dev cycle
            • Closed

            Sub-task - The sub-task of the issue MODDATAIMP-603 Investigate possibilities to remove Kafka cache from mod-inventory

            • P3 - low priority level, item will be considered for inclusion in the next dev cycle
            • Closed

            Story - Created by Jira Software - do not edit or delete. Issue type for a user story. MODDATAIMP-622 Test DI performance with Lotus improvements for Create scenarios

            • P3 - low priority level, item will be considered for inclusion in the next dev cycle
            • Closed

            Task - A task that needs to be done. MODDATAIMP-623 Remove Kafka cache initialization and Maven dependency

            • P2 - normal priority level, must be included in the current development cycle
            • Closed

            Story - Created by Jira Software - do not edit or delete. Issue type for a user story. MODDATAIMP-644 Test DI performance with Lotus improvements for Update scenarios

            • P3 - low priority level, item will be considered for inclusion in the next dev cycle
            • Closed

            Task - A task that needs to be done. MODDICONV-217 Kiwi R3 2021 - Log4j vulnerability verification and correction

            • P1 - highest priority item, drop everything else before this is resolved, reserved for critical bugfixes
            • Closed

            Story - Created by Jira Software - do not edit or delete. Issue type for a user story. MODDICORE-253 Add logic to skip sending DI_ERROR from mod-inventory in case for duplicates - LOTUS Bugfix

            • P3 - low priority level, item will be considered for inclusion in the next dev cycle
            • Closed

            Story - Created by Jira Software - do not edit or delete. Issue type for a user story. MODINV-401 Spike: KafkaCache Memory not released (after a long time)

            • P3 - low priority level, item will be considered for inclusion in the next dev cycle
            • Closed

            Bug - A problem which impairs or prevents the functions of the product. MODINV-444 Spike: CPU spikes related to events_cache topic

            • P2 - normal priority level, must be included in the current development cycle
            • Closed

            Task - A task that needs to be done. MODINV-598 Remove Kafka cache from update handlers

            • P3 - low priority level, item will be considered for inclusion in the next dev cycle
            • Closed

            Task - A task that needs to be done. MODINV-603 Remove Kafka cache by handling Constraint Violation Exceptions for DataImportKafkaHandler

            • P3 - low priority level, item will be considered for inclusion in the next dev cycle
            • Closed

            Story - Created by Jira Software - do not edit or delete. Issue type for a user story. MODINV-642 [Perf only] Inventory-storage could not found suitable instance type id

            • P2 - normal priority level, must be included in the current development cycle
            • Closed

            Story - Created by Jira Software - do not edit or delete. Issue type for a user story. MODINV-646 Add logic to skip sending DI_ERROR from mod-inventory in case for duplicates - Lotus Bugfix

            • P3 - low priority level, item will be considered for inclusion in the next dev cycle
            • Closed

            Story - Created by Jira Software - do not edit or delete. Issue type for a user story. MODINVSTOR-878 Upgrade RMB/Vertx versions that contain fixes for the connection pool

            • P2 - normal priority level, must be included in the current development cycle
            • Closed

            Task - A task that needs to be done. MODSOURCE-420 Add possibility to adjust parameter connectionPoolSize for configuring connection pool

            • P3 - low priority level, item will be considered for inclusion in the next dev cycle
            • Closed

            Task - A task that needs to be done. MODSOURCE-421 Add possibility to adjust parameter DB_MAXPOOLSIZE for configuring connection pool (Juniper HF5)

            • P2 - normal priority level, must be included in the current development cycle
            • Closed

            Task - A task that needs to be done. MODSOURCE-422 Upgrade Vertx from 4.0.0 to 4.1.2 (Juniper HF5)

            • P2 - normal priority level, must be included in the current development cycle
            • Closed

            Story - Created by Jira Software - do not edit or delete. Issue type for a user story. MODSOURCE-461 Upgrade RMB and Vertx versions that contain fixes for the connection pool

            • P3 - low priority level, item will be considered for inclusion in the next dev cycle
            • Closed

            Story - Created by Jira Software - do not edit or delete. Issue type for a user story. MODSOURCE-462 [Perf only] SRS container restarted during import of large files in new perf env

            • P2 - normal priority level, must be included in the current development cycle
            • Closed

            Story - Created by Jira Software - do not edit or delete. Issue type for a user story. MODSOURCE-464 [Perf only] SRM container restarted during import of large files in new perf env

            • P2 - normal priority level, must be included in the current development cycle
            • Closed

            Story - Created by Jira Software - do not edit or delete. Issue type for a user story. MODSOURCE-467 Reduce amount of logging of KafkaProducerConfig, KafkaConsumerConfig by moving them to WARN level

            • P3 - low priority level, item will be considered for inclusion in the next dev cycle
            • Closed

            Task - A task that needs to be done. MODSOURMAN-468 Create a new API and db table that should store and represent information for DataImport landing page.

            • P3 - low priority level, item will be considered for inclusion in the next dev cycle
            • Closed

            Task - A task that needs to be done. MODSOURMAN-469 Change data-import progress mechanism with a new plain DB table counter and background job

            • P3 - low priority level, item will be considered for inclusion in the next dev cycle
            • Closed

            Story - Created by Jira Software - do not edit or delete. Issue type for a user story. MODSOURMAN-616 Optimize DI landing page query

            • P2 - normal priority level, must be included in the current development cycle
            • Closed

            Story - Created by Jira Software - do not edit or delete. Issue type for a user story. MODSOURMAN-617 DI landing page: Push notifications instead of Polling for them

            • P2 - normal priority level, must be included in the current development cycle
            • Closed

            Task - A task that needs to be done. MODSOURMAN-630 Test and merge PRs on changes for DI landing page

            • P3 - low priority level, item will be considered for inclusion in the next dev cycle
            • Closed

            Task - A task that needs to be done. MODSOURMAN-631 Create a migration script for job_executions

            • P3 - low priority level, item will be considered for inclusion in the next dev cycle
            • Closed

            Task - A task that needs to be done. MODSOURMAN-638 Remove Kafka cache by handling Constraint Violation Exceptions for StoreRecordsChunksKafkaHandler

            • P2 - normal priority level, must be included in the current development cycle
            • Closed

            Task - A task that needs to be done. MODSOURMAN-639 Improve performance of saving journal records during import

            • P2 - normal priority level, must be included in the current development cycle
            • Closed

            Task - A task that needs to be done. MODSOURMAN-640 Remove Kafka cache by handling Constraint Violation Exceptions for DataImportJournalKafkaHandler

            • P2 - normal priority level, must be included in the current development cycle
            • Closed

            Task - A task that needs to be done. MODSOURMAN-641 Remove Kafka cache by handling Constraint Violation Exceptions

            • P2 - normal priority level, must be included in the current development cycle
            • Closed

            Task - A task that needs to be done. MODSOURMAN-644 Add the required index: job_executions.jsonb->>'status'

            • P3 - low priority level, item will be considered for inclusion in the next dev cycle
            • Closed

            Task - A task that needs to be done. MODSOURMAN-668 Restructure job_execution_progress table for DataImportKafkaHandler

            • P2 - normal priority level, must be included in the current development cycle
            • Closed

            Story - Created by Jira Software - do not edit or delete. Issue type for a user story. MODSOURMAN-694 Improve sql query for retrieving job execution source chunks

            • P3 - low priority level, item will be considered for inclusion in the next dev cycle
            • Closed

            Story - Created by Jira Software - do not edit or delete. Issue type for a user story. MODSOURMAN-695 Upgrade RMB and Vertx versions that contain fixes for the connection pool

            • P3 - low priority level, item will be considered for inclusion in the next dev cycle
            • Closed

            Story - Created by Jira Software - do not edit or delete. Issue type for a user story. MODSOURMAN-699 Fix Can`t map 'RECORD' or/and 'MARC_BIBLIOGRAPHIC' statements from logs

            • P3 - low priority level, item will be considered for inclusion in the next dev cycle
            • Closed

            Story - Created by Jira Software - do not edit or delete. Issue type for a user story. MODSOURMAN-700 SQL timeouts in SRM connected with calculation job execution progress

            • P3 - low priority level, item will be considered for inclusion in the next dev cycle
            • Closed

            Task - A task that needs to be done. MODSOURMAN-712 Timeout when importing MARC files - Lotus BF

            • P2 - normal priority level, must be included in the current development cycle
            • Closed

            Story - Created by Jira Software - do not edit or delete. Issue type for a user story. MODSOURMAN-751 Improve sql query used by UI to know is processing completed - Lotus BF

            • P3 - low priority level, item will be considered for inclusion in the next dev cycle
            • Closed

            Task - A task that needs to be done. RMB-901 Add ability to set reconnectAttempts and reconnectInterval for PgConnectOptions

            • P2 - normal priority level, must be included in the current development cycle
            • Closed

            Task - A task that needs to be done. UIDATIMP-918 Use new API for DataImport landing page

            • P3 - low priority level, item will be considered for inclusion in the next dev cycle
            • Closed

            Task - A task that needs to be done. UIDATIMP-1069 Use new API for DataImport ViewAll page

            • P3 - low priority level, item will be considered for inclusion in the next dev cycle
            • Closed

            Task - A task that needs to be done. UIPFIMP-45 Update 'source-manager-job-executions' dependency versions

            • P3 - low priority level, item will be considered for inclusion in the next dev cycle
            • Closed
            relates to

            Bug - A problem which impairs or prevents the functions of the product. MODSOURMAN-622 DI Job marked as completed too soon?

            • P3 - low priority level, item will be considered for inclusion in the next dev cycle
            • Closed
            mentioned in

            Page Loading...

            Activity

              People

                abreaux Ann-Marie Breaux
                Taisiya Trunova Taisiya Trunova (Inactive)
                Votes:
                0 Vote for this issue
                Watchers:
                5 Start watching this issue

                Dates

                  Created:
                  Updated:
                  Resolved:

                  TestRail: Runs

                    TestRail: Cases