Uploaded image for project: 'UX Product'
  1. UX Product
  2. UXPROD-3261

NFR: R1 2022 Lotus Data import performance work



    • Lotus R1 2022
    • Out of scope
    • Medium
    • Jumbo: > 45 days
    • Folijet
    • 124


      Team estimation - 45 days

      UXPROD-3135 was split into UXPROD-3193 for stability and reliability and UXPROD-3191 for performance; abreaux to close UXPROD-3135 once all issues moved from it to the new features

      Current situation or problem:

      1.Kafka producer closed after sending

      2.WARN message when no handler found

      3. Kafka cache resource consumption
      4. Data import impacts other processes

      5. High resource consumption to get job(s) status/progress

      NOTE: Several stories in this feature are related to Spring; those may move to Morning Glory; need input from architects on whether Spring Kafka Template can be use in DI modules. Probably will need an architecture spike in Lotus, and then the Spring stories may move to Morning Glory

      Proposed solution/stories
      *1.*Create pool of active producers. Start pool on module launch, close on shutdown. Reuse connections. Add max/min pool sizes.
      2. Do not subscribe to messages you're not going to process OR Lower log lever for this type of messages.

      *3.*Remove Kafka cache. Modules that do not do persistent changes will sometimes (on duplicates read) do unnecessary calls. Can be optimized further upon adding distributed in-memory cache (ex hazelcast) (blocked by 6 PUT LINK TO FEATURE in p.6)
      4. SPIKE REQ.: Need investigation (possible solution - configure rate limiter). Relates to High CPU/Memory consumption on modules
      5. Add some kind of caching for progress tracking (database or in-memory)

      Links to additional information:

      Data Import Stabilization plan - Vladimir Shalaev - FOLIO Wiki


      TestRail: Results


          Issue Links



                abreaux Ann-Marie Breaux
                Taisiya Trunova Taisiya Trunova (Inactive)
                0 Vote for this issue
                5 Start watching this issue



                  TestRail: Runs

                    TestRail: Cases