Uploaded image for project: 'UX Product'
  1. UX Product
  2. UXPROD-3261

NFR: R1 2022 Lotus Data import performance work

    XMLWordPrintable

    Details

    • Template:
      UXPROD features
    • Release:
      Lotus R1 2022
    • Front End Estimate:
      Out of scope
    • Back End Estimate:
      Jumbo: > 45 days
    • Front-End Confidence factor:
      Medium
    • Development Team:
      Folijet
    • Calculated Total Rank:
    • PO Rank:
      124

      Description

      Team estimation - 45 days

      UXPROD-3135 was split into UXPROD-3193 for stability and reliability and UXPROD-3191 for performance; Ann-Marie Breaux to close UXPROD-3135 once all issues moved from it to the new features

      Current situation or problem:

      1.Kafka producer closed after sending

      2.WARN message when no handler found

      3. Kafka cache resource consumption
      4. Data import impacts other processes

      5. High resource consumption to get job(s) status/progress

      NOTE: Several stories in this feature are related to Spring; those may move to Morning Glory; need input from architects on whether Spring Kafka Template can be use in DI modules. Probably will need an architecture spike in Lotus, and then the Spring stories may move to Morning Glory

      Proposed solution/stories
      *1.*Create pool of active producers. Start pool on module launch, close on shutdown. Reuse connections. Add max/min pool sizes.
      2. Do not subscribe to messages you're not going to process OR Lower log lever for this type of messages.

      *3.*Remove Kafka cache. Modules that do not do persistent changes will sometimes (on duplicates read) do unnecessary calls. Can be optimized further upon adding distributed in-memory cache (ex hazelcast) (blocked by 6 PUT LINK TO FEATURE in p.6)
      4. SPIKE REQ.: Need investigation (possible solution - configure rate limiter). Relates to High CPU/Memory consumption on modules
      5. Add some kind of caching for progress tracking (database or in-memory)

      Links to additional information:

      Data Import Stabilization plan - Vladimir Shalaev - FOLIO Wiki

      Questions

        TestRail: Results

          Attachments

            Issue Links

              Activity

                People

                Assignee:
                abreaux Ann-Marie Breaux
                Reporter:
                Taisiya Trunova Taisiya Trunova
                Votes:
                0 Vote for this issue
                Watchers:
                5 Start watching this issue

                  Dates

                  Created:
                  Updated:
                  Resolved:

                    TestRail: Runs

                      TestRail: Cases