Uploaded image for project: 'mod-source-record-manager'
  1. mod-source-record-manager
  2. MODSOURMAN-872

Spike: Data Import Processing Calls mod-data-import-converter-storage Excessively

    XMLWordPrintable

Details

    • Folijet Sprint 151
    • 5
    • Folijet
    • Orchid (R1 2023)

    Description

      During any CREATE import job (MARC BIB or Authority), mod-data-import-converter-storage v1.14.1 is being called over and over again to get the job profile snapshot for the same jobProfileSnapshotID.  This causes mod-data-import-cs service to use a lot of CPU and affecting performance negatively of not only the ongoing data import job but also everything else as well because calling this API (/data-import-profiles/jobProfileSnapshots/{ID}) uses shared resources such as the database and on the EC2 instance that hosts mod-data-import-cs and other module services as well.  

      Some raw facts:

      1) The API /data-import-profiles/jobProfileSnapshots/{ID} was invoked 96,233 times out of 96241 times in a MARC Authority Import of 50K records for the same ID

      2) The API /data-import-profiles/jobProfileSnapshots/{ID} was invoked 79,698 times out of 79706 times in a MARC Bib import of 50K records for the same ID

      3) The module's CPU level for the MARC Authority Import job is around 121% and over 400% for the MARC BIB import. 

      Could it be redesigned to "cache" any same call to any external service (here it's mod-data-import-cs) for the duration of the import so that no wasteful calls are made? 

       

       

      TestRail: Results

        Attachments

          Issue Links

            Activity

              People

                afedasiuk Aliaksandr Fedasiuk
                mtraneis Martin Tran
                Votes:
                0 Vote for this issue
                Watchers:
                5 Start watching this issue

                Dates

                  Created:
                  Updated:
                  Resolved:

                  TestRail: Runs

                    TestRail: Cases