Details
-
Task
-
Status: Closed (View Workflow)
-
P2
-
Resolution: Done
-
None
-
None
-
-
Folijet Sprint 151
-
5
-
Folijet
-
Orchid (R1 2023)
Description
During any CREATE import job (MARC BIB or Authority), mod-data-import-converter-storage v1.14.1 is being called over and over again to get the job profile snapshot for the same jobProfileSnapshotID. This causes mod-data-import-cs service to use a lot of CPU and affecting performance negatively of not only the ongoing data import job but also everything else as well because calling this API (/data-import-profiles/jobProfileSnapshots/{ID}) uses shared resources such as the database and on the EC2 instance that hosts mod-data-import-cs and other module services as well.
Some raw facts:
1) The API /data-import-profiles/jobProfileSnapshots/{ID} was invoked 96,233 times out of 96241 times in a MARC Authority Import of 50K records for the same ID
2) The API /data-import-profiles/jobProfileSnapshots/{ID} was invoked 79,698 times out of 79706 times in a MARC Bib import of 50K records for the same ID
3) The module's CPU level for the MARC Authority Import job is around 121% and over 400% for the MARC BIB import.
Could it be redesigned to "cache" any same call to any external service (here it's mod-data-import-cs) for the duration of the import so that no wasteful calls are made?
TestRail: Results
Attachments
Issue Links
- defines
-
UXPROD-3725 NFR: Data Import Technical, NFR, & Misc work (Orchid R1 2023)
-
- Closed
-
- relates to
-
MODSOURCE-556 Fix behaviour of the cache in JobProfileSnapshotCache class
-
- Closed
-