Details
-
Bug
-
Status: Closed (View Workflow)
-
P2
-
Resolution: Done
-
None
-
Folio Q4 2019
17 node K8s cluster
Dedicated 24-core, 64GB memory Postgres v10.11 DB node
7 each of Okapi/mod-SRS/mod-SRM/mod-inv/mod-inv-storage
-
-
Folijet
Description
The degradation of performance based upon number of records loaded is observed in the Q4-2019 release of mod-SRS. We ran a migration for a few hours - and the times started at under 15 seconds for a jobExecution of 500 records to be imported. It is 500 records per POST adding to a jobExecution that finishes at 10000. Now we’re over 90 seconds per POST... And the times just kept steadily increasing.
During all of this, the CPU of the database node in our cluster is taking an extremely nasty hit. It’s a 24-core CPU/64GB of mem node and it’s pegged out at 30GHz. I had to limit it, as it nearly ran away with one of our VMWare ESXI hosts that the node is running on.
We are batch posting the 500 records to the endpoint: http://okapi:9130/change-manager/jobExecutions/<UUID>/records
A tail of the postgresql daily logs shows why there might be some slowdown, this script is being called on every save: mod-source-record-storage-server/src/main/resources/templates/db_scripts/create_get_records_generation_function.sql
postgresql daily log:
bash-4.2$ tail postgresql-Fri.log 2020-02-07 20:47:55 UTC [16437]: [326-1] user=tern_mod_source_record_storage,db=okapi_modules,app=[unknown],client=10.42.28.68LOG: statement: select get_highest_generation('d9c51040-701b-4bcd-b312-392b180e7733', '138bd685-319d-4b0c-a17a-acd71f50bb3d'); 2020-02-07 20:47:55 UTC [16421]: [115-1] user=tern_mod_source_record_storage,db=okapi_modules,app=[unknown],client=10.42.28.68LOG: statement: select get_highest_generation('af9a8af4-a7b7-4db0-b227-c3c56cdb584e', '138bd685-319d-4b0c-a17a-acd71f50bb3d'); 2020-02-07 20:47:56 UTC [16403]: [116-1] user=tern_mod_source_record_storage,db=okapi_modules,app=[unknown],client=10.42.28.68LOG: duration: 11056.625 ms 2020-02-07 20:47:56 UTC [16403]: [117-1] user=tern_mod_source_record_storage,db=okapi_modules,app=[unknown],client=10.42.28.68LOG: statement: select get_highest_generation('f00865eb-67f0-4d03-86ce-98874df43463', '138bd685-319d-4b0c-a17a-acd71f50bb3d'); 2020-02-07 20:47:56 UTC [16416]: [141-1] user=tern_mod_source_record_storage,db=okapi_modules,app=[unknown],client=10.42.28.68LOG: duration: 9899.306 ms 2020-02-07 20:47:56 UTC [16416]: [142-1] user=tern_mod_source_record_storage,db=okapi_modules,app=[unknown],client=10.42.28.68LOG: statement: select get_highest_generation('c6cb9a6a-e8e0-43b3-9c57-df6ac84e8067', '138bd685-319d-4b0c-a17a-acd71f50bb3d'); 2020-02-07 20:47:56 UTC [16412]: [2539-1] user=tern_mod_source_record_storage,db=okapi_modules,app=[unknown],client=10.42.28.68LOG: duration: 11538.123 ms 2020-02-07 20:47:56 UTC [16412]: [2540-1] user=tern_mod_source_record_storage,db=okapi_modules,app=[unknown],client=10.42.28.68LOG: statement: select get_highest_generation('635b177e-5316-4f5d-8407-58675463ba2a', '138bd685-319d-4b0c-a17a-acd71f50bb3d'); 2020-02-07 20:47:56 UTC [16406]: [141-1] user=tern_mod_source_record_storage,db=okapi_modules,app=[unknown],client=10.42.28.68LOG: duration: 11704.982 ms 2020-02-07 20:47:56 UTC [16406]: [142-1] user=tern_mod_source_record_storage,db=okapi_modules,app=[unknown],client=10.42.28.68LOG: statement: select get_highest_generation('2c7672d5-8fe4-4394-ba54-e121f914d378', '138bd685-319d-4b0c-a17a-acd71f50bb3d');
TestRail: Results
Attachments
Issue Links
- defines
-
MODSOURCE-110 Improve performance when ingesting MARC records into SRS, including bulk loads
-
- Closed
-
-
UXPROD-2363 SRS Performance Improvements
-
- Closed
-
- relates to
-
MODSOURCE-107 Don't calculate generation id for Record
-
- Closed
-