Uploaded image for project: 'mod-data-export-worker'
  1. mod-data-export-worker
  2. MODEXPW-218

eHoldings: rework export job not to store all job data in-memory

    XMLWordPrintable

Details

    • Spitfire Sprint 149, Spitfire Sprint 150, Spitfire Sprint 151
    • 5
    • Spitfire
    • Nolana (R3 2022)
    • TBD

    Description

      Purpose/Overview:

      Currently eHoldings job looks like following:

      And the problem is that on "Save eHoldings step" all job related data is read from database and is stored in-memory till the end of the step. Furthermore, even bigger object is constructed during this step in order to support required export format. This uses huge amount of RAM during big exports (f.e. package with some notes + 10k titles).

      Earlier this was the only way we can calculate the number of columns in result file, as we need to query all titles first to count maximum amount of notes they have in order to define number of columns.

      As of https://issues.folio.org/browse/MODEXPW-198 postgres database is introduced and RAM usage can be decreased significantly.

      Approach:

      Don't store job-related data (packages, titles, notes) in job context.

      Save job-related data in postgres after each read.

      Calculate desired number of columns by postgres query.

      Create result file with necessary column headers.

      Read data from postgres in batches and write it to a file in a single step.

      Clean up job data in database.

      Acceptance criteria:

      • Functionality implemented
      • Result file remains the same
      • Covered with unit tests
      • Huge jobs successfully finish even on environments with limited resources

      TestRail: Results

        Attachments

          Issue Links

            blocks

            Story - Created by Jira Software - do not edit or delete. Issue type for a user story. MODEXPW-257 eholdings export: Package > Add Provider level token to export file

            • P3 - low priority level, item will be considered for inclusion in the next dev cycle
            • Closed

            Bug - A problem which impairs or prevents the functions of the product. MODEXPW-263 EHoldings export: Missing .csv file format in file name.

            • P3 - low priority level, item will be considered for inclusion in the next dev cycle
            • Closed
            defines

            New Feature - New feature requests UXPROD-177 Export eHoldings (EBSCO KB customers only)

            • P3 - low priority level, item will be considered for inclusion in the next dev cycle
            • Closed
            has to be done after

            Task - A task that needs to be done. MODEXPW-198 Removing shared resources

            • P2 - normal priority level, must be included in the current development cycle
            • Closed
            has to be done before

            Story - Created by Jira Software - do not edit or delete. Issue type for a user story. MODEXPW-230 Spike: eholdings export | Only keep successful jobs for 45 days

            • P3 - low priority level, item will be considered for inclusion in the next dev cycle
            • Closed

            Story - Created by Jira Software - do not edit or delete. Issue type for a user story. MODKBEKBJ-645 PTF - Export eholdings - load testing (single and multi-environment)

            • P3 - low priority level, item will be considered for inclusion in the next dev cycle
            • Closed
            relates to

            Task - A task that needs to be done. MODEXPW-221 Resolve memory issues in mod-data-export-worker with eHoldings

            • P3 - low priority level, item will be considered for inclusion in the next dev cycle
            • Closed

            Story - Created by Jira Software - do not edit or delete. Issue type for a user story. MODEXPW-224 Spike: eHoldings: create a plan to migrate data that is stored in job context to database

            • P3 - low priority level, item will be considered for inclusion in the next dev cycle
            • Closed

            Bug - A problem which impairs or prevents the functions of the product. MODEXPW-288 The "NullPointer" error appears when user exporting a "Title+Package" record at the same time as the same "Package" record.

            • P3 - low priority level, item will be considered for inclusion in the next dev cycle
            • Closed

            Bug - A problem which impairs or prevents the functions of the product. MODEXPW-290 Export from "eHoldings" app failed with error "column eholdingsp0_.e_provider does not exist"

            • P3 - low priority level, item will be considered for inclusion in the next dev cycle
            • Closed

            Bug - A problem which impairs or prevents the functions of the product. MODEXPW-293 The "Resource" data transfers between packages when they are exporting

            • TBD - To be determined
            • Closed

            Bug - A problem which impairs or prevents the functions of the product. MODEXPW-295 "Titles" are missed in exported ".csv" when applying sort to "Package"

            • P2 - normal priority level, must be included in the current development cycle
            • Closed

            Bug - A problem which impairs or prevents the functions of the product. MODEXPW-297 Sorting of "Title+Package" data by "Title Name" column is case-sensitive

            • P3 - low priority level, item will be considered for inclusion in the next dev cycle
            • Closed

            Bug - A problem which impairs or prevents the functions of the product. MODEXPW-298 Export of "Package" / "Title+Package" with more than 255 characters in "Title" field is failed

            • TBD - To be determined
            • Closed

            Bug - A problem which impairs or prevents the functions of the product. MODEXPW-299 "Subject" field value from exported ".csv" file doesn't match the value in "Title+Package" record

            • P3 - low priority level, item will be considered for inclusion in the next dev cycle
            • Closed

            Bug - A problem which impairs or prevents the functions of the product. MODEXPW-300 "Custom Coverage" field is filled with "Managed Coverage" field value in the exported ".csv" file

            • TBD - To be determined
            • Closed
            mentioned in

            Page Loading...

            Activity

              People

                GurleenKaur Gurleen Kaur
                viacheslav_kolesnyk Viacheslav Kolesnyk
                Votes:
                0 Vote for this issue
                Watchers:
                4 Start watching this issue

                Dates

                  Created:
                  Updated:
                  Resolved:

                  TestRail: Runs

                    TestRail: Cases