Uploaded image for project: 'mod-data-import'
  1. mod-data-import
  2. MODDATAIMP-392

SPIKE: Improve Uploading MARC files

    XMLWordPrintable

Details

    • 5
    • Folijet
    • Not Scheduled

    Description

      There are two ways that uploading MARC files can be improved in mod-data-import:

      1. Allow importing compressed files. Currently the 100K records file size is 203MB, and 835MB for the 500K records file. Uploading them without compression is slow. If they can be compressed and uncompressed on the server it will improve performance. 
      2. Instead of consuming the huge MARC records file all at once, can it consume the file bit by bit so that the Docker container does not need to be created with 2GB of memory? With 1M records, for example, then the container would need to have 4GB of memory. Also, the current way does not allow for concurrent imports because it is using too much memory in one import job already.

      abreaux re-confirm the largest files and how often, which will help us determine whether we need to proceed with this, plus perhaps testing the option for external storage of the files

      CC: abreaux  Kateryna Senchenko mage.air

      TestRail: Results

        Attachments

          Issue Links

            Activity

              People

                Unassigned Unassigned
                mtraneis Martin Tran
                Votes:
                0 Vote for this issue
                Watchers:
                5 Start watching this issue

                Dates

                  Created:
                  Updated:

                  TestRail: Runs

                    TestRail: Cases