Details
-
Story
-
Status: Blocked (View Workflow)
-
P3
-
Resolution: Unresolved
-
None
-
None
-
-
5
-
Folijet
-
Not Scheduled
Description
There are two ways that uploading MARC files can be improved in mod-data-import:
- Allow importing compressed files. Currently the 100K records file size is 203MB, and 835MB for the 500K records file. Uploading them without compression is slow. If they can be compressed and uncompressed on the server it will improve performance.
- Instead of consuming the huge MARC records file all at once, can it consume the file bit by bit so that the Docker container does not need to be created with 2GB of memory? With 1M records, for example, then the container would need to have 4GB of memory. Also, the current way does not allow for concurrent imports because it is using too much memory in one import job already.
abreaux re-confirm the largest files and how often, which will help us determine whether we need to proceed with this, plus perhaps testing the option for external storage of the files
TestRail: Results
Attachments
Issue Links
- defines
-
UXPROD-4257 Data Import Architecture Refinements
-
- Open
-
- relates to
-
MODDATAIMP-606
Refactor mod-data-import to use Spring instead of Vertx
-
- Draft
-
-
MODDATAIMP-607
Spike: Refactor mod-data-import to act as a Saga coordinator in DI process - DRAFT
-
- Draft
-
-
MODDATAIMP-637
SPIKE 2: review folio-spring-base and prep for mod-data-import moving to Spring
-
- Open
-
-
MODDATAIMP-640
SPIKE 3: review folio-spring-base and prep for mod-srs and mod-srm moving to Spring
-
- Open
-
-
MODDATAIMP-692
SPIKE 1: Review recent architectural proposals for DI refactoring - DRAFT
-
- Draft
-
-
PERF-565
SPIKE: Slice large data import files into chunks
-
- In progress
-