Details
-
Story
-
Status: Open (View Workflow)
-
P3
-
Resolution: Unresolved
-
None
-
None
-
-
Prokopovych
Description
Overview
Users must be allowed to download Instance UUIDs, when searching and filtering from Inventory App
We need to build an API which would download a file with all relevant UUIDs
The number of UUIDs can span several thousands to several millions
Given that each UUID is 36 characters, with 1 Million UUIDs the file size would be ~37 MB
For now the largest known data set is UC with ~8Millions Bibs, so file size could be ~300 MB
- Note: In the future , they should be allowed to download HRIDs, but that is out of scope of this story.
Approach
- An API can return the file as an attachment (not as JSON).
- RMB has implemented new streaming mechanism from DB: PostgresClient.streamGet https://issues.folio.org/browse/RMB-500
- Allow to reduce the size of the UUID download file by either
a) returning a binary file where the 128 bits of a UUID are sent as 16 bytes or
b) returning a CSV file where each UUID is encoded in Base64 (resulting in 22 bytes) and the UUIDs are separated by commas.
Notes for Base64 implementation:
Example: f31a36de-fcf8-44f9-87ef-a55d06ad21ae,7fbd5d84-62d1-44c6-9c45-6cb173998bbd results in 8xo23vz4RPmH7wAApV0GrQ,f71dhGLRRMacRQAAbLFzmQ
https://stackoverflow.com/a/29836273
https://stackoverflow.com/a/55474589
https://sakthipriyan.com/2017/04/02/creating-base64-uuid-in-java.html
https://www.npmjs.com/package/uuid-encoder
https://www.fileformat.info/tool/guid-base64.htm
Note for HRIDs: HRIDs cannot be encoded as binary file or as CSV file with Base64 encoded values. HRIDs must always been sent as raw values in a CSV file.
Acceptance Criteria
- API is developed, and allows chunked download
- Unit tests are added
TestRail: Results
Attachments
Issue Links
- clones
-
MODINVSTOR-439 Create bulk download endpoint to save id UUIDs
-
- Closed
-
- is blocked by
-
RMB-500 PostgresClient.streamGet with total hits and error handling
-
- Closed
-
- relates to
-
MODINVSTOR-438 stream GET for /instance, /holdings and /items
-
- Closed
-
-
RMB-554 PgUtil.getAllIds
-
- Open
-
-
RMB-559 Add stream get utility that produces HTTP result
-
- Closed
-
-
RMB-581 Extend PgUtil.streamGet() to support manual selection of a set of columns
-
- Open
-
-
UIDEXP-1 Save Instance UUIDs of the Inventory search results
-
- Closed
-
-
UIDEXP-12 Disable saving Instance UUIDs option if no records displayed in result pane
-
- Closed
-
-
UIDEXP-28 Enable downloading Instance UUIDs for more than 30 records
-
- Closed
-
-
UXPROD-1928 APIs for batch downloads (exports)
-
- Closed
-