Uploaded image for project: 'mod-inventory-storage'
  1. mod-inventory-storage
  2. MODINVSTOR-452

Create size optimized bulk download endpoint to save id UUIDs

    XMLWordPrintable

Details

    • Story
    • Status: Open (View Workflow)
    • P3
    • Resolution: Unresolved
    • None
    • None
    • Prokopovych

    Description

      Overview

      Users must be allowed to download Instance UUIDs, when searching and filtering from Inventory App
      We need to build an API which would download a file with all relevant UUIDs
      The number of UUIDs can span several thousands to several millions
      Given that each UUID is 36 characters, with 1 Million UUIDs the file size would be ~37 MB
      For now the largest known data set is UC with ~8Millions Bibs, so file size could be ~300 MB

      • Note: In the future , they should be allowed to download HRIDs, but that is out of scope of this story.

      Approach

      • An API can return the file as an attachment (not as JSON).
      • RMB has implemented new streaming mechanism from DB: PostgresClient.streamGet https://issues.folio.org/browse/RMB-500
      • Allow to reduce the size of the UUID download file by either
        a) returning a binary file where the 128 bits of a UUID are sent as 16 bytes or
        b) returning a CSV file where each UUID is encoded in Base64 (resulting in 22 bytes) and the UUIDs are separated by commas.

      Notes for Base64 implementation:
      Example: f31a36de-fcf8-44f9-87ef-a55d06ad21ae,7fbd5d84-62d1-44c6-9c45-6cb173998bbd results in 8xo23vz4RPmH7wAApV0GrQ,f71dhGLRRMacRQAAbLFzmQ
      https://stackoverflow.com/a/29836273
      https://stackoverflow.com/a/55474589
      https://sakthipriyan.com/2017/04/02/creating-base64-uuid-in-java.html
      https://www.npmjs.com/package/uuid-encoder
      https://www.fileformat.info/tool/guid-base64.htm

      Note for HRIDs: HRIDs cannot be encoded as binary file or as CSV file with Base64 encoded values. HRIDs must always been sent as raw values in a CSV file.

      Acceptance Criteria

      • API is developed, and allows chunked download
      • Unit tests are added

      TestRail: Results

        Attachments

          Issue Links

            Activity

              People

                Unassigned Unassigned
                julianladisch Julian Ladisch
                Votes:
                0 Vote for this issue
                Watchers:
                13 Start watching this issue

                Dates

                  Created:
                  Updated:

                  TestRail: Runs

                    TestRail: Cases