Uploaded image for project: 'UX Product'
  1. UX Product
  2. UXPROD-1928

APIs for batch downloads (exports)

    XMLWordPrintable

Details

    • XXL < 30 days
    • Includes API design, RMB support and rollout to a selected module (PoC). Excludes rollout across all FOLIO modules.
    • Core: Platform
    • 9
    • R4
    • R1
    • R1
    • R4
    • R1
    • R2
    • R1
    • R1
    • R1
    • R1

    Description

      Problem statement

      FOLIO's API are focused on fetching (GET) single record at a time and while this process can be tuned through various HTTP optimisation techniques (keepalive, pipelining) it's generally tricky to ensure that all clients are configured properly.

      To ensure good performance, it is much easier to provide dedicated batch APIs for batch fetching (downloads).

      Implementation wise, batch downloads should support streaming of data directly from the HTTP connection into a DB connection.

      Streaming downloads

      There is an example on how to implement streaming downloads with RMB in mod-users. Requires code review and maybe polishing.
      https://github.com/folio-org/mod-users/compare/user-stream-example

      TestRail: Results

        Attachments

          Issue Links

            is blocked by

            New Feature - New feature requests CQLPG-88 Implement id > [uuid] for pagination

            • P3 - low priority level, item will be considered for inclusion in the next dev cycle
            • Closed

            Task - A task that needs to be done. FOLIO-1774 Scalable stream support

            • P3 - low priority level, item will be considered for inclusion in the next dev cycle
            • Closed

            Umbrella - MODINVSTOR-288 roll out RMB support for streaming responses in mod-inventory-storage

            • P2 - normal priority level, must be included in the current development cycle
            • Open

            New Feature - New feature requests MODUSERS-114 GET users as a stream

            • P3 - low priority level, item will be considered for inclusion in the next dev cycle
            • Closed

            Task - A task that needs to be done. RMB-367 Documentation about downloads using GET with streaming

            • P3 - low priority level, item will be considered for inclusion in the next dev cycle
            • Open
            relates to

            New Feature - New feature requests MODINVSTOR-295 streaming POST holdings

            • P3 - low priority level, item will be considered for inclusion in the next dev cycle
            • Blocked

            New Feature - New feature requests MODINVSTOR-296 streaming POST items

            • P3 - low priority level, item will be considered for inclusion in the next dev cycle
            • Closed

            Story - Created by Jira Software - do not edit or delete. Issue type for a user story. MODINVSTOR-439 Create bulk download endpoint to save id UUIDs

            • P2 - normal priority level, must be included in the current development cycle
            • Closed

            Story - Created by Jira Software - do not edit or delete. Issue type for a user story. MODINVSTOR-452 Create size optimized bulk download endpoint to save id UUIDs

            • P3 - low priority level, item will be considered for inclusion in the next dev cycle
            • Open

            New Feature - New feature requests RMB-389 PgUtil.delete by CQL

            • P3 - low priority level, item will be considered for inclusion in the next dev cycle
            • Closed

            New Feature - New feature requests FOLIO-1839 Enable incremental requests for all records since last request

            • P2 - normal priority level, must be included in the current development cycle
            • Open

            Story - Created by Jira Software - do not edit or delete. Issue type for a user story. MODINV-119 Create endpoint to save collection of instances

            • P3 - low priority level, item will be considered for inclusion in the next dev cycle
            • Closed

            New Feature - New feature requests MODINVSTOR-294 streaming POST instance

            • P3 - low priority level, item will be considered for inclusion in the next dev cycle
            • Closed

            New Feature - New feature requests UXPROD-1826 APIs for batch uploads (imports)

            • P3 - low priority level, item will be considered for inclusion in the next dev cycle
            • Closed

            Activity

              People

                jakub Jakub Skoczen
                jakub Jakub Skoczen
                Votes:
                0 Vote for this issue
                Watchers:
                7 Start watching this issue

                Dates

                  Created:
                  Updated:
                  Resolved:

                  TestRail: Runs

                    TestRail: Cases