Uploaded image for project: 'mod-inventory-storage'
  1. mod-inventory-storage
  2. MODINVSTOR-353

implement batch upload for /instances, /items and /holdings

    XMLWordPrintable

Details

    • CP: sprint 75, CP: sprint 76
    • 5
    • Core: Platform

    Description

      The batch endpoints should work as follows:

      • the client bundles a set of records into a single JSON array structure called a "batch". It's up to the client to split the entire collection of records into multiple batches.
      • the byte size of the of the batch is determined by the client through the Content-Length header with the upper limit decided on the server-side by the module. If the upper limit is reached the module will report a 413 Payload Too Large error to the client. The server may also handle requests with missing or wrong Content-Length and report this error during processing when the limit is reached.
      • the DB transaction is opened when the processing of a batch is started and is committed when all records from the batch have been successfully stored in the DB at which point the server responds with 201 Created.
      • in case of errors, the server rollbacks the transaction and responds with 422 Unprocessable Entity (all-or-nothing approach)
      • the server must use a method to store the records in the DB efficiently: e.g the saveBatch method in the RMB PostgresClient, based on the old vertx pg-mysql driver, or batch from https://vertx.io/docs/vertx-pg-client/java/

      Note on the API

      mod-inventory-storage already includes partial batch update API called inventory-storage-batch. The semantics of this API are not entirely compatible with what has been proposed above. The new API should be introduced as a new interface, independent of this existing API for backwards compatibility reasons. The new interface should be called inventory-storage-batch-sync (to differentiate from an async API that may be introduced at a later stage)

      Not implemented

      Content-Length/413 Payload Too Large and using a streaming method without loading the complete batch into mod-inventory-storage memory requires RMB support that doesn't exist yet. Therefore it cannot be implemented within the scope of this issue. It this has been split into RMB-505.
      The implementation uses PostgresClient.saveBatch that does not support streaming.

      TestRail: Results

        Attachments

          Issue Links

            Activity

              People

                julianladisch Julian Ladisch
                jakub Jakub Skoczen
                Votes:
                0 Vote for this issue
                Watchers:
                9 Start watching this issue

                Dates

                  Created:
                  Updated:
                  Resolved:

                  TestRail: Runs

                    TestRail: Cases