Uploaded image for project: 'mod-data-import'
  1. mod-data-import
  2. MODDATAIMP-39

Create service for file chunking

    XMLWordPrintable

Details

    • EPAM BatchLoader Sprint 6, EPAM BatchLoader Sprint 7
    • 8
    • Folijet

    Description

      FileUploadEngine component of mod-data-import should provide functionality for splitting the MARC file into raw records and sending it in chunks to mod-source-record-manager for further processing.

      1. Add an endpoint to initiate processing.
      2. Implement call to mod-source-record-manager to update JobExecution setting jobProfile data and Status="IMPORT_IN_PROGRESS"
        Ticket for parsing: MODDATAIMP-45
      3. After the last chunk is sent, the service should send another request to mod-source-record-manager to notify that there is no more data. alternatively it can be a request parameter to distinguish that.
      4. The size of a chunk should be configurable.
      5. In case of error - send an error message with a description to the 'Source manager'.
      6. Send total quantity of records.

      For more details refer to the technical design - Sequence diagram of chunking the file .

      Acceptance Criteria

      • Parsing and sending the chunks is a non-blocking process
      • New endpoint exists
      • API is covered with tests (add sample files for testing)

      TestRail: Results

        Attachments

          Issue Links

            is blocked by

            Story - Created by Jira Software - do not edit or delete. Issue type for a user story. MODSOURMAN-3 Create technical design for Change Manager component from mod-source-record-manager

            • P3 - low priority level, item will be considered for inclusion in the next dev cycle
            • Closed

            Story - Created by Jira Software - do not edit or delete. Issue type for a user story. MODSOURMAN-9 Create ChangeManager component, expose API for JobExecutions creation

            • P3 - low priority level, item will be considered for inclusion in the next dev cycle
            • Closed
            relates to

            Story - Created by Jira Software - do not edit or delete. Issue type for a user story. MODDATAIMP-45 Implement MARC file reader for chunking service

            • P3 - low priority level, item will be considered for inclusion in the next dev cycle
            • Closed

            Story - Created by Jira Software - do not edit or delete. Issue type for a user story. MODSOURMAN-41 Add endpoints for updating JobExecution status and getting JobExecution children by id

            • P2 - normal priority level, must be included in the current development cycle
            • Closed

            Umbrella - MODSOURMAN-10 ChangeManager: Add functionality for saving chunks of raw data

            • P3 - low priority level, item will be considered for inclusion in the next dev cycle
            • Closed

            Story - Created by Jira Software - do not edit or delete. Issue type for a user story. MODSOURMAN-11 Create ChangeEngine component for parsing raw records

            • P3 - low priority level, item will be considered for inclusion in the next dev cycle
            • Closed

            Story - Created by Jira Software - do not edit or delete. Issue type for a user story. MODSOURMAN-18 Create endpoint for ChangeManager to update JobExecution

            • P3 - low priority level, item will be considered for inclusion in the next dev cycle
            • Closed

            New Feature - New feature requests UXPROD-656 Data Import (Batch Importer for Bib Acq) Infrastructure and Planning

            • P3 - low priority level, item will be considered for inclusion in the next dev cycle
            • Closed

            Activity

              People

                Igor_Gorchakov Igor Gorchakov
                Kateryna Senchenko Kateryna Senchenko
                Votes:
                0 Vote for this issue
                Watchers:
                1 Start watching this issue

                Dates

                  Created:
                  Updated:
                  Resolved:

                  TestRail: Runs

                    TestRail: Cases