Uploaded image for project: 'UX Product'
  1. UX Product
  2. UXPROD-665

Import Inventory Instances and MARCcat bib records in MARC format

    XMLWordPrintable

    Details

    • Template:
    • Epic Link:
    • Analysis Estimate:
      Large < 10 days
    • Front End Estimate:
      Small < 3 days
    • Back End Estimate:
      XXL < 30 days
    • Confidence factor:
      Low
    • Estimation Notes and Assumptions:
      Hide
      Wayne and Shale (and MARCcat) may already have many of the bits and pieces for this already, but I'm not sure how much we got and how much is missing.

      I'm assuming analysis and implementation work to be done in the area of batch loading on top of existing records.

      To the extend this is to be user configurable, the estimate assumes this is covered by some of the other UXPROD issues.
      Show
      Wayne and Shale (and MARCcat) may already have many of the bits and pieces for this already, but I'm not sure how much we got and how much is missing. I'm assuming analysis and implementation work to be done in the area of batch loading on top of existing records. To the extend this is to be user configurable, the estimate assumes this is covered by some of the other UXPROD issues.
    • Development Team:
      Folijet
    • Calculated Total Rank:
      130
    • Rank: Chalmers (Impl Aut 2019):
      R1
    • Rank: Chicago (MVP Sum 2020):
      R1
    • Rank: Cornell (Full Sum 2021):
      R1
    • Rank: Duke (Full Sum 2021):
      R1
    • Rank: 5Colleges (Full Jul 2021):
      R1
    • Rank: FLO (MVP Sum 2020):
      R1
    • Rank: GBV (MVP Sum 2020):
      R1
    • Rank: hbz (TBD):
      R1
    • Rank: Lehigh (MVP Summer 2020):
      R1
    • Rank: Leipzig (Full TBD):
      R1
    • Rank: Leipzig (ERM Aut 2019):
      R1
    • Rank: MO State (MVP June 2020):
      R1
    • Rank: TAMU (MVP Jan 2021):
      R1
    • Rank: U of AL (MVP Oct 2020):
      R1

      Description

      Q2 2019 Data Import Priority 1 of 8

      Note: While UI input is preferred, this will only cover command line input to start, and using the standard MARC data mapper

      Requirements:

      1. CLI ability to load MARC bibs to SRS (and some instructions)
      2. For now, use the existing MARC mapper for Instance. In UXPROD-1577, will update that mapping to the newer default MARC bib-to-Instance mapping
      3. Sort out the 001/003/035/999 handling for newly created SRS records?
        • Move non-FOLIO 001 to 035 $a field (only if unique 035)
        • Move non-FOLIO 003 to prefix on 035 $a field? e.g. 001 12345 and 003 OCoLC becomes 035 (OCoLC)12345 [A-M doublechecking with DI workgroup]
        • Create 001 using the sequential HRID generator
        • Create 003 using the tenant default (new MARC settings area, with tenant-level 003 and subfield preferences)
        • Create 999 field with $s [SRS UUID] (will add 999 $i with Instance UUID and 999 $m with MARCcat UUID when they are created), so complete 999 field will look like: 999 $i [Instance UUID] $m [MARCcat UUID] $s [SRS UUID] (see MODSOURMAN-95 and MODSOURMAN-96)
      4. Figure out what happens with the old mod-data-loader - retire? repurpose?

      Per Harry, aim for at least these performance metrics:

      • 100 records/second
      • or 1,000 records/second on empty system - turn off indexing, load, and then reindex

      May need to review mod-data-loader and decide if it is replaced by the data-import-loader, or if the mod-data-loader mapping just needs updating or what

      Note that MARCcat will not happen until SRS Integration with MARCcat (UXPROD-1595)

        TestRail: Results

          Attachments

            Issue Links

            defines

            Epic - Created by Jira Software - do not edit or delete. Issue type for a big user story that needs to be broken down. UXPROD-47 Batch Importer (Bib/Acq)

            • P3 - low priority level, item will be considered for inclusion in the next dev cycle
            • Analysis Complete
            is defined by

            Task - A task that needs to be done. MODDATAIMP-105 SPIKE: Event payload container

            • P3 - low priority level, item will be considered for inclusion in the next dev cycle
            • Closed

            Task - A task that needs to be done. MODDATAIMP-106 SPIKE: Job processing engine

            • P3 - low priority level, item will be considered for inclusion in the next dev cycle
            • Closed

            Task - A task that needs to be done. MODDATAIMP-107 SPIKE: Create event handler

            • P3 - low priority level, item will be considered for inclusion in the next dev cycle
            • Closed

            Task - A task that needs to be done. MODDATAIMP-110 SPIKE: Find "MARC -> Inventory Instance" mapping source code.

            • P3 - low priority level, item will be considered for inclusion in the next dev cycle
            • Closed

            Task - A task that needs to be done. MODDATAIMP-111 Set status for upload definition as non draft once the file processing has started

            • P3 - low priority level, item will be considered for inclusion in the next dev cycle
            • Closed

            Umbrella - MODDATAIMP-112 Data Import to Inventory integration for Chalmers PoC

            • P3 - low priority level, item will be considered for inclusion in the next dev cycle
            • Closed

            Task - A task that needs to be done. MODDATAIMP-113 SPIKE: Investigate Inventory API (mod-inventory) to perform CRUD for Instances

            • P3 - low priority level, item will be considered for inclusion in the next dev cycle
            • Closed

            Task - A task that needs to be done. MODDATAIMP-114 SPIKE: Check Data Import performance.

            • P3 - low priority level, item will be considered for inclusion in the next dev cycle
            • Closed

            Task - A task that needs to be done. MODDATAIMP-115 SPIKE: Check Inventory performance.

            • P3 - low priority level, item will be considered for inclusion in the next dev cycle
            • Closed

            Task - A task that needs to be done. MODDATAIMP-116 Update fileDefinition status on upload fail

            • P3 - low priority level, item will be considered for inclusion in the next dev cycle
            • Closed

            Task - A task that needs to be done. MODDATAIMP-117 Finalize PoC with event approach and add examples of usage and integrations

            • P3 - low priority level, item will be considered for inclusion in the next dev cycle
            • Closed

            Story - Created by Jira Software - do not edit or delete. Issue type for a user story. MODDATAIMP-126 Add readers for all files that can contain MARC data

            • P3 - low priority level, item will be considered for inclusion in the next dev cycle
            • Closed

            Sub-task - The sub-task of the issue MODDICONV-11 Populate sample data

            • P3 - low priority level, item will be considered for inclusion in the next dev cycle
            • Closed

            Story - Created by Jira Software - do not edit or delete. Issue type for a user story. MODSOURCE-40 Inventory instance’s data structure should be extended to hold a reference to a MARC record which has affected this instance.

            • P3 - low priority level, item will be considered for inclusion in the next dev cycle
            • Closed

            Story - Created by Jira Software - do not edit or delete. Issue type for a user story. MODSOURMAN-93 Design and develop a bespoke processor to handle parsed MARC records, create instances and store them in the mod-inventory-storage

            • P3 - low priority level, item will be considered for inclusion in the next dev cycle
            • Closed

            Story - Created by Jira Software - do not edit or delete. Issue type for a user story. MODSOURMAN-94 Integrate extracted logic from data-loader into the ChangeManager

            • P3 - low priority level, item will be considered for inclusion in the next dev cycle
            • Closed

            Task - A task that needs to be done. MODSOURMAN-95 SPIKE: Figure out 999 Instance and SRS UUID handling

            • P2 - normal priority level, must be included in the current development cycle
            • Closed

            Story - Created by Jira Software - do not edit or delete. Issue type for a user story. MODSOURMAN-99 MARC 999 field: Put SRS UUID

            • P3 - low priority level, item will be considered for inclusion in the next dev cycle
            • Closed

            Story - Created by Jira Software - do not edit or delete. Issue type for a user story. UIDATIMP-185 Create temporary MARC Bib Load option on Choose jobs pane

            • P3 - low priority level, item will be considered for inclusion in the next dev cycle
            • Closed

            Story - Created by Jira Software - do not edit or delete. Issue type for a user story. UIDATIMP-186 Landing page changes when temporary MARC Bib Load option button pushed

            • P3 - low priority level, item will be considered for inclusion in the next dev cycle
            • Closed

            Story - Created by Jira Software - do not edit or delete. Issue type for a user story. UIDATIMP-187 Add Log button to log entries in second pane of landing page when temporary MARC Bib Load option button pushed

            • P3 - low priority level, item will be considered for inclusion in the next dev cycle
            • Closed

            Sub-task - The sub-task of the issue UIDATIMP-198 Handle broken tests issue caused by stripes-components update

            • P3 - low priority level, item will be considered for inclusion in the next dev cycle
            • Closed

            Bug - A problem which impairs or prevents the functions of the product. UIDATIMP-204 MARC Bib load results in invalid date and no log

            • P2 - normal priority level, must be included in the current development cycle
            • Closed

            Story - Created by Jira Software - do not edit or delete. Issue type for a user story. UIDATIMP-209 Show all records in log including error ones

            • P2 - normal priority level, must be included in the current development cycle
            • Closed
            relates to

            New Feature - New feature requests UXPROD-1479 Simple UI for edit of the default MARC mapping of bibliographic records

            • P3 - low priority level, item will be considered for inclusion in the next dev cycle
            • Open

            New Feature - New feature requests UXPROD-1805 SRS MARC-Inventory Instance relationship refinement

            • P2 - normal priority level, must be included in the current development cycle
            • Closed

            New Feature - New feature requests UXPROD-2078 SRS MARC-Inventory Instance relationship refinement, part 2

            • P2 - normal priority level, must be included in the current development cycle
            • Closed

            New Feature - New feature requests UXPROD-2207 SRS MARC-Inventory Instance relationship refinement, part 3

            • P3 - low priority level, item will be considered for inclusion in the next dev cycle
            • Closed

            Bug - A problem which impairs or prevents the functions of the product. MODSOURMAN-115 After Instances are created the field 999 does not appear in SRS

            • P3 - low priority level, item will be considered for inclusion in the next dev cycle
            • Closed

            New Feature - New feature requests UXPROD-191 Manage approval plan purchases in FOLIO: MARC Load

            • P3 - low priority level, item will be considered for inclusion in the next dev cycle
            • Open

            New Feature - New feature requests UXPROD-1397 Generating a MARC bibliographic record from an Inventory Instance record

            • P1 - highest priority item, drop everything else before this is resolved, reserved for critical bugfixes
            • Closed

            New Feature - New feature requests UXPROD-1499 Accept files in JSON for intital migration for input and output

            • P3 - low priority level, item will be considered for inclusion in the next dev cycle
            • Closed

            New Feature - New feature requests UXPROD-1806 NFR: Data Import Pub-Sub (Event Driven) Infrastructure, part 1

            • P2 - normal priority level, must be included in the current development cycle
            • Closed

            New Feature - New feature requests UXPROD-2012 NFR: Data Import Pub-Sub (Event Driven) Infrastructure, part 2

            • P2 - normal priority level, must be included in the current development cycle
            • Closed

            New Feature - New feature requests UXPROD-2115 Define human readable identifiers (HRIDs) for SRS MARC Bib records

            • P2 - normal priority level, must be included in the current development cycle
            • Closed

              Activity

                People

                Assignee:
                abreaux Ann-Marie Breaux
                Reporter:
                abreaux Ann-Marie Breaux
                Analysis Estimator:
                Niels Erik Gilvad Nielsen Niels Erik Gilvad Nielsen
                Front End Estimator:
                Viktor Soroka Viktor Soroka
                Back End Estimator:
                Taras Spashchenko Taras Spashchenko
                Votes:
                0 Vote for this issue
                Watchers:
                7 Start watching this issue

                  Dates

                  Created:
                  Updated:
                  Resolved:

                    TestRail: Runs

                      TestRail: Cases