Details
-
Epic
-
Status: Analysis Complete (View Workflow)
-
P3
-
Resolution: Unresolved
-
None
-
None
-
-
Batch Importer (Bib/Acq)
-
Folijet
-
-
47.25
-
ghx-label-7
Description
Allow for the batch upload and importing of MARC metadata, holdings, and authority records, to affect various records in FOLIO (inventory instance, inventory holdings, inventory items, MARCcat bib record, MARCcat authority record, order record, invoice record). Authority capabilities have dependency on Authority support in general (see UXPROD-787).
Frontend: mod-acqcat-loader (UIDATIMP)
Backend: ui-acqcat-loader (MODDATAIMP)
Feature from v1 roadmap: Import data: Batch Import of metadata records
Note that holdings data may be embedded in the bibliographic records or separate files.
Older post: https://discuss.folio.org/t/page-facing-up-arrow-right-zap-august-31st-meeting-batch-import-discussion/1166
Workshop #1 (12/18/2018) - see documentation on Google drive
Source: https://discuss.folio.org/t/page-facing-up-zap-importing-marc-approaches-english/1389
[A-M: replace all the following with links to the Migration loader epic and features once created]
The following requirements to become features depending on outcome of meetings in MM and SysOps and are also documented here:
(https://docs.google.com/document/d/1oXbEE48zd889lGD87dP7cF3GfuuKwllI_MDp2zGwTRg/edit):
The loader to handle initial data migration and population of FOLIO for instance, holdings, and item data must meet the following requirements:
- User needs proper roles and permissions to access needed files and programs.
- Must be able to accept bibliographic data, possibly with embedded holdings and items, in Standard MARC-21, MARCXML or already mapped JSON file. Flat files for item data must also be accepted. (Common with Batch Loader, need to be able to ingest MARCXML or binary MARC.) -
UXPROD-665 - The loader must be able to map data from MARC tags and fields correctly to the relevant fields in FOLIO inventory module. In mapping MARC/MARCXML, we should use only one transformation method, common to migration and batch loading. (Common with Batch Loader)
- When the source data is extracted from the previous system as a separate record, whether in JSON or in the MARC Format for Holdings Data (MFHD), the loader must be able to map the holdings appropriately in the FOLIO system, preserving links among holdings, instances, and items. (Common with Batch Loader-preserve links among bibs, holdings, items)
UXPROD-665,UXPROD-659,UXPROD-658 - We should only need to load records once and the loader should arrange for the data to land in all the right places: inventory module, storage module, and MARCcat.
- Administrative metadata for bib, holding, and item records must be preserved. This includes but is not limited to information such as staff-only flag, fast-add flag (for bibs or items), create date and operator, last updated date and operator, bibliographic record status, status change date and operator, notes on the different records. These data are not typically contained in the bib/holdings/item records themselves but are associated metadata about those records.
- The loaders must retain the legacy systems’ req key IDs and map them to the relevant FOLIO UUID. Depending on the legacy system, mapping may be required to link instance, holdings and item records in FOLIO based on the legacy ID numbers.
- The loader should allow the user the choice of how to create UUIDs: Either they will be transferred from the designated field in the source data and retained in that form; or a new UUID is created in FOLIO and linked back to the legacy ID and creates the corresponding UUIDs. (Common with Batch Loader-those batch operations will always want to preserve the UUIDs, UUIDs must be persistent once established.)
- If a user chooses to continue to use their source IDs, they have to be able to specify both the pattern and the starting point for the IDs. (Common with Batch Loader)
- The loaders must create a datastore for both the incoming source data and the FOLIO target data.???? Q: is this only for the inventory module or for all the places the Finventory data is relevant. (Common with Batch Loader-the full MARC record must be preserved when desired.)
- The loader must perform quickly and efficiently, so that [4,000,000 bibs] can be loaded [per hour] (Common with Batch Loader)
- The loader requires a CLI. The interface should allow the kick off or scheduling of a load process. (Common with Batch Loader)
- The loader should also provide User Interface and should allow the kick off or scheduling of a load process, view progress, see status, see errors and issues. (Common with Batch Loader)
- The loader should allow the ability to truncate the database, and control when indexes get built.
- Logging and specific error reporting is required. There should be a way to subscribe to these notification. Exception logging should have a way of linking to the source file. (Common with Batch Loader)
*Exceptions: - If the source data fails to load, the record should be written to an exceptions file where it could be corrected and reloaded.
TestRail: Results
Attachments
Issue Links
- is defined by
-
MODDATAIMP-767 Data Import MARC Modification Mapping for MARC Fixed Fields, LDR, 001
-
- Open
-
-
UXPROD-130 Parse MARC21 format data records
-
- Closed
-
-
UXPROD-145 Batch upload and import of MARC metadata, holding, and authority records (when authority is supported)
-
- Closed
-
-
UXPROD-149 Create Instance records from a MARC bibliographic record
-
- Closed
-
-
UXPROD-185 Import Orders in MARC format
-
- In progress
-
-
UXPROD-655 Import Orders in Delimited format
-
- Draft
-
-
UXPROD-656 Data Import (Batch Importer for Bib Acq) Infrastructure and Planning
-
- Closed
-
-
UXPROD-657 Import Inventory Items in Delimited format
-
- Draft
-
-
UXPROD-658 Import Inventory Items from MARC Bibs: Create
-
- Closed
-
-
UXPROD-659 Import Inventory Holdings from MARC Bibs: Create
-
- Closed
-
-
UXPROD-660 Import Inventory Holdings in Delimited format
-
- Draft
-
-
UXPROD-661 Create interactive log-summary of batchload results, initial prep
-
- Closed
-
-
UXPROD-662 Automated FTP and Scheduling for batch files
-
- Draft
-
-
UXPROD-663 Import Invoices based on data supplied in MARC Bibliographic records
-
- Draft
-
-
UXPROD-664 Batch upload and import of delimited metadata
-
- Draft
-
-
UXPROD-665 Import Inventory Instances and MARCcat bib records in MARC format
-
- Closed
-
-
UXPROD-666 Import Inventory Instances and MARC bib records in delimited files
-
- Draft
-
-
UXPROD-669 Ability to do a test load (preview), then revise before final run/commit
-
- Draft
-
-
UXPROD-674 Import Invoices in delimited format
-
- Draft
-
-
UXPROD-1038 ISBN normalization: Preparation, and Initial Development
-
- Closed
-
-
UXPROD-1376 CRUD for File Extensions
-
- Closed
-
-
UXPROD-1377 CRUD for Job Profiles
-
- Closed
-
-
UXPROD-1378 CRUD for Match Profiles
-
- Closed
-
-
UXPROD-1379 CRUD for Action Profiles
-
- Closed
-
-
UXPROD-1380 CRUD for MARC Field Mapping Profiles, part 1
-
- Closed
-
-
UXPROD-1381 CRUD for EDIFACT Invoice Field Mapping Profiles
-
- Closed
-
-
UXPROD-1382 CRUD for Delimited Data Mapping Profiles
-
- Draft
-
-
UXPROD-1383 Ability to roll back a load if problems
-
- Draft
-
-
UXPROD-1384 Data Import permissions - Additional Enhancements - DRAFT
-
- Draft
-
-
UXPROD-1394 ISBN normalization: Refinement
-
- Closed
-
-
UXPROD-1405 Data Import (Batch Importer for Bib Acq) - Misc navigation & screen issues
-
- Draft
-
-
UXPROD-1447 Instance record with an underlying MARC record. Block edit of given elements if SRS MARC is linked (Part 2 of Connecting SRS, Inventory, and MARCCat for Bibs/Instances)
-
- Closed
-
-
UXPROD-1448 Connect SRS and Inventory for MARC Holdings | Holdings record with an underlying MARC Holdings record. Block edit of Inventory Holdings MARC-related elements - Part 2
-
- Closed
-
-
UXPROD-1479 Simple UI for edit of the default MARC mapping of bibliographic records
-
- Draft
-
-
UXPROD-1575 quickMARC | Store and Display MARC Holdings records via Inventory App
-
- Closed
-
-
UXPROD-1576 Connect MARCcat and SRS for MARC Authority records
-
- Closed
-
-
UXPROD-1577 Make Inventory Instance rely on SRS MARC Bib (Part 1 of Connecting SRS, Inventory, and MARCCat for Bibs/Instances)
-
- Closed
-
-
UXPROD-1595 Connect SRS and MARCcat - Folijet work (Part 4 of Connecting SRS, Inventory, and MARCCat for Bibs/Instances)
-
- Closed
-
-
UXPROD-1596 SRS-MARCcat integration for Holdings (Part 4 of Connect MARCcat, SRS, and Inventory for MARC Holdings)
-
- Closed
-
-
UXPROD-1769 Remaining Job and Match profile work, part 1
-
- Closed
-
-
UXPROD-1805 SRS MARC-Inventory Instance relationship refinement
-
- Closed
-
-
UXPROD-1806 NFR: Data Import Pub-Sub (Event Driven) Infrastructure, part 1
-
- Closed
-
-
UXPROD-1860 Migration. Simple UI for edit of the default MARC mapping of holdings records
-
- Draft
-
-
UXPROD-2012 NFR: Data Import Pub-Sub (Event Driven) Infrastructure, part 2
-
- Closed
-
-
UXPROD-2074 Bring Action Profiles to life: prep work
-
- Closed
-
-
UXPROD-2075 MARC Bib Field Mapping Profiles Details for Inventory records
-
- Closed
-
-
UXPROD-2076 NFR: Data Import (Batch Importer for Bib Acq) Q4 2019 Technical, NFR, & Misc bug work
-
- Closed
-
-
UXPROD-2077 Remaining Job and Match profile work, part 2
-
- Closed
-
-
UXPROD-2078 SRS MARC-Inventory Instance relationship refinement, part 2
-
- Closed
-
-
UXPROD-2098 Add a way to create job profiles on-the-fly on the "choose job" screen - DRAFT
-
- Draft
-
-
UXPROD-2099 Ability to export Data Import profile details
-
- Draft
-
-
UXPROD-2152 Additional MARC-Instance relationship refinement - DRAFT
-
- Draft
-
-
UXPROD-2164 Supply default Job, Match, Action, Mapping profiles with the Data Import app
-
- Draft
-
-
UXPROD-2196 NFR: PubSub enhancements (BE)
-
- Open
-
-
UXPROD-2197 NFR: Data Import (Batch Importer for Bib Acq) Q1 2020 Technical, NFR, & Misc bug work
-
- Closed
-
-
UXPROD-2200 Create interactive log-summary of batchload results, next steps
-
- Draft
-
-
UXPROD-2207 SRS MARC-Inventory Instance relationship refinement, part 3
-
- Closed
-
-
UXPROD-2208 Remaining Job and Match profile work, part 3
-
- Closed
-
-
UXPROD-2209 Bring Action Profiles to life: next work
-
- Closed
-
-
UXPROD-2220 NFR: Data Import (Batch Importer for Bib Acq) & PubSub Q2 2020 Technical, NFR, & Misc bug work
-
- Closed
-
-
UXPROD-2273 Connect SRS and quickMARC for MARC Bibs (Folijet work)
-
- Closed
-
-
UXPROD-2281 quickMARC | Define human readable identifiers (HRIDs) and UUIDs for SRS MARC Authority records
-
- Closed
-
-
UXPROD-2282 MARC Field Mapping Profiles Details for Order records
-
- Closed
-
-
UXPROD-2286 MARC Field Mapping Profiles Details for Updating SRS MARC Bib records
-
- Closed
-
-
UXPROD-2322 Import MARC authority records - Create action (backend support only)
-
- Closed
-
-
UXPROD-2334 NFR: Data Import (Batch Importer for Bib Acq) Lower-priority Technical, NFR, & Misc bug work
-
- Draft
-
-
UXPROD-2338 Remaining Job and Match profile refinements
-
- Closed
-
-
UXPROD-2350 Support for the Delete action during Data Import
-
- Draft
-
-
UXPROD-2351 Import Inventory Holdings from MARC Bibs: Update
-
- Closed
-
-
UXPROD-2353 Import Inventory Items from MARC Bibs: Update
-
- Closed
-
-
UXPROD-2354 Import Inventory Instances from MARC Bibs: Update
-
- Closed
-
-
UXPROD-2355 Import SRS MARC Bib Records: Update
-
- Closed
-
-
UXPROD-2363 SRS Performance Improvements
-
- Closed
-
-
UXPROD-2372 NFR: Data Import accessibility bugfixes - spans multiple releases
-
- Closed
-
-
UXPROD-2446 Finish & cleanup MARC Field Mapping Profiles Details for Inventory records
-
- Closed
-
-
UXPROD-2550 MARC Bib Field Mapping Profiles Details for Inventory Item Status
-
- Closed
-
-
UXPROD-2551 NFR: Data Import (Batch Importer for Bib Acq) & PubSub Q3 2020 Technical, NFR, & Misc bug work
-
- Closed
-
-
UXPROD-2614 NFR: Data Import (Batch Importer for Bib Acq) & PubSub R1 2021 Technical, NFR, & Misc bug work
-
- Closed
-
-
UXPROD-2615 Ability to kill a long-running or errored data import
-
- Closed
-
-
UXPROD-2641 Blocked Field mapping profile updates for Inventory records
-
- Draft
-
-
UXPROD-2659 NFR: Refactor data-import flow to increase reliability
-
- Closed
-
-
UXPROD-2661 Load MARC records to create/update various record types, but do not store MARC
-
- Draft
-
-
UXPROD-2697 NFR: Create integration tests for key Data Import functionality using Karate (Kiwi)
-
- Closed
-
-
UXPROD-2701 Tech Debt: Migrate UI automated unit tests from BigTest to React/JEST - R1 2021 Iris
-
- Closed
-
-
UXPROD-2741 Import of MARC Bibs to create/update multiple holdings and items
-
- Draft
-
-
UXPROD-2742 MARC-MARC matching, part 2
-
- Blocked
-
-
UXPROD-2743 NFR: Reuse components from the data-transfer-components repository
-
- Closed
-
-
UXPROD-2753 Data Import Log - "lite version"
-
- Closed
-
-
UXPROD-2790 NFR: Make some revisions to Source Record Storage to improve performance for data migration
-
- Closed
-
-
UXPROD-2802 NFR: Handle implicit/explicit actions in job profiles
-
- Draft
-
-
UXPROD-2858 Define human readable identifiers (HRIDs) and UUIDs for SRS MARC Holdings records
-
- Closed
-
-
UXPROD-2929 NFR: Increase security of Kafka for Data Import and PubSub
-
- Closed
-
-
UXPROD-2934 Job profile-level Data import MARC field protections
-
- Draft
-
-
UXPROD-2939 Data Import Log - refine error handling
-
- Closed
-
-
UXPROD-2990 Import MARC authority records - Update action
-
- Closed
-
-
UXPROD-2998 Data Import: Add support for Importing Notes app Notes
-
- Draft
-
-
UXPROD-2999 Data Import: Add support for Importing Tags
-
- Draft
-
-
UXPROD-3000 Data Import: Add support for tokens in field mapping profiles
-
- Draft
-
-
UXPROD-3001 Data Import: Implement keyboard shortcuts
-
- Closed
-
-
UXPROD-3023 NFR: R2 2021 Juniper Data Import Stabilization and Reliability work
-
- Closed
-
-
UXPROD-3041 NFR: Data Import (Batch Importer for Bib Acq) R3 2021 Kiwi Technical, NFR, & Misc bug work
-
- Closed
-
-
UXPROD-3074 Create monitoring task for Data Import app
-
- Closed
-
-
UXPROD-3097 Refinements to Data Import MARC Field Protection
-
- Draft
-
-
UXPROD-3099 Add the ability to print MARC records from FOLIO UI
-
- In progress
-
-
UXPROD-3105 Tech Debt: Migrate UI unit automated tests from BigTest to React/JEST - R2 2021 Juniper
-
- Closed
-
-
UXPROD-3106 Tech Debt: Migrate UI automated unit tests from BigTest to React/JEST - R3 2021 Kiwi
-
- Closed
-
-
UXPROD-3120 Folijet - R3 2022 Kiwi Module releases, RMB/Stripes updates, migration scripts verification
-
- Closed
-
-
UXPROD-3121 NFR: Create integration tests for key Data Import functionality using Karate (Lotus)
-
- Closed
-
-
UXPROD-3135 NFR: R3 2021 Kiwi Data Import Stabilization and Reliability work
-
- Closed
-
-
UXPROD-3145 Remove SRS and Data Import Debris
-
- Analysis Complete
-
-
UXPROD-3146 Add if/then logic to the field mapping profiles - DRAFT
-
- Draft
-
-
UXPROD-3173 Folijet support work for Inventory Optimistic Locking (Data Import and SRS)
-
- Closed
-
-
UXPROD-3174 Analyze Folijet support work for Inventory Bound-with feature (Data Import)
-
- Blocked
-
-
UXPROD-3191 NFR: R3 2021 Kiwi Data import performance work
-
- Closed
-
-
UXPROD-3193 NFR: R3 2021 Kiwi Data import Stability/Reliability work
-
- Closed
-
-
UXPROD-3194 Assess the work for supporting the UNIMARC Bibliographic format in FOLIO
-
- Draft
-
-
UXPROD-3261 NFR: R1 2022 Lotus Data import performance work
-
- Closed
-
-
UXPROD-3262 NFR: Data Import R1 2022 Lotus Technical, NFR, & Misc work
-
- Closed
-
-
UXPROD-3263 NFR: UI end-to-end tests for Data Import (Lotus)
-
- Closed
-
-
UXPROD-3361 Prep for Importing Orders in MARC format
-
- Closed
-
-
UXPROD-3363 Data Import Log - refine error handling, part 2
-
- Draft
-
-
UXPROD-3419 NFR: UI end-to-end tests for Folijet (Morning Glory)
-
- Closed
-
-
UXPROD-3463 NFR: Data Import R1 2022 Lotus Support Bug Work
-
- Closed
-
-
UXPROD-3464 NFR: Data Import R2 2022 Morning Glory Support Bug work
-
- Closed
-
-
UXPROD-3465 NFR: Morning Glory R2 2022 Module releases, RMB/Stripes updates, migration scripts verification
-
- Closed
-
-
UXPROD-3575 NFR: Karate API integration tests for key Data Import functionality (Nolana R3 2022)
-
- Closed
-
-
UXPROD-3576 NFR: Data Import Support Bug work (Nolana R3 2022)
-
- Closed
-
-
UXPROD-3577 NFR: Module releases, RMB/Stripes updates, Migration scripts (Nolana R3 2022)
-
- Closed
-
-
UXPROD-3623 Refine Data Import Field protection
-
- Closed
-
-
UXPROD-3628 Redesign approach for handling external IDs/HRIDs for entities created by DI to allow removal of post-processing logic. - DRAFT
-
- Draft
-
-
UXPROD-3667 Change Data Import to use a Saga execution coordinator in DI processes - DRAFT
-
- Draft
-
-
UXPROD-3694 NFR: UI end-to-end tests for Folijet (Nolana R3 2022)
-
- Closed
-
-
UXPROD-3695 Additional Changes to the Data Import log - DRAFT
-
- Draft
-
-
UXPROD-3696 Refine Update individual fields functionality - DRAFT
-
- Closed
-
-
UXPROD-3717 Import Invoices in MARC format - DUPLICATE
-
- Closed
-
-
UXPROD-3726 NFR: UI end-to-end tests tech debt for Folijet (Orchid R1 2023)
-
- In progress
-
-
UXPROD-3727 NFR: Data Import Support Bug work (Orchid R1 2023)
-
- In progress
-
-
UXPROD-3728 NFR: Module releases, RMB/Stripes updates, Migration scripts (Orchid R1 2023)
-
- Open
-
-
UXPROD-3729 Planning for Import of MARC Bibs to create/update multiple holdings and items
-
- In progress
-
-
UXPROD-3782 Import of MARC Bibs to create MARC Bibs and MARC Holdings - DRAFT
-
- Draft
-
-
UXPROD-3814 Import Orders in MARC format - Refinements
-
- Analysis Complete
-
-
UXPROD-3831 Allow user to assign various job profiles to Inventory Single Record Imports
-
- In progress
-
-
UXPROD-3838 NFR: Data Import Support Bug work (Poppy R2 2023)
-
- Open
-
-
UXPROD-3839 NFR: UI end-to-end tests tech debt for Folijet (Poppy R2 2023)
-
- Open
-
-
UXPROD-3840 NFR: Data Import Technical, NFR, & Misc work (Poppy R2 2023)
-
- Open
-
-
UXPROD-3841 NFR: Module releases, RMB/Stripes updates, Migration scripts (Poppy R2 2023)
-
- Open
-
-
UXPROD-3969 Extend solution that will update Instances when default mapping rules have changed
-
- Open
-
- relates to
-
UXPROD-211 Single MARC Record Integration with OCLC
-
- Closed
-
-
UXPROD-593 Workflow Services (Back End for Workflows and More)
-
- Open
-
-
UXPROD-597 Files App
-
- Open
-
-
UXPROD-652 Metadata Record Export
-
- In progress
-
-
UXPROD-868 Bulk Edit
-
- In progress
-
-
UXPROD-3081 NFR: UI end-to-end tests for Data Import (Kiwi)
-
- Closed
-
-
FAT-27 SPIKE: Investigate if API Karate tests integrated with TestRail are applicable for data-import related flows
-
- Closed
-
-
FAT-65 ui-data-import: UI tests replacement with RTL/Jest
-
- Closed
-
-
FAT-66 ui-plugin-find-import-profile: UI tests replacement with RTL/Jest
-
- Closed
-
-
FAT-117 Data Import: Define Application Workflow for performance test - DRAFT
-
- Closed
-
-
UIDATIMP-656 Update stripes to v5
-
- Closed
-
-
UXPROD-559 Migrate Bibliographic & Holdings data
-
- Closed
-
-
UXPROD-600 Workflow App with To-Do App Integration
-
- In Review
-
-
UXPROD-850 Migration Tools
-
- Open
-
-
UXPROD-949 Rewrite Cornell's Web based LS-Tools
-
- Open
-
-
UXPROD-1220 Ability to assign Tags to Data Import profiles
-
- Closed
-