Status: Analysis Complete (View Workflow)
Affects Version/s: None
Fix Version/s: Q1 2019
Epic Name:Batch Importer (Bib/Acq)
Calculated Total Rank:0
Allow for the batch upload and importing of MARC metadata, holdings, and authority records, to affect various records in FOLIO (inventory instance, inventory holdings, inventory items, MARCcat bib record, MARCcat authority record, order record, invoice record). Authority capabilities have dependency on Authority support in general (see UXPROD-787).
Frontend: mod-acqcat-loader (UIDATIMP)
Backend: ui-acqcat-loader (MODDATAIMP)
Feature from v1 roadmap: Import data: Batch Import of metadata records
Note that holdings data may be embedded in the bibliographic records or separate files.
Older post: https://discuss.folio.org/t/page-facing-up-arrow-right-zap-august-31st-meeting-batch-import-discussion/1166
Workshop #1 (12/18/2018) - see documentation on Google drive
[A-M: replace all the following with links to the Migration loader epic and features once created]
The following requirements to become features depending on outcome of meetings in MM and SysOps and are also documented here:
The loader to handle initial data migration and population of FOLIO for instance, holdings, and item data must meet the following requirements:
- User needs proper roles and permissions to access needed files and programs.
- Must be able to accept bibliographic data, possibly with embedded holdings and items, in Standard MARC-21, MARCXML or already mapped JSON file. Flat files for item data must also be accepted. (Common with Batch Loader, need to be able to ingest MARCXML or binary MARC.) -
- The loader must be able to map data from MARC tags and fields correctly to the relevant fields in FOLIO inventory module. In mapping MARC/MARCXML, we should use only one transformation method, common to migration and batch loading. (Common with Batch Loader)
- When the source data is extracted from the previous system as a separate record, whether in JSON or in the MARC Format for Holdings Data (MFHD), the loader must be able to map the holdings appropriately in the FOLIO system, preserving links among holdings, instances, and items. (Common with Batch Loader-preserve links among bibs, holdings, items)
UXPROD-665, UXPROD-659, UXPROD-658
- We should only need to load records once and the loader should arrange for the data to land in all the right places: inventory module, storage module, and MARCcat.
- Administrative metadata for bib, holding, and item records must be preserved. This includes but is not limited to information such as staff-only flag, fast-add flag (for bibs or items), create date and operator, last updated date and operator, bibliographic record status, status change date and operator, notes on the different records. These data are not typically contained in the bib/holdings/item records themselves but are associated metadata about those records.
- The loaders must retain the legacy systems’ req key IDs and map them to the relevant FOLIO UUID. Depending on the legacy system, mapping may be required to link instance, holdings and item records in FOLIO based on the legacy ID numbers.
- The loader should allow the user the choice of how to create UUIDs: Either they will be transferred from the designated field in the source data and retained in that form; or a new UUID is created in FOLIO and linked back to the legacy ID and creates the corresponding UUIDs. (Common with Batch Loader-those batch operations will always want to preserve the UUIDs, UUIDs must be persistent once established.)
- If a user chooses to continue to use their source IDs, they have to be able to specify both the pattern and the starting point for the IDs. (Common with Batch Loader)
- The loaders must create a datastore for both the incoming source data and the FOLIO target data.???? Q: is this only for the inventory module or for all the places the Finventory data is relevant. (Common with Batch Loader-the full MARC record must be preserved when desired.)
- The loader must perform quickly and efficiently, so that [4,000,000 bibs] can be loaded [per hour] (Common with Batch Loader)
- The loader requires a CLI. The interface should allow the kick off or scheduling of a load process. (Common with Batch Loader)
- The loader should also provide User Interface and should allow the kick off or scheduling of a load process, view progress, see status, see errors and issues. (Common with Batch Loader)
- The loader should allow the ability to truncate the database, and control when indexes get built.
- Logging and specific error reporting is required. There should be a way to subscribe to these notification. Exception logging should have a way of linking to the source file. (Common with Batch Loader)
- If the source data fails to load, the record should be written to an exceptions file where it could be corrected and reloaded.