Uploaded image for project: 'mod-source-record-storage'
  1. mod-source-record-storage

SPIKE: Figure out the details of migrating MARC bib records from Inv to SRS



    • Template:
    • Sprint:
      EPAM BatchLoader Sprint 5, EPAM BatchLoader Sprint 6, EPAM BatchLoader Sprint 7, EPAM BatchLoader Sprint 8
    • Story Points:
    • Development Team:


      Currently MARC bib records are being stored in Inventory, and linked to related Inventory Instances. In the future, they will be stored in Source Record Storage (SRS). Once SRS is ready to receive them, then we need to have an understanding of what work will be involved in changing the existing configuration to the new SRS configuration.

      Section 3 in this diagram lays out the high level architecture of Data Import, with links to technical diagrams, including SRS:

      Stakeholders (set up as watchers)
      Inventory: Charlotte Whitt, Niels Erik Gilvad Nielsen, Wayne Schneider Marc Johnson
      Data Import: Ann-Marie, Taras, Alex, Tetyana, anyone else?
      OAI-PMH: Craig McNally, Harry Kaplanian
      Misc: Jakub Skoczen, Vince Bareau
      MARCcat: Tiziana Possemato, Christian Chiama, Zeno Tajoli, Mirko Fonzo
      EBSCO dev: Mark Veksler
      Anyone else? Julian Ladisch

      Details to figure out

      1. When will SRS be ready to accept data?
      2. Does Source Record Manager (SRM) need to be ready as well? If so, when will it be ready?
      3. Questions/dev review of the SRS/SRM APIs that will be used to link other apps to SRS data [Taras Spashchenko will add details]
      4. What is involved in removing existing MARC Bibs from Inventory?
        1. Per Charlotte, MARC Bibs are loaded to the various FOLIO servers, into mod-inventory-storage, every morning (or every time a FOLIO instance is built). Removing the MARC bibs from mod-inventory-storage is thus achieved by stop loading them.
      5. How will current linkage between Inv MARC and Inv Instance need to change?
        1. Per Charlotte, all Inventory needs is for the data loader to continue to set the property Instance.sourceRecordFormat when it loads a MARC source and creates an Instance in Inventory from SRS. UI-inventory needs this to determine whether to display a "Display source" button on the instance in the UI. See: MODINVSTOR-162, and UIIN-93.
      6. Are there any other apps or components interacting with Inv MARC Bibs currently? e.g. OAI-PMH?
        1. Per Craig, OAI-PMH will be retrieving records from SRS. The two teams are working on integrating.already.
      7. There's an inventory setting to suppress output for discovery for all 3 record types (instances, holdings, items). When MARC moves to SRS (instances, then holdings), how will that be handled?
      8. What is the best timing for the change?
        1. Per Charlotte, anytime, but it must be coordinated with
          1. Wayne Schneider who loads the data
          2. Niels Erik Gilvad Nielsen who needs to change the link to the 'View source' button on Instance detail view
      9. Is MARCcat storage built and ready to accept records yet?
        1. Yes, but not yet linked to Inventory or SRS.
      10. Will we need an interim solution and then refactor when MARCcat storage ready to accept records?
      11. How will this affect folio-testing, folio-snapshot, folio-snapshot-stable?
        1. Per Taras, the environments are created from scratch each time, so will need to reload the data to SRS, Inv, and MARcat
        2. Once we have the ID environments working, perhaps try demo.folio.org next, since there are lots of MARC records there?
      12. What happens with the sample data each day when the environments are rebuilt? Do we have to re-load to SRS, or can it just stay there and not have to be re-loaded each time?
        1. Per Taras, the answer is yes, because as far as I know environments are created from scratch. but we need more details for each environment.
      13. Which dev team(s) build which portion(s)?
      14. Build wiki page with details, so that all related devs and POs can track decisions and progress? Or just track here in this spike and whatever Jira stories come from it?
      15. How will we test?

      From Taras in the comments section:
      The first steps I propose to take to start off:
      1. SRS API documentation is available at https://s3.amazonaws.com/foliodocs/api/mod-source-record-storage/source-record-storage.html#
      to get MARC records use GET request to the /source-storage/result endpoint https://s3.amazonaws.com/foliodocs/api/mod-source-record-storage/source-record-storage.html#source_storage_result_get
      2. SRS module for now does not contain MARC data. There is a story in Folijet backlog https://issues.folio.org/browse/MODSOURCE-5 “Develop a CLI tool to allow initial filling with data for testing purposes.” To address this.
      3. Find the person, who is responsible for the daily MARC loading into mod-inventory storage. Add the same jobs for SRS using SRS-CLI tool.
      4. Change mod-inventory implementation for ExternalStorageCollections classes to read MARC records from SRS module instead of mod-inventory-storage. Maybe it make sense to preserve existing implementation as a fallback and make it configurable.

        TestRail: Results


            Issue Links



                Taras_Spashchenko Taras Spashchenko
                abreaux Ann-Marie Breaux
                0 Vote for this issue
                16 Start watching this issue



                    TestRail: Runs

                      TestRail: Cases