Uploaded image for project: 'mod-pubsub'
  1. mod-pubsub
  2. MODPUBSUB-114

Data Import stops when trying to load a large file to folio-snapshot-load

    XMLWordPrintable

Details

    • Folijet Sprint 92, Folijet Sprint 93
    • 5
    • Folijet
    • Q2 2020

    Description

      Overview: In folio-snapshot-load, importing a large MARC file using a job profile causes the import to stop.

      Steps to Reproduce:

      1. Log into folio-snapshot-load as diku_admin
      2. Go to Settings/Data import/Field mapping profiles
      3. Create a field mapping profile with Incoming record MARC Bib and FOLIO record Instance. No additional mappings
      4. Create an action profile with action Create and FOLIO record = Instance, and link the above field mapping profile to it
      5. Create a job profile and link the above action profile to it
      6. Upload a 100MB+ MARC file (you can get some large files from here: https://drive.google.com/drive/folders/1ydR_KIdas9MI0iIhy7CLqwf_Tx8JFMTV?usp=sharing)
      7. On the choose jobs page, select the newly created job profile and run the import

      Expected Results: The import should complete

      Actual Results: It gets stuck

      See attached video, plus screenshot from yesterday. I was able to reproduce the problem on multiple days. The import completes when using the (non-pubsub) secret button (took 24 minutes for 100,000 records), but not when using a pubsub job profile

      Ideally we get back to 100,000 records. 10,000 records is acceptable for Q2, so long as we work toward 100,000 by the end of Q3. 1,000 records - not sure if this would be acceptable for the Q2 release, then aim to get back to 10K with a Q2 hotfix release. 100 records is not OK.

      TestRail: Results

        Attachments

          Issue Links

            Activity

              People

                ruslan_lavrov Ruslan Lavrov
                abreaux Ann-Marie Breaux
                Votes:
                0 Vote for this issue
                Watchers:
                5 Start watching this issue

                Dates

                  Created:
                  Updated:
                  Resolved:

                  TestRail: Runs

                    TestRail: Cases