Uploaded image for project: 'mod-data-export'
  1. mod-data-export
  2. MDEXP-393

Data export job fails to export if location id is invalid

    XMLWordPrintable

Details

    • Bug
    • Status: Closed (View Workflow)
    • P3
    • Resolution: Done
    • None
    • None
    • None
    • Firebird Sprint 121
    • 3
    • Firebird

    Description

      Purpose/Overview:

      Run data export job which will export 100K Instances, Holdings, and Items to mrc file. Data export job fails when an instance has an invalid location id or when the location id is null.

       

      This was tested in PTF env against following Iris module versions:
      mod-data-export-4.0.1
      mod-source-record-storage-5.0.0
      mod-source-record-manager-3.0.2
      okapi-4.7.2
       
      Requirements/Scope:

      Approach:

      Extract 100K instance-ids whose source=MARC. Below is the SQL query:

      SELECT id FROM fs09000000_mod_inventory_storage.instance where jsonb->>'source'='MARC' LIMIT 100000; 

      Save ids in instance-record-ids.csv Use this file to run the data export job.

       

      Actual Result:

      Data export job fails after exporting 49350 records. The following error is found in the CloudWatch log:

      2021-04-19T19:15:33,485 ERROR [export-thread-worker-1   ] TranslationsFunctionHolder Location is not found by the given id: null 

      Attached log below.

      Error is seen in UI console log:

      {
          "id" : "745062aa-4ed2-41d9-86f7-027597e04f17",
          "hrId" : 7752,
          "exportedFiles" : [ {
            "fileId" : "545b380a-fce5-420d-b738-52370c125c3f",
            "fileName" : "instance-record-ids-7752.mrc"
          } ],
          "jobProfileId" : "cd4abceb-e145-4ecf-b93e-fa672cc4b125",
          "jobProfileName" : "source record storage, holdings and items",
          "progress" : {
            "exported" : 49350,
            "failed" : 0,
            "total" : 100000
          },
          "completedDate" : "2021-04-19T19:15:33.485+00:00",
          "lastUpdatedDate" : "2021-04-19T19:15:33.485+00:00",
          "startedDate" : "2021-04-19T17:44:26.251+00:00",
          "runBy" : {
            "firstName" : "folio",
            "lastName" : "folio"
          },
          "status" : "FAIL"
        } 

      Expected Result:

      Data export should export all 100K records successfully. Code should handle invalid location id.

      Acceptance criteria:
      *The exception should be handled gracefully

      • The affected record should not be exported
      • The affected record should be logged so that the user can clean the data.
      • If the error occurs on the Holdings record level the error should contain elements as described in MDEXP-259
      • If the error occurs on the item record level the error should contain elements as described in MDEXP-260
      • The error should read "Invalid or missing location: " followed by the invalid value or null.

      TestRail: Results

        Attachments

          1. image-2021-04-20-12-59-07-418.png
            image-2021-04-20-12-59-07-418.png
            82 kB
          2. job execution 1 instance.png
            job execution 1 instance.png
            72 kB
          3. job execution 2 insatnces, items with invalid locations.PNG
            job execution 2 insatnces, items with invalid locations.PNG
            117 kB
          4. job execution 2 instances.png
            job execution 2 instances.png
            47 kB
          5. mod-data-export.log
            58 kB
          6. report.png
            report.png
            130 kB
          7. screenshot-1.png
            screenshot-1.png
            81 kB
          8. screenshot-2.png
            screenshot-2.png
            142 kB
          9. screenshot-3.png
            screenshot-3.png
            47 kB
          10. screenshot-4.png
            screenshot-4.png
            67 kB
          11. screenshot-5.png
            screenshot-5.png
            157 kB

          Issue Links

            Activity

              People

                TotoroDev Illia Daliek
                varunjavalkar Varun Javalkar
                Votes:
                0 Vote for this issue
                Watchers:
                4 Start watching this issue

                Dates

                  Created:
                  Updated:
                  Resolved:

                  TestRail: Runs

                    TestRail: Cases