Uploaded image for project: 'mod-data-export'
  1. mod-data-export
  2. MDEXP-394

Memory leaks in mod-data-export

    XMLWordPrintable

Details

    • Story
    • Status: Closed (View Workflow)
    • P2
    • Resolution: Done
    • None
    • 4.2.0
    • None
    • Firebird Sprint 119, Firebird Sprint 120, Firebird Sprint 121, Firebird Sprint 122, Firebird Sprint 123, Concorde - Sprint 117, Concorde - Sprint 118
    • 5
    • Firebird
    • R3 2021

    Description

      Purpose/Overview:

      During data export testing of 100K instances, memory leaks were found.

       

      Memory allocated to mod-data-export - 512MB

       

      This was tested in PTF env against following Iris module versions:
      mod-data-export-4.0.1
      mod-source-record-storage-5.0.0
      mod-source-record-manager-3.0.2
      okapi-4.7.2

       

      Approach:
      Use custom mapping profile with items/holdings fields included
      Extract 100K instance-ids whose source=MARC. Below is the SQL query:

      SELECT id FROM fs09000000_mod_inventory_storage.instance where jsonb->>'source'='MARC' LIMIT 100000; 

      Save ids in instance-record-ids.csv Use this file to run the data export job.

       

      Actual results:

      Memory leaks were found in objects:

      • sun.util.calendar.ZoneInfoFile
      • java.util.zip.ZipFile$Source
      • jdk.internal.loader.ClassLoaders$AppClassLoader
      • io.vertx.core.impl.VertxThread export-thread-worker-0

       

      Please see attached LeakSuspects pdf and biggest object.

       

      Expected results:

      No memory leaks found

       

      Acceptance criteria:

      • The above objects should not leave behind any traces of memory leaks

      TestRail: Results

        Attachments

          1. DE-log-2.html
            4.29 MB
          2. export custom mp after 10h.png
            export custom mp after 10h.png
            78 kB
          3. export custom mp after 22h.png
            export custom mp after 22h.png
            78 kB
          4. export custom mp after 3h.png
            export custom mp after 3h.png
            79 kB
          5. export custom mp after 5h.png
            export custom mp after 5h.png
            62 kB
          6. image-2021-07-31-21-35-53-021.png
            image-2021-07-31-21-35-53-021.png
            436 kB
          7. image-2021-07-31-21-53-14-636.png
            image-2021-07-31-21-53-14-636.png
            769 kB
          8. image-2021-08-23-10-30-18-230.png
            image-2021-08-23-10-30-18-230.png
            10 kB
          9. image-2021-08-31-10-09-26-702.png
            image-2021-08-31-10-09-26-702.png
            578 kB
          10. instances-ids.csv
            3.62 MB
          11. java saw custom mp 10h.PNG
            java saw custom mp 10h.PNG
            106 kB
          12. java saw custom mp 22h.PNG
            java saw custom mp 22h.PNG
            106 kB
          13. java saw custom mp 26h.PNG
            java saw custom mp 26h.PNG
            49 kB
          14. java saw custom mp 3h.PNG
            java saw custom mp 3h.PNG
            43 kB
          15. java saw custom mp 5h.PNG
            java saw custom mp 5h.PNG
            100 kB
          16. large-logs.png
            large-logs.png
            159 kB
          17. LeakSuspects-mod-data-export.pdf
            148 kB
          18. memory_after.png
            memory_after.png
            201 kB
          19. memory-leaks-biggest-objects.png
            memory-leaks-biggest-objects.png
            522 kB
          20. Memory-utilization-graph.png
            Memory-utilization-graph.png
            81 kB
          21. mod-data-export-Leak-Suspects.pdf
            136 kB
          22. mod-data-export-new-changes.log
            524 kB
          23. module startup memory analysis.png
            module startup memory analysis.png
            52 kB
          24. problem suspects custom mp after 26h.PNG
            problem suspects custom mp after 26h.PNG
            61 kB
          25. screenshot-1.png
            screenshot-1.png
            479 kB
          26. screenshot-2.png
            screenshot-2.png
            1.78 MB
          27. Screen Shot 2021-08-22 at 9.29.38 PM.png
            Screen Shot 2021-08-22 at 9.29.38 PM.png
            610 kB
          28. Screenshot 2021-09-14 at 15.04.35.png
            Screenshot 2021-09-14 at 15.04.35.png
            519 kB
          29. Screenshot 2021-09-14 at 15.05.18.png
            Screenshot 2021-09-14 at 15.05.18.png
            96 kB
          30. screenshot-3.png
            screenshot-3.png
            425 kB
          31. screenshot-4.png
            screenshot-4.png
            542 kB
          32. screenshot-5.png
            screenshot-5.png
            523 kB
          33. screenshot-6.png
            screenshot-6.png
            1.06 MB
          34. screenshot-7.png
            screenshot-7.png
            1.35 MB
          35. screenshot-8.png
            screenshot-8.png
            565 kB
          36. screenshot-9.png
            screenshot-9.png
            446 kB

          Issue Links

            Activity

              People

                khandramai Viachaslau Khandramai
                varunjavalkar Varun Javalkar
                Votes:
                0 Vote for this issue
                Watchers:
                4 Start watching this issue

                Dates

                  Created:
                  Updated:
                  Resolved:

                  TestRail: Runs

                    TestRail: Cases