Uploaded image for project: 'mod-inventory-storage'
  1. mod-inventory-storage
  2. MODINVSTOR-614

inventory-storage crashes after grabbing all available DB connections

    XMLWordPrintable

Details

    • Bug
    • Status: Closed (View Workflow)
    • P1
    • Resolution: Done
    • 19.4.2
    • 19.4.3, 20.0.0
    • None
    • Refer to ptf-honeysuckle-module-versions.txt

    • Core: F - Sprint 101
    • 5
    • Prokopovych
    • Q3 2020 Bug Fix

    Description

      Overview:
      While performance testing mod-rtac's POST /rtac-batch API in PTF env, mod-inventory-storage grabs all available DB connections during a 30 minute JMeter test and crashes. POST /rtac-batch API internally makes a call to POST /inventory-hierarchy/items-and-holdings API, that's where it starts failing

      Steps to Reproduce:
      Run JMeter script for 30 minutes for 5 virtual users which calls POST /rtac-batch API with instanceId having 300+ holdings. I have attached the JMeter script for reference(mod_rtac_listInstanceIds.jmx)

      Log into some FOLIO environment as User X
      Log into PTF Honeysuckle env with folio user. From backend, login can be done by POST /authn/login

      Expected Results:
      POST /inventory-hierarchy/items-and-holdings should succeed with response 200

      Actual Results:
      POST /inventory-hierarchy/items-and-holdings fails with Internal Server Error 500

      Additional Information:
      From mod-inventory-storage logs, after multiple calls to invoking postInventoryHierarchyItemsAndHoldings, there is warning and error before 500 response

      1604434933007,"03 Nov 2020 20:22:13:007 WARN  NoticeResponse [] Backend notice: severity='WARNING', code='57P02', message='terminating connection because of crash of another server process', detail='The postmaster has commanded this server process to roll back the current transaction and exit, because another server process exited abnormally and possibly corrupted shared memory.', hint='In a moment you should be able to reconnect to the database and repeat your command.', position='null', internalPosition='null', internalQuery='null', where='SQL function ""get_items_and_holdings_view"" statement 1', file='postgres.c', line='2643', routine='quickdie', schema='null', table='null', column='null', dataType='null', constraint='null'"
      1604434933012,03 Nov 2020 20:22:13:012 ERROR AbstractInstanceRecordsAPI [] io.vertx.core.VertxException: closed
      

      Attached below are:

      Interested parties:
      All consumers of mod-inventory-storage

      TestRail: Results

        Attachments

          1. DB-connection-error.png
            DB-connection-error.png
            39 kB
          2. DB-connection-error-nov6.png
            DB-connection-error-nov6.png
            36 kB
          3. inventory-storage-nov-6.log
            345 kB
          4. mod_rtac_listInstanceIds.jmx
            19 kB
          5. mod-inventory-storage-db-connection-error.log
            271 kB
          6. mod-inventory-storage-ERROR.log
            111 kB
          7. mod-rtac.log
            70 kB
          8. mod-rtac-nov-6-500.log
            44 kB
          9. ptf-honeysuckle-module-versions.txt
            4 kB
          10. RDS-DB-connection-fixed.png
            RDS-DB-connection-fixed.png
            33 kB
          11. screenshot-1.png
            screenshot-1.png
            154 kB
          12. screenshot-2.png
            screenshot-2.png
            133 kB

          Issue Links

            Activity

              People

                bohdan-suprun Bohdan Suprun
                varunjavalkar Varun Javalkar
                Votes:
                0 Vote for this issue
                Watchers:
                5 Start watching this issue

                Dates

                  Created:
                  Updated:
                  Resolved:

                  TestRail: Runs

                    TestRail: Cases