Uploaded image for project: 'RAML Module Builder'
  1. RAML Module Builder
  2. RMB-863

Free streamGet PreparedStatement fixing Aggressive DB Memory Consumption

    XMLWordPrintable

Details

    • Bug
    • Status: Closed (View Workflow)
    • P2
    • Resolution: Done
    • 33.0.0
    • 33.1.0
    • None
    • CP: sprint 119, CP: sprint 123
    • 5
    • Core: Platform

    Description

      Overview:
      When running a check-in/out test with the Juniper release for 30 minutes, the database consumes memory aggressively. Depending on the load, the DB could use up anywhere from 800MB (1 user) to 2GB of memory (20 users). Running the test for more than 3 hours would exhaust all the available memory that the DB started out with, 7GB. Investigate the cause of this aggressive memory consumption.

      Steps to Reproduce A:
      Run the check-in/out test for 30 minutes with at least 5 concurrent users and observe the DB's memory usage pattern in Cloudwatch

      Expected Results A:
      Memory consumption would be steady and does not drop much more as long as it has claimed enough memory for the current load.

      Actual Results A:
      Memory keeps on being claimed but not returned during the test duration.

      Steps to reproduce B:
      Build and run mod-configuration 5.7.0 jar:

      java -jar mod-configuration-server/target/mod-configuration-server-fat.jar
      

      POST /_/tenant to create schema for diku and run GET requests against the empty configuration table:

      curl -w"\n" -s -H "Content-type: application/json" -H "X-Okapi-Tenant: diku" -d '{}' http://localhost:8081/_/tenant
      for i in {1..99999} then; do curl -s -H "X-Okapi-Tenant: diku" http://localhost:8081/configurations/entries > /dev/null; done
      

      There is no record and each GET returns an empty set.
      Run pg_top (or ps -F ax) to watch the memory consumption of the database process of the connections of mod-configuration.

      Expected result B:
      The database memory usage stays below 20 MB per connection.

      Actual result B:
      The database memory usage for one of the four mod-configuration connections increases by 90 MB per minute.

      Additional Information:
      See https://wiki.folio.org/pages/viewpage.action?pageId=65116448 for more details.

      The memory issue is caused by RMB's streamGet that doesn't close PreparedStatement after use. The bug has been introduced by https://github.com/folio-org/raml-module-builder/commit/d38030b9d8929b0d497bd48167fffd05c560bc10 and is in RMB >= 33.0.0.

      streamGet is used by only 4 modules (GitHub search):

       After RMB has been released with this fix these 4 modules should upgrade RMB and release a new version.

        Juniper w/o fix (HotFix#2) Juniper with fix (HotFix#3) Kiwi w/o fix Kiwi with fix
      mod-configuration 5.7.0 5.7.1 MODCONF-94 5.7.1
      mod-inventory-storage 21.0.6 21.0.8 MODINVSTOR-795 22.0.0
      mod-permissions 5.14.0 5.14.2 MODPERMS-155 5.14.2
      mod-users 18.0.0 18.0.1 MODUSERS-273 18.1.0 18.1.1

      Interested parties:
      julianladisch jakub

      TestRail: Results

        Attachments

          Issue Links

            Activity

              People

                julianladisch Julian Ladisch
                mtraneis Martin Tran
                Votes:
                0 Vote for this issue
                Watchers:
                5 Start watching this issue

                Dates

                  Created:
                  Updated:
                  Resolved:

                  TestRail: Runs

                    TestRail: Cases