Uploaded image for project: 'RAML Module Builder'
  1. RAML Module Builder
  2. RMB-863

Free streamGet PreparedStatement fixing Aggressive DB Memory Consumption

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Closed (View Workflow)
    • Priority: P2
    • Resolution: Done
    • Affects Version/s: 33.0.0
    • Fix Version/s: 33.1.0
    • Labels:
      None
    • Template:
      Standard Bug Write-Up Format
    • Sprint:
      CP: sprint 119, CP: sprint 123
    • Story Points:
      5
    • Development Team:
      Core: Platform

      Description

      Overview:
      When running a check-in/out test with the Juniper release for 30 minutes, the database consumes memory aggressively. Depending on the load, the DB could use up anywhere from 800MB (1 user) to 2GB of memory (20 users). Running the test for more than 3 hours would exhaust all the available memory that the DB started out with, 7GB. Investigate the cause of this aggressive memory consumption.

      Steps to Reproduce A:
      Run the check-in/out test for 30 minutes with at least 5 concurrent users and observe the DB's memory usage pattern in Cloudwatch

      Expected Results A:
      Memory consumption would be steady and does not drop much more as long as it has claimed enough memory for the current load.

      Actual Results A:
      Memory keeps on being claimed but not returned during the test duration.

      Steps to reproduce B:
      Build and run mod-configuration 5.7.0 jar:

      java -jar mod-configuration-server/target/mod-configuration-server-fat.jar
      

      POST /_/tenant to create schema for diku and run GET requests against the empty configuration table:

      curl -w"\n" -s -H "Content-type: application/json" -H "X-Okapi-Tenant: diku" -d '{}' http://localhost:8081/_/tenant
      for i in {1..99999} then; do curl -s -H "X-Okapi-Tenant: diku" http://localhost:8081/configurations/entries > /dev/null; done
      

      There is no record and each GET returns an empty set.
      Run pg_top (or ps -F ax) to watch the memory consumption of the database process of the connections of mod-configuration.

      Expected result B:
      The database memory usage stays below 20 MB per connection.

      Actual result B:
      The database memory usage for one of the four mod-configuration connections increases by 90 MB per minute.

      Additional Information:
      See https://wiki.folio.org/pages/viewpage.action?pageId=65116448 for more details.

      The memory issue is caused by RMB's streamGet that doesn't close PreparedStatement after use. The bug has been introduced by https://github.com/folio-org/raml-module-builder/commit/d38030b9d8929b0d497bd48167fffd05c560bc10 and is in RMB >= 33.0.0.

      streamGet is used by only 4 modules (GitHub search):

       After RMB has been released with this fix these 4 modules should upgrade RMB and release a new version.

        Juniper w/o fix (HotFix#2) Juniper with fix (HotFix#3) Kiwi w/o fix Kiwi with fix
      mod-configuration 5.7.0 5.7.1 MODCONF-94 5.7.1
      mod-inventory-storage 21.0.6 21.0.8 MODINVSTOR-795 22.0.0
      mod-permissions 5.14.0 5.14.2 MODPERMS-155 5.14.2
      mod-users 18.0.0 18.0.1 MODUSERS-273 18.1.0 18.1.1

      Interested parties:
      Julian Ladisch Jakub Skoczen

        TestRail: Results

          Attachments

            Issue Links

              Activity

                People

                Assignee:
                julianladisch Julian Ladisch
                Reporter:
                mtraneis Martin Tran
                Votes:
                0 Vote for this issue
                Watchers:
                6 Start watching this issue

                  Dates

                  Created:
                  Updated:
                  Resolved:

                    TestRail: Runs

                      TestRail: Cases