Details
-
Bug
-
Status: Closed (View Workflow)
-
P2
-
Resolution: Done
-
33.0.0
-
None
-
CP: sprint 119, CP: sprint 123
-
5
-
Core: Platform
Description
Overview:
When running a check-in/out test with the Juniper release for 30 minutes, the database consumes memory aggressively. Depending on the load, the DB could use up anywhere from 800MB (1 user) to 2GB of memory (20 users). Running the test for more than 3 hours would exhaust all the available memory that the DB started out with, 7GB.
Investigate the cause of this aggressive memory consumption.
Steps to Reproduce A:
Run the check-in/out test for 30 minutes with at least 5 concurrent users and observe the DB's memory usage pattern in Cloudwatch
Expected Results A:
Memory consumption would be steady and does not drop much more as long as it has claimed enough memory for the current load.
Actual Results A:
Memory keeps on being claimed but not returned during the test duration.
Steps to reproduce B:
Build and run mod-configuration 5.7.0 jar:
java -jar mod-configuration-server/target/mod-configuration-server-fat.jar
POST /_/tenant to create schema for diku and run GET requests against the empty configuration table:
curl -w"\n" -s -H "Content-type: application/json" -H "X-Okapi-Tenant: diku" -d '{}' http://localhost:8081/_/tenant for i in {1..99999} then; do curl -s -H "X-Okapi-Tenant: diku" http://localhost:8081/configurations/entries > /dev/null; done
There is no record and each GET returns an empty set.
Run pg_top (or ps -F ax) to watch the memory consumption of the database process of the connections of mod-configuration.
Expected result B:
The database memory usage stays below 20 MB per connection.
Actual result B:
The database memory usage for one of the four mod-configuration connections increases by 90 MB per minute.
Additional Information:
See https://wiki.folio.org/pages/viewpage.action?pageId=65116448 for more details.
The memory issue is caused by RMB's streamGet that doesn't close PreparedStatement after use. The bug has been introduced by https://github.com/folio-org/raml-module-builder/commit/d38030b9d8929b0d497bd48167fffd05c560bc10 and is in RMB >= 33.0.0.
streamGet is used by only 4 modules (GitHub search):
- https://github.com/folio-org/mod-configuration/blob/v5.7.0/mod-configuration-server/src/main/java/org/folio/rest/impl/ConfigAPI.java#L50
- https://github.com/folio-org/mod-inventory-storage/blob/v21.0.6/src/main/java/org/folio/rest/impl/InstanceStorageAPI.java#L93
- https://github.com/folio-org/mod-permissions/blob/v5.14.0/src/main/java/org/folio/rest/impl/PermsAPI.java#L120
- https://github.com/folio-org/mod-users/blob/v18.0.0/src/main/java/org/folio/rest/impl/UsersAPI.java#L149
After RMB has been released with this fix these 4 modules should upgrade RMB and release a new version.
Juniper w/o fix (HotFix#2) | Juniper with fix (HotFix#3) | Kiwi w/o fix | Kiwi with fix | |
---|---|---|---|---|
mod-configuration | 5.7.0 | 5.7.1 |
— | 5.7.1 |
mod-inventory-storage | 21.0.6 | 21.0.8 |
— | 22.0.0 |
mod-permissions | 5.14.0 | 5.14.2 |
— | 5.14.2 |
mod-users | 18.0.0 | 18.0.1 |
18.1.0 | 18.1.1 |
Interested parties:
julianladisch jakub
TestRail: Results
Attachments
Issue Links
- blocks
-
MODCONF-87 MODCONF (mod-configuration) release for 2021 R3
-
- Closed
-
-
MODCONF-92 Update RMB to 33.1.1 and Vert.x to 4.1.4
-
- Closed
-
-
MODINVSTOR-795 Backport database memory fix to Juniper (RMB-863, RMB v33.1.1)
-
- Closed
-
-
MODPERMS-151 MODPERMS (mod-permissions) release for 2021 R3
-
- Closed
-
-
MODUSERS-270 MODUSERS (mod-users) release for 2021 R3
-
- Closed
-
-
RMB-870 RMB (raml-module-builder) release for 2021 R3
-
- Closed
-
- is blocked by
-
MODCONF-86 RMB 33.0.3, Vert.x 4.1.2
-
- Closed
-
-
RMB-867 Release RMB 33.0.3
-
- Closed
-
- relates to
-
MODUSERS-271 Release v18.1.1 for Kiwi R3 2021
-
- Closed
-
-
RMB-724 totalRecords=none/auto
-
- Closed
-
-
MODCONF-94 Backport database memory fix to Juniper (RMB-863, RMB v33.1.1)
-
- Closed
-
-
MODINVSTOR-793 Update RMB to 33.1.1 and Vert.x to 4.1.4
-
- Closed
-
-
MODPERMS-155 Backport database memory fix to Juniper (RMB-863, RMB v33.1.1)
-
- Closed
-
-
MODUSERS-273 Backport database memory fix to Juniper (RMB-863, RMB v33.1.1)
-
- Closed
-
-
RMB-664 PoC: implement one connection pool for all tenants
-
- Closed
-
- mentioned in
-
Page Loading...