Details
-
Bug
-
Status: Closed (View Workflow)
-
P2
-
Resolution: Done
-
30.2.0, 30.2.1
-
CP: sprint 91, CP: sprint 92
-
0.5
-
Core: Platform
-
Q2 2020
Description
Overview:
A database connection is cancelled when it takes more than 60 seconds before the first record is sent or if a pause of 60 seconds between two record.
While testing bulk instance id download, noticed that mysterious 500 closed error. It turns out that the default 60s connectionReleaseDelay parameter kicked in. While the parameter can be adjusted when starting mod-inventory-storage module, it would be nice if the connection used by streaming operation is not treated as idle.
Steps to Reproduce:
This problem was observed in bugfest-golden env, but you can easily reproduce it by doing
- Use a larger dataset. For example, nightly perf data set.
- Start mod-inventory-storage with a smaller connectionReleaseDelay value like 10s
- Make API call to /instance-bulk/ids?limit=2147483647&query=(keyword="") sortby title
Expected Results:
API call finishes OK
Actual Results:
{ "ids": [], "totalRecords": 2760630, "resultInfo": { "totalRecords": 2760630, "facets": [], "diagnostics": [ { "code": "500", "message": "closed" } ] } }
Database has log entry about broken pipe
2020-06-26 21:24:53.376 UTC [720] LOG: could not send data to client: Broken pipe 2020-06-26 21:24:53.376 UTC [720] STATEMENT: SELECT jsonb,id FROM supertenant_mod_inventory_storage.instance WHERE true ORDER BY left(lower(f_unaccent(instance.jsonb->>'title')),600), lower(f_unaccent(instance.jsonb->>'title')) 2020-06-26 21:24:53.376 UTC [720] FATAL: connection to client lost 2020-06-26 21:24:53.376 UTC [720] STATEMENT: SELECT jsonb,id FROM supertenant_mod_inventory_storage.instance WHERE true ORDER BY left(lower(f_unaccent(instance.jsonb->>'title')),600), lower(f_unaccent(instance.jsonb->>'title'))
Cause
RMB incorrectly implements connectionReleaseDelay by cancelling connections that are in use.
Solution
Disable the current wrong implementation of connectionReleaseDelay.
Consequently connections that are no longer in use will stay in the connection pool forever (but can be re-used).
See RMB-663 for the correct implementation.
Note
This bug does not affect the long running database tasks when upgrading a module to a new version.
TestRail: Results
Attachments
Issue Links
- blocks
-
MODINVSTOR-527 OAI-PMH view doesn't respond in Bug Fest
-
- Closed
-
- relates to
-
RMB-620 Honor connectionReleaseDelay
-
- Closed
-
-
RMB-633 Check DB_CONNECTIONRELEASEDELAY for normal ops and upgrades
-
- Closed
-
-
RMB-784 Reenable DB_CONNECTIONRELEASEDELAY
-
- Closed
-
-
MODINVSTOR-465 Bulk download Instance UUIDs OOM
-
- Closed
-
-
OKAPI-866 Okapi uses excessive memory or OOM for bulk instance id download
-
- Closed
-
-
RMB-663 patch idle connection release in the upstream vertx-sql-client driver
-
- Closed
-
-
RMB-739 patch idle connection release in the forked vertx-sql-client driver
-
- Closed
-