Details
-
Bug
-
Status: Closed (View Workflow)
-
P1
-
Resolution: Done
-
3.1.1
-
CP: sprint 91, CP: sprint 92
-
Core: Platform
Description
Overview:
While testing Okapi 3.1.1 with bulk instance id download, noticed that Okapi uses excessive memory usage. If the memory is setting a bit low like 1g, it will get OOM error.
Steps to Reproduce:
- Use Okapi 3.1.1 (set -Xmx to 1g for example) with latest modules code
- Use nightly perf data set. It has about 2.7m instance records
- Make API call to /instance-bulk/ids?limit=2147483647&query=(keyword="") sortby title*
- Check Okapi log
Expected Results:
No error in the log
Actual Results:
OOM error
20:49:53 INFO ProxyContext 465546/instance-bulk REQ 140.234.253.9:21737 supertenant GET /instance-bulk/ids mod-authtoken-2.6.0-SNAPSHOT.73 mod-inventory-storage-19.3.0-SNAPSHOT.440 20:49:53 INFO ProxyContext 465546/instance-bulk;355490/users REQ 172.31.6.98:39356 supertenant GET /users/8cef003e-a9bd-409f-8ea4-8d395b55e415 mod-authtoken-2.6.0-SNAPSHOT.73 mod-users-17.1.0-SNAPSHOT.136 20:49:53 INFO ProxyContext 465546/instance-bulk;355490/users RES 202 15719us mod-authtoken-2.6.0-SNAPSHOT.73 http://172.31.6.98:9201/users/8cef003e-a9bd-409f-8ea4-8d395b55e415 20:49:53 INFO ProxyContext 465546/instance-bulk;355490/users RES 200 12436us mod-users-17.1.0-SNAPSHOT.136 http://172.31.6.98:9227/users/8cef003e-a9bd-409f-8ea4-8d395b55e415 20:49:53 INFO ProxyContext 465546/instance-bulk;975356/perms REQ 172.31.6.98:39358 supertenant GET /perms/users mod-authtoken-2.6.0-SNAPSHOT.73 mod-permissions-5.12.0-SNAPSHOT.82 20:49:53 INFO ProxyContext 465546/instance-bulk;975356/perms RES 202 5139us mod-authtoken-2.6.0-SNAPSHOT.73 http://172.31.6.98:9201/perms/users?query=userId==8cef003e-a9bd-409f-8ea4-8d395b55e415 20:49:53 INFO ProxyContext 465546/instance-bulk;975356/perms RES 200 17055us mod-permissions-5.12.0-SNAPSHOT.82 http://172.31.6.98:9220/perms/users?query=userId==8cef003e-a9bd-409f-8ea4-8d395b55e415 20:49:53 INFO ProxyContext 465546/instance-bulk;150874/perms REQ 172.31.6.98:39358 supertenant GET /perms/users/be69019d-fdb4-421d-ab14-9ad4c2c7ec56/permissions mod-authtoken-2.6.0-SNAPSHOT.73 mod-permissions-5.12.0-SNAPSHOT.82 20:49:53 INFO ProxyContext 465546/instance-bulk;150874/perms RES 202 3804us mod-authtoken-2.6.0-SNAPSHOT.73 http://172.31.6.98:9201/perms/users/be69019d-fdb4-421d-ab14-9ad4c2c7ec56/permissions?expanded=true 20:49:53 INFO ProxyContext 465546/instance-bulk;150874/perms RES 200 23457us mod-permissions-5.12.0-SNAPSHOT.82 http://172.31.6.98:9220/perms/users/be69019d-fdb4-421d-ab14-9ad4c2c7ec56/permissions?expanded=true 20:49:53 INFO ProxyContext 465546/instance-bulk RES 202 183318us mod-authtoken-2.6.0-SNAPSHOT.73 http://172.31.6.98:9201/instance-bulk/ids?limit=2147483647&query=%28keyword=%22%2A%22%29%20sortby%20title 20:50:45 INFO ProxyContext 465546/instance-bulk RES 200 51941389us mod-inventory-storage-19.3.0-SNAPSHOT.440 http://172.31.6.98:9213/instance-bulk/ids?limit=2147483647&query=%28keyword=%22%2A%22%29%20sortby%20title 20:52:44 WARN ? Thread Thread[vert.x-eventloop-thread-1,5,main] has been blocked for 2493 ms, time limit is 2000 ms 20:53:04 INFO jFactory$Log4jLogger [172.17.0.2]:5701 [dev] [3.12.2] processors=2, physical.memory.total=7.5G, physical.memory.free=5.4G, swap.space.total=0, swap.space.free=0, heap.memory.used=720.9M, heap.memory.free=189.6M, heap.memory.total=910.5M, heap.memory.max=910.5M, heap.memory.used/total=79.17%, heap.memory.used/max=79.17%, minor.gc.count=0, minor.gc.time=0ms, major.gc.count=0, major.gc.time=0ms, load.process=56.61%, load.system=53.50%, load.systemAverage=0.88, thread.count=62, thread.peakCount=63, cluster.timeDiff=0, event.q.size=0, executor.q.async.size=0, executor.q.client.size=0, executor.q.client.query.size=0, executor.q.client.blocking.size=0, executor.q.query.size=0, executor.q.scheduled.size=0, executor.q.io.size=0, executor.q.system.size=0, executor.q.operations.size=0, executor.q.priorityOperation.size=0, operations.completed.count=582, executor.q.mapLoad.size=0, executor.q.mapLoadAllKeys.size=0, executor.q.cluster.size=0, executor.q.response.size=0, operations.running.count=0, operations.pending.invocations.percentage=0.00%, operations.pending.invocations.count=0, proxy.count=0, clientEndpoint.count=0, connection.active.count=0, client.connection.count=0, connection.count=0 20:53:16 WARN ? Thread Thread[vert.x-eventloop-thread-1,5,main] has been blocked for 2704 ms, time limit is 2000 ms 20:53:24 INFO jFactory$Log4jLogger [172.17.0.2]:5701 [dev] [3.12.2] processors=2, physical.memory.total=7.5G, physical.memory.free=5.3G, swap.space.total=0, swap.space.free=0, heap.memory.used=792.5M, heap.memory.free=118.0M, heap.memory.total=910.5M, heap.memory.max=910.5M, heap.memory.used/total=87.03%, heap.memory.used/max=87.03%, minor.gc.count=0, minor.gc.time=0ms, major.gc.count=0, major.gc.time=0ms, load.process=87.23%, load.system=86.38%, load.systemAverage=1.20, thread.count=63, thread.peakCount=63, cluster.timeDiff=0, event.q.size=0, executor.q.async.size=0, executor.q.client.size=0, executor.q.client.query.size=0, executor.q.client.blocking.size=0, executor.q.query.size=0, executor.q.scheduled.size=0, executor.q.io.size=0, executor.q.system.size=0, executor.q.operations.size=0, executor.q.priorityOperation.size=0, operations.completed.count=582, executor.q.mapLoad.size=0, executor.q.mapLoadAllKeys.size=0, executor.q.cluster.size=0, executor.q.response.size=0, operations.running.count=0, operations.pending.invocations.percentage=0.00%, operations.pending.invocations.count=0, proxy.count=0, clientEndpoint.count=0, connection.active.count=0, client.connection.count=0, connection.count=0 20:53:29 WARN ? Thread Thread[vert.x-eventloop-thread-1,5,main] has been blocked for 2279 ms, time limit is 2000 ms 20:53:32 WARN ? Thread Thread[vert.x-eventloop-thread-1,5,main] has been blocked for 2965 ms, time limit is 2000 ms 20:53:34 WARN ? Thread Thread[vert.x-eventloop-thread-1,5,main] has been blocked for 2252 ms, time limit is 2000 ms Exception in thread "vertx-blocked-thread-checker" java.lang.OutOfMemoryError: GC overhead limit exceeded java.lang.OutOfMemoryError: GC overhead limit exceeded 20:57:14 WARN jFactory$Log4jLogger [172.17.0.2]:5701 [dev] [3.12.2] BroadcastOperationControlTask delayed 175872 ms java.lang.OutOfMemoryError: GC overhead limit exceeded java.lang.OutOfMemoryError: GC overhead limit exceeded java.lang.OutOfMemoryError: GC overhead limit exceeded
TestRail: Results
Attachments
Issue Links
- relates to
-
RMB-657 Default 60s connectionReleaseDelay prevents streaming large data
-
- Closed
-
-
MODINVSTOR-465 Bulk download Instance UUIDs OOM
-
- Closed
-
-
OKAPI-870 writeQueueFull , drainHandler thread blocked
-
- Closed
-
-
OKAPI-887 Unit tests sporadically crash
-
- Closed
-
-
RMB-665 PgUtil.streamGet should not use plain write
-
- Open
-