Uploaded image for project: 'mod-inventory'
  1. mod-inventory
  2. MODINV-401

Spike: KafkaCache Memory not released (after a long time)



    • 5
    • Folijet
    • Lotus R1 2022


      KS: Investigate possibility of removing Kafka cache. Modules that do not do persistent changes will sometimes (on duplicates read) do unnecessary calls. Can be optimized further upon adding distributed in-memory cache (ex hazelcast)

      When running consecutive imports and/or running a long import, 10 io.kcache.KafkaCache instances seem to accumulate memory along the way, eventually causes the container to crash. In a better outcome, these 10 objects seem to release the memory on its own but not after a very long or unpredictable time. This object contains the KafkaTopicReader object, which is a major contributor to the overall size of KafkaCache. Investigate how to enable garbage collection on these objects timely so that it would not eat up all the memory.

      Note that data import behaves unpredictably when the container crashes. Sometimes the job gets stuck, sometimes the job continues on as if nothing has happened, sometimes a few to hundred of records not getting created. So, it's essential that this problem be figured out.

      Steps to Reproduce:
      Run repeated data-import jobs of 5K MARC records or more and watch mod-inventory's memory consumption increases.

      Expected Results:
      Memory consumed and released in a timely manner such that the container would not run out of memory in a 24 hour time span.

      Actual Results:
      As described - Objects retained memory for a long time before releasing it.

      Additional Information:
      In this diagram mod-inventory's container crashed twice during the imports run. The crash events are indicated by the letter "C". But at 8:00, mod-inventory released the memory on its own, indicated by the letter "R".

      Shows the memory taken at around 6:00 on the graph above.
      Shows the memory taken at around 12:00

      At another period of time Memory's analyzer shows KafakCache taking up a lot of memory:

      Contact Roman Fedynyshyn for heap dumps

      Interested parties:
      OleksiiKuzminov abreaux

      TestRail: Results


          Issue Links



                Unassigned Unassigned
                mtraneis Martin Tran
                0 Vote for this issue
                4 Start watching this issue



                  TestRail: Runs

                    TestRail: Cases