Team estimation - 45 days
Current situation or problem:
1.Kafka producer closed after sending
2.WARN message when no handler found
3. Kafka cache resource consumption
4. Data import impacts other processes
5. High resource consumption to get job(s) status/progress
NOTE: Several stories in this feature are related to Spring; those may move to Morning Glory; need input from architects on whether Spring Kafka Template can be use in DI modules. Probably will need an architecture spike in Lotus, and then the Spring stories may move to Morning Glory
*1.*Create pool of active producers. Start pool on module launch, close on shutdown. Reuse connections. Add max/min pool sizes.
2. Do not subscribe to messages you're not going to process OR Lower log lever for this type of messages.
*3.*Remove Kafka cache. Modules that do not do persistent changes will sometimes (on duplicates read) do unnecessary calls. Can be optimized further upon adding distributed in-memory cache (ex hazelcast) (blocked by 6 PUT LINK TO FEATURE in p.6)
4. SPIKE REQ.: Need investigation (possible solution - configure rate limiter). Relates to High CPU/Memory consumption on modules
5. Add some kind of caching for progress tracking (database or in-memory)
Links to additional information: