Team estimation - 45 days
Current situation or problem:
1.Kafka producer closed after sending
2.WARN message when no handler found
3. Kafka cache resource consumption
4. Data import impacts other processes
5. High resource consumption to get job(s) status/progress
Out of scope
*1.*Create pool of active producers. Start pool on module launch, close on shutdown. Reuse connections. Add max/min pool sizes.
2. Do not subscribe to messages you're not going to process OR Lower log lever for this type of messages.
*3.*Remove Kafka cache. Modules that do not do persistent changes will sometimes (on duplicates read) do unnecessary calls. Can be optimized further upon adding distributed in-memory cache (ex hazelcast) (blocked by 6 PUT LINK TO FEATURE in p.6)
4. SPIKE REQ.: Need investigation (possible solution - configure rate limiter). Relates to High CPU/Memory consumption on modules
5. Add some kind of caching for progress tracking (database or in-memory)
Links to additional information: