Excessive CPU usage in a system with no user activity

Seen in vagrant box folio/testing - version 5.0.0-20210412.6124 - but also for earlier releases.

Module versions: mod-source-record-manager-3.1.0-SNAPSHOT.388  , mod-inventory-16.4.0-SNAPSHOT.358 .

In my test I am using vb.cpus=12 and vb.memory=18000. No swapping happening.

The system at boot deploys all modules.. This takes a couple of minutes. After this point, no user is logged on. However, the CPU load is 2 (200%) in total for the system. What is measured here is "TIME" .. Ie the total CPU usage user+system for a process.

After 10 minutes, the system is surely turned to "idle" state and the top 4 contenders are: mod-source-record-manager 3:44 (3 minutes 44 seconds), mod-inventory 3:39, kafka 2:07, mod-pubsub-server 0:57. .. further down is list is Okapi 0:17.

After one hour the stats are as follows: mod-source-record-manager 24:22, mod-inventory 24:01, Kafka 11:41, docker-proxy 5:50, mod-pubsub-server 3:25. Down the list is okapi with 0:37.

Would expected that mod-inventory, mod-source-record-manager would not use any significant CPU after 10 minutes. Okapi is running mod-circulation jobs every minute.. And has done work (20 seconds ouf of 3000 seconds in total ~ 0.03% ).

mod-source-record-manafer OTOH has used abt 20 minutes of CPU of in 50 minutes (40%).

System load like this is not directly seen by users, but it will slow the system down as a whole.. Not to mention, add costs to hosting.

There is no log in okapi.log to indicate what these modules - mod-source-record-manager and mod-inventory would be doing in a system with no importing, exporting or user activity.

The following command line was executed to gather information:

while true; do
  uptime >>stat2
  ps -w -a -A -o pid,user,time,args --sort time >>stat2
  sleep 60
done

Attached is stat2 for 2 hours of operation.

It's not clear whether is an issue in mod-inventory or mod-source-record-manager.. One of these modules initiates a message quite often (or something).

How it looks like on top:

top - 11:11:13 up  2:14,  1 user,  load average: 1.70, 1.97, 1.99
Tasks: 468 total,   1 running, 467 sleeping,   0 stopped,   0 zombie
%Cpu(s):  6.0 us,  4.0 sy,  0.0 ni, 89.1 id,  0.1 wa,  0.0 hi,  0.8 si,  0.0 st
MiB Mem :  17578.1 total,   3592.0 free,  11755.8 used,   2230.3 buff/cache
MiB Swap:    980.0 total,    980.0 free,      0.0 used.   5495.0 avail Mem 

    PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+ COMMAND                                                                                                                    
  14864 vagrant   20   0 2314012 251620  16268 S  45.5   1.4  55:00.09 java                                                                                                                       
   7255 vagrant   20   0 2309856 194572  15612 S  44.9   1.1  54:16.52 java                                                                                                                       
  15785 root      20   0 6438640 640892  11768 S  25.7   3.6  25:46.61 java                                                                                                                       
  15396 root      20   0 1216856  13932   2812 S  12.9   0.1  13:20.60 docker-proxy                                                                                                               
  12130 vagrant   20   0 1945208 277152  17484 S   5.3   1.5   7:04.01 java                                                                                                                       
  13077 vagrant   20   0 1798936 233004  16380 S   3.6   1.3   5:20.91 java