Uploaded image for project: 'mod-pubsub'
  1. mod-pubsub
  2. MODPUBSUB-76

mod-pubsub filling container filesystem

    XMLWordPrintable

    Details

    • Template:
    • Sprint:
      EPAM BatchLoader Sprint 39
    • Story Points:
      3
    • Development Team:
      Folijet

      Description

      Problem: pubsub appears to be saving files to /tmp and not cleaning them up resulting in a full disk on some reference environments.

      Report: Noticed folio-snapshot ran out of disk space today (4/1/2020). Cause seems like mod pubsub filling its container filesystem. Inspecting the disk on folio-snapshot found the container's filesystem was 17GB:

      ubuntu@ip-10-36-1-22:~$ sudo du -h -d 1 /var/lib/docker/overlay2/6d231c0e0cbdd53b0e26694d4a59e507a40a2e16eeea6e82cee28c3eb9084820
      8.0K	/var/lib/docker/overlay2/6d231c0e0cbdd53b0e26694d4a59e507a40a2e16eeea6e82cee28c3eb9084820/work
      8.4G	/var/lib/docker/overlay2/6d231c0e0cbdd53b0e26694d4a59e507a40a2e16eeea6e82cee28c3eb9084820/diff
      8.5G	/var/lib/docker/overlay2/6d231c0e0cbdd53b0e26694d4a59e507a40a2e16eeea6e82cee28c3eb9084820/merged
      17G	/var/lib/docker/overlay2/6d231c0e0cbdd53b0e26694d4a59e507a40a2e16eeea6e82cee28c3eb9084820
      

      Attaching a shell to mod-pubsub container to inspect, it appears there's a bunch of liquibase directories saved to the /tmp directory that are not getting cleaned up:

      bash-5.0# du -h -d 1 /tmp
      325.1M	/tmp/liquibase-unzip7628116248472040077.dir
      325.1M	/tmp/liquibase-unzip1457664510988781026.dir
      325.1M	/tmp/liquibase-unzip7541852650204654778.dir
      325.1M	/tmp/liquibase-unzip2339504978943753794.dir
      325.1M	/tmp/liquibase-unzip2094135029201299360.dir
      46.7M	/tmp/liquibase-unzip3754848262343354531.dir
      325.1M	/tmp/liquibase-unzip6181878362686715827.dir
      325.1M	/tmp/liquibase-unzip1630660329798279237.dir
      325.1M	/tmp/liquibase-unzip2477919249106612832.dir
      325.1M	/tmp/liquibase-unzip1419275139438205172.dir
      8.0K	/tmp/vertx-cache
      325.1M	/tmp/liquibase-unzip586509806846540356.dir
      36.0K	/tmp/hsperfdata_folio
      325.1M	/tmp/liquibase-unzip1036824275861968162.dir
      325.1M	/tmp/liquibase-unzip8490716457988981753.dir
      325.1M	/tmp/liquibase-unzip238084761907544779.dir
      325.1M	/tmp/liquibase-unzip3465586917602159626.dir
      325.1M	/tmp/liquibase-unzip5043329286219450538.dir
      325.1M	/tmp/liquibase-unzip1118294144892395644.dir
      325.1M	/tmp/liquibase-unzip1852706735294912377.dir
      325.1M	/tmp/liquibase-unzip7375525321773166756.dir
      325.1M	/tmp/liquibase-unzip3829576104331642158.dir
      325.1M	/tmp/liquibase-unzip891181331508888512.dir
      325.1M	/tmp/liquibase-unzip6236507222543511760.dir
      325.1M	/tmp/liquibase-unzip3258783700955607024.dir
      325.1M	/tmp/liquibase-unzip2853907133213379056.dir
      325.1M	/tmp/liquibase-unzip4289679006255011442.dir
      325.1M	/tmp/liquibase-unzip6446489014551918778.dir
      325.1M	/tmp/liquibase-unzip5351569439684215205.dir
      8.3G	/tmp
      

      Logs from pubsub attached

        TestRail: Results

          Attachments

            Issue Links

              Activity

                People

                Assignee:
                OleksiiKuzminov Oleksii Kuzminov
                Reporter:
                ian Ian Hardy
                Votes:
                0 Vote for this issue
                Watchers:
                4 Start watching this issue

                  Dates

                  Created:
                  Updated:
                  Resolved:

                    TestRail: Runs

                      TestRail: Cases