Uploaded image for project: 'mod-pubsub'
  1. mod-pubsub
  2. MODPUBSUB-76

mod-pubsub filling container filesystem

    XMLWordPrintable

Details

    • EPAM BatchLoader Sprint 39
    • 3
    • Folijet

    Description

      Problem: pubsub appears to be saving files to /tmp and not cleaning them up resulting in a full disk on some reference environments.

      Report: Noticed folio-snapshot ran out of disk space today (4/1/2020). Cause seems like mod pubsub filling its container filesystem. Inspecting the disk on folio-snapshot found the container's filesystem was 17GB:

      ubuntu@ip-10-36-1-22:~$ sudo du -h -d 1 /var/lib/docker/overlay2/6d231c0e0cbdd53b0e26694d4a59e507a40a2e16eeea6e82cee28c3eb9084820
8.0K	/var/lib/docker/overlay2/6d231c0e0cbdd53b0e26694d4a59e507a40a2e16eeea6e82cee28c3eb9084820/work
8.4G	/var/lib/docker/overlay2/6d231c0e0cbdd53b0e26694d4a59e507a40a2e16eeea6e82cee28c3eb9084820/diff
8.5G	/var/lib/docker/overlay2/6d231c0e0cbdd53b0e26694d4a59e507a40a2e16eeea6e82cee28c3eb9084820/merged
17G	/var/lib/docker/overlay2/6d231c0e0cbdd53b0e26694d4a59e507a40a2e16eeea6e82cee28c3eb9084820

      Attaching a shell to mod-pubsub container to inspect, it appears there's a bunch of liquibase directories saved to the /tmp directory that are not getting cleaned up:

      bash-5.0# du -h -d 1 /tmp
325.1M	/tmp/liquibase-unzip7628116248472040077.dir
325.1M	/tmp/liquibase-unzip1457664510988781026.dir
325.1M	/tmp/liquibase-unzip7541852650204654778.dir
325.1M	/tmp/liquibase-unzip2339504978943753794.dir
325.1M	/tmp/liquibase-unzip2094135029201299360.dir
46.7M	/tmp/liquibase-unzip3754848262343354531.dir
325.1M	/tmp/liquibase-unzip6181878362686715827.dir
325.1M	/tmp/liquibase-unzip1630660329798279237.dir
325.1M	/tmp/liquibase-unzip2477919249106612832.dir
325.1M	/tmp/liquibase-unzip1419275139438205172.dir
8.0K	/tmp/vertx-cache
325.1M	/tmp/liquibase-unzip586509806846540356.dir
36.0K	/tmp/hsperfdata_folio
325.1M	/tmp/liquibase-unzip1036824275861968162.dir
325.1M	/tmp/liquibase-unzip8490716457988981753.dir
325.1M	/tmp/liquibase-unzip238084761907544779.dir
325.1M	/tmp/liquibase-unzip3465586917602159626.dir
325.1M	/tmp/liquibase-unzip5043329286219450538.dir
325.1M	/tmp/liquibase-unzip1118294144892395644.dir
325.1M	/tmp/liquibase-unzip1852706735294912377.dir
325.1M	/tmp/liquibase-unzip7375525321773166756.dir
325.1M	/tmp/liquibase-unzip3829576104331642158.dir
325.1M	/tmp/liquibase-unzip891181331508888512.dir
325.1M	/tmp/liquibase-unzip6236507222543511760.dir
325.1M	/tmp/liquibase-unzip3258783700955607024.dir
325.1M	/tmp/liquibase-unzip2853907133213379056.dir
325.1M	/tmp/liquibase-unzip4289679006255011442.dir
325.1M	/tmp/liquibase-unzip6446489014551918778.dir
325.1M	/tmp/liquibase-unzip5351569439684215205.dir
8.3G	/tmp

      Logs from pubsub attached

      TestRail: Results

        Attachments

          Issue Links

            blocks

            Task - A task that needs to be done. MODPUBSUB-81 Release v1.1.4

            • P3 - low priority level, item will be considered for inclusion in the next dev cycle
            • Closed
            defines

            New Feature - New feature requests UXPROD-2220 NFR: Data Import (Batch Importer for Bib Acq) & PubSub Q2 2020 Technical, NFR, & Misc bug work

            • P2 - normal priority level, must be included in the current development cycle
            • Closed
            relates to

            Task - A task that needs to be done. MODPUBSUB-74 Switch off by default logging of event payload

            • P1 - highest priority item, drop everything else before this is resolved, reserved for critical bugfixes
            • Closed

            Activity

              People

                OleksiiKuzminov Oleksii Kuzminov
                ian Ian Hardy
                Votes:
                0 Vote for this issue
                Watchers:
                4 Start watching this issue

                Dates

                  Created:
                  Updated:
                  Resolved:

                  TestRail: Runs

                    TestRail: Cases