LinuxQuestions.org - Docker with 2 containers growing out exceedingly

- Linux - Containers (https://www.linuxquestions.org/questions/linux-containers-122/)

- - Docker with 2 containers growing out exceedingly (https://www.linuxquestions.org/questions/linux-containers-122/docker-with-2-containers-growing-out-exceedingly-4175707075/)

Docker with 2 containers growing out exceedingly

So I installed Docker on a Rocky Linux VM with a 10GB hard drive (runs on Proxmox) for two small docker applications (rsshub & tinytinyrss). The applications were not taking much more than 2.5GB of hard drive space at the moment of installation, and the VM had plenty of space left in /

Fast forward today, I keep getting notifications from this VM that the hard drive is nearly full (the VM is configured to send an email notification via cron if a storage layer reaches 80% or more)... df -H reveals:

Code:

Filesystem                                      Size  Used Avail Use% Mounted on

devtmpfs                                        490M    0  490M  0% /dev

tmpfs                                            509M    0  509M  0% /dev/shm

tmpfs                                            509M  52M  457M  11% /run

tmpfs                                            509M    0  509M  0% /sys/fs/cgroup

/dev/mapper/rl_vm--template-root                8.6G  6.9G  1.8G  80% /

/dev/vda1                                        1.1G  341M  724M  32% /boot

overlay                                          8.6G  6.9G  1.8G  80% /var/lib/docker/overlay2/0770d9a34e34bd2b7d6af52755cd3b1d9a6c3f36b7b2c2964dd56ecb15d28b5c/merged

overlay                                          8.6G  6.9G  1.8G  80% /var/lib/docker/overlay2/62cd457a4cb51385cfc66c8296559abfa75bb5e5d66335ddf6d99843033278ce/merged

overlay                                          8.6G  6.9G  1.8G  80% /var/lib/docker/overlay2/3321797212b44e1d23ec2d438dfc5997145e28fa6051c7ae71d2b261a79d7ae1/merged

overlay                                          8.6G  6.9G  1.8G  80% /var/lib/docker/overlay2/06b6a4f4cfbb0bc7a80bb1e103e320b10cf90fbdb6e2b580afa71ff7304d8718/merged

tmpfs                                            102M    0  102M  0% /run/user/1000

Examining / I believe the issue is inside "/var/lib/docker/overlay2/" but to be transparent, googling this issue sent me in so many different directions that I am not sure. I know very little about docker, and I believe it uses virtual FS's to perform storage, and some users have pointed out that even if overlay2 seems to be the culprit, in reality it could be some other files occupying all the space somewhere else within the docker virtual FS... Could be bull**** too...

Anyways, /var/lib/docker is indeed 5.8GB

Code:

du -hs /var/lib/docker

5.8G        /var/lib/docker

Looking at docker's FS usage stats:

Code:

docker system df

TYPE            TOTAL    ACTIVE    SIZE      RECLAIMABLE

Images          4        4        2.657GB  0B (0%)

Containers      4        4        174.4MB  0B (0%)

Local Volumes  2        2        146.1kB  0B (0%)

Build Cache    0        0        0B        0B

Looking at docker's container sizes:

Code:

docker ps --size

CONTAINER ID  IMAGE                                  COMMAND                  CREATED        STATUS        PORTS                                                          NAMES                  SIZE

f260f3a94b0f  diygod/rsshub                          "dumb-init -- npm ru…"  2 months ago  Up 13 hours  0.0.0.0:1200->1200/tcp, :::1200->1200/tcp                      rsshub_rsshub_1        174MB (virtual 408MB)

d2e03bd28b6e  redis:alpine                            "docker-entrypoint.s…"  2 months ago  Up 13 hours  6379/tcp                                                        rsshub_redis_1        0B (virtual 32.4MB)

064474c5af54  browserless/chrome:1.43-chrome-stable  "./start.sh"            2 months ago  Up 13 hours  3000/tcp                                                        rsshub_browserless_1  11B (virtual 2.14GB)

3ebad0761681  portainer/portainer-ce                  "/portainer"            2 months ago  Up 13 hours  8000/tcp, 9443/tcp, 0.0.0.0:9000->9000/tcp, :::9000->9000/tcp  portainer              0B (virtual 252MB)

Finally trying to find the largest folders within /var/lib/docker:

Code:

du -a /var/lib/docker | sort -n -r | head -n 20

6054432        /var/lib/docker

5864496        /var/lib/docker/overlay2

2181636        /var/lib/docker/overlay2/62cd457a4cb51385cfc66c8296559abfa75bb5e5d66335ddf6d99843033278ce

2181624        /var/lib/docker/overlay2/62cd457a4cb51385cfc66c8296559abfa75bb5e5d66335ddf6d99843033278ce/merged

1755720        /var/lib/docker/overlay2/62cd457a4cb51385cfc66c8296559abfa75bb5e5d66335ddf6d99843033278ce/merged/usr

1349052        /var/lib/docker/overlay2/de6aaef229cdce3712f113911ef72859e10980217a24901976f87f87eb80fb7c

1349044        /var/lib/docker/overlay2/de6aaef229cdce3712f113911ef72859e10980217a24901976f87f87eb80fb7c/diff

1300180        /var/lib/docker/overlay2/de6aaef229cdce3712f113911ef72859e10980217a24901976f87f87eb80fb7c/diff/usr

957564        /var/lib/docker/overlay2/62cd457a4cb51385cfc66c8296559abfa75bb5e5d66335ddf6d99843033278ce/merged/usr/lib

863272        /var/lib/docker/overlay2/de6aaef229cdce3712f113911ef72859e10980217a24901976f87f87eb80fb7c/diff/usr/lib

609260        /var/lib/docker/overlay2/06b6a4f4cfbb0bc7a80bb1e103e320b10cf90fbdb6e2b580afa71ff7304d8718

480908        /var/lib/docker/overlay2/62cd457a4cb51385cfc66c8296559abfa75bb5e5d66335ddf6d99843033278ce/merged/usr/lib/x86_64-linux-gnu

446508        /var/lib/docker/overlay2/de6aaef229cdce3712f113911ef72859e10980217a24901976f87f87eb80fb7c/diff/usr/lib/x86_64-linux-gnu

437464        /var/lib/docker/overlay2/06b6a4f4cfbb0bc7a80bb1e103e320b10cf90fbdb6e2b580afa71ff7304d8718/merged

415556        /var/lib/docker/overlay2/9127947de0ee3a35e55a3139facf3b3d1f8c248adcfc1ec31cfdbfcfd129ecb4

415548        /var/lib/docker/overlay2/9127947de0ee3a35e55a3139facf3b3d1f8c248adcfc1ec31cfdbfcfd129ecb4/diff

335492        /var/lib/docker/overlay2/9127947de0ee3a35e55a3139facf3b3d1f8c248adcfc1ec31cfdbfcfd129ecb4/diff/usr

335284        /var/lib/docker/overlay2/9127947de0ee3a35e55a3139facf3b3d1f8c248adcfc1ec31cfdbfcfd129ecb4/diff/usr/src/app

335284        /var/lib/docker/overlay2/9127947de0ee3a35e55a3139facf3b3d1f8c248adcfc1ec31cfdbfcfd129ecb4/diff/usr/src

335284        /var/lib/docker/overlay2/62cd457a4cb51385cfc66c8296559abfa75bb5e5d66335ddf6d99843033278ce/merged/usr/src/app

I thought the logs and json files were the issue but looking at the total space used by each of these file types (find . -name "*.log" | xargs du -sch):
-logs < 519MB
-JSON < 7.5MB
-DEB < 139MB
....

1. What is causing the filesystem to grow out of control?
2. How to clean up Docker to free up HDD space?

I hope some can help me with this.

Thanks

As far as I know docker and overlayfs is ok, it is the app inside the docker container (most probably the cache of chrome or something similar).

You may be onto something... I have created 2 more env variables for rsshub as per their wiki (https://docs.rsshub.app/en/install/#docker-image) in the hope the cache will clear itself at some point but I am not expecting anything at this moment...

For the chrome browserless docker app, I also believe it may be the root cause... I however cant find how to clear cache, or limit its growth...

If anybody has info please share.

theoretical you may go inside the docker image and look around, as if it was a real VM. But that require some knowledge about them.
you may get some ideas (for example) here: https://phoenixnap.com/kb/how-to-ssh...cker-container

There might be unused/dangling objects.

Have you tried to prune unused objects of docker ?

check this official document : https://docs.docker.com/config/pruning/

This will help you cleanup if any unused image/container/network/volumes present in overlay directory.

"Unused objects" won't account for this. Something inside the containerized environment must be the culprit in this case.

Quote:

Originally Posted by sundialsvcs (Post 6327502)

"Unused objects" won't account for this. Something inside the containerized environment must be the culprit in this case.

Fully agree with you. I tried pruning the system, and it deleted a few unused images and cleared up 233MB of space, barely doing a dent in the problem.

"docker image ls" gives:

Code:

REPOSITORY              TAG                  IMAGE ID      CREATED        SIZE

diygod/rsshub            latest              f40e0167073d  5 weeks ago    234MB

portainer/portainer-ce  latest              ad0ecf974589  3 months ago    252MB

redis                    alpine              5c08f13a2b92  3 months ago    32.4MB

browserless/chrome      1.43-chrome-stable  ec7bb30f39c5  15 months ago  2.14GB

clearly the problem is "ec7bb30f39c5".

ssh into the container, I could see that /usr is taking 1.7GB, with /usr/src and /usr/lib taking most of that space...

Code:

0        /bin

0        /boot

0        /dev

0        /lib

0        /lib32

0        /lib64

0        /libx32

0        /media

0        /mnt

0        /proc

0        /root

0        /sbin

0        /srv

0        /sys

4.0K        /home

8.0K        /run

2.2M        /etc

26M        /var

86M        /tmp

228M        /opt

1.7G        /usr

Am I wrong to say that there's little to do with this unless screwing around in the container trying to save space, when perhaps its the container that is NOT built intelligently (to clean itself up)?

These days, I've encountered a lot of software (specifically including "Chrome") that is extremely wasteful of disk space probably because the authors thereof figure that it doesn't matter. It is entirely correct to presume that the "stock" container definition also did not bother. Really, the only thing that you can do at this point is to enter the container and have a look around ... look at the configuration files, zero-in on exactly what is in (say ...) "/opt," and then exactly why.

Incidentally – this has become one of the specific reasons why I have come to dislike "Docker." In my view, it leaves you too far disconnected from what is actually going on. "The price of convenience" may turn out to be too much. However, that's just me . . .

Quote:

Originally Posted by lpallard (Post 6334785)

Yes, you are wrong. First of all you can drop the whole container and start over with a new one.
Next you can use ncdu or other software to see what is in /usr and clean that manually (if possible).
This is not the container, it cannot clean itself (as a VM or any other OS cannot clean the storage), but the software running inside (=chrome) which eats up your disk.
Docker (and other container solutions) will encapsulate the running process by having its own environment and virtually locking out anything else. Docker can also control the used resources like RAM or CPU, see for example here: https://docs.docker.com/engine/refer...-per-container