No space left on device dokku error
October 15, 2020
I have a few projects running on dokku. One day I deployed a new release and unexpectedly was greeted with an error message at the end of the commit: remote: fatal: cannot create directory at 'frontend/src': No space left on device and ! [remote rejected] master -> master (pre-receive hook declined).
The error message is clear. The is not enough space for the new build. The question is how is there not enough space? I have only four apps running each of which shouldnt be more than a gig with the container and everything included.
Time to find out what was robbing the space.
How to check what directories take up the space?
du -sh /*du is a standard Unix command. I suppose it means disk usage. The flags after it are -s for getting the total space usage of the directory instead of single files and -h for human-readable formatting. The last argument /* is the path.
After running the command on the root level, I noticed that /var was easily the biggest directory. The output was something like this:
15M /bin
|
|
|
965M /usr
20G /varThe next step was drilling down to /var and run the same command again:
du -sh /var/*The process needs to continue until you find the root cause. In my case that was found in /var/lib. Docker folder was the culprit. Probably something wrong with the containers.
Another common cause could be /var/log. You might have some excessive logging going on.
Checking the health of the Docker daemon
docker system dfdocker system df shows the amount of disk space used by the Docker daemon. The output in my case was this:
TYPE TOTAL ACTIVE SIZE RECLAIMABLE
Images 47 18 9.575GB 5.974GB (62%)
Containers 52 5 290.7MB 290.2MB (99%)
Local Volumes 2 1 0B 0B
Build Cache 0 0 0B 0BA whopping 47 images and 48 containers. That can’t be right. I have only 4 apps!
docker container ls -adocker container ls -a gives you a list of all docker containers present in the system. My ouput looked something like this:
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
715de26d52cf ee77575140d7 "/bin/sh -c 'npm run…" 2 hours ago Exited (254) 2 hours ago infallible_sammet
b3b8648b5d62 ba4d32a75345 "/bin/sh -c 'npm ins…" 2 hours ago Exited (228) 2 hours ago trusting_goldberg
cb59f6185361 a6d8a8a9fea0 "/bin/sh -c 'npm ins…" 6 hours ago Exited (1) 6 hours ago nostalgic_austin
f8fc8512def0 dokku/snw:latest "node index.js" 2 days ago Up 2 days 5000/tcp snw.web.1
952a363d7e43 dokku/playerfan-landing:latest "/start web" 3 days ago Up 3 days playerfan-landing.web.1
57fd98cfb3d3 b91d2e5651dd "/bin/bash -c 'mkdir…" 3 days ago Exited (0) 3 days ago relaxed_kilby
461a99095b4e 906276725301 "/bin/bash -c 'mkdir…" 3 days ago Exited (0) 3 days ago stoic_tharp
7fb994596c1a d74c56ccc252 "/bin/bash -c 'mkdir…" 3 days ago Exited (0) 3 days ago blissful_curran
|
|
|
5d68cd37084e 165031171fbc "/build" 2 weeks ago Exited (0) 2 weeks ago nervous_elgamal
db3db0614c35 5c23278731f7 "/bin/bash -c 'cat >…" 2 weeks ago Exited (0) 2 weeks ago busy_mendeleev
034ed44c8617 bc547b6cd03e "/bin/bash -c 'mkdir…" 2 weeks ago Exited (0) 2 weeks ago lucid_tharp
ca88cc2fcf37 gliderlabs/herokuish:latest "/bin/bash -c 'mkdir…" 2 weeks ago Exited (0) 2 weeks ago boring_meninskyThe output was full of exited containers that had been left running there. We will get rid of those later. Let’s run the next command first.
docker image lsdocker image ls lists all the docker images present in the system. The output was something like this:
REPOSITORY TAG IMAGE ID CREATED SIZE
<none> <none> ee77575140d7 2 hours ago 943MB
<none> <none> ba4d32a75345 2 hours ago 944MB
<none> <none> a6d8a8a9fea0 6 hours ago 1.21GB
dokku/snw latest 2d15588b1d6e 2 days ago 1.46GB
dokku/playerfan-landing latest 29e8b4a9dfad 3 days ago 1.47GB
dokku/playerfan latest 47350dbae961 11 days ago 2.04GB
<none> <none> c1aa2ac74fc8 11 days ago 2.04GB
<none> <none> 1453b7f3ee55 11 days ago 2.04GB
<none> <none> 4f906504e071 11 days ago 2.04GB
|
|
|
<none> <none> 0178b0b978df 12 days ago 1.09GB
<none> <none> a646869a410c 12 days ago 1.09GB
<none> <none> 1d361291e72d 12 days ago 1.09GBAlso here, the system seemed to be full of useless <none> images that take up 2GB space each!
It was clear that Docker was leaving some development related cruft behind and a serious cleanup was needed.
Cleaning up the system
Dokku comes with a one useful command: dokku cleanup. This command should delete all dangling images and exited containers. However, it doesn’t always do the job perfectly. You may have to delete some containers and images manually. This was my case.
After running dokku cleanup I reran docker container ls -a. There was still some exited containers. I reran docker image ls and still found a lot of dangling <none> images. At this point, I ran docker system prune. Running this is risky. You may delete something you didn’t intend to delete. When you ran the command, it gives you this warning:
WARNING! This will remove:
- all stopped containers
- all networks not used by at least one container
- all dangling images
- all dangling build cacheSo, if you have, for example, an app container stopped for whatever reason, that will get deleted. You may not want that.
There is a way to delete the containers and images one-by-one, which is a lot safer. For containers, you can use this command:
docker container rm CONTAINER_ID1 CONTAINER_ID2...The container ids’ can be found in the output of docker container ls -a.
For the images you maybe use the following command:
docker image rm IMAGE_ID1 IMAGE_ID2...The image ids’ can be found in the output of docker image ls.
Wrap it up
I had a while back when deploying to dokku. The problem was that dokku was leaving behind some cruft from the development process. The commands used to do the cleanup, especially docker system prune must be used with care. If you don’g have a gazillion images and container to delete, just delete them one by one by using docker container rm and docker image rm.