No space left on device.

Sometimes we can be fooled by error messages. For example one sunny day you see that for some reason your web or mail server doesn’t work. So you go to check the logs and find something similar to this:

2016/12/28 09:02:37 [crit] 24668#24668: *472674 open() "/var/cache/nginx/client_temp/0020878597" failed (28: No space left on device), client: 192.168.1.1, server: www.domain.com, request: "GET /cart/add/uenc/aHR0cDovL3d3dy5hYmNob21lLmNvbS9zaG9wL2xvdi1vcmdhbmljLWxvdi1pcy1iZWF1dGlmdWwtdGVh/product/19471/form_key/N8l3OyVkC1el9T8q/?product=19471&related_product=&send_to_friend=%2F%2Fwww.domain.com%2Fshop%2Fsendfriend%2Fproduct%2Fsend%2Fid%2F19471%2F&form_key=N8l3OyVkC1el9T8q&super_group%5B19425%5D=1&super_group%5B19424%5D= HTTP/1.1", host: "www.domain.com", referrer: "http://www.domain.com/shop/organic-tea"

Then when you check the free space you see that you have more than enough, and all kind of irrational thoughts start flowing into your mind, when it is the simple inodes space.

Usually it is just that there is not enough inodes left free on your files system, simple as that, but is easy to overlook as for some people this doesn’t happen often (and it shouldn’t).

[root@hostname client_temp]# df -i
Filesystem Inodes IUsed IFree IUse% Mounted on
/dev/mapper/os-root 1703936 1703103 833 100% /
tmpfs 1524264 4 1524260 1% /dev/shm
/dev/sda1 51000 50 50950 1% /boot
/dev/mapper/os-tmp 131072 2155 128917 2% /tmp
/dev/mapper/data-data
19660800 578302 19082498 3% /data