On one server there was space missing

[root@lxrd_refsms /]# pwd
/
[root@lxrd_refsms /]# df -h
Filesystem           Size  Used Avail Use% Mounted on
devtmpfs              13G     0   13G   0% /dev
tmpfs                 13G   22M   13G   1% /dev/shm
tmpfs                 13G  1.4G   12G  11% /run
tmpfs                 13G     0   13G   0% /sys/fs/cgroup
/dev/mapper/cl-root  118G  118G  198M 100% /
/dev/sda1            976M  357M  552M  40% /boot
tmpfs                2.6G   12K  2.6G   1% /run/user/42
tmpfs                2.6G  4.0K  2.6G   1% /run/user/1001
tmpfs                2.6G  4.0K  2.6G   1% /run/user/1002
tmpfs                2.6G     0  2.6G   0% /run/user/0
[root@lxrd_refsms /]# du -hx --max-depth=1
34M     ./etc
620M    ./root
1.6G    ./var
5.6G    ./usr
3.5G    ./home
0       ./media
0       ./mnt
3.6M    ./opt
0       ./srv
36K     ./tmp
35G     ./u01
23G     ./u02
69G     .

As you can see / size is 118G but when you look at the data size in / you find 69G

49G are missing

The reason is that there are files deleted but still referenced by some process: In Unix/Linux, when a file is deleted (rm), its inode (file structure) remains on the disk as long as a process has it open.

[root@lxrd_refsms /]# lsof +L1
COMMAND       PID           USER   FD   TYPE DEVICE    SIZE/OFF NLINK      NODE NAME
systemd-u     855           root    8r   REG  253,0     9253600     0  25739281 /var/lib/sss/mc/passwd (deleted)
systemd-u     855           root    9r   REG  253,0     6940392     0  25739302 /var/lib/sss/mc/group (deleted)
auditd        977           root    4r   REG  253,0     6940392     0  25739302 /var/lib/sss/mc/group (deleted)
.........
.........
restart.s 2704120          orhus    1w   REG  253,0 51596331082     0 162323224 /home/orhus/scripts/restart.log (deleted)
restart.s 2704120          orhus    2w   REG  253,0 51596331082     0 162323224 /home/orhus/scripts/restart.log (deleted)
stopTomca 2704157          orhus    1w   REG  253,0 51596331082     0 162323224 /home/orhus/scripts/restart.log (deleted)
stopTomca 2704157          orhus    2w   REG  253,0 51596331082     0 162323224 /home/orhus/scripts/restart.log (deleted)

The four last lines are suspicious because they show deleted files that are very big: 51596331082 bytes each.

[root@lxrd_refsms /]# ps -ef | grep 2704157
root      822019  470357  0 17:23 pts/1    00:00:00 grep --color=auto 2704157
orhus    2704157 2704120 12 Apr08 ?        11-04:36:05 /bin/bash /home/orhus/scripts/stopTomcat.sh restart

We can see that the process who hold the deleted file is started since april and we are in july. The stopTomcat.sh should have finished its task.

Solution: kill the process or restart the server. And the space will be freed.

explanation on the lsof +L1 command:

+L<N>: lists all open files with fewer than <N> hard links. Then +L1 means to list all open files with fewer than 1 hard link, in other words, it means to list all open files deleted.

Also you can detect deleted open files bigger than 100MB with this command:

lsof +L1 | awk '{if ($7 > 104857600) print $0}'