CentOS: solution for server disk is full and cannot free space

The hard disk shows that it is full, but the sum of the hard disk space occupied by du -sh /* is far less than the size of the hard disk

Use df -h to check that the disk space is indeed full, that is, it cannot be found how the hard disk partition is full.

After understanding the cause and effect, I learned that the log file of this WEB server was deleted. The purpose is to clear the large amount of disk space occupied by the log file, but when the file is being written all the time, we cannot completely delete it. .

Therefore, such a problem arises. Let me simulate it below:

[root@oldboyedu test]# dd if=/dev/zero of=/dev/sdc bs=10M count=10
10+0 records in
10+0 records out
104857600 bytes (105 MB) copied, 2.18347 s, 48.0 MB/s
[root@oldboyedu test]# ll -hi /dev/sdc
48657 -rw-r--r--. 1 root root 100M Jul 14 19:08 /dev/sdc
[root@oldboyedu test]# mkfs -t ext4 /dev/sdc
mke2fs 1.41.12 (17-May-2010)
/dev/sdc is not a block special device.
Proceed anyway? (y,n) y
Filesystem label=
OS type: Linux
Block size=1024 (log=0)
Fragment size=1024 (log=0)
Stride=0 blocks, Stripe width=0 blocks
25688 inodes, 102400 blocks
5120 blocks (5.00%) reserved for the super user
First data block=1
Maximum filesystem blocks=67371008
13 block groups
8192 blocks per group, 8192 fragments per group
1976 inodes per group
Superblock backups stored on blocks: 
    8193, 24577, 40961, 57345, 73729

Writing inode tables: done                            
Creating journal (4096 blocks): done
Writing superblocks and filesystem accounting information: done

This filesystem will be automatically checked every 22 mounts or
180 days, whichever comes first.  Use tune2fs -c or -i to override.
[root@oldboyedu test]# tune2fs -c -1 /dev/sdc
tune2fs 1.41.12 (17-May-2010)
Setting maximal mount count to -1
[root@oldboyedu test]# mkdir /log
[root@oldboyedu test]# mount -o loop /dev/sdc /log
[root@oldboyedu log]# dd if=/dev/zero of=/log/1.txt bs=10M count=8
8+0 records in
8+0 records out
83886080 bytes (84 MB) copied, 0.287821 s, 291 MB/s
[root@oldboyedu log]# echo "dddd">>1.txt
[root@oldboyedu log]# df -h
Filesystem      Size  Used Avail Use% Mounted on
/dev/sda3        19G  1.5G   17G   8% /
tmpfs           491M     0  491M   0% /dev/shm
/dev/sda1       190M   35M  146M  19% /boot
/dev/sdc         93M   82M  6.4M  93% /log

I reopen a terminal and execute the following commands:

[root@oldboyedu ~]# tail -f /log/1.txt
dddd

Then prepare to delete, see the results

[root@oldboyedu log]# rm 1.txt -f
[root@oldboyedu log]# df -h
Filesystem      Size  Used Avail Use% Mounted on
/dev/sda3        19G  1.5G   17G   8% /
tmpfs           491M     0  491M   0% /dev/shm
/dev/sda1       190M   35M  146M  19% /boot
/dev/sdc         93M   82M  6.4M  93% /log
[root@oldboyedu log]# ll
total 12
drwx------. 2 root root 12288 Jul 14 19:11 lost+found
[root@oldboyedu log]# du -sh /log
13K    /log
[root@oldboyedu log]# du -sh /log/*
12K    /log/lost+found
[root@oldboyedu log]# df -h
Filesystem      Size  Used Avail Use% Mounted on
/dev/sda3        19G  1.5G   17G   8% /
tmpfs           491M     0  491M   0% /dev/shm
/dev/sda1       190M   35M  146M  19% /boot
/dev/sdc         93M   82M  6.4M  93% /log
[root@oldboyedu log]# lsof |grep deleted
tail      3029      root    3r      REG                7,0 83886085         12 /log/1.txt (deleted)
[root@oldboyedu log]# kill -15 3029
[root@oldboyedu log]# df -h
Filesystem      Size  Used Avail Use% Mounted on
/dev/sda3        19G  1.5G   17G   8% /
tmpfs           491M     0  491M   0% /dev/shm
/dev/sda1       190M   35M  146M  19% /boot
/dev/sdc         93M  1.6M   87M   2% /log
[root@oldboyedu log]# lsof |grep deleted
[root@oldboyedu log]#