Android du and df very different: File System Corrupted?

Question

On an Android Nougat (7.1.1) set top box, du and df of my /data partition are very different:

$ adb shell du -sh /data

1.0G
 /data

$ adb shell df -H /data

Filesystem

Size  Used Avail Use% Mounted on

/dev/block/sda9  10G
10G  469M  96% /data

I donâ€™t think that anything else is mounted under /data:

$ adb shell mount | grep "\/data"

/dev/block/sda9 on /data type ext4 (rw,seclabel,nosuid,nodev,relatime,discard,noauto_da_alloc,data=ordered)

$ adb shell mount | grep "sda9"

/dev/block/sda9 on /data type ext4 (rw,seclabel,nosuid,nodev,relatime,discard,noauto_da_alloc,data=ordered)

lsof indicates to me that there are dozens of processes holding hundreds of small deleted files of the sort

init

 1

 root
 3w

CHR

1,11

 0t0

14517 /dev/__kmsg__ (deleted)

ueventd
  391

 root
 3w

CHR

1,11

 0t0

14593 /dev/__kmsg__ (deleted)

  . . .

main

  756

 root  mem
unknown

  /dev/ashmem/dalvik-large object space allocation (deleted)

main

  756

 root  mem
unknown

  /dev/ashmem/dalvik-large object space allocation (deleted)

  . . .

omm.times  3934
  system  mem
unknown

  /dev/ashmem/dalvik-mark sweep sweep array free buffer (deleted)

omm.times  3934
  system  mem
unknown

  /dev/ashmem/dalvik-mark sweep sweep array free buffer (deleted)

Using busybox, results are different (df shows much less usage), but there is still a large discrepancy between du and df:

$ adb shell busybox du -sh /data

389.4M  /data

$ adb shell busybox df -h /data

Filesystem

 Size

Used Available Use% Mounted on

/dev/block/bootdevice/by-name/userdata


  9.7G

8.6G

1.1G  89% /data

And using toybox, the results are similar to busybox:

$ adb shell toybox du -sh /data

389M
 /data

$ adb shell toybox df -h /data

Filesystem

Size  Used Avail Use% Mounted on

/dev/block/sda9  10G  8.6G  1.1G  89% /data

It think that it's important to note that these numbers remain similar after a reboot.

Also, I wish to note that I cannot OTA update this device's Android due to running out of disk space, even though the OTA image is only about 1GB in size. This fact leads me to believe that the results of df are accurate in terms of actual available disk space.

All adb commands are run as root i.e., I did adb root before executing them. But at risk of making this question be too verbose, here's everything run at the adb prompt:

$ adb shell

Z:/ # whoami

root

Z:/ # du -sh /data

389M
 /data

Z:/ # df -h /data

Filesystem

Size  Used Avail Use% Mounted on

/dev/block/sda9  10G  8.6G  1.1G  89% /data

10.197.12.14:/ # busybox du -sh /data

389.4M  /data

Z:/ # busybox df -h /data

Filesystem

 Size

Used Available Use% Mounted on

/dev/block/bootdevice/by-name/userdata


  9.7G

8.6G

1.1G  89% /data

Z:/ # toybox du -sh /data

389M
 /data

Z:/ # toybox df -h /data

Filesystem

Size  Used Avail Use% Mounted on

/dev/block/sda9  10G  8.6G  1.1G  89% /data

Z:/ #

A user in comments has asked for the following:

Z:/ # while read num; do (( sum += num ));
 done <<< $(find /data -type f -exec stat -c%b {

}
 +);
 expr $sum / 2048

393

I understand that du reports free space by scanning reachable nodes, and that this could therefore mean that my file system is corrupt. However, unfortunately, this is for devices in the field, and I donâ€™t want to have to (and donâ€™t think I can) run fsck or e2fsck.

What might be causing the huge difference between du and df on this device, and how might this problem be resolved? Iâ€™m happy to provide further logs.

— zippy | Sources

Answer

The /data partition was indeed corrupted.

After running e2fsck -p /dev/block/sda9, here was the updated du and df (again, after doing an adb root):

$ adb shell du -s /data

1.3G
 /data

$ adb shell df -H /data

Filesystem

Size  Used Avail Use% Mounted on

/dev/block/sda9  10G  1.5G  8.9G  15% /data

After fixing the file system, the device ran properly and updated successfully thereafter.

(Note that I found that I had to run e2fsck with the -p option; it wouldn't run interactively for some reason.)

A portion of the output of the above e2fsck command is

data contains a file system with errors, check forced.

data: Deleted inode 163845 has zero dtime.  FIXED.

data: Deleted inode 163847 has zero dtime.  FIXED.

data: Deleted inode 163849 has zero dtime.  FIXED.


. . . [snip ] . . .

data: Deleted inode 213003 has zero dtime.  FIXED.

data: Deleted inode 262223 has zero dtime.  FIXED.

data: Deleted inode 303108 has zero dtime.  FIXED.

data: 5203/655360 files (2.0% non-contiguous), 186831/2621440 blocks

There were a total of 140 lines of the sort data: Deleted inode...

FWIW, we are guessing that this problem was possibly caused by the filesystem becoming full with logs, which then caused corruption during the next attempted download of OTA updates: Those downloads failed, but thereafter left the file system in a corrupted state. Subsequent attempts at those downloads seemed to further corrupt the file system.

Thanks very much to @alecxs for your persistence and relevant details. (However, please note that to run e2fsck, the partition /dev/block/sda9 had to be UNmounted.)

— zippy | Sources

Android du and df very different: File System Corrupted?

Topics