7 ms·

What does the 'single' profile do in btrfs? I thought it was equivalent to RAID-0 mode, but apparently it isn't, because they are separate options and btrfs sw

What does the 'single' profile do in btrfs? I thought it was equivalent to RAID-0 mode, but apparently it isn't, because they are separate options and btrfs switched from RAID-0 mode to 'single' mode by default, although the 'mkfs.btrfs' man page doesn't explain what's the difference between them nor why btrfs did that. It has helpful diagrams for the RAID modes, but not 'single' mode. So how does btrfs allocate data among disks with this profile? This is not explained anywhere I could find. Is it just like concatenating a bunch of disks together? If so, yes, you would expect corrupting a single disk would completely corrupt a corresponding percentage of the files, but almost nothing on others, assuming the files were 1) written perfectly in order (i.e. were not fragmented) and 2) they were filling all of the disks. In this scenario, completely corrupting one of N disks would correspondingly completely corrupt 1/Nth of files (approximately speaking). If you were using RAID-0 mode, however, it would be different: it would corrupt almost all (large enough) files, but only 1/Nth of their contents. The results, however, could vary a lot depending on how file data actually ends up being allocated on disk, of course. That said, note that the results of this experiment can be completely misleading due to most of the data still being cached in RAM. I would say that to conduct this experiment more meaningfully you should: 1. "umount" the filesystem after having written all data to it. 2. Corrupt the disk. 3. Run "echo 3 > /proc/sys/vm/drop_caches" (a couple of times, just in case). This should get rid of the still-cached blocks in RAM, just in case. Or better yet, just reboot your machine :) 4. Mount the filesystem and check for corruption. At least, that's what you should do if you were performing this test on ZFS, otherwise corrupting the disks would not affect ZFS's cache and you wouldn't necessarily observe the effects of corruption when accessing the corrupted files normally (i.e. not as part of a scrub). Admittedly, in btrfs this might not be necessary because btrfs' cached file data might be exactly the same as the underlying disks' cached blocks. This is not true with ZFS, however, as ZFS's caches and the Linux kernel one don't share data (at least, not the vast majority of it). It would also be helpful to know the distribution of file sizes that were copied into the filesystem, otherwise the "successfully read the entire file" vs "got an error reading the entire file" statistics are not as helpful as it might seem at first sight (e.g. if there is a significant percentage of very small files).

wizeman 4y ago
> At least, that's what you should do if you were performing this test on ZFS, Just to clarify: in ZFS you shouldn't just 'umount' the filesystem, you should also export the pool, otherwise the cache would still be in use.
xk3 4y ago
This is very good advice. I did the same preparation, here is the distribution of files before the degraded state: Number of successful reads: 280 Number of IO errors: 0 Successful read files size: sum 82648303047 max 4884066696 average 295172511 then I unmounted the fs, deleted disk 2, echo 3 > /proc/sys/vm/drop_caches, and remounted the fs. sudo umount /mnt/loop echo 3 | sudo tee /proc/sys/vm/drop_caches echo 3 | sudo tee /proc/sys/vm/drop_caches echo 3 | sudo tee /proc/sys/vm/drop_caches dmesg --human --nopager --decode --level emerg,alert,crit,err,warn,notice,info kern :info : [Jan22 13:18] tee (215899): drop_caches: 3 kern :info : [ +3.232287] tee (215931): drop_caches: 3 kern :info : [ +0.775697] tee (215953): drop_caches: 3 rm d2.img sudo mount "$ld1" /mnt/loop I am surprised that mounting worked without error but I guess the device is still active via losetup. I'm assuming this would be similar to an actual disk failure though, if the device weren't there maybe btrfs will complain and ask to be mounted with the `-o degraded` flag. There was nothing exciting in dmesg kern :info : [ +14.363762] BTRFS info (device loop0): using crc32c (crc32c-intel) checksum algorithm kern :info : [ +0.000004] BTRFS info (device loop0): using free space tree Oohh weird... Number of successful reads: 280 Number of IO errors: 0 Successful read files size: sum 82648303047 max 4884066696 average 295172511 sudo btrfs scrub status /mnt/loop/ UUID: a57027e5-feb8-4f58-9022-f5dc0a5c67ac Scrub started: Sun Jan 22 13:33:49 2023 Status: finished Duration: 0:00:28 Total to scrub: 77.25GiB Rate: 2.76GiB/s Error summary: no errors found Okay turns out the deleted file is still connected to the loopback device. sudo losetup -d $ld2 sudo umount /mnt/loop echo 3 | sudo tee /proc/sys/vm/drop_caches Now we get some interesting stuff in dmesg sudo mount -o degraded "$ld1" /mnt/loop mount: /mnt/loop: wrong fs type, bad option, bad superblock on /dev/loop0, missing codepage or helper program, or other error. dmesg(1) may have more information after failed mount system call. kern :info : [Jan22 13:37] tee (222135): drop_caches: 3 kern :info : [ +16.362674] BTRFS info (device loop0): using crc32c (crc32c-intel) checksum algorithm kern :info : [ +0.000004] BTRFS info (device loop0): using free space tree kern :err : [ +0.000419] BTRFS error (device loop0): devid 2 uuid 1b352839-f719-499f-b9a7-25ed4d06e2be is missing kern :err : [ +0.000003] BTRFS error (device loop0): failed to read chunk tree: -2 kern :err : [ +0.000183] BTRFS error (device loop0): open_ctree failed kern :info : [ +11.713125] BTRFS info (device loop0): using crc32c (crc32c-intel) checksum algorithm kern :info : [ +0.000004] BTRFS info (device loop0): allowing degraded mounts kern :info : [ +0.000001] BTRFS info (device loop0): using free space tree kern :warn : [ +0.000167] BTRFS warning (device loop0): devid 2 uuid 1b352839-f719-499f-b9a7-25ed4d06e2be is missing kern :warn : [ +0.007647] BTRFS warning (device loop0): chunk 2177892352 missing 1 devices, max tolerance is 0 for writable mount kern :warn : [ +0.000002] BTRFS warning (device loop0): writable mount is not allowed due to too many missing devices kern :err : [ +0.000155] BTRFS error (device loop0): open_ctree failed But we can still mount it as read-only sudo mount -o ro,degraded "$ld1" /mnt/loop And the results are Number of successful reads: 219 Number of IO errors: 61 Successful read files size: sum 21798190683 max 2122064756 average 99535117 IO error files size: sum 60850112364 max 4884066696 average 997542825 In this test about 26% of data is still fully readable (21798190683 / (21798190683+60850112364)). I also tried another variant of the experiment where I did all of the above but ran this command before removing the disk: sudo rm /mnt/loop/file # a 500 mb file that was included the above tests. I deleted this to give btrfs defrag some room to work sudo btrfs fi defrag -v -r -czstd /mnt/loop/ and the results are not much better... in fact they are worse 20% lol Number of successful reads: 199 Number of IO errors: 80 Successful read files size: sum 16695157031 max 2122064756 average 83895261 IO error files size: sum 65428858016 max 4884066696 average 817860725

wizeman 4y ago

> At least, that's what you should do if you were performing this test on ZFS, Just to clarify: in ZFS you shouldn't just 'umount' the filesystem, you should also export the pool, otherwise the cache would still be in use.

xk3 4y ago

This is very good advice. I did the same preparation, here is the distribution of files before the degraded state: Number of successful reads: 280 Number of IO errors: 0 Successful read files size: sum 82648303047 max 4884066696 average 295172511 then I unmounted the fs, deleted disk 2, echo 3 > /proc/sys/vm/drop_caches, and remounted the fs. sudo umount /mnt/loop echo 3 | sudo tee /proc/sys/vm/drop_caches echo 3 | sudo tee /proc/sys/vm/drop_caches echo 3 | sudo tee /proc/sys/vm/drop_caches dmesg --human --nopager --decode --level emerg,alert,crit,err,warn,notice,info kern :info : [Jan22 13:18] tee (215899): drop_caches: 3 kern :info : [ +3.232287] tee (215931): drop_caches: 3 kern :info : [ +0.775697] tee (215953): drop_caches: 3 rm d2.img sudo mount "$ld1" /mnt/loop I am surprised that mounting worked without error but I guess the device is still active via losetup. I'm assuming this would be similar to an actual disk failure though, if the device weren't there maybe btrfs will complain and ask to be mounted with the `-o degraded` flag. There was nothing exciting in dmesg kern :info : [ +14.363762] BTRFS info (device loop0): using crc32c (crc32c-intel) checksum algorithm kern :info : [ +0.000004] BTRFS info (device loop0): using free space tree Oohh weird... Number of successful reads: 280 Number of IO errors: 0 Successful read files size: sum 82648303047 max 4884066696 average 295172511 sudo btrfs scrub status /mnt/loop/ UUID: a57027e5-feb8-4f58-9022-f5dc0a5c67ac Scrub started: Sun Jan 22 13:33:49 2023 Status: finished Duration: 0:00:28 Total to scrub: 77.25GiB Rate: 2.76GiB/s Error summary: no errors found Okay turns out the deleted file is still connected to the loopback device. sudo losetup -d $ld2 sudo umount /mnt/loop echo 3 | sudo tee /proc/sys/vm/drop_caches Now we get some interesting stuff in dmesg sudo mount -o degraded "$ld1" /mnt/loop mount: /mnt/loop: wrong fs type, bad option, bad superblock on /dev/loop0, missing codepage or helper program, or other error. dmesg(1) may have more information after failed mount system call. kern :info : [Jan22 13:37] tee (222135): drop_caches: 3 kern :info : [ +16.362674] BTRFS info (device loop0): using crc32c (crc32c-intel) checksum algorithm kern :info : [ +0.000004] BTRFS info (device loop0): using free space tree kern :err : [ +0.000419] BTRFS error (device loop0): devid 2 uuid 1b352839-f719-499f-b9a7-25ed4d06e2be is missing kern :err : [ +0.000003] BTRFS error (device loop0): failed to read chunk tree: -2 kern :err : [ +0.000183] BTRFS error (device loop0): open_ctree failed kern :info : [ +11.713125] BTRFS info (device loop0): using crc32c (crc32c-intel) checksum algorithm kern :info : [ +0.000004] BTRFS info (device loop0): allowing degraded mounts kern :info : [ +0.000001] BTRFS info (device loop0): using free space tree kern :warn : [ +0.000167] BTRFS warning (device loop0): devid 2 uuid 1b352839-f719-499f-b9a7-25ed4d06e2be is missing kern :warn : [ +0.007647] BTRFS warning (device loop0): chunk 2177892352 missing 1 devices, max tolerance is 0 for writable mount kern :warn : [ +0.000002] BTRFS warning (device loop0): writable mount is not allowed due to too many missing devices kern :err : [ +0.000155] BTRFS error (device loop0): open_ctree failed But we can still mount it as read-only sudo mount -o ro,degraded "$ld1" /mnt/loop And the results are Number of successful reads: 219 Number of IO errors: 61 Successful read files size: sum 21798190683 max 2122064756 average 99535117 IO error files size: sum 60850112364 max 4884066696 average 997542825 In this test about 26% of data is still fully readable (21798190683 / (21798190683+60850112364)). I also tried another variant of the experiment where I did all of the above but ran this command before removing the disk: sudo rm /mnt/loop/file # a 500 mb file that was included the above tests. I deleted this to give btrfs defrag some room to work sudo btrfs fi defrag -v -r -czstd /mnt/loop/ and the results are not much better... in fact they are worse 20% lol Number of successful reads: 199 Number of IO errors: 80 Successful read files size: sum 16695157031 max 2122064756 average 83895261 IO error files size: sum 65428858016 max 4884066696 average 817860725