5 ms·

How to free up and automatically manage disk space for WSL

67 points by twilight-code 1y ago

Dylan16807 1y ago
I think Optimize-VHD will do the same thing as messing with diskpart. It's a shame you can't just turn on live TRIM support. This kind of disk image is a bunch of multi-megabyte blocks of data, plus a list of where each block goes on the virtual disk. Implementations can support TRIM by deleting the block that's zeroed and moving the block at the end of the file into that spot. VirtualBox can do this, shrinking the file when the guest OS TRIMs, but Hyper-V can't.
- wtallis 1y ago
  It seems like just using a sparse file would make things simpler. I guess there would be potential inefficiency when you try to move the VM to another drive, but for just storing and using the VM in place it doesn't seem like this has to be hard: a TRIM inside the VM gets translated into a hole punch in the disk image file, which gets translated into a TRIM sent to the underlying storage.
  - Dylan16807 1y ago
    Sparse files would be useful but I'm pretty sure sparse files on NTFS always reserve their full size. Eating 1TB of free space for my ext4.vhdx is a non-starter.
    - wtallis 1y ago
      Based on https://learn.microsoft.com/en-us/windows/win32/fileio/sparse-files https://learn.microsoft.com/en-us/windows/win32/fileio/spars... and subsequent links, it sounds like sparse files count against disk quotas as if they weren't sparse, but otherwise sparse files actually work on NTFS. If sparse files didn't save on actual space allocations, it would simply be a lie to call them sparse files. Granted, that's totally something Microsoft would do, and has done for other features they lack, but that doesn't seem to be the case here.
    - banana_giraffe 1y ago
      https://gist.github.com/Q726kbXuN/c80028aa1714032546d3b442c1d7f1c3 https://gist.github.com/Q726kbXuN/c80028aa1714032546d3b442c1... Nope, they don't eat up space. One of the fun side effects of sparse files on NTFS is calling WriteFile on the sparse file can lead to an ERROR_DISK_QUOTA_EXCEEDED error, which applications tend to not expect.
  - bayindirh 1y ago
    rsync in Linux can move sparse files as efficiently as possible, but I don't know if there's an equivalent in Windows. It'd be very strange if there is not.
    - mjmas 1y ago
      as far as i know robocopy should
- vrighter 1y ago
  and the vhdx format already allows for this too. You can mark a block as "present, but zeroed out" (contents of never accessed blocks is undefined). These types of blocks don't have an actual block of data in the file, but still have well defined semantics.
rr808 1y ago
I'm done with WSL. It just makes life too hard, dual disk is a problem, two network endpoints two firewalls, two different hypervisors you can choose from where some things expect one some the other. I used to think a windows laptop would be better for hardware management, maybe it is but I just gave up and installed Linux. My life is so much easier.
- rogerrogerr 1y ago
  Wish there was a good option in corporate environments where I can’t just install Linux.
  - rr808 1y ago
    At work I asked for a remote server and my pc is just a terminal to use ssh & xwindows.
    - pxc 1y ago
      I have something like that at work for testing Windows things, since my work machine is a Mac. It's pretty miserable in terms of latency and window management. If you can get this to work well with NX or X forwarding, that's cool, but it's a second class offering when that's your only option.
      - lostmsu 1y ago
        Hm, are you using native Windows RDP? That one has pretty low latency even in the default configuration (assuming a capable client).
        mook 1y ago
        For me using Microsoft's RDP client to a Windows machine a few states over is rather frustrating; using a slower machine locally was much better.
        vladvasiliu 1y ago
Eridrus 1y ago
Someone should really make a utility to make this easy if Microsoft won't do it.
- twilight-code 1y ago
  There are two options mentioned out there: - https://github.com/brooks-code/WSL-VHDX-Compact https://github.com/brooks-code/WSL-VHDX-Compact - https://github.com/okibcn/wslcompact https://github.com/okibcn/wslcompact and probably (much) more on GitHub :)
jve 1y ago
Actually great intro for step-by-step powershell scripting. I thought I didn't know about "Throw-And-Exit" command, but no, my powershell doesn't have that and google also doesn't know a thing, so wonder what's up with that. And instead of manual confirmations, one can write script with ShouldProcess support to have support for builtin -Confirm/-WhatIf/-Force parameters. And it would actually be a great use case for that scheduled task to run with -Force parameter instead of changing code to strip out confirmations. https://learn.microsoft.com/en-us/powershell/scripting/learn/deep-dives/everything-about-shouldprocess?view=powershell-7.5 https://learn.microsoft.com/en-us/powershell/scripting/learn...
- TacticalMalice 1y ago
  Throw-And-Exit is a custom function in the script @ https://github.com/brooks-code/WSL-VHDX-Compact/blob/main/wsl_compactor.ps1 https://github.com/brooks-code/WSL-VHDX-Compact/blob/main/ws...
  - jve 1y ago
    Well in that case, a regular throw can be used. Don't see a need for custom function. From docs: > The throw keyword causes a terminating error. You can use the throw keyword to stop the processing of a command, function, or script.
    - twilight-code 1y ago
      good point
Saavedro 1y ago
You also may need to run fstrim inside WSL to make all the free space actually compactible
acemarke 1y ago
I recently used https://github.com/okibcn/wslcompact https://github.com/okibcn/wslcompact to shrink a couple of WSL VHDX files, and it worked great. Looking at the source, it seems to use a `wsl --export` option rather than `diskpart`.
- twilight-code 1y ago
  Exporting and re-importing the WSL distro is simpler and usually preferred. The tutorial and script are based on this conversation: https://superuser.com/questions/1606213/how-do-i-get-back-unused-disk-space-from-ubuntu-on-wsl2#1612289 https://superuser.com/questions/1606213/how-do-i-get-back-un... which mentions some issues with the re-import that includes the need to reset the default username.
eviks 1y ago
By the way, is WSL1, which doesn't use vhds and so doesn't suffer from such issues (but has slowed io), effectively abandoned by MS?
- moondev 1y ago
  You can run wsl1 and wsl2 side by side

67 points by twilight-code 1y ago

I think Optimize-VHD will do the same thing as messing with diskpart. It's a shame you can't just turn on live TRIM support. This kind of disk image is a bunch of multi-megabyte blocks of data, plus a list of where each block goes on the virtual disk. Implementations can support TRIM by deleting the block that's zeroed and moving the block at the end of the file into that spot. VirtualBox can do this, shrinking the file when the guest OS TRIMs, but Hyper-V can't.

wtallis 1y ago
It seems like just using a sparse file would make things simpler. I guess there would be potential inefficiency when you try to move the VM to another drive, but for just storing and using the VM in place it doesn't seem like this has to be hard: a TRIM inside the VM gets translated into a hole punch in the disk image file, which gets translated into a TRIM sent to the underlying storage.
- Dylan16807 1y ago
  Sparse files would be useful but I'm pretty sure sparse files on NTFS always reserve their full size. Eating 1TB of free space for my ext4.vhdx is a non-starter.
  - wtallis 1y ago
    Based on https://learn.microsoft.com/en-us/windows/win32/fileio/sparse-files https://learn.microsoft.com/en-us/windows/win32/fileio/spars... and subsequent links, it sounds like sparse files count against disk quotas as if they weren't sparse, but otherwise sparse files actually work on NTFS. If sparse files didn't save on actual space allocations, it would simply be a lie to call them sparse files. Granted, that's totally something Microsoft would do, and has done for other features they lack, but that doesn't seem to be the case here.
  - banana_giraffe 1y ago
    https://gist.github.com/Q726kbXuN/c80028aa1714032546d3b442c1d7f1c3 https://gist.github.com/Q726kbXuN/c80028aa1714032546d3b442c1... Nope, they don't eat up space. One of the fun side effects of sparse files on NTFS is calling WriteFile on the sparse file can lead to an ERROR_DISK_QUOTA_EXCEEDED error, which applications tend to not expect.
- bayindirh 1y ago
  rsync in Linux can move sparse files as efficiently as possible, but I don't know if there's an equivalent in Windows. It'd be very strange if there is not.
  - mjmas 1y ago
    as far as i know robocopy should
vrighter 1y ago
and the vhdx format already allows for this too. You can mark a block as "present, but zeroed out" (contents of never accessed blocks is undefined). These types of blocks don't have an actual block of data in the file, but still have well defined semantics.

rr808 1y ago

I'm done with WSL. It just makes life too hard, dual disk is a problem, two network endpoints two firewalls, two different hypervisors you can choose from where some things expect one some the other. I used to think a windows laptop would be better for hardware management, maybe it is but I just gave up and installed Linux. My life is so much easier.

rogerrogerr 1y ago
Wish there was a good option in corporate environments where I can’t just install Linux.
- rr808 1y ago
  At work I asked for a remote server and my pc is just a terminal to use ssh & xwindows.
  - pxc 1y ago
    I have something like that at work for testing Windows things, since my work machine is a Mac. It's pretty miserable in terms of latency and window management. If you can get this to work well with NX or X forwarding, that's cool, but it's a second class offering when that's your only option.
    - lostmsu 1y ago
      Hm, are you using native Windows RDP? That one has pretty low latency even in the default configuration (assuming a capable client).
      - mook 1y ago
        For me using Microsoft's RDP client to a Windows machine a few states over is rather frustrating; using a slower machine locally was much better.
      - vladvasiliu 1y ago

Eridrus 1y ago

Someone should really make a utility to make this easy if Microsoft won't do it.

twilight-code 1y ago
There are two options mentioned out there: - https://github.com/brooks-code/WSL-VHDX-Compact https://github.com/brooks-code/WSL-VHDX-Compact - https://github.com/okibcn/wslcompact https://github.com/okibcn/wslcompact and probably (much) more on GitHub :)

jve 1y ago

Actually great intro for step-by-step powershell scripting. I thought I didn't know about "Throw-And-Exit" command, but no, my powershell doesn't have that and google also doesn't know a thing, so wonder what's up with that. And instead of manual confirmations, one can write script with ShouldProcess support to have support for builtin -Confirm/-WhatIf/-Force parameters. And it would actually be a great use case for that scheduled task to run with -Force parameter instead of changing code to strip out confirmations. https://learn.microsoft.com/en-us/powershell/scripting/learn/deep-dives/everything-about-shouldprocess?view=powershell-7.5 https://learn.microsoft.com/en-us/powershell/scripting/learn...

TacticalMalice 1y ago
Throw-And-Exit is a custom function in the script @ https://github.com/brooks-code/WSL-VHDX-Compact/blob/main/wsl_compactor.ps1 https://github.com/brooks-code/WSL-VHDX-Compact/blob/main/ws...
- jve 1y ago
  Well in that case, a regular throw can be used. Don't see a need for custom function. From docs: > The throw keyword causes a terminating error. You can use the throw keyword to stop the processing of a command, function, or script.
  - twilight-code 1y ago
    good point

Saavedro 1y ago

You also may need to run fstrim inside WSL to make all the free space actually compactible

acemarke 1y ago

I recently used https://github.com/okibcn/wslcompact https://github.com/okibcn/wslcompact to shrink a couple of WSL VHDX files, and it worked great. Looking at the source, it seems to use a `wsl --export` option rather than `diskpart`.

twilight-code 1y ago
Exporting and re-importing the WSL distro is simpler and usually preferred. The tutorial and script are based on this conversation: https://superuser.com/questions/1606213/how-do-i-get-back-unused-disk-space-from-ubuntu-on-wsl2#1612289 https://superuser.com/questions/1606213/how-do-i-get-back-un... which mentions some issues with the re-import that includes the need to reset the default username.

eviks 1y ago

By the way, is WSL1, which doesn't use vhds and so doesn't suffer from such issues (but has slowed io), effectively abandoned by MS?

moondev 1y ago
You can run wsl1 and wsl2 side by side