7 ms·

A neat trick I was told is to always have ballast files on your systems. Just a few GiB of zeros that you can delete in cases like this. This won't fix the prob

by flanfly 6mo ago

A neat trick I was told is to always have ballast files on your systems. Just a few GiB of zeros that you can delete in cases like this. This won't fix the problem, but will buy you time and free space for stuff like lock files so you can get a working system.

jaapz 6mo ago
Love the simplicity and pragmatism of this solution
- d4lt4 6mo ago
  [dead]
omarqureshi 6mo ago
Surely a 50% warning alarm on disk usage covers this without manual intervention?
- jcims 6mo ago
  If the alarms are reliably configured, confirmed to be working, low noise enough to be actioned, etc etc. And of course there's nothing to say that both of these things can't be done simultaneously.
- theshrike79 6mo ago
  Depends. A Kubernetes container might have only a few megabytes of disk space, because it shouldn't need it. Except that one time when .NET decides that the incoming POST is over some magic limit and it doesn't do the processing in-memory like before, but instead has to write it to disk, crashing the whole pod. Fun times. Also my Unraid NAS has two drives in "WARNING! 98% USED" alert state. One has 200GB of free space, the other 330GB. Percentages in integers don't work when the starting number is too big :)
  - majormajor 5mo ago
    The "ballast file" idea doesn't really change that spill-to-disk crash, as far as I can tell. You have to delete it manually; it already crashed by the time you realize it. Seems like the sort of thing that only makes sense in a "I know my cheapskate boss won't have larger drives ready to go (or be willing to pay to expand it in a cloud scenario), and he insists that the alarm not go off until 95%, but it'll be my fault if we have a bad incident we can't recover quickly from, so I'm gonna give myself some headroom by padding things a bit" extra-paranoid scenario.
- dspillett 6mo ago
  If the alarm works. And it actioned not just snoozed too much or just dismissed entirely. Defence in depth is a good idea: proper alarms, and a secondary measure in case they don't have the intended effect.
  - pixl97 6mo ago
    Alarms are great, but when something goes wrong SSDs can fill up amazingly fast!
  - n4r9 6mo ago
    Surely there are pitfalls either way. A ballast file can be deleted too readily, or someone could forget to re-add it.
    - dspillett 5mo ago
      Yep. That is why doing both can be beneficial. Alerts are more proactive if acted upon, but often too easy to ignore meaning ballast is more fail-safe in that respect.
  - jamiemallers 6mo ago
    [dead]
- coredog64 6mo ago
  You don't want an alarm on a usage threshold, you want a linear regression that predicts when utilization will cross a threshold. Then you set your alarms for "How long does it take me to remediate this condition?"
  - dotancohen 5mo ago
    That's far more complicated and fragile. Where are you storing this log of disk usage? If you already have some external time series database then this is already a solved problem. But for a single server, desktop, or embedded device you'll need a database or text log, a cron job to measure it, and another script to parse, make predictions, and then raise alerts. And a single large dump to disk, like some daemon suddenly bugging out and writing incessantly to logs, will render all that moot anyway.
- evil-olive 5mo ago
  > Surely a 50% warning alarm on disk usage covers this without manual intervention? surely you don't need a fire extinguisher in your kitchen, if you have a smoke detector? a "warning alarm" is a terrible concept, in general. it's a perfect way to lead to alert fatigue. over time, you're likely to have someone silence the alarm because there's some host sitting at 57% disk usage for totally normal reasons and they're tired of getting spammed about it. even well-tuned alert rules (ones that predict growth over time rather than only looking at the current value) tend to be targeted towards catching relatively "slow" leaks of disk usage. there is always the possibility for a "fast" disk space consumer to fill up the disk more quickly than your alerting system can bring it to your attention and you can fix it. at the extreme end, for example, a standard EBS volume has a throughput of 125mb/sec. something that saturates that limit will fill up 10gb of free space in 80 seconds.
  - ssl-3 5mo ago
    50% is probably unrealistic. Nobody really wants to diminish their storage by 50%. Let's set a fixed threshold -- 100GB, say -- and play out both methods. Method A: One or more ballast files are created, totalling 100GB. The machine runs out of storage and grinds to a halt. Hopefully someone notices soon or gets a generic alert that it has ceased, remembers that there's ballast files, and deletes one or more of them. They then poke it with a stick and get it going again, and set forth to resolve whatever was causing the no-storage condition (adding disk, cleaning trash, or whatever). Method B: A specific alert that triggers with <100GB of free space. Someone sees this alert, understands what it means (because it is descriptive instead of generic), and logs in to resolve the low-storage condition (however that is done -- same as Method A). There is no stick-poking. Method C: The control. We do nothing, and run out of space. Panic ensues. Articles are written. --- Both A and B methods have an equal number of alerts for each low-disk condition (<100GB). Both methods work, in that they can form the impetus to free up some space. But Method A relies on a system to crash, while Method B does not rely upon a crash at all. I think that the lack of crash makes Method B rather superior all on its own. (Method C sucks.)
    - tempestn 5mo ago
fifilura 6mo ago
I did this too, but i also zipped the file, turns out it had great packing ratio!
- saagarjha 6mo ago
  Personally I just keep the file on a ramdisk so you can avoid having to fetch it from slow storage
  - 3form 6mo ago
    Neat! I optimized for my own case, and I'm storing my ramdisk on SSD to gain persistence.
ninalanyon 6mo ago
This is why I never empty the Rubbish Bin/trash Can on my Linux laptop until the disk fills.
- zrm 5mo ago
  That's not a great idea for three different reasons: Filesystems have to do ugly things when they're almost full like split files into many small blocks and store more metadata to keep track of them all, SSDs get slower and have compromised wear leveling when they're almost full, and it makes you more likely to subject yourself to perils of fully running out which can cause random non-temporary problems even if it only happens temporarily.
testplzignore 6mo ago
Would another way be to drop the reserved space (typically 1% to 5% on an ext file system)?
- bombcar 6mo ago
  Reserved space doesn't protect you against root, who is often the user to blame for the last used MB.
dspillett 6mo ago
Similarly, I always leave some space unallocated on LMV volume groups. It means that I can temporarily expand a volume easily if needed. It also serves to leave some space unused to help out the wear-levelling on the SSDs on which the RAID array that is the PV¹ for LVM. I'm, not 100% sure this is needed any more² but I've not looked into that sufficiently so until I do I'll keep the habit. -------- [1] if there are multiple PVs, from different drives/arrays, in the VG, then you might need to manually skip a bit on each one because LVM will naturally fill one before using the next. Just allocate a small LV specially on each and don't use it. You can remove one/all of them and add the extents to the fill LV if/when needed. Giving it a useful name also reminds you why that bit of space is carved out. [2] drives under-allocate by default IIRC
- justsomehnguy 5mo ago
  Not needed. All your unused/unfilled space is that space for wear-leveling. It wasn't needed even back then besides some corner cases. And most importantly 10% of the drive in ~2010 were 6-12GB, nowadays it's 50-100GB at least.
  - Dylan16807 5mo ago
    Empty space is good for wear-leveling but enforcing a few percent extra helps. > And most importantly 10% of the drive in ~2010 were 6-12GB, nowadays it's 50-100GB at least. Back then you were paying about $2 per gigabyte. Right now SSDs are 1/15th as expensive. If we use the prices from last year they're 1/30th, and if we also factor in inflation it's around 1/50th. So while I would say to use a lower percentage as space increases, 50-100GB is no problem at all.
    - justsomehnguy 5mo ago
      > but enforcing a few percent extra helps. Only if you fill the drive up to 95-99% and do this often. Otherwise it's just a cargo-cult. > So while I would say to use a lower percentage as space increases If your drive is over-provisioned (eg 960GB instead of 1024GB) then it's not needed. If not and you fill your drive to the full and just want to be sure then you need the size of the biggest write you would do plus some leeway, eg if you often write 20GB video files for whatever reason then 30-40GB would be more than enough. Leaving 100GB of 1TB drive is like buying a sneakers but not wearing them because they would wear.
Chaosvex 6mo ago
Similar to the old game development trick of hiding some memory away and then freeing it up near the end of development when the budget starts getting tight.
bombcar 6mo ago
Some filesystems can be unable to delete a file if full. Something to be a bit worried about.
- 6031769 6mo ago
  Please name and shame those filesystems so that we will all be forewarned.
  - deleted 5mo ago
    [deleted]
  - SAI_Peregrinus 5mo ago
    Any Copy-on-Write filesystem can run into this. There's always some way around it, but it can be problematic if you only have one device, can't remember the steps to fix a full filesystem, and can't look up the steps because you can't launch a browser without it trying to make some files!
    - _carbyau_ 5mo ago
      > and can't look up the steps because you can't launch a browser My early days of computing got easier when I had a second computer to look up the issues of the first computer.
- seized 5mo ago
  Instead of deleting the ballast file you can just truncate it. That works on ZFS when you fill the pool and delete starts failing.
  - nagaiaida 5mo ago
    under what circumstances does deleting files fail on a full pool? i have one that fills up semiregularly and i've never had issues that required me to truncate files
throw0101d 6mo ago
> A neat trick I was told is to always have ballast files on your systems. ZFS has a "reservation" mechanism that's handy: > The minimum amount of space guaranteed to a dataset, not including its descendants. When the amount of space used is below this value, the dataset is treated as if it were taking up the amount of space specified by refreservation. The refreservation reservation is accounted for in the parent datasets' space used, and counts against the parent datasets' quotas and reservations. * https://openzfs.github.io/openzfs-docs/man/master/7/zfsprops.7.html#refreservation https://openzfs.github.io/openzfs-docs/man/master/7/zfsprops... Quotas prevent users/groups/directories (ZFS datasets) from using too much space, but reservations ensure that particular areas always have a minimum amount set aside for them.
- dizhn 6mo ago
  Also if you VMs on a disk backed by ZFS it's trivial to extend those disks provided you actually do have space on the real disk. (Even automatic with LXC).
  - rubatuga 5mo ago
    Please explain!
    - dizhn 5mo ago
      ZFS supports instant resizing of datasets. When that dataset is the virtual disk for a VM you can just increase its size on the hypervisor, then it's a simple growfs operation for the VM to see the increased size. On LXC the dataset is usually mounted directly so the resize operation is reflected immediately. I use Proxmox as the hypervisor, and the ZFS resize part is supported on the GUI and it's trivial to use. Let me know if you need more details.
- throw0101d 5mo ago
  Typo; link should be: * https://openzfs.github.io/openzfs-docs/man/master/7/zfsprops.7.html#reservation https://openzfs.github.io/openzfs-docs/man/master/7/zfsprops... Addendum: there's also the built-in compression functionality: > When set to on (the default), indicates that the current default compression algorithm should be used. The default balances compression and decompression speed, with compression ratio and is expected to work well on a wide variety of workloads. Unlike all other settings for this property, on does not select a fixed compression type. As new compression algorithms are added to ZFS and enabled on a pool, the default compression algorithm may change. The current default compression algorithm is either lzjb or, if the lz4_compress feature is enabled, lz4. * https://openzfs.github.io/openzfs-docs/man/master/7/zfsprops.7.html#compression https://openzfs.github.io/openzfs-docs/man/master/7/zfsprops...
dijit 6mo ago
I always called it a “bit-mass”. Like a thermal mass used in freezers in places where the power is not very stable. I knew I didn’t invent the concept, as there’s so many systems that cannot recover if the disk is totally full. (a write may be required in many systems in order to execute an instruction to remove things gracefully). The latest thing I found with this issue is Unreal Engines Horde build system, its so tightly coupled with caches, object files and database references: that a manual clean up is extremely difficult and likely to create an unstable system. But you can configure it to have fewer build artefacts kept around and then it will clear itself out gracefully. - but it needs to be able to write to the disk to do it. Now that I think about it, I don’t do this for inodes, but you can run out of those too and end up in a weird “out of disk” situation despite having lots of usable capacity left.
layer8 6mo ago
Better fill those files with random bytes, to ensure the filesystem doesn’t apply some “I don’t actually have to store all-zero blocks” sparse-file optimization. To my knowledge no non-compressing file system currently does this, but who knows about the future.
- ape4 6mo ago
  If I recall correctly: dd if=/dev/urandom of=/home/myrandomfile bs=1 count=N
  - tdeck 5mo ago
    Fwiw you can also do this with head -c 1G /dev/urandom > /home/myrandomfile And not have to remember dd's bizarre snowflake command syntax.
  - fragmede 5mo ago
    bs=1 is a recipe for waiting far longer than you have to because of the overhead of the system calls. Better bs=N count=1
    - __david__ 5mo ago
      That’s also not great if you’re trying to make a 10 gigabyte file. In that case, use bs=1M and count=SizeInMB.
      - marcosdumay 5mo ago
        Modern computers are crazily overengineered... Most current desktops (smaller than your usual server) won't have any problem with the GP's command. Yours is still better, of course.
    - deleted 5mo ago
      [deleted]
  - Twirrim 5mo ago
    If you want to do it really quickly openssl enc -aes-256-ctr -pbkdf2 -pass pass:"$(date '+%s')" < /dev/zero | dd of=/home/myrandomfile bs=1M count=1024 Almost all CPUs have AES native instructions so you'll be able to produce pseudorandom junk really fast. Even my old system will produce it at about 3Gb/s. Much faster than urandom can go.
dj0k3r 6mo ago
I did this recently, aka, docker images prune. Can confirm, saved the day.
HoldOnAMinute 5mo ago
Sounds like something straight out of Dilbert
happycrappy 5mo ago
Interesting strategy, can't believe I've never heard of this one before. Would it be more pragmatic to allocate a swap file instead? Something that provides a theoretical benefit in the short term vs a static reservation.
- prmoustache 5mo ago
  Because adding swap file is instantaneous, removing one that is in use can take a longtime unless you reboot the OS so you can't just nuke it quickly.
  - esseph 5mo ago
    Let's get crazy 1. swapoff 2. drop disk cache (1) 3. panik!?! hrm, seems ok
    - esseph 5mo ago
      Actually there should be a 'sync' as step 2 :)
    - genewitch 5mo ago
      it has been 3 days since esseph has nuked prod
      - esseph 5mo ago
        "That's fine let's just restore from the backups." "..." "Oh no, restore failed" (I feel like this is about 9 out of 10 companies these days) Reminder to regularly test backup restoration!
jasonpeacock 5mo ago
> A neat trick I was told is to always have sleep statements in your code. Just a few sleep statements that you can delete in cases like this. This won't fix the problem, but will buy you time and free up latency for stuff like slow algorithms so you can get faster code. FTFY ;)
klaushardt 5mo ago
This is my snippet i used alot. In doubt when even rm wont work just reboot. Disc Space Insurance File fallocate -l 8G /tmp/DELETE_IF_OUT_OF_SPACE.img https://gist.github.com/klaushardt/9a5f6b0b078d28a23fd968f7549f6bba#disc-space-insurance-file https://gist.github.com/klaushardt/9a5f6b0b078d28a23fd968f75...
- drybjed 5mo ago
  Make sure your /tmp is on disk and not a tmpfs, like in recent Linux distrubitions.
bguebert 5mo ago
This saved us a couple times. At least until I had time to add monitoring to their old system to track disk usage. It was also helpful to use a tool called ncdu. It helps you visualize where most disk space is getting used up to track down the problem.
- greedo 5mo ago
  ncdu is a lifesaver...
reddalo 5mo ago
This trick is actually used by some banking apps. They fill app their mobile apps with junk data just to make the APK/IPA bigger. So if they need to push an urgent update, they won't have users that can't update because their phones are full to the brim. I know two Italian banks that do it, Unicredit and Intesa. The latter was on the news when a user found out that one of the filler files was a burp recording [1]. [1] https://www.ilfattoquotidiano.it/2024/12/20/intesa-san-paolo-file-rutto-mp3/7811551/ https://www.ilfattoquotidiano.it/2024/12/20/intesa-san-paolo... (in Italian)
- Dylan16807 5mo ago
  But you still need a bunch of extra space to download and unpack the new version, and there are so many apps that need to share space, and a banking app should only need about 0.1% of a phone's storage... Whoever gave them that idea was doing a bad deed.
  - reddalo 5mo ago
    I know and I agree with you. It doesn't seem that smart. And you can tell by the fact that the filler data is called "burp.mp3" and things like that.
- ikr678 5mo ago
  Doesnt this create an arms race situation where every 'critical' app claims a larger diskspace than necessary, just in case, and accelerates the issue?
  - bentcorner 5mo ago
    Kinda sorta, but there's a limit where users will typically install X apps and apps of Y size need Z extra space to update. User content would fill up the rest. I would imagine a typical 256 gb phone is probably over this limit and people who take lots of videos/photos just need to clean up their phone a little more often.
  - deleted 5mo ago
    [deleted]
snthpy 5mo ago
Great idea, thanks!
johntash 5mo ago
ext2/ext3/ext4 all automatically reserve an amount of space on a partition. 5% iirc
totetsu 5mo ago
Also good for stopping phone-home auto firmware updates
PunchyHamster 5mo ago
Just use LVM and don't allocate all of it to LV. We have a script that basically slowly expands volume when demand grows, up to a limit. So we don't have to think on stuff like "does the logs partition need to be 1 or 10GB", it will expand to the sane limit, and if it hits that we get disk usage alert before it finishes so we can either see what's going on (app shat in logs), or take a look for the one in the 10 apps that need some special tuning there
functional_dev 5mo ago
Such a simple trick, but effective!

by flanfly 6mo ago

jaapz 6mo ago

Love the simplicity and pragmatism of this solution

d4lt4 6mo ago
[dead]

omarqureshi 6mo ago

Surely a 50% warning alarm on disk usage covers this without manual intervention?

jcims 6mo ago
If the alarms are reliably configured, confirmed to be working, low noise enough to be actioned, etc etc. And of course there's nothing to say that both of these things can't be done simultaneously.
theshrike79 6mo ago
Depends. A Kubernetes container might have only a few megabytes of disk space, because it shouldn't need it. Except that one time when .NET decides that the incoming POST is over some magic limit and it doesn't do the processing in-memory like before, but instead has to write it to disk, crashing the whole pod. Fun times. Also my Unraid NAS has two drives in "WARNING! 98% USED" alert state. One has 200GB of free space, the other 330GB. Percentages in integers don't work when the starting number is too big :)
- majormajor 5mo ago
  The "ballast file" idea doesn't really change that spill-to-disk crash, as far as I can tell. You have to delete it manually; it already crashed by the time you realize it. Seems like the sort of thing that only makes sense in a "I know my cheapskate boss won't have larger drives ready to go (or be willing to pay to expand it in a cloud scenario), and he insists that the alarm not go off until 95%, but it'll be my fault if we have a bad incident we can't recover quickly from, so I'm gonna give myself some headroom by padding things a bit" extra-paranoid scenario.
dspillett 6mo ago
If the alarm works. And it actioned not just snoozed too much or just dismissed entirely. Defence in depth is a good idea: proper alarms, and a secondary measure in case they don't have the intended effect.
- pixl97 6mo ago
  Alarms are great, but when something goes wrong SSDs can fill up amazingly fast!
- n4r9 6mo ago
  Surely there are pitfalls either way. A ballast file can be deleted too readily, or someone could forget to re-add it.
  - dspillett 5mo ago
    Yep. That is why doing both can be beneficial. Alerts are more proactive if acted upon, but often too easy to ignore meaning ballast is more fail-safe in that respect.
- jamiemallers 6mo ago
  [dead]
coredog64 6mo ago
You don't want an alarm on a usage threshold, you want a linear regression that predicts when utilization will cross a threshold. Then you set your alarms for "How long does it take me to remediate this condition?"
- dotancohen 5mo ago
  That's far more complicated and fragile. Where are you storing this log of disk usage? If you already have some external time series database then this is already a solved problem. But for a single server, desktop, or embedded device you'll need a database or text log, a cron job to measure it, and another script to parse, make predictions, and then raise alerts. And a single large dump to disk, like some daemon suddenly bugging out and writing incessantly to logs, will render all that moot anyway.
evil-olive 5mo ago
> Surely a 50% warning alarm on disk usage covers this without manual intervention? surely you don't need a fire extinguisher in your kitchen, if you have a smoke detector? a "warning alarm" is a terrible concept, in general. it's a perfect way to lead to alert fatigue. over time, you're likely to have someone silence the alarm because there's some host sitting at 57% disk usage for totally normal reasons and they're tired of getting spammed about it. even well-tuned alert rules (ones that predict growth over time rather than only looking at the current value) tend to be targeted towards catching relatively "slow" leaks of disk usage. there is always the possibility for a "fast" disk space consumer to fill up the disk more quickly than your alerting system can bring it to your attention and you can fix it. at the extreme end, for example, a standard EBS volume has a throughput of 125mb/sec. something that saturates that limit will fill up 10gb of free space in 80 seconds.
- ssl-3 5mo ago
  50% is probably unrealistic. Nobody really wants to diminish their storage by 50%. Let's set a fixed threshold -- 100GB, say -- and play out both methods. Method A: One or more ballast files are created, totalling 100GB. The machine runs out of storage and grinds to a halt. Hopefully someone notices soon or gets a generic alert that it has ceased, remembers that there's ballast files, and deletes one or more of them. They then poke it with a stick and get it going again, and set forth to resolve whatever was causing the no-storage condition (adding disk, cleaning trash, or whatever). Method B: A specific alert that triggers with <100GB of free space. Someone sees this alert, understands what it means (because it is descriptive instead of generic), and logs in to resolve the low-storage condition (however that is done -- same as Method A). There is no stick-poking. Method C: The control. We do nothing, and run out of space. Panic ensues. Articles are written. --- Both A and B methods have an equal number of alerts for each low-disk condition (<100GB). Both methods work, in that they can form the impetus to free up some space. But Method A relies on a system to crash, while Method B does not rely upon a crash at all. I think that the lack of crash makes Method B rather superior all on its own. (Method C sucks.)
  - tempestn 5mo ago

fifilura 6mo ago

I did this too, but i also zipped the file, turns out it had great packing ratio!

saagarjha 6mo ago
Personally I just keep the file on a ramdisk so you can avoid having to fetch it from slow storage
- 3form 6mo ago
  Neat! I optimized for my own case, and I'm storing my ramdisk on SSD to gain persistence.

ninalanyon 6mo ago

This is why I never empty the Rubbish Bin/trash Can on my Linux laptop until the disk fills.

zrm 5mo ago
That's not a great idea for three different reasons: Filesystems have to do ugly things when they're almost full like split files into many small blocks and store more metadata to keep track of them all, SSDs get slower and have compromised wear leveling when they're almost full, and it makes you more likely to subject yourself to perils of fully running out which can cause random non-temporary problems even if it only happens temporarily.

testplzignore 6mo ago

Would another way be to drop the reserved space (typically 1% to 5% on an ext file system)?

bombcar 6mo ago
Reserved space doesn't protect you against root, who is often the user to blame for the last used MB.

dspillett 6mo ago

Similarly, I always leave some space unallocated on LMV volume groups. It means that I can temporarily expand a volume easily if needed. It also serves to leave some space unused to help out the wear-levelling on the SSDs on which the RAID array that is the PV¹ for LVM. I'm, not 100% sure this is needed any more² but I've not looked into that sufficiently so until I do I'll keep the habit. -------- [1] if there are multiple PVs, from different drives/arrays, in the VG, then you might need to manually skip a bit on each one because LVM will naturally fill one before using the next. Just allocate a small LV specially on each and don't use it. You can remove one/all of them and add the extents to the fill LV if/when needed. Giving it a useful name also reminds you why that bit of space is carved out. [2] drives under-allocate by default IIRC

justsomehnguy 5mo ago
Not needed. All your unused/unfilled space is that space for wear-leveling. It wasn't needed even back then besides some corner cases. And most importantly 10% of the drive in ~2010 were 6-12GB, nowadays it's 50-100GB at least.
- Dylan16807 5mo ago
  Empty space is good for wear-leveling but enforcing a few percent extra helps. > And most importantly 10% of the drive in ~2010 were 6-12GB, nowadays it's 50-100GB at least. Back then you were paying about $2 per gigabyte. Right now SSDs are 1/15th as expensive. If we use the prices from last year they're 1/30th, and if we also factor in inflation it's around 1/50th. So while I would say to use a lower percentage as space increases, 50-100GB is no problem at all.
  - justsomehnguy 5mo ago
    > but enforcing a few percent extra helps. Only if you fill the drive up to 95-99% and do this often. Otherwise it's just a cargo-cult. > So while I would say to use a lower percentage as space increases If your drive is over-provisioned (eg 960GB instead of 1024GB) then it's not needed. If not and you fill your drive to the full and just want to be sure then you need the size of the biggest write you would do plus some leeway, eg if you often write 20GB video files for whatever reason then 30-40GB would be more than enough. Leaving 100GB of 1TB drive is like buying a sneakers but not wearing them because they would wear.

Chaosvex 6mo ago

Similar to the old game development trick of hiding some memory away and then freeing it up near the end of development when the budget starts getting tight.

bombcar 6mo ago

Some filesystems can be unable to delete a file if full. Something to be a bit worried about.

6031769 6mo ago
Please name and shame those filesystems so that we will all be forewarned.
- deleted 5mo ago
  [deleted]
- SAI_Peregrinus 5mo ago
  Any Copy-on-Write filesystem can run into this. There's always some way around it, but it can be problematic if you only have one device, can't remember the steps to fix a full filesystem, and can't look up the steps because you can't launch a browser without it trying to make some files!
  - _carbyau_ 5mo ago
    > and can't look up the steps because you can't launch a browser My early days of computing got easier when I had a second computer to look up the issues of the first computer.
seized 5mo ago
Instead of deleting the ballast file you can just truncate it. That works on ZFS when you fill the pool and delete starts failing.
- nagaiaida 5mo ago
  under what circumstances does deleting files fail on a full pool? i have one that fills up semiregularly and i've never had issues that required me to truncate files

throw0101d 6mo ago

> A neat trick I was told is to always have ballast files on your systems. ZFS has a "reservation" mechanism that's handy: > The minimum amount of space guaranteed to a dataset, not including its descendants. When the amount of space used is below this value, the dataset is treated as if it were taking up the amount of space specified by refreservation. The refreservation reservation is accounted for in the parent datasets' space used, and counts against the parent datasets' quotas and reservations. * https://openzfs.github.io/openzfs-docs/man/master/7/zfsprops.7.html#refreservation https://openzfs.github.io/openzfs-docs/man/master/7/zfsprops... Quotas prevent users/groups/directories (ZFS datasets) from using too much space, but reservations ensure that particular areas always have a minimum amount set aside for them.

dizhn 6mo ago
Also if you VMs on a disk backed by ZFS it's trivial to extend those disks provided you actually do have space on the real disk. (Even automatic with LXC).
- rubatuga 5mo ago
  Please explain!
  - dizhn 5mo ago
    ZFS supports instant resizing of datasets. When that dataset is the virtual disk for a VM you can just increase its size on the hypervisor, then it's a simple growfs operation for the VM to see the increased size. On LXC the dataset is usually mounted directly so the resize operation is reflected immediately. I use Proxmox as the hypervisor, and the ZFS resize part is supported on the GUI and it's trivial to use. Let me know if you need more details.
throw0101d 5mo ago
Typo; link should be: * https://openzfs.github.io/openzfs-docs/man/master/7/zfsprops.7.html#reservation https://openzfs.github.io/openzfs-docs/man/master/7/zfsprops... Addendum: there's also the built-in compression functionality: > When set to on (the default), indicates that the current default compression algorithm should be used. The default balances compression and decompression speed, with compression ratio and is expected to work well on a wide variety of workloads. Unlike all other settings for this property, on does not select a fixed compression type. As new compression algorithms are added to ZFS and enabled on a pool, the default compression algorithm may change. The current default compression algorithm is either lzjb or, if the lz4_compress feature is enabled, lz4. * https://openzfs.github.io/openzfs-docs/man/master/7/zfsprops.7.html#compression https://openzfs.github.io/openzfs-docs/man/master/7/zfsprops...

dijit 6mo ago

I always called it a “bit-mass”. Like a thermal mass used in freezers in places where the power is not very stable. I knew I didn’t invent the concept, as there’s so many systems that cannot recover if the disk is totally full. (a write may be required in many systems in order to execute an instruction to remove things gracefully). The latest thing I found with this issue is Unreal Engines Horde build system, its so tightly coupled with caches, object files and database references: that a manual clean up is extremely difficult and likely to create an unstable system. But you can configure it to have fewer build artefacts kept around and then it will clear itself out gracefully. - but it needs to be able to write to the disk to do it. Now that I think about it, I don’t do this for inodes, but you can run out of those too and end up in a weird “out of disk” situation despite having lots of usable capacity left.

layer8 6mo ago

Better fill those files with random bytes, to ensure the filesystem doesn’t apply some “I don’t actually have to store all-zero blocks” sparse-file optimization. To my knowledge no non-compressing file system currently does this, but who knows about the future.

ape4 6mo ago
If I recall correctly: dd if=/dev/urandom of=/home/myrandomfile bs=1 count=N
- tdeck 5mo ago
  Fwiw you can also do this with head -c 1G /dev/urandom > /home/myrandomfile And not have to remember dd's bizarre snowflake command syntax.
- fragmede 5mo ago
  bs=1 is a recipe for waiting far longer than you have to because of the overhead of the system calls. Better bs=N count=1
  - __david__ 5mo ago
    That’s also not great if you’re trying to make a 10 gigabyte file. In that case, use bs=1M and count=SizeInMB.
    - marcosdumay 5mo ago
      Modern computers are crazily overengineered... Most current desktops (smaller than your usual server) won't have any problem with the GP's command. Yours is still better, of course.
  - deleted 5mo ago
    [deleted]
- Twirrim 5mo ago
  If you want to do it really quickly openssl enc -aes-256-ctr -pbkdf2 -pass pass:"$(date '+%s')" < /dev/zero | dd of=/home/myrandomfile bs=1M count=1024 Almost all CPUs have AES native instructions so you'll be able to produce pseudorandom junk really fast. Even my old system will produce it at about 3Gb/s. Much faster than urandom can go.

dj0k3r 6mo ago

I did this recently, aka, docker images prune. Can confirm, saved the day.

HoldOnAMinute 5mo ago

Sounds like something straight out of Dilbert

happycrappy 5mo ago

Interesting strategy, can't believe I've never heard of this one before. Would it be more pragmatic to allocate a swap file instead? Something that provides a theoretical benefit in the short term vs a static reservation.

prmoustache 5mo ago
Because adding swap file is instantaneous, removing one that is in use can take a longtime unless you reboot the OS so you can't just nuke it quickly.
- esseph 5mo ago
  Let's get crazy 1. swapoff 2. drop disk cache (1) 3. panik!?! hrm, seems ok
  - esseph 5mo ago
    Actually there should be a 'sync' as step 2 :)
  - genewitch 5mo ago
    it has been 3 days since esseph has nuked prod
    - esseph 5mo ago
      "That's fine let's just restore from the backups." "..." "Oh no, restore failed" (I feel like this is about 9 out of 10 companies these days) Reminder to regularly test backup restoration!

jasonpeacock 5mo ago

> A neat trick I was told is to always have sleep statements in your code. Just a few sleep statements that you can delete in cases like this. This won't fix the problem, but will buy you time and free up latency for stuff like slow algorithms so you can get faster code. FTFY ;)

klaushardt 5mo ago

This is my snippet i used alot. In doubt when even rm wont work just reboot. Disc Space Insurance File fallocate -l 8G /tmp/DELETE_IF_OUT_OF_SPACE.img https://gist.github.com/klaushardt/9a5f6b0b078d28a23fd968f7549f6bba#disc-space-insurance-file https://gist.github.com/klaushardt/9a5f6b0b078d28a23fd968f75...

drybjed 5mo ago
Make sure your /tmp is on disk and not a tmpfs, like in recent Linux distrubitions.

bguebert 5mo ago

This saved us a couple times. At least until I had time to add monitoring to their old system to track disk usage. It was also helpful to use a tool called ncdu. It helps you visualize where most disk space is getting used up to track down the problem.

greedo 5mo ago
ncdu is a lifesaver...

reddalo 5mo ago

This trick is actually used by some banking apps. They fill app their mobile apps with junk data just to make the APK/IPA bigger. So if they need to push an urgent update, they won't have users that can't update because their phones are full to the brim. I know two Italian banks that do it, Unicredit and Intesa. The latter was on the news when a user found out that one of the filler files was a burp recording [1]. [1] https://www.ilfattoquotidiano.it/2024/12/20/intesa-san-paolo-file-rutto-mp3/7811551/ https://www.ilfattoquotidiano.it/2024/12/20/intesa-san-paolo... (in Italian)

Dylan16807 5mo ago
But you still need a bunch of extra space to download and unpack the new version, and there are so many apps that need to share space, and a banking app should only need about 0.1% of a phone's storage... Whoever gave them that idea was doing a bad deed.
- reddalo 5mo ago
  I know and I agree with you. It doesn't seem that smart. And you can tell by the fact that the filler data is called "burp.mp3" and things like that.
ikr678 5mo ago
Doesnt this create an arms race situation where every 'critical' app claims a larger diskspace than necessary, just in case, and accelerates the issue?
- bentcorner 5mo ago
  Kinda sorta, but there's a limit where users will typically install X apps and apps of Y size need Z extra space to update. User content would fill up the rest. I would imagine a typical 256 gb phone is probably over this limit and people who take lots of videos/photos just need to clean up their phone a little more often.
- deleted 5mo ago
  [deleted]

snthpy 5mo ago

Great idea, thanks!

johntash 5mo ago

ext2/ext3/ext4 all automatically reserve an amount of space on a partition. 5% iirc

totetsu 5mo ago

Also good for stopping phone-home auto firmware updates

PunchyHamster 5mo ago

Just use LVM and don't allocate all of it to LV. We have a script that basically slowly expands volume when demand grows, up to a limit. So we don't have to think on stuff like "does the logs partition need to be 1 or 10GB", it will expand to the sane limit, and if it hits that we get disk usage alert before it finishes so we can either see what's going on (app shat in logs), or take a look for the one in the 10 apps that need some special tuning there

functional_dev 5mo ago

Such a simple trick, but effective!