5 ms·

> I wasn't intending to include transient filesystems in the index. There's absolutely no difference between transient and persistent filesystems in regards to

> I wasn't intending to include transient filesystems in the index. There's absolutely no difference between transient and persistent filesystems in regards to that problem. Every time a filesystem gets mounted, you have no idea what you're going to get. The last time it was mounted there could have been 13 million files on it and now when you mount it all of them could be gone or renamed. This is also super common on modern Linux systems, because many of them boot into a minimal boot environment to perform system updates and hence alter the filesystem heavily while such daemons as a file system monitor isn't running. So the question is: how do you know, whether /some/random/file has been modified while your daemon or application wasn't running or the filesystem wasn't mounted on your system, without performing a stat call on it? If you don't have an answer to that, which also needs to be orders of magnitudes faster, then you'll never match the performance of Everything. And that's not some uncommon situation, because your daemon/app has to figure that out every time it gets launched for every file and folder.

lelanthran 3y ago
> So the question is: how do you know, whether /some/random/file has been modified while your daemon or application wasn't running or the filesystem wasn't mounted on your system, without performing a stat call on it? If you don't have an answer to that, which also needs to be orders of magnitudes faster, then you'll never match the performance of Everything. Well, my intention is to match the feature list of Everything, but on Linux, and as far as I knew, Everything did not have full support for external drives - you'd have to convert them to NTFS, or add them to be indexed manually. The use-case I've seen for Everything has always been for a local user searching their local PC; I wasn't even sure until now that Everything can sometimes search transient filesystems because know one I ever saw using it used it for files on a transient filesystem. You're correct; what I cannot do is monitor transient filesystems; but doing permanent filesystems at a speed better than or equal to Everything is still better than anything I've used on Linux, many of which don't even search system files, nevermind transient filesystems. And they all use the locate db which is always a day or so out of date. And yes, it can be done purely by monitoring filesystem changes. Sure, a full index needs to be built the first time, but that's a one-off cost - index updates after that should be fast enough to do for each write/remove/move operation that you can update the index dozens of times per second. For non-transient filesystems, performance should be the same as, or better than, Everything.
- wander_homer 3y ago
  > And yes, it can be done purely by monitoring filesystem changes. Sure, a full index needs to be built the first time, but that's a one-off cost And how do you build the full index initially without recursively walking the filesystem? Otherwise you're not going to match Everything's performance on initial index creation. And regarding the second crucial question: How do you know that a file you saw the last time your app or daemon was running, hasn't been modified in the meantime? You still haven't answered those two fundamental questions. Everything else are solved issues anyway. > index updates after that should be fast enough to do for each write/remove/move operation that you can update the index dozens of times per second Like I already said, that has never been a problem. My app can currently update the index several thousand times per second and there's still a lot of room for improvements with many low hanging fruits. > For non-transient filesystems, performance should be the same as, or better than, Everything. You keep saying that, but you're also not giving an answer to how you're going to solve the two major and pretty much only issues.
  - lelanthran 3y ago
    > And how do you build the full index initially without recursively walking the filesystem? Otherwise you're not going to match Everything's performance on initial index creation. I wasn't planning to; it's a once-off cost - the user experience while using any software isn't degraded by the installation time, is it? > And regarding the second crucial question: How do you know that a file you saw the last time your app or daemon was running, hasn't been modified in the meantime? It's a daemon. if it isn't running while the user is using their desktop system, then it's not working or the user has turned it off. In any case, if a component of the software is not running, then the software is not running. I mean, seriously, even during regular updates, daemons still run. Even during distro upgrades daemons are still running. The rare cases where files are removed/changed/moved while daemons are turned off are fractions of fractions of a percentage. My desktop system currently has 2.5m files. There are maybe a dozen files which will be modified during a maintenance-mode bootup, which has happened exactly zero times in the last decade. For a Linux desktop file finding utility, monitoring all file writes, moves and deletes pretty much puts you ahead of any game in town right now, right? > You keep saying that, but you're also not giving an answer to how you're going to solve the two major and pretty much only issues. Perfect is the enemy of good. Issue 1 - Initial index creation: I will create the index during the s/ware installation process and never create it again unless it is missing. To speed the creation during installation, I will use the mlocate.db file if it is found. Issue 2 - Files that are changed/moved/removed when daemon is turned off: I don't really care, mostly. Those files a) have such a small probability of both existing and being of interest to the desktop user that lottery jackpots have a higher chance of happening to the user, and b) After an MVP, if the userbase requests those files, I'll either hardcode their locations and always check only for those dozens of files that can possibly be changed when daemons are turned off, or allow the user to specify via configuration, the pathname patterns to always check. I believe that this is enough to satisfy my original claim[1] of " "similar in performance and query capabilities""[2]. [1] https://news.ycombinator.com/item?id=38686022 https://news.ycombinator.com/item?id=38686022 [2] I don't recall making any claim along the lines of "walking the filesystem tree is never used".

lelanthran 3y ago

> So the question is: how do you know, whether /some/random/file has been modified while your daemon or application wasn't running or the filesystem wasn't mounted on your system, without performing a stat call on it? If you don't have an answer to that, which also needs to be orders of magnitudes faster, then you'll never match the performance of Everything. Well, my intention is to match the feature list of Everything, but on Linux, and as far as I knew, Everything did not have full support for external drives - you'd have to convert them to NTFS, or add them to be indexed manually. The use-case I've seen for Everything has always been for a local user searching their local PC; I wasn't even sure until now that Everything can sometimes search transient filesystems because know one I ever saw using it used it for files on a transient filesystem. You're correct; what I cannot do is monitor transient filesystems; but doing permanent filesystems at a speed better than or equal to Everything is still better than anything I've used on Linux, many of which don't even search system files, nevermind transient filesystems. And they all use the locate db which is always a day or so out of date. And yes, it can be done purely by monitoring filesystem changes. Sure, a full index needs to be built the first time, but that's a one-off cost - index updates after that should be fast enough to do for each write/remove/move operation that you can update the index dozens of times per second. For non-transient filesystems, performance should be the same as, or better than, Everything.

wander_homer 3y ago
> And yes, it can be done purely by monitoring filesystem changes. Sure, a full index needs to be built the first time, but that's a one-off cost And how do you build the full index initially without recursively walking the filesystem? Otherwise you're not going to match Everything's performance on initial index creation. And regarding the second crucial question: How do you know that a file you saw the last time your app or daemon was running, hasn't been modified in the meantime? You still haven't answered those two fundamental questions. Everything else are solved issues anyway. > index updates after that should be fast enough to do for each write/remove/move operation that you can update the index dozens of times per second Like I already said, that has never been a problem. My app can currently update the index several thousand times per second and there's still a lot of room for improvements with many low hanging fruits. > For non-transient filesystems, performance should be the same as, or better than, Everything. You keep saying that, but you're also not giving an answer to how you're going to solve the two major and pretty much only issues.
- lelanthran 3y ago
  > And how do you build the full index initially without recursively walking the filesystem? Otherwise you're not going to match Everything's performance on initial index creation. I wasn't planning to; it's a once-off cost - the user experience while using any software isn't degraded by the installation time, is it? > And regarding the second crucial question: How do you know that a file you saw the last time your app or daemon was running, hasn't been modified in the meantime? It's a daemon. if it isn't running while the user is using their desktop system, then it's not working or the user has turned it off. In any case, if a component of the software is not running, then the software is not running. I mean, seriously, even during regular updates, daemons still run. Even during distro upgrades daemons are still running. The rare cases where files are removed/changed/moved while daemons are turned off are fractions of fractions of a percentage. My desktop system currently has 2.5m files. There are maybe a dozen files which will be modified during a maintenance-mode bootup, which has happened exactly zero times in the last decade. For a Linux desktop file finding utility, monitoring all file writes, moves and deletes pretty much puts you ahead of any game in town right now, right? > You keep saying that, but you're also not giving an answer to how you're going to solve the two major and pretty much only issues. Perfect is the enemy of good. Issue 1 - Initial index creation: I will create the index during the s/ware installation process and never create it again unless it is missing. To speed the creation during installation, I will use the mlocate.db file if it is found. Issue 2 - Files that are changed/moved/removed when daemon is turned off: I don't really care, mostly. Those files a) have such a small probability of both existing and being of interest to the desktop user that lottery jackpots have a higher chance of happening to the user, and b) After an MVP, if the userbase requests those files, I'll either hardcode their locations and always check only for those dozens of files that can possibly be changed when daemons are turned off, or allow the user to specify via configuration, the pathname patterns to always check. I believe that this is enough to satisfy my original claim[1] of " "similar in performance and query capabilities""[2]. [1] https://news.ycombinator.com/item?id=38686022 https://news.ycombinator.com/item?id=38686022 [2] I don't recall making any claim along the lines of "walking the filesystem tree is never used".