would like to see work like this, but for datasets in the hundreds of TB or single-digit PB but i definitely agree about this point > Cluster fatigue is real

would like to see work like this, but for datasets in the hundreds of TB or single-digit PB but i definitely agree about this point > Cluster fatigue is real imo, the concept of “extremely ephemeral query workers” is under-explored stateless, maintenance-free, burstable fleets of query workers is what I would like to see more of in the future. it’s how we do it, and it gives us full-text search on multi-hundred terabyte data sets in S3, where queries finish in a handful of seconds. our approach: https://docs.scanner.dev/scanner/what-and-why/how-it-works/how-scanner-achieves-fast-queries https://docs.scanner.dev/scanner/what-and-why/how-it-works/h... anyone else doing ephemeral query workers fleets?