There is experimental support for distributed query execution with spill-to-disk between stages to support larger than memory datasets. This is implemented in t

There is experimental support for distributed query execution with spill-to-disk between stages to support larger than memory datasets. This is implemented in the Ballista crate, which extends DataFusion. https://github.com/apache/arrow-datafusion/tree/master/ballista https://github.com/apache/arrow-datafusion/tree/master/balli...