I am toying with something similar. However my approach would be to use duckdb and S3 over lambda. Leaving many of the concerns to the infrastructure. Like ba

I am toying with something similar. However my approach would be to use duckdb and S3 over lambda. Leaving many of the concerns to the infrastructure. Like basically no OOM. No need to manage servers.